A Joint Subspace View to Convolutional Neural Networks
Authors: Ze Wang
Xiuyuan Cheng
Guillermo Sapiro
Qiang Qiu
Venue: ICLR 2022 Submitted (2022)

Abstract: Motivated by the intuition that important image regions remain important across different layers and scales in a CNN, we propose in this paper a joint subspace view to convolutional filters across network layers. When we construct for each layer a filter subspace by decomposing convolutional filters over a small set of layer-specific filter atoms, we observe a low-rank structure within subspace coefficients across layers. The above observation matches widely-known cross-layer filter correlation and redundancy. Thus, we propose to jointly model filter subspace across different layers by enforcing cross-layer shared subspace coefficients. In other words, a CNN is now reduced to layers of filter atoms, typically a few hundred of parameters per layer, with a common block of subspace coefficients shared across layers. We further show that such subspace coefficient sharing can be easily extended to other network sub-structures, from sharing across the entire network to sharing within filter groups in a layer. While significantly reducing the parameter redundancy of a wide range of network architectures, the proposed joint subspace view also preserves the expressiveness of CNNs, and brings many additional advantages, such as easy model adaptation and better interpretation. We support our findings with extensive empirical evidence. 

Under review as a conference paper at ICLR 2022
A JOINT SUBSPACE VIEW TO CONVOLUTIONAL NEU- RAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Motivated by the intuition that important image regions remain important across different layers and scales in CNN, we propose in this paper joint subspace view convolutional filters across network layers. When we construct for each layer filter subspace by decomposing convolutional filters over small set of layer-specific filter atoms, we observe low-rank structure within subspace coef- ficients across layers. The above observation matches widely-knows cross-layer filter correlation and redundancy. Thus, we propose to jointly model filter sub- space across different layers by enforcing enfforcing cross-layer shared subspace coefficients. In other words, a CNN is now reduced to layers of filter atoms, typically few hundred of parameters per layer, with a common block of subspace coefficients shared across layers. We further show that such subspace coefficient sharing be easily extended to other network sub-structures, from sharing across the entire network sharing within filter groups in layer. While significantly reducing the parameter redundancy of a wide range of network architectures, the proposed joint subspace view also preserves the expressiveness of CNNs, and brings many ad- ditional advantages, such as easy model adaptation and better interpretation. We support our findings with extensive empirical evidence.
INTRODUCTION
Modern developments of deep network components (loffe & Szegedy 2013 Xu et al. 2015). struc- tures (He et al. 2016:; Huang et al., (017), and training techniques (Kingma & Ba 2014 Zoph & Le 2017) have enabled deep convolutional neural networks to consistently grow deeper and wider, with hundreds of millions of parameters. The trend of growing network scales leads to both severe challenges and urgent needs to fast adaptations (Finn et al. 2017 Lopez-Paz Ranzato 2017 Shin
et al. 2017). parameter efficiency (Cheng et al. 2017 Han et al. 2015a: Luo et al. 201 7| Savarese
2018b). and interpretabiliity (Selvaraju et al. 2017 & Maire| 2019, Yang et al. 2019; Wang et al.
Zhou et al. 2016). While recent works on efficient convolution operations (Chollet 2017 Howard
al. 2017 Ma et al. 2018 Zhang al. 2018) alleviate the long recognized over- areer-arametrization
problem of deep CNNs, and practical methods on modeling the shared components across CNN fil-
ters have been proposed (Ha al. 2016 Savarese & Maire 2019), there still lack of a principled view to exploiting the common structure of the filters within a CNN.
While preliminary studies have revealed the redundancies in deep network parameters (Michel et al. 2019 Raghu et al.) 2017). and the fact that deep features across layers are highly correlated under certain linear transforms (Kornblith et al.) [2199] Morcos al.] (0188 Raghu et al. (2017). exploiting this potential redundancy remains challenging. We suspect that the spatial patterns of convolutional filters, which correspond to different semantics levels, the main reason why we cannot observe obvious cross-layer filter low-rankness. To validate our hypothesis, we decouple the spatial patterns and channel mixing of a convolutional filter by decomposing it over anyer-speiffic 2D filter atoms, linearly combined using atom coefficients. Then with proper alignments the layer outputs, we finally observe a low-rank structure among those atom coefficients across different layers. This highly non-trivial observation, which our knowledge has never been reported before, hints the feasibility of enforcing a common block of atom coefficients across layers and suggests joint subspace view to CNNs.
Applying to wide range of network architectures, this joint subspace view leads subsequently to a novel CNN architecture in which the majority of parameters shared across layers common

---

Under review as a conference paper at ICLR 2022
Conv 3 K₃ E xxxxxi
Atoms
D3 E mmxb
Conv 2 K2 E IR XIXI
Shared Coefficients EBexe'xm
Atoms 2 D2 E mxxxi
Conv K, E Ie xc'xixi
Atoms
D2 E Rmxiit
Figure I: Atom-coefficient decomposition disentangles spatial convolution and channel mixing. So that each feature map after atom convolution now preserves spatial correspondence the original image; and channels that describe important region features, e.g., wings, beak, and tail of bird, will highly likely be consistently assigned with weights across layers, order to carry those critical information. Thus, atom coefficients, i.e., weights to linearly combine channels, now become shareable across layers to permit joint subspace view. Such joint subspace view leads naturally to a novel CNN architecture in which the majority parameters are shared across layers as a common block of atom coefficients, with only few hundred parameters remaining specific to each layer as filter atoms.
block of atom coefficients, with only a few hundred parameters remaining specific to each layer as filter atoms shown in Figure This matches the intuition visualized in Figure that important image regions, e.g., the wings, beak, and tail of a bird, remain important across different layers and scales output informative deep representation. Atom-coefficient decomposition disentangles spatial convolution and channel mixing. Thus, channels that describe important region features will highly likely be consistently assigned with large weights across layers, in order to carry on those critical information. Thus, atom coefficients, i.c., weights to linearly combine channels, now become shareable across layers to permit joint subspace CNN modeling.
Our approach accepts many easily constructed variants, e-g., using different filter atom numbers and sharing substructures, to allow highly flexible trade-offs between parameter reduction and model expressiveness. This simple and plug-and-play joint subspace view opens the door to better inter- preting, training, adapting, and compacting deep models. We support our findings with extensive empirical evidence in this paper. By using variants of ACDC as plug-and and--aay replacements the standard convolutions, we observe comparable and even better performance on challenging image classification datasets with orders magnitude smaller models. We further demonstrate with few shot learning experiments that ACDC improves the adaptation of deep networks on novel tasks with limited supervisions. We end our paper with model interpretability discussion.
Our main contributions are summarized as follows:
We introduce a joint subspace view to CNN, which is compatible to a wide range of net- work architectures, to provide structural regularization over CNN filters for better model interpretability, training, adaptation and compaction.
Highlighting the remarkable flcxibility and compatibility, we introduce various model vari- ants constructed casily by sharing coefficient within different network sub-structures.
We validate the effectiveness of our approach by architectures for various real-world tasks.
them into modern CNN
2 RELATED WORK
Before presenting the connections of our method with the literature, let us comment on the com- plementary efforts on network compression. Contrary to recent efforts on network compression and efficient architectures, tions among convolutional layers within given CNN. We decompose the convolutional kernels among layers, and enforce shared composition coefficients within sub-structures of CNN. This
instead explore and exploit the underlying correspondence and correla-
2

---

Under review as a conference paper at ICLR 2022
plug-and and-play reparametrization of convolutional kemels significantly reduces the network redun- dancy while maintaining performance, improving adaptation, providing additional interpretability, reducing computational cost, and adding compression (as by-product) comparable to many mod- ern heavily tailored light-weight networks. We consider directions such as neural architecture search (Howard et al. (209). post processing methods to deep networks (Phan et al. [2020] Stock et al.)
2020 Li et al. 2019a Wu et. 2018 Son et al. 2018), and heavily tailored efficient networks and training Tan & Le 2019) complementary to our efforts.
CNN architectures. The tremendous success of applying convolutional neural networks (CNNs) on numerous tasks has stimulated rapid developments for more effective and efficient network ar- chitectures in both hand-crafted (Chen al. 2017 He et al., 2016; [Howard et al. 2017 landola
iet al. 2016 Sandler et al. 2018) and automatically discovered (Elsken et al. 2018 8 Liu et al. 2018
Pham et al. 2018 Zoph & Le 2017) manners. We consider our work orthogonal to such topology-
based methods, as the plug-am and play property of the proposed ACDC allows it to added to all the replacement to the standard convolution. Besides efforts efficient network architectures, methods for network compression and pruming (Han et al., 2015a, 2016 2015b] He et al. 2017; Li et al., 2016 Luo et al., 2017) have been extensively studied for decreasing the model size by pruning the inconsequential connections and weights of network. Methods (Ha et al. 2016: Savarese & Maire, 2019) align with our direction as they are also insensi- tive to network topologies. And as shown in the experiments, ACDC can achieve higher performance in terms of parameter reduction and classification accuracy with greater flexibility.
aforementioned methods
studying
Kernel decomposition in CNNS Convolutional kernel decomposition has been studied for various objectives. (Sosnovik et al. 2019) utilizes kernel decomposition as a tool for constructing same kernel with multiple receptive fields. DCFNet (Qiu al.. 2018) is proposed
principle way of regularizing the convolutional filter structures by decomposing convolutional filters in CNN as a truncated expansion with pre-fixed bases. Split-wise decomposition over basis elements is intro- duced in (Li et al. 2019a) to retrain a CNN with reduced parameters.
Weight sharing in networks Exploiting sharing weight within CNNs is discussed in (Son et al. 2018 Wu et al. 2018) as a post-prose post-procsssing step for model compression. Hypernetworks (Ha et al, 2016 adopt weight generation from shared network (denoted as Hypernet) to exploit correlations among weights empirically.
JOINT SUBSPACE CNN MODELING
In this section, we start with brief introduction atom-coefficient filter decomposition, which builds the foundation of our joint subspace CNN modeling (Section 3.J). We then present the empiri- cal support for the joint subspace view CNN filters through a motivating experiment (Section We show in this motivating experiment that, with ttom-coefficiem decomposition, a jointly low-rank structure can be clearly observed among coefficients trained independently in different layers of CNN under certain linear transformations. This observation, together with the well recognized over- parametrization problem of CNNs, leads to the idea of subspace coefficients sharing enforced across network sub-structures, which can then be constructed as plug-and and-play replacements to the standard convolutions (Section 3.3 and Section 3.4).
3.1 CONVOLUTIONAL FILTER DECOMPOSITION
Previous works have shown that a convolutional filter in a CNN can be decomposed as a linear com- bination of fieef basis (Qiu et al.) 2018). In our approach, we adopt similar decomposition as shown in Figure 2,, in which a convolutional filter is represented as a linear combination of trainable 2D filter atoms. After decomposition, convolution layer with e-channel output Y and c'-channel input X becomes
= X K=
(1)
where * denotes the convolution operation. As illustrated in Figure 2 in T.. a convolutional filter K E Ieexe' xixl, which can be seen as stack of x c' 2D convolutional filters with the size of I 1, is reconstructed by multiplying 2D filter atoms, denoted collectively D Rmxtxl with the corresponding atom coefficients A Rexe' xm Note that square filters are assumed here for simplicity, while all filter shapes are supported. Since both convolution and tensor multiplication commutative, convolutional layer now be decomposed into two:
3

---

Under review as a conference paper at ICLR 2022
A atom sub-layer where each atom involves spatial-only convolution with the filter atoms,
i..., Z pRd'mxx
D x X;
A coefficient sub-layer that linearly combines feature channels from the atom sub-layer: Y E Roxhxx = AZ. Note that here denotes atom sub-layer outputs, and stride 1 and same padding assumed for the sake of discussion.
3.2 THE MOTIVATION BEHIND
Deep CNNs are long recognized to be wver-parametrized. The very deep layers in modern CNN structures (He et at... 2016 Huang at. 2017 Zagoruyko & Komodakis 2016 and the high-dimensional filters with little struc- tural regularizations lead to hundreds of millions of pa- rameters. Such over- wver-paramtrization problem also ob- served in the studies of deep representations (Raghu et al.) 2017), and empirically alleviated by new network struc- tures (Chollet, 2017, Howard et al. 2017), network com- pression, and parameter reduction methods (Ha al.
K
Figure 2: Atom-coefficient filter de- composition. A convolutional kernel K is decomposed
set of m 2D fil- ter atoms linearly combined by atom coefficients A.
2016 Savarese & Maire 2019).
Meanwhile, recent studies on deep representations (Kornblith et al. 2019 Morcos al. 2018 Raghu et al. 2017) have shown that there exists obvious correlations in features across layers within CNN alter proper linear transformations. The correlation of features across network layers moli- vates us to explore and exploit correlations across filters for structural regularizations.
We present here motivating experiment MNIST by applying CCA alignments in (Raghu al. 2017) to the atom sub-layer outputs and the coefficient sub-layer outputs of layer i and layer j. Note that atom coefficient sharing is yet imposed here, and the network reports the same testing accu- racy before and after filter decomposition. Formally, c, m, d, and hw denote the number of channels, number of filter atoms, test set size, and the 2D feature dimensions, respectively. The atom sub-layer outputs of the i-th and j-th layer, Z; and Zj E gem gewwddww are firstly aligned by lincar transforma- tions P, and Pj- E IR cmmm that maximize the correlation Pa max corr(P.Z, P,Zz). And P..P,
similarly, the coefficient sub-layer outputs of both layers, Y, and Y, E Rexdhw are aligned by Q: and Q2 Bexe that maximize the correlation p max corr Q Y, Q2Y,). Omitting the layer Q.Q,
indexes, the feed forwards of both layers can be rewritten as
Y = QPD
(2)
By merging the transform into the coefficients A by À = QAP-1 we obtain 'aligned coefficients' A and Aj. that reside in a low rank structure reflected by the very similar effective ranks of A, and [A;, A, For example, our MNIST experiment using a 4-layer 32-channel CNN, out of the possible (i,j) pairs, the average effective rank of A, and [A,, A are 31.98 and 38.56, respectively. Our observations agree with and further support recent studies on cross-layer feature correlations (Krmblihh et al.) 2019 Morcos et al. 2018 Raghu et al.) 2017). This observation hints that, in- stead of treating filters in each layers independently, it feasible to exploit such intrinsic low-rank structure of coefficients across layers, and impose a joint subspace view to filters in a CNNs. And practically, this observation supports the feasibility of directly reducing the parameter redundancy of a CNN by enforcing shared atom coefficients across layers.
3.3 COEFFICIENTS SHARING ACROSS LAYERS
Based on the observations above and [Tp. joint subspace CNN modeling is constructed by enforcing common atom coefficients A across layers in CNN, as illustrated in Figure Formally, given N-layers CNN, the n-th convolutional filter is constructed by
K, AD,, n. Vn I,...,N. N.
Assuming for now all layers have identical channel number with é' = c, the amount of parameters is reduced from c²22N to e2m. + N
4

---

Under review as a conference paper at ICLR 2022
Fiio grater
Claszel Stafee
Ye 2 lptkem
Ye R (ysten
Yeel Sprntoo
X
Alexza
Alssn a Pr ARRtNC2
Keruln I ABl Maliellins
Kemel 2 Bustiiill
Kemalln Kayaltioliin
Dg ARp06o
B (gater
Chased Grraging
Ianari
NG jehilov
Figure 4: atom coefficient sharing with three groups layer The input feature first equally divided into groups (denoted as boxes with different grey scales), each of which is convolved with one group of filters reconstructed by multiplying the corresponding filter filter atoms and the shared coefficients. The output of three groups are com-- bined by channel shuffle.
Figure 3: Illustration on how coefficients are shared across three layers with increas- ing numbers channels. The shared co- efficients are initialized with the largest di- mensions required.
In practice, convolution layers within a network can have different numbers of channels. When sharing coefficients across layers with different channels numbers, we initialize the dimensions of the shared coefficients to be the largest dimensions needed by the corresponding layers. For ex- ample, given N-layer CNN with convolutional kemels {Kn E R X6,, xix! 1....,
the network is constructed by initializing the shared coefficient as A E 185mexXC Xm where .N. 1....,N)
The filters with fewer channels are reconstructed by multiplying the filter atoms with a subset of the shared coefficients K₂ A [ : Cn. 1 c, 1 m Dn,- A 3-layer illustration with progressively increased channels is shown in Figure 3 Such design choice is motivated by multi-scale decomposition (Mallat 1999), and proves be highly effective with our extensive experimental validation.
Cmaz = max(c,;; n 1, :..... N} and Cmaz max
Note that a relaxed version can be formulated as follows: Instead of enforcing common coefficients across all layers, we allow the sharing to happen among few consecutive layers in network. We refer a group of consecutive layers with identical number of output channels and identical feature sizes as block in deep network, and enforce coefficient sharing in each block. For example, VGG16 (Simonyan & Zisserman 2014) can be implemented by sharing coefficients within 5 groups, each of which consists of conv layers with 64, 128, 256, 512, 512 channels, respectively.
3.4 COEFFICIENTS SHARING ACROSS FILTER GROUPS
Motivated by the observation in (Raghu et al. 2017) that the representation learned at a layer is not fully determined by the number of neurons in the layer, and the existence of parameter redun- dancy within a single layer as studied in (Michel et al.) 2019), we suggest to further extend our joint subspace view from coefficient sharing across layers to sharing across groups of filters in a convolutional layer.
Practically, we further break down the smallest sharing unit from a layer to part of layer. A high-dimensional convolutional layer can now be separated into several groups with identical sizes, and sharing coefficients are imposed across groups. Formally, given convolutional layer with d' input channels and e output channels, respectively, we divide input channels into identical-sized
groups, and each group is convolved with a convolution kernel K, E R x xixl .j 1, 11...,.. g-
After grouping, we decompose {Kjij = 1, 1......) o} into shared coefficients A A R and independent sets of filter atoms (D, E _ = 1.....g). In this way. the number of shared coefficients is reduced by g" times, and the number of filter atoms is increased by g times. Since filter atoms have orders of magnitude smaller dimension comparing to the coefficients, we achieve further parameter reduction. Since each K, only convolves with a subset of the input feature, this method reduces the overall computations.
However, directly deploying this sharing mechanism breaks the network into several paralleled sub- networks with no feature passing and gradient propagation among them. To remedy this without adding any additional parametric components, we utilize channel shuffle (Zhang et al.) 2018) that enables information to be efficiently propagated among groups in a -arameemetti way. An illus- tration is presented in Figure 4 Since the size of the shared coefficient now does not depend on the
5

---

Under review as a conference paper at ICLR 2022
largest feature dimension of a network but the size of the groups, obtain compact model. In practice, to handle the expanding channels numbers in typical CNNs, we can easily allow each group in the shallow layer to feed the features to multiple groups in the deep layer. Therefore the proposed coefficient sharing across filter groups permits remarkable parameter sharing with little modification to the original network structures such as the numbers of channels and layers.
Table 1: Comparisons on CIFAR-IO with parameter sizes, parameter reduction rates, and error. Those with higher than baseline accuracy but fewer parameters are marked in bold.
VGG16
ResNet18
WRN-40-4
Architectures
Size
Error
Size
Error
Size
Error
Baseline
14.72M
6.20
11.17M
5.81
8.90M
4.97
2. (85, (557%%
5.67
2.28M (79.6% 6%%))
5.98
0,58M
(93.5%) (9
4.85
ACDC-net
16
4.21M (71.4%
5.44
4.38M (60.8%1)
5.43
1.1 (87.5%))
4.42
8
4.89M (66.8%..)
5,47
2.96M (73.5%
5.50
0.74M (91. (5%..)
4.46
ACDC-block
16
9.78M (33.6%..)
5.40
5.76M (48.4%%.)
4.92
1.43M (33.9%..)
4.38
32
0.03M 99.8%%)
10.24
0.20M 998.2%%.)
7.30
0.07M (99.2 L)
8.20
ACDC-g-net
16
64
0.08M (99.5%) (9)
9.87
0,26M (97.7%.)
7.91
0.13M (98.5% [)
6.85
8
32
0.06M (99.6%..)
9.71
0.22M (98.0%:L)
7.48
0.09M (99.0%..)
8.92
ACDC-g-block
16
64
0.35M (97.6%.)
6.63
0.45M (96.0%%.)
7.22
0.26M (97.1%..
6.88
32
0.13M (99.1 (%-.)
6.68
0.89M (92.0%;L)
5.23
0.36M (96.0%.) 06..
5.02
ACDC-g-layer
16
64
0.80M (94.6%%.)
5.67
0.60M (94.658.))
6.20
1.98M (77.8%.))
4.23
4 EXPERIMENTS
In this section, we support our findings with extensive empirical evidence, and evaluate variants of our joint subspace CNN modeling, referred to as ACDC, plug-and-play replacements to the standard convolution, with several different levels of atom coefficient sharing:
ACDC-ner: Sharing across all layers.
ACDC-block: Sharing across blocks of layers.
ACDC-g: Sharing across filter groups.
ACDC-g naturally further leads three variants by allowing sharing within the entire network, blocks within a network, and layers within a network, and are thus named as ACDCCg-net, ACDC- g g-block, and ACDC-g -layer, respectively. We use and to denote the number of filter atoms and grouping size, respectively.
4.1 IMAGE CLASSIHCATION
In this section, we perform standard image classification experiments, and validate our joint sub- space CNN modeling with extensive experiments on CIFAR-10, CIFAR-IOU 100, TinylmagNNet, and ImageNet.
Evaluating variants of ACDC. first report on CIFAR-10 self-comparisons on variants of ACDC constructed with different numbers of filter atoms as well as grouping sizes. We present performance terms of both parameter size and classification error in Table VGG16 (Simonyan & Zisserman 2014), ResNet18 (He et al. 2016). and Wide ResNet (WRN) (Zagoruyko & Ko- modakis 2016) are adopted as the underlying network architectures in order to show the remarkable compatibility of the proposed approach. This set of experiments show our joint subspace model- ing maintains or improves CNN performance with a significantly more compact model, e.g., 98% reduction with comparable performance, and 70% reduction with even higher accuracy.
CIFAR-10, CIFAR-100 and TinyImageNet. We further present experiment results on CIFAR-10 100 CIFAR- 100, and TinylmageNet in Table 2 We compare exampled variants of ACDC against Hy- perNetworks (Ha al. 2016) and Soft Parameter Sharing (Savarese & Maire 2019). both of which serve as plug-and-play replacements to standard convolutions as well. Though HyperNetworks (Ha et al. 2016) achieves remarkable parameter reduction, ACDC is able to achieve higher accuracies

---

Under review as a conference paper at ICLR 2022
Table 2: Classification performances on CIFAR-10, CIFAR-100 and TinylmageNet datasets. Per-
formance on sfate-o--the-nd light CNN architectures are listed in the upper block. The middle block shows the performance results of variants of ACDC in the bottom block. Performance obtained by our reproductions are marked with
auugannd-pay methods with parameter sharing in CNNs. We present the
Methods
Size
CIFAR-10
CIFAR-100
TinyImageNet
SqueezeNel (andolaa et al. 2016
2.36M
6,98
29.56'
48.22'
ShuffleNet Zhang el al. 2018
0.91M
7.89*
29.94*
54.72'
ShuffleNet- V2 ci al. 2018
1.3M
8,96
29.68'
51.13*
MohilcNet-Ve V2 (Sandler cf al. 2018
2.36M
5.52"
30.02"
48.22"
NASNet (Zoph al.
2018
3.1M
3.59
21.77"
47.17*
DCFNet (K=6, FB) VGG16 (Qiu 2018
9.84M
5.98
32.75*
57.70"
Learning filter bases VGG16
20196
3.21M
6.23
LegoNet- (GG16--(0044mm=0255)
Yang
2019
0.9M
8.65
30.11*
55.22
LegoNet- VGGGle-VGGlle V-NNNN--GG-----0022222000 (- Yang cl al. 2019
3.7M
6.77
29.45'
48.42*
WRN-40-1 HyperNets (Ha el al.
2016
0.10M
8.02
WRN-40-2 HyperNets (Ha el al.
2016
2.24M
7.23
SWRN 28- 10-1 (Savarese Maire
2019
12M
4.01
19.73
43.05*
SWRN 28- 10-2
(Savarese Maire
2019
17M
3.75
18.37
41.12"
VGG16 ACDC-g- C-yyye m8 #32
0.13M
6.68
28.81
49.96
VGG16 ACDC-net m8
2.11M
5.67
22.13
43.34
WRN-40-1 ACDC-block m8
0.043M
7.19
30.23
51.47
WRN-40-1 ACDC-block m22
0.114M
7.02
28.14
49.05
WRN-40-4 ACDC-g-layer m16 s32
0.67M
4.38
20.04
45.87
WRN-28-10 ACDC-g-block m24 s160
2.27M
4.25
19.64
41.24
WRN-28-10 ACD(-net m12
5.21M
3.52
18.81
39.96
WRN-28-100 ACDC-blook m24
13.20M
3.26
17.85
38.74
Table 3: Performance ImageNet. Parameters, top-1 and top-5 errors are reported. Numbers ob- tained by our reproductions are marked with *. We use ResNet34 and ResNe150 as baselines.
mintImageNett
70%
60%
Methods
Top-1
Top-5
55%
ResNet34
21.28M
27.42*
9.02*
50%
Structured Conv A
9.82M
27.19
Structured Conv
5.60M
30.56
45%
ACDC-g- g-layer 2s61 (ours)
1666
32.82
12.14
Conv6
18
ACDC-net m12 (ours)
3.34M
30.85
11.25
ACDC-stage (ours)
5.77M
27.78
9.72
SWIS
MAML-ACTC.5WIS
ResNet50
25.26M
24.17*
7.82"
MAML SW5S
MAML-ACIX
ChPrune
17.89M
27.7
9.2
Figure 5: Few-shot image classification with deeper CNN architectures. 5WIS and 5W5S de- note 5-way 1-shot and 5-way 5-shol experiments, respectively. Performance (Y-axis) evaluated by averaging 3,000 rounds of randomly sampled test- ing tasks.
LegoNet-w
Versatile
11.0M
25.5
8.2
Structured Conv
13.49M
24.35
Structured Cony
8.57M
26.59
ACDC-net m8 (ours)
14,29M
25.38
8.08
ACDC-stage stage (aurs)
with even fewer parameters. The parameter reductions in Soft Parameter Sharing (Savarese & Maire 2019) are highly restricted by the large scale elements in the filter bank. For example, SWRN 28- I, as the smallest variant of Soft Parameter Sharing on WRN, adopts single template per sharing group. and can only achieve 66% of parameter reduction. By adopting ACDC-net and ACDC- to WRN, we are able to achieve both higher parameter reductions and accuracy. Though model compaction is not mere objective,
include here references some state-of-the-ant light CNN architectures (landola et al.] 2016 ZZhang et al.] 2018 Ma et al. 2018) Sandler et al., (018)) and architecture based on neural architecture search (Zoph et al. 2018).. Additional comparisons against network compression and pruning methods are in supplementary material Section
ImageNet. To fully validate joint subspace CNN modeling, we perform further experiments on the large-scale ImageNet dataset (Deng et al. 2009). We use ResNet34 and ResNet50 as baseline models, and compare ACDC against LegoNet Yang al.) 2019), Versatile (Wang al. 2018h).

---

obtained by Under review as a conference paper at ICLR 2022
Performance with Different Number of Atoms
Atom-drop
91.8%
LIM
10M
96.00%
N..8%
LJIM
1370
96 95.00%
90.0%
95.0%
17.8%
1/11
0.08
0.1
62
Nanber
20
24
Atom Drap Rale
Figure 6: Accuracy with different number of Figure 7: Accuracy with different atom drop rate filter atoms m. Parameter sizes are denoted p. as #M.
network pruning method ChPrune (He 2017). ). and recently proposed Structured Conv (Bhalgat et al. 2020). Results are presented in Table 3] For ResNet34, one variant of ACDC with grouping layers uses only 1.66 million parameters with acceptable performance decrease, For ResNet50, since large proportion of the parameters are in the 1 x 1 conv layers, ACDC achieves fair parameter reductions with maintained performance.
4.2 FEW-SHOT EXPERIMENTS
We further demonstrate that joint subspace modeling improves deep model adaptation novel tasks with limited supervisions, which is validated by few-shot classification using commonly adopted experimental settings. Specifically, we adopt ACDC-net the model-agnostic meta- learning (MAML) (Finn et al.) 2017) algorithm, which method that adapts to a novel task by tuning the entire network from a learned initialization. Although MAML is designed to be model- agnostic, we consistently observe that it struggles for further performance improvements when using deeper and wider networks. Same observations are reported in (Chen et al. 2019). We show that such limitation can be alleviated by structural regularizations with ACDC. We follow the same ex- perimental settings as (Chen et al. (2019) and perform both 5-way 1-shot and 5-way 5-shot image classifications on minIImageNet dataset. The comparisons are shown in Figure 5 Though adopt- ing ResNet10 with MAML achieves improvements over simple network with few stacked layers, the performance drops with more residual layers shown by the results ResNet18. By using ACDC-net with deeper ResNets, performance is not only maintained but also improved when more layers used, which thus permits larger models in MAML.
Table 4: Comparisons of FLOPs and speed with different variants of ACDC with grouping.
Networks
Baseline
ACDC m8 s64
ACDC m16 s64
ACDC m8 s32
ACDC m16 s32
VGG16
125.66B 222.4B 8.9
64.15B 116.76B 5.8
64.17B 116.78B 5.8
32.29B 60.13B 4.2
32.30B
ResNet18
60.14B
Speed (ms/batch)
4.3
4.3 COMPUTATIONAL EFFICIENCY
ACDC enjoys another merit of being computationally efficient when using sharing with grouping. Since after grouping, each group of convolutional filters only convolves with subset of input fea- tures, ACDC-g-block and ACDC-g-nel substantially reduce the number of FLOPs and accelerate the speed. We report comparisons with ResNet18 8 and VGG16 in Table 4 All numbers feeding the network typical batch with 100 64 x 64 images. It is clearly shown that by using small groups, the computation the speed the total FLOPs slightly. We present further speed comparisons against recent efficient network structures in Appendix Table
be reduced dramatically, and larger number of filter atoms only effeets
Number of filter atoms One additional hyperparameter introduced is the number of filter atoms m per sub-structure. As shown in Figure 6 having more filter atoms in each sub-structure leads to performance improvements that saturate at m = 32. More filter atoms also result larger parameter sizes, which unfavourable.
8

---

Under review as a conference paper at ICLR 2022
Regularization by atom-drop. To improve the robustness of the filter atoms and the correspond- ing reconstructed kernels, we further propose a structural regularization to filter atoms named atom- drop inspired by widely used dropout. Specifically, when training the network, we randomly drop a filter atom with probability p, which referred as alom drop rale, by temporarily setting values of the dropped filter atoms to 0, and the values of all other remained filter atoms are multiplied by in order to maintain consistent scales of the reconstructed convolutional kernels. At test time, all filter atoms are presented with no dropping. As shown in Figure 7 atom-drop improves gen- eralization when p < 0.1. Higher values of p potentially degrade the performance as the training becomes unstable. Thus we use p 0.1 as the default setting. We show Appendix Section further comparison against typical dropout methods.
T-p
Standard CNN
ACDC-net
Standard CNN
ACDC-net
Figure 8: Illustration on extending CAM to all layers (shallow + deep) with standard CNN and ACDC-net. While CAM is originally introduced to explain the feature at the final convolutional layer, we show that sharing coefficients allows CAM better explain the shallow layers. The network is progressively extracting features that attend to the discriminative regions, e.g., the wings or the head of bird.
CONCLUSION AND INTERPRETABILITY DISCUSSION
In this paper, we introduced a joint subspace approach to CNNs. We presented observations that, due to the underlying cross-layer correlations, atom coefficients in the decomposed convolution layers across a CNN reside low-rank structure. We explicitly exploited such observations by enforcing atom coefficients to be shared within sub-structures of CNNs. Variants of the proposed method can be constructed with different sharing structures, number of atoms, and grouping sizes. We reported extensive experiment results that show the effectiveness of the proposed method on standard image classification and adaptations.
This joint subspace modeling has the potential for better interpretability of CNNs, due to the cross- layer shared coefficients. Class activation mapping (CAM) (Zhou et al. 2016). is method that is originally proposed to explain the importance of image regions only at the final convolution layer, and is calculated by weighted averaging the features of the final convolution layer by the weight vector of a particular class. We close our paper with an illustration in Figure that extends class activation mapping (CAM) to all layers of a ResNet- 18 and a ResNet- 18 with ACDC-net. The networks are trained from scratch on CUB-200 (Wah et al., (011) high-resolution (448 448) fine-grained bird classification dataset.
clearly shown that, in ACDC-net, while the class activation maps for shallow layers are evitably noisy due to the limited receptive fields, features in deeper layers are progressively refined the discriminative regions. However, in standard CNNs, due no explicit correspondence among filters across layers, CAM can only explain the feature of the last Conv layer. This shows the great potential for better interpretability with ACDC, and will keep this direction of future effort.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Yash Bhalgat, Yizhe Zhang, Jamie Lin, and Fatih Porikli. Structured convolutions for efficient neural network design. arXiv preprint arXiv:2008. 02454, 2020.
Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. arXiv preprint arXiv: 1904.0423, 2019.
Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. In Advances in Neural Information Processing Systems, pp. 4467-4475, 2017.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint urXiv:1710.092:0 09222, 2017.
Francois Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, July 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 255. leee, 2009.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. arXiv preprint xrxiv: /008 053777 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135. JMLR. org, 2017.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv: 1609.0906, 2016.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 15 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pp. 1135-1143, 2015b.
Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Enhao Gong, Shijian Tang, Erich Elsen, Peter Va- jda, Manohar Paluri, John Tran, et al. Dsd: panse-sparse-dense training for deep neural networks. arXiv preprint
2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukum Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE International Conference on Computer Vision, pp. 1314-1324, 2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Mareo Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint /7X 2016.
10

---

Under review as a conference paper at ICLR 2022
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. arXiv preprint arXiv: http//90500044.. 2019.
Namhoon Lee. Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning
based connection sensitivity, International Conference Learning Representations, 2018.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv: 160Kvv:6008.08800 1608.68700, 2016.
Yawei Li, Shuhang Gu, Luc Van Gool, and Radu Timofte. Learning filter basis for convolutional neural network compression. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5623-5632, 2019a.
Yawei Li, Shuhang Gu, Luc Van Gool, and Radu Timofie. Learning filter basis for convolutional neural network compression. Proceedings of the IEEE International Conference on Computer Vision, pp. 5623-5632, 2019b.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. Proceed- ings of the European Conference Computer Vision, pp. 19-34, 2018.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruming method for deep neural network compression. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5058-5066, 2017.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European Conference on Computer Vision, pp. 116-131, 2018
Stéphane Mallat. A wavelet tour of signal processing. Elsevier, 1999.
Paul Michel, Omer Levy, and Graham Neubig, Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, pp. 14014-14024, 2019.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems, pp. 5727-5736, 2018
Anton Obukhov, Maxim Rakhuba, Stamatios Georgoulis, Menelaos Kanakis, Dengxin Dai, and Luc Van Gool. T-basis: a compact representation for neural networks. International Conference on Machine Learning, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems, pp- 8026-8037, 2019.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural architecture search via paramcter sharing. arXiv preprint arXiv:1802.03266 2018
Anh-Huy Phan, Konstantin Sobolev, Konstantin Sozykin, Dmitry Ermilov, Julia Gusak, Petr Tichavský, Valeriy Glukhov, Ivan Oseledets, and Andrzej Cichocki. Stable low-rank tensor de- composition for compression of convolutional neural network. In Proceedings of the European Conference Computer Vision, pp. 522-539. Springer, 2020.
11

---

Under review as a conference paper at ICLR 2022
Qiang Qiu, Xiuyuan Cheng, Robert Calderbank, and Guillermo Sapiro. DCFNet: Deep neural network with decomposed convolutional filters. International Conference on Machine Learning. 2018.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, pp.
2017.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- -hich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Pedro Savarese and Michael Maire. Learning implicitly recurrent cnns through parameter sharing. arXiv preprint arXiv: /00.00900,.0901,,, 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference Computer Vision, pp. 618-626, 2017.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-2999, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Sanghyun Son, Seungjun Nah, and Kyoung Mu Lee. Clustering convolutional kernels to compress deep neural networks. In Proceedings of the European Conference on Computer Vision, pp. 216- 232. 2018.
Ivan Sosnovik, Michal Szmaja, and Arnold Smeulders. Scale-equivariant steerable networks. arXiv preprint arXiv: 1093, 2019.
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. And the bit goes down: Revisiting the quantization of neural networks. International Conference on Learning Representations, 2020.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint 19Xiv:195..11944 2019.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucse birds-200-2011 dataset. 2011.
Chaoqi Wang, Guodong Zhang, and Roger Grosse Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
Wenqi Wang. Yifan Sun, Brian Eriksson, Wenlin Wang, and Vaneet Aggarwal. Wide compres- sion: Tensor ring nets. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018a.
Yunhe Wang, Chang Xu, XU Chunjing, Chao Xu, and Dacheng Tao. Learning versatile filters for efficient convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 1608-1618, 2018b.
Junru Wu, Yue Wang, Zhenyu Wu, Zhangyang Wang, Ashok Veeraraghavan, and Yingyan Lin. Deep k-means: Re-training and parameter sharing with harder cluster assignments for compressing deep convolutions. International Conference on Machine Learning, 2018.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in 2015.
convolutional network. arXiv preprint
Zhaohui Yang. Yunhe Wang, Chuanjian Liu, Hanting Chen, Chunjing Xu, Boxin Shi, Chao Xu, and Chang Xu. LegoNet: Efficient convolutional neural networks with lego filters. In International Conference Machine Learning, pp. 7005-7014, 2019.
12

---

Under review as a conference paper at ICLR 2022
Sergey Zagonuyko and Nikos Komodakis. arXiv: 2016.
Wide residual networks.
arXiv preprint
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6848-6856, 2018
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep
features for discriminative localization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2921-2929, 2016.
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. International Conference on Learning Representations, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697-8710, 2018
13

---

Under review as a conference paper at ICLR 2022
APPENDIX
A IMPLEMENTATION DETAILS
All experiments are conducted on a server with 8 Nvidia RTX 3090 graphic cards, and each has 24GB memory. Experiments with few-shot image classification and CIFAR trained and tested single card. Experiments with ImageNet are trained with 8 cards in parallel. The machine is also equipped with 512GB memory and two AMD EPYC 7502 CPUs.
All experiments are performed using Py ToTch Paszke et al. 2019
Initialization. We train every ACDC network from scratch. We use orthogonal initialization for all filter atoms, and Kaiming normal initialization for all coefficients.
Training details. All networks for CIFAR 10 and CIFARIO0 are trained for 350 epochs. The initial learning rate to be 0.I. The learning rate decays by factor 10 the 150-th and the 250-th epoch. All ACDC networks are trained with a weight decay of 10-4 The reported numbers are averaged over 5 runs. Networks for ImageNet are trained for 90 epochs. The initial learning rate is 0.1 and decays by factor of 10 every 30 epochs. The weight decay for ImageNet experiments are set to be 2.5 10-5, We consistently observe that lower values for weight decay yield better results on ACDC networks. We believe the reason is that the structural regularization of ACDC already reduces the risk of overfitting.
B COMPARISONS OF ATOM-DROP
The proposed atom-drop shares the similar motivation with dropout, which is introduced to im- prove the robustness of networks weights by randomly closing activations in the training of the the deep network. Atom-drop introduced in ACDC a method to improve the robustness of the learned atoms at each layer. Most importantly, since atom-drop is performed on atoms, which are network parameters, and dropout performed on feature activations, they can be applied simulta- neously. We present in Table A the performance of ACDC-net m16 with WRN-40-4 and different dropping regularizations. Dropout- ld and dropout-2d correspond to randomly dropping pixels and feature channels, respectively. Using drop-atom alone delivers better performance comparing to both dropout- 1d and dropout-2d, and drop-atom is compatible with dropout- and dropout-2d for better performance.
Table A: Performance comparisons with different dropping regularizations.
dropout-14 ld
dropout-2d
alom-drop atom-drop + dropout-
atom-drop dropout-2d
Error
4.92
4.90
4.85
4.83
4.82
C COMPARISONS AGAINST NETWORK COMPRESSION AND PRUNING METHODS
We present here further comparisons against network compression and pruning methods. Perfor- mance is measured with error rates and proportion of remained parameters (both lower the better).
Note that different from post-processing using compression and pruning, ACDC is primarily pro- posed as a structural regularization and plug-and-plle replacement to standard convolutional lay-
ers, so the networks with ACDC remain trained end-to-end. We include SNIP (Lee et al.. 2018). TR (Wang et al.) (2018)), and recently proposed T-Basis (Obukhov al. 2020) and GraSP (Wang al.) 2020) into the comparisons. Results shown in Figure A Alclaaryy demonstrate that ACDC can achieve even smaller parameter size with negligent scarification to accuracy.
D COMPARISONS ON SPEED
We present the comparisons on speed against some recent efficient network structures in Table All numbers obtained by average of 10,000 runs. Each contains batch of 100 samples

---

Under review as a conference paper at ICLR 2022
22.5
ACDC-net m8
20.0
ACDC-g-net m8 s64
ACDC.g-black m16 s64 T-Basis ICML2020 CVPRR2018
17.5
Error
ICLR2018
12.5
GraSP ICLR2020
10.0
7.5
5.0
10-²
10-1
10
% of parameter
Figure Comparisons against network compression and pruning methods on CIFAR-10 dataset. Performance is measured with error rates and proportion of remained parameters (both lower the bet- Aes).As a plug-and-play method, ACDC can outperform network compression and pruning methods in terms of both parameter reduction rate and prediction accuracy.
with a resolution of 224 x 224, which saturate the GPU cores. While fast speed is not the primary goal of the paper, our joint subspace modeling allows comparably fast speed with the state-of-thes the-art efficient network structures.
Networks
Speed (ms)
ResNet18 (He et al. 2016
25.8
MobileNet-v3 (Howard all
2019)
23.4
ShuffleNet Zhang et al. (0088
13.3
SqueezeNet (landola ct al
2016)
25.9
ResNet-18 ACDC s64
18.6
ResNet-18 18 ACDC m16
16.4
Table B: Comparisons on speed against some recent efficient network structures.
Training stages
1, ACDC-net net m=8 2. ACDC-block m=8
3. ACDC-block m= 12
Parameters Error
3.57
4.59
6.74
41.76
40.28
39.06
Table C: Experiments with expanding network capacity.
E ARCHITECTURE ADAPTIVITY
ACDC offers a variety of variants with varying degrees sharing and performance. The selection of the variants can be determined based on the factors like the size and difficulty of the datasets at hand. Furthermore, the shared atom-coefficients decomposition for all ACDC variants allows for easy transfer and, as result, an adaptive model adaptation scheme. For example, one start training the network with coefficients sharing across all layers (ACDC-net) with relatively small number of atoms. And the network capacity can be gradually increased by relating the coefficient sharing blocks only and adding more atoms. Meanwhile, the majority of the parameters can be inherited from the earlier trained model to guarantee training efficiency. To show this, we use WideResNet (WRN-28-101 as the backbone and train an expanding model on the TinyImageNet dataset. As shown in Table C in stage 1, we start with training ACDC-net m=8. When further performance improvement is desired, we can relax the sharing by using the parameters obtained with ACDC-net m=8 to initialize ACDC-stage m=8 and continue training the network at stage 2. Similarly, further expand the network ACDCCstage m= /2 with most of the parameter inherited from ACDC-stage m=8.
2


Reviews:
[{"review_id": "pqqbaJeCg3I", "reviewer": "Reviewer_r8DN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n1.\tMotivation is well defined. By suggesting motivating experiment, the method of the paper could be verified theoretically as well as experimentally.\n2.\tThe method is simple but very strong. The method is just decomposing the shared atom coefficient of the layers in CNN, then they convolute images only with layer-specific coefficients followed after matrix-vector multiplication with respect to shared coefficients. Furthermore, by using group convolution, they accelerate the speed of the algorithm as in Tab.4. Even with fast computation, it shows good performance at network reduction, with at least maintaining or improving the error ratio.\n3.\tThe method can be compatible with various case. Since there are additional methods to be combined with the proposed such as block (section 2.3.) and group (section 2.4.) approaches, it can have a lot of variants.\n\nCons\n1.\tThere is a lack of reasons for using maximum value of C_in and C_out in the section 2.3. If you already have ablation study on that, it would be better to have in text, somewhere.\n\nMinors\n1.\tThe caption of Figure 1 is same as the third paragraph of the Introduction (replica). If the authors provide additional description of the figure, it will be clearer.\n2.\tThe authors need to explain about the bottom block in table 2. The description of third paragraph of section 3.1 has no referred table or figure. (The former is likely results about your methods and the latter seems to be an analysis about table 2.)  Also, the author needs to clarify the meaning of y-axis in figure 5.\n3.\tThere is a typo in Fig. 4 : D_{n_1}\\in\\mathbb{R}^{k\\times l\\times l} -> D_{n_1}\\in\\mathbb{R}^{m\\times l\\times l}\n4.\tThe organization of the manuscript is uncommon; a section of related work is located after result section.\n", "summary_of_the_paper": "This paper introduces a joint subspace view between the layers of convolutional neural network. Since the important features across the layers in deep CNN is maintained, the filters can be decomposed into shared coefficients and layer-specific coefficients. Motivated by multi-scale decomposition, they compose the shared coefficient in order to exploit correlations between features maximally. Also, by using group convolution, they reduce the amount of parameters as well as maintain the performance. This method is easily compatible to modern CNN architectures.", "summary_of_the_review": "Overall, the paper is hard to follow the logic because of unusual organization. However, the novelty of the proposed method is fully convinced with supported experiments and their explanations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "otcrh7WvBI2", "reviewer": "Reviewer_vUb9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Depending on the low-rank structure observation, this paper proposes to jointly model filter subspace across different layers by enforcing cross-layer shared subspace coefficients. The key idea is that all convolutional filters are decomposed into a shared coefficient submatrix and special filter atoms. \n\n+ves: \n+ The idea of the shared subspace coefficient for reducing the parameter redundancy seems interesting. \n+ This paper is well-written and is easy to follow.\n\nConcerns: \n\n- The novelty of this paper is weak. In fact, the idea that the convolutional filter is considered a multiplication of two small (low-dimensional) submatrixes has been proposed in [ref-1]. Of course, this paper proposed a shared coefficient submatrix. However, this novelty is not enough for the standard of ICLR.\n- The experimental results are not surprising. For the small datasets (e.g., CIFAR-10, CIFAR-100, and TinyImageNet), they only need a small deep model essentially although the proposed method is to reduce a large number of the parameters. For the large datasets (e.g., ImageNet), the proposed method is to reduce a small number of the parameters. Moreover, many related works (e.g., ShuffleNet-V2,MobleNet-V2) are missing to compared with the proposed method on ImageNet.\n-I am sincerely hoping that the authors deeply think more important problems in the topic of speeding up deep networks.\n\n[ref-1] Speeding up Convolutional Neural Networks with Low Rank Expansions, BMVC 2014.\n", "summary_of_the_paper": "To reduce the parameters of the deep networks, this paper proposes a joint subspace view to convolutional filters across network layers. Experimental results verify the effectiveness of the proposed method.", "summary_of_the_review": "I recommend the borderline rejection due to the limited novelty and unsurprising experimental results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jkh-Pa_Qk5_", "reviewer": "Reviewer_FvfC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n(1)\tThe idea of coefficient sharing across different settings (e.g., the whole networks, the convolution stages, convolution layers, and groups of filters) seem interesting.\n(2)\tThe experimental results show the proposed ACDC is superior or competitive to state-of-the-art methods.\n(3)\tThe paper is clear written.\n\nWeaknesses:\n(1)\tMy first concern is the settings of coefficient sharing. For CNN models, different convolution stages capture various kinds of features. For example, the preceding layers or blocks extract the fundamental cues (e.g., color or boundaries), while the latter layers or blocks learn the high-level semantic features. Although there exist some correlations in features across layers, correlations among different-level settings may be clearly different. Therefore, the best settings (e.g., the whole networks, the convolution stages, convolution layers, and groups of filters) of coefficient sharing may vary in various backbones and tasks. So, could the authors give the guideline or develop an adaptive method?\n(2)\tAs shown in Table 1, coefficient sharing across groups of filters have great effect on the final performance. Could the authors give more detailed discussion? Furthermore, the original backbone models with only groups of filters could be compared as baselines for verifying the effectiveness of ACDC-g. How about the performance of ACDC-g on ImageNet?\n(3)\tThe authors claim that the proposed method can provide better interpretation for CNN. However, it seems not very clear for me. Could the authors give more detailed discussion?  \n\nOther Comments:\n\n1.\tCompared to 2D convolutions, 3D convolutions need more optimization due to high computation complexity. Therefore, do the proposed method adopt to 3D convolutions (spatial and temporal information are coupled)?  Could the authors provide some tiny examples about 3D convolutions?\n2.\tSome typos, e.g., 13.49-> 13.49M and 8.57 -> 8.57M in Table 3.\n", "summary_of_the_paper": "This work first shows empirical observations that there exist obvious correlations in features across layers within a CNN after proper linear transformations. Accordingly, the authors propose coefficient sharing strategies across layers or across groups of filters, which aim to decompose convolution filters for reducing model sizes. The experiments are conducted on four image benchmarks using several CNN backbones, and the results show the effectiveness of the proposed ACDC on image classification and few-shot image classification tasks.", "summary_of_the_review": "The idea of coefficient sharing across different settings seem interesting, and a ok experimental results. However, there exist some weaknesses in this work (see main comments above). Therefore, my current rating is borderline accept.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T-cOBiz-UiO", "reviewer": "Reviewer_v8NG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The factorization approach for convolutional layers is exactly that proposed in DCFNet [Qui et al., 2018].  The difference is in terms of choosing to share channel mixing coefficients between multiple layers.  However, [Qui et al., 2018] focus on another means of parameter reduction -- namely, choosing a fixed basis for the filter atoms, and report accuracy-parameter efficiency improvements.\n\nAs the paper makes use of the same factorization as DCFNet [Qui et al., 2018], I would expect an experimental comparison between DCFNet and the proposed approached.  The paper also cites [Li et al., 2019] as another previous work that uses a filter basis factorization.  Moreover, Section 4.2 of [Li et al., 2019] proposes sharing the same basis across multiple convolutional layers.  Given this similarity in terms of ideas, it also seems necessary to experimentally compare to [Li et al., 2019] and provide further discussion of how the proposed approach and/or experimental results differ from these two prior works.\n\nFurthermore, though the paper does compare results to LegoNet [Yang et al, 2019] in Table 2, that comparison is uninformative as it applies the filter parameterization scheme of LegoNet to VGG, while applying its proposed scheme to Wide ResNet.  VGG vs Wide ResNet is a significant change in the base network architecture, which is a confound for the question of which filter parameterization is better; each method could be applied to either base architecture and the comparison should be made by choosing the same base architecture.\n", "summary_of_the_paper": "This paper proposes a scheme for partially sharing CNN parameter across layers.  Specifically, convolutional layer parameters are factored into the product of layer-specific filter atoms (which define spatial filter structure) and a set of shared coefficient that mix features across channels.  Training CNNs reparameterized in this manner yields improvements along the accuracy-parameter tradeoff curve, compared to baseline models on image classification tasks.", "summary_of_the_review": "This paper proposes a cross-layer parameter sharing scheme for CNNs that demonstrates some benefits on image classification tasks.  However, the proposed method is similar to filter factorization approaches in several previous published works, for which experimental comparison is missing or incomplete.\n\nUpdate (after rebuttal):\nThe author response addresses some of the experimental concerns from my initial review.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MTu_86zkFd1", "reviewer": "Reviewer_FSGC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n\n* The idea to use a low-rank decomposition of convolutional kernels and to consequently use the coefficients as sharable weights throughout a network is really interesting. This is especially true for the filter groups networks as the authors show a significant reduction in parameters and improvement in model performance (Table 1). Performance of the method when compared against a good number of _light_ CNN architectures on CIFAR-10/1000 + TinyImageNet is good and further demonstrates the performance.\n\n[Weaknesses]\n\n* The main motivation for the joint subspace (Section 2.2) is quite confusing in the opinion of this reviewer. It is difficult to understand why exactly one can exploit \"intrinsic, low-rank structure...across layers\" from the explanation and the pilot MNIST experiment. This is an important section that needs to be reworked.\n\n* The idea to share filter basis' across layers is not entirely new some and _shares_ similarites to https://arxiv.org/pdf/2006.05066.pdf. (Deeply Shared Filter Bases for Parameter-Efficient Convolutional Neural Networks). In that paper, they also learn a filter basis that can be shared across layers despite a different methodology. \n\n[Other]\n\n* In the filter groups model, how exactly does ACDC work? Is there a limitation that you need to have equal amounts of filter groups at every layer as the dimension of the shared coefficient is contingent on the groups _g_?\n\n* The decomposition method that is based on Qiu et al. 2018 is presented in Section 2.1. How much does this differ? Why not just use Qiu et al?\n\n* I would double check the notation for all subscripts as some $\\times$ symbols seem to be missing", "summary_of_the_paper": "This paper proposes a novel method to decrease redundancy of parameters in a CNN through a joint subspace view. The authors use a low-rank decomposition of the convolutional kernel (A * D) into shared coefficients (A) and atoms (D). The authors propose a variety of proposal methods for their method termed Atom Coefficient Decomposed Convolution. Notably by sharing A across all layers, sharing A across groups of layers and by utilising the idea of filter groups. The authors demonstrate that ACDC-x outperforms baselines at moderate to significant reductions in model size. Further, they show that their method can alleviate issues of model degradation as deeper networks are used.", "summary_of_the_review": "The idea to learn a joint subspace view to share weights across layers in a CNN is interesting especially with filter groups. They show good results on a variety of datasets against a solid number of baselines. Despite this, it is to my feeling that the idea is not entirely novel as i) low rank decomposition of convolution kernels has already been performed and ii) the idea to share filter basis/weights across layers has also been proposed. Further, the motivation/pilot experiment for the insight to sharing shared coefficients over layers needs to be worked on. I therefore propose 6 for now.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper eventually got 5 "marginally above the threshold" after rebuttal. Such scores testify to that the paper is a borderline one. By reading the post-rebuttal comments, it is evident that most of the reviewers still deemed that the novelty is incremental. One of the reviewer (vUb9) raised the score simply to "encourage the authors to think more important problems", rather than acknowledging the merits of the paper. The AC also read through the paper and had the following opinions:
1. The paper is actually about DNN compression, based on the "new finding" that the weights across layers are low-rank. However, the authors would not write the paper in the way of DNN compression, but put more emphasis on the "new finding", which has no theoretical support at all (only some heuristic reasoning). The AC would deem that the "new finding" is only an assumption.
2. Actually the "new finding" is not new at all. For example,

[*] Zhong et al., ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decomposition, Neural Networks, 2019, 

used a shared core tensor (which could be regarded as the common dictionary) across all layers for higher compression rates. More recent references that use tensors and consider shared information across layers for compression can be easily found as well.

So the AC thanked the authors for preparing the rebuttals carefully, but regretfully the paper is not good enough for ICLR.
Decision: Reject