Deeply Shared Filter Bases for  Parameter-Efficient  Convolutional Neural Networks
Authors: Woochul Kang
Daeyeon Kim
Venue: NeurIPS 2021 Poster (2021)
Keywords: parameter sharing
recursive networks
model compression
CNN
convolution networks
regularization
parameter-efficiency
efficiency

Abstract: Modern convolutional neural networks (CNNs) have massive identical convolution blocks, and, hence, recursive sharing of parameters across these blocks has been proposed to reduce the amount of parameters.  However, naive sharing of parameters poses many challenges such as limited representational power and the vanishing/exploding gradients problem of recursively shared parameters. In this paper, we present a recursive convolution block design and training method, in which a recursively shareable part, or a filter basis, is separated and learned while effectively avoiding the vanishing/exploding gradients problem during training. We show that the unwieldy vanishing/exploding gradients problem can be controlled by enforcing the elements of the filter basis orthonormal, and empirically demonstrate that the proposed orthogonality regularization improves the flow of gradients during training. Experimental results on image classification and object detection show that our approach, unlike previous parameter-sharing approaches, does not trade performance to save parameters and consistently outperforms over parameterized counterpart networks. This superior performance demonstrates that the proposed recursive convolution block design and the orthogonality regularization not only prevent performance degradation, but also consistently improve the representation capability while a significant amount of parameters are recursively shared.

Deeply Shared Filter Bases for Parameter-Efficiet Convolutional Neural Networks
Woochul Kang
Daeyeon Kim Dain Technology. Inc. dustin@@ouumdbleeeaatth... com
Department of Embedded Systems Engincering Incheon National University
Yeonsu-gu, South Korea 22012
vchkang@inu.as
Abstract
Modern convolutional neural networks (CNNs) have massive identical convolu- tion blocks, and, hence, recursive sharing of parameters across these blocks has been proposed reduce the amount of parameters. However, naive sharing of parameters poses many challenges such as limited representational power and the vanishing/exploding gradients problem recursively shared parameters. In this paper, we present a recursive convolution block design and training method, in which recursively shareable part, or filter basis, is separated and learned while effectively avoiding the vanishing/exploding gradients problem during training. We show that the unwieldy vanishing/explodingg gradients problem can be controlled by enforcing the elements of the filter basis orthonormal, and empirically demonstrate that the proposed orthogonality regularization improves the flow of gradients dur- ing training. Experimental results image classification and object detection show that our approach, unlike previous parameter srrameter--saring approaches, does not trade performance save parameters and consistently outperforms overparameterized counterpart networks. This superior performance demonstrates that the proposed recursive convolution block design and the orthogonality regularization not only prevent performance degradation, but also consistently improve the representation capability while significant amount of parameters recursively shared.
Introduction
Modern convolutional neural networks (CNNs) such as ResNets have massive identical convolution blocks and recent analytic studies (Jastrzebski et al. 2018) show that these blocks perform mostly iterative refinement of features rather than learning new features. Inspired by these results, recursive sharing of weights has been studied as a promising direction to parameter-efficien CNNs @astrzebski jet al., 2018 Guo al. 2019 Savarese Maire 2019). However, naive sharing of parameters across many convolution layers incurs several problems. First of all, recursive sharing of parameters can result in the vanishing and the exploding gradients problem, which is one of the main reasons that recurrent neural networks (RNNs) are so hard to train properly (Pascanu al., 2013 Jastrzebski (ct al. 2018). Another problem is that overall representation power can be limited by iterative sharing of parameters. Due these challenges, most compression approaches based on parameter-sharing suffer from performance degradation.
In this work, we conjecture that convolution layers or blocks can be separated into inherently shareable parts and oon-shareeble parts, and can be trained effectively by avoiding the vanishing and the exploding gradients problem. To achieve this, for full convolution operator, we first replace with a factorized version that splits the convolution operator into two separate operators; one
W. Kang and D. Kim contributed equally. This work conducted Incheon National University.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

operator with inherently shareable filters, called a filter basis, and the other operator with non-shared filters, called coefficients. When successive convolution blocks share a common filter basis, they are positioned in the same vector subspace. However, their representation capability is retained through non-shared coefficients that learn diverse features by linearly combining the shared filter basis.
By separating shareable parts from non-shareable parts, we can impose desirable properties on the shared parameters. To avoid performance degradation from recursive sharing of parameters, we propose the orthogonality regularization, in which the vanishing/explodee controlled by enforcing the elements of shared filter basis orthonormal during training. We both theoretically and empirically show that the proposed orthogonality regularization improves the flow of the gradients during training and reduces the redundancy in parameters effectively.
gradients problem is
For efficient CNNs such as MobileNets (Howard et al. 2017), we do not need to factorize convolution operators uncover shared filter basis since these networks already have factorized convolution block structures for computational efficiency. For such networks, our approach can be applied simply by identifying one or two convolution operators of repeating convolution blocks as a filter basis that shares weights across the repeating blocks. Other convolution operators in each convolution block become block-specific non-shared coefficients.
Since our focus is not on pushing the ofate-of-the-ae performance, we demonstrate the effectiveness of our work using widely-used models base models on image classification and object detection. Without bells and whistles, simply applying the proposed convolution block design and the orthog- onality regularization saves significant amount of parameters while consistently outperforming over-parameterized counterpart networks. For example, our method can save up to 46.0% of param- eters of ResNets while consistently achieving lower test errors. Even in compact models, such as MobileNetV2, our approach can achieve further 8-21% % parameter savings while outperforming the original models. This superior performance demonstrates that the proposed recursive convolution blocks and orthogonality regularization enables effective learning of better feature representations while a significant amount of parameters are shared recursively.
2 Related Work
Recursive networks and parameter sharing: Recurrent neural networks (RNNs) Graves et al.
2013) have been well-studied for temporal and sequential data. As a generalization of RNNs, recursive variants of CNNs are used extensively for visual tasks (Socher et al.] 2011 Liang &
Hu 2015 Xingjian al.] 2013) Kim et al.] 2016:] Zamir et al., (017). For instance, Eigen al.
(2014) explore recursive convolutional architectures that share filters across multiple convolution
layers. They show that recurrence with deeper layers tends to increase performance. However, their recursive architecture shows worse performance than independent convolution layers due to overfitting. contrast, we share only fundamentally shareable parts, or filter bases, that are separated from conventional convolution operators. By separating sharcable parts from non-shareable parts, we can impose desirable properties on the shared parameters that prevents the vanishing/cxploding gradients problems without damaging representational capability of the networks.
More recently, Jastrzebski al. (2018) show that iterative refinement of features in ResNets suggests that deep networks can potentially leverage intensive parameter sharing. Guo et al. (2019) introduce a gate unit to determine whether to jump out of the recursive loop of convolution blocks save computational resources. These works show that training recursive networks with naively shared blocks leads to bad performance due to the problem of gradient explosion and vanish like RNN (Pascanu et al. [2013] Vorontsov et al., (017)). In order to deal with the problem of gradient explosion and vanish, they suggest unshared batch normalization strategy. In our work, we propose the orthogonality regularization of shared filter bases to further address this problem.
few recent works generate convolution filters by combining shared building blocks (Bhalgat et al. 2020 (Savarese & Maire 2019: Yang et al., 2019 Qiu et al. 2018). Although these approaches are different details, they are similar to our work since they all share set of filters and combine them to build layer-specific filters. However, they save parameters at the cost of accuracy loss. For example, in ResNet50 on ImageNet, Yang et al. 22019 work incurs 0.9% accuracy drop, while our method achieves performance improvement. We believe that this performance gap comes from the gradients issue of shared parameters and these previous works could benefit more from the improved flow of gradients of our approach.
2

---

W
W basis
n times
iterate
Conv
basis
Shared filter basis
Conv
!
non-shared coefficients
(a) Full convolution
(b) filter basis and coefficients
(c) Iterative sharing a filter basis
Figure 1: A full convolution operator in (a) with filter W can be replaced by two separate convolution operators, as in (b): one operator with a filter basis (aste)) and the other operator with coefficients (as). A filter basis is inherently shareable for convolution operators in the same vector subspace, and, hence, a recursive architecture is suggested, as in (c). The filter basis shared by recursive convolution layers be learned through typical gradient- based training while avoiding potential vanishing/exploding gradients problem by enforeing its elements orthonormal.
Model compression and efficient convolution block design: Reducing storage and inference time of CNNs has been an important research topic for both resource constrained mobile/embeddee systems and energy- -hugrry data centers. A number of research techniques have been developed such as filter pruning (LeCun et al., 1990 Polyak & Wolf 2015 Li et al., [017 He et al., (017b)) low-rank factorization (Denton et al.) 2014; Jaderberg et al. 2014). and quantization (Han al. 2016). to
name few. Several model compression techniques factorize trained filters to reduce the computation complexity (Zhang et al., 2013 Li al. 2019a). Our proposed block structure also benefit from the reduced computational complexity too factorizing convolution filters. However, most previous compression techniques have been suggested as post-processing steps that are applied after initial training. Therefore, their accuracy is usually bounded by the approximated base models. In contrast, our primary goal in factorizing convolution operators is to find common filter basis shared by iterative convolution layers, the weights of the shared filter basis are learned from scratch, not from decomposing pretrained filters. Hence, unlike previous compression techniques, the performance of our approach is not limited by the original models, and the experimental results show that our parameter- sharring approach consistently outperforms the overparameterized counterpart networks on various datasets and tasks.
3 Proposed Method
In this section, we describe the design of recursive convolution blocks and discuss how to train them to deal with the vanishing/exploding gradients problem.
3.1 Recursive Sharing of a Filter Basis
Naive sharing convolution filters across repetitive layers degrades the overall performance due to limited representation power of individual layers. We argue that convolution filters be separated into fundamentally shareable components and non-shareable components, and the performance degradation can be prevented by sharing only fundamentally shareable parts.
More formally, we consider a convolution layer, shown in Figure T (a), that has S input channels, T output channels, and set of filters W = [W, E R ,t [..T]]. Each filter W, can be decomposed into a filter basis Whaeis and coefficients a:
W, = Σ 1=1
of WT Wa
(I)
where Whnsis {WT E x E R.. is filter basis, and = {af E R, E [I..R]
[I..T]] is scalar coefficients. In Equation 1. R is the rank of the basis. In a typical convolution layer, output feature maps V, Rwxhx? [1.TT] are obtained by the convolution between input feature maps U x and filters W., e [1.T]. With Equation this convolution can be rewritten as

---

follows:
1 U W U
aj W
basis
(U Wbasis): where [1.1]]
In Equation 2 the order of the convolution operation and the linear combination of filter basis is reordered according to the linearity of convolution operators. Therefore, a conventional convolution layer can be replaced with two successive convolution layers shown in Figure (b).
This factorized convolution filters suggest recursive architecture, shown in Figure (c), in which a common filter basis is used for iterative convolution layers. Since all filters in the recursive loop have a common filter basis, they are positioned in the same vector subspace. However, their representation capability is still retained since their coordinates in the subspace are diversified by their respective non-shared coefficients.
3.2 Orthonormality of Shared Filter Bases
T For recursive sharing of a filter basis, as in Figure (c), we need to find an optimal filter basis that be used by iterative convolution layers without loss of performance. Although this optimal filter basis can be searched by typical gradient-basee that the exploding/vanishin search of optimization space (Pascanu et al., 2013).
optimization such as SGD, one major problem is gradients problem of recursively shared filter bases prevent proper
More formally, we consider series of N factorized convolution blocks, in which a filter basis W Winsis shared N times, as in Figure (c). Let x be the input of the i-th convolution block, and a++ be the output of the convolution of x' with the filter basis Wbasis
at (x(x-1)
W Whasiss
W busii TR X is reshaped filter basis that has basis elements at its columns. We assume that input x is properly adapted (e.g., with im2col operations) to express convolutions using matrix-matrix multiplication. Since W Wassi is shared by N recursive convolution blocks, the gradient of Whosis for some loss function L is:
In Equation 3
aL
N
aL
N
0da+1
Oa'
II
(4)
aw basis
Nu
dui
OW WWasss
where
0a
0as+1
Ox!
Ox
basis
(5)
da
0x
Oai
Oa
If we plug Weasis doy gradients unstable since Wbasis multiplied many times. This aaploding/vanishhngg gradients can be controlled to large extent by keeping W Whesis close to orthogonal (Vorontsov et al. 2017). For
Ox in Equation into Equation 4 we can see that II 0a
8o? is the term that makes
instance, if W basis admits eigendecomposition, [W NNN can be rewritten as follows:
[Wbaeis] = [QAQ-IJ QANQ=1
(6)
where A is diagonal matrix with the eigenvalues placed the diagonal and is matrix composed of the corresponding cigenvectors. If Whesis is orthogonal, [Waests]* neither explodes nor vanishes, since all the cigenvalues of an orthogonal matrix have absolute value of 1. This result shows that a shared filter basis with orthonormal elements ensures that forward and backward signals neither explode nor vanish.
Based on this result, we propose the orthogonality regularization to enforce orthonormality to shared filter bases during training. For instance, when convolution operators in each residual block group of a ResNet shares a filter basis, the objective function LR can be defined to have the orthogonality regularization term in addition the original loss L:
-R= +
[Iw(() T. W
basis basss 111
TTecniiaally, the columns (or rows) of orthogonal matrix form an orthonormal basis. However, we use the term orthogonality regularization since orthonormal bases have proper term for their matrix-fomi

---

X
COTY 3x3
Shared filter basis
BN
Shared fllter
convix1
convix1
Shared filter
BN
ReLU
conv
ReLU
depthwise
conv 3x3
conv 3x3
3x3
BN. ReLU
Shared repeating blocks
BN ReLU
Shared by
Shared by repeating conv. blocks
convixt
convixt
repeating
convix1
BN
conv. layers
ReLU
(a) Shared block for ResNet34
(b) Shared block for ResNet50
(c) Shared block for MobileNetV2
Figure 2: The structure of residual blocks with a shared filter basis. For ResNet34, two full convo- lution operators are factorized to uncover shared filter basis. For ResNets with bottleneck blocks (ResNet50/101) and MobileNetV2, one or two computationally expensive convolution operators of each convolution block are selected as shared filter bases and the other convolution operators are designated as non-shared coefficients, In (a), the filter basis has non-shared elements to further increase representation capability.
where Wo(a) basis is shared filter basis for g-th residual block group and a hyperparameter.
In Equations 4 and 5 we also need to ensure that the norm of 8x Buy is bounded for stability during forward and backward passes (Pascanu et al.) 2013). is shown that batch normalization after non- linear activation at each convolution layer ensures healthy norms (loffe & Szegedy) 2015 Guo et al.) 2019 Dastrzebski et al., (2018). In Section 4.4], we empirically show that the proposed orthogonality regularization and batch normalization similarly improve the flow of gradients during training.
3.3 Enhancing Representation Capability
When convolution operators share common filter basis, they all in the same vector subspace. Therefore, if the rank of the filter basis is too low, it might limit the representation capability of the convolution operators sharing the filter basis. Conversely, if the rank of the shared filter basis is too high (e.g.. R 2 T), the computational gain of factorized structure mitigated. One way to increase the representational power of each convolution operator, while still maintaining its computational complexity low, is placing the convolution operators in different subspace by adding small number of non-shared elements the filter basis. For instance, we build filter basis Whasis by combining shared elements and non-shared elements:
W Wassis Wbs_shared Wbs uniquer
(8)
where Wos Wbs_shared (Wr
bs shared E RR , E [1..n]} are shared filter basis elements, and Wh Wbs_unique {W7r unique R , E [n+11..R] are non-shared filter basis elements. For example, Figure (a) shows that the filter basis of ResNet34 is compose of both shared and non- shared elements. One disadvantage
this hybrid scheme is that non-shared filter basis elements require parameters. The ratio of non-shared basis elements
be varied to control the tradeoffs. But, our results in Section show that providing only a few non-shared elements to a filter basis is enough to achieve high performance.
4 Experiments
In this section, we perform series of experiments on image classification and object detection using several modern networks base models. We also analyze the effect of the orthogonality regularization.
For ResNets with conventional convolution filters (e.g. ResNet34), filter is replaced by the proposed factorized convolution block as shown in 2a). filter basis shared within a residual block group having same kernel dimensions. Throughout the experiments, we denote by ResNet a ResNet with L, layers that has a filter basis with a s-to-u ratio of shared elements and non-shared elements.
5

---

ResNets with bottleneck blocks (e.g.. ResNet50) and MobileNetV2 already have decomposed block structures. Therefore, we designate one or two convolution operators with the largest parameters in each block as filter bases sharing weights across iterative blocks, shown in Figure (b) and -(c). During the training, the proposed orthogonality regularization applied to enforce the shared filter bases orthonormal. In these models, we leave at least one convolution operator as non-shared coefficients to improve expression ability. In Section 4.1 both filter basis and coefficients in residual groups.
elements of
we explore the effect of sharing
During experiments, programs for training and evaluation run on PCs equipped either with four RTX 2080Ti GPUs or two RTX 3090 GPUs and an Intel i9- 10900X CPU @3.7GHz.
4.1 Image Classification on ImageNet
We evaluate method on the ILSVRC2012 dataset (Russakovsky et al. 2015) that has 1000 classes. The dataset consists 1.28M training and 50K validation images. We use ResNets and MobileNetV2 as base models. We train the ResNet-derived models for 150 epochs with SGD optimizer with mini- batch size of 256, a weight decay le-4, and a momentum of 0.9. The learning rate starts with 0.1 and decays by 0.1 60-th, 100-th, and 140-th epochs. MobileNetV2 and our MobileNetV2-SShane models are trained for 300 epochs with a weight decay of le-5. Its learning rate starts with 0.1 and decays by 0.1 at 150-th, 225-th, and 285-th epochs.
Table 1: Error (%) ImageNet. In ResNet50/101 SShared and MobileNetV2-SSheee -h-hhrred" first two convolution operators each residual block are designated as recursively shared filter bases. In ResNet50-Sharee
both filter basis and coefficients are shared recursively across blocks of each residual group. In DesNet50-Shared-NoOrti LesNet50-Shared during training. Latency is measured the Nvidia Jetson TX2 embedded board (GPU, batch size = 1).
orthogonality regulaization is not applied to
Baseline
Model
Params
FLOPs
top-1
top-5
Latency
ResNer34 (baseline)
21.80 21.80M
3.67G
26.70
8.58
33.6ms
ResNet34
Filter Pruning (Li ct al. 2017
19.30M
2.76G
27.83
ResNet34-S40e (ours)
11.79M
3.26G
26.67
8.54
38.6ms
ResNe150 (haseline)
25.56M
4.11G
23.85
7.13
43.8ms
Versatile-Re<Ne5500 (Wang et al.
2018)
19.0M
3.2G
24.5
7.6
FSNet (Yang al. 2020
13.9M
26.89
8.63
ResNet50
ResNeeSU-Shared (ours)
20.5
4.11G
23.64
6.98
43.3ms
SelNet50-Sharede (ours)
18.26M
4.11G
23.95
7.14
43.3ms
SAANet5--Shared
16.02M
4.11G
24.35
7.41
SanNet5O-Shared -oored-N
20.5
4.11G
24.19
7.34
ResNet101 (baseline)
44.55M
7.83G
22.63
6.44
73.2ms
ResNet101
ResNet101-Shann (ours)
29.47M
7.83G
22.31
6.47
72.9ms
MobileNetV2 (baseline)
3,50M
0.33G
28.0
9.71
18.4ms
DR- RR-MobileNert2 (Guo et al. 2019
2.96M
0.27G
28.2
9.72
MobileNetV2
MobileNetV2-Sharee (ours)
3.24M
0.33G
27.61
9.34
17.9ms
MobileNerV2-Sharede (ours)
2.98M
0.33G
28.21
9.85
17.8ms
In Table [ we compare our results with competing techniques. Most parameter-sharing and com-
pression techniques save parameters at the cost of performance. Unlikely, the results in Table T show that our approach consistently outperforms the counterpart networks while saving significant amount of parameters. For example, ResNet34-S48UI outperforms the counterpart ResNet34 while only using 54.0% parameters. Since our models derived from ResNet50/10 101 and MobileNetV2 already have factorized convolution blocks, their overall parameter-saving not high as ResNet34- S48U1's. However, they still save about 19.8%, 33.9% and 7.5% parameters, respectively, while outperforming the baselines. The benefit of our method is more pronounced deeper networks. In ResNet101, for example, filter basis is shared by up to 23 recursive convolution layers, resulting in 35.7% reduction of parameters. In Table T ResNet50-Shareee Shared-NoOrrtool not applying the orthogonality regularization on ResNet50-Sharee accuracy drops to 75.81 %, that is 0.34% lower than the counterpart ResNet50. In contrast, when the orthogonality regularization is applied, the same model achieves 76.35%, which is 0.21% higher than
shows the effect of Our result shows that its top1

---

the counterpart ResNet50. This 0.55% improvement is obtained simply by applying orthogonality regularization. In Table SosNNet0--Share-AA All shows the effect of sharing not only a filter basis but also coefficients recursively within residual groups. The result shows that we can save further 18% parameters (reduced from 20.51M to 16.02M) of BesNee50-Shhaee but its top-1 (top-5) accuracy drops to 75.65% (92.59%), which is 0.5% lower than the counterpart ResNer50. This result shows that having unshared coefficients is important for high performance. Yang et al. (2019) also demonstrates that sharing coefficients drops the accuracy by additional 2%.
Although CSsNee34-S48UT requires lower FLOPs than the counterpart ResNet34, it takes 14% longer latency on Jetson TX2. This overhead mostly comes from separated convolution operations for shared and non-shared filter basis elements. Current GPU-based deep learning libraries are not optimized to process such separated operations efficiently. Unlikely, ResNet50/101 -Shared and MobileNerV2-Shacce do not have such separated convolution operations, and, hence, their latency slightly lower on the device that is constrained by limited cache and memory.
4.2 Image Classification on CIFAR
We evaluate the effectiveness of our method various modern CNNs with the CIFAR dataset that has 50,000 and 10,000 32 x 32 images for training and testing. respectively. For training networks, we follow similar training scheme in He et al. (2016). Standardized data-augmentation and normalization are applied to input data. Networks are trained for 300 epochs with SGD optimizer with weight decay of 5e-4 and momentum of 0.9. The learning rate is initialized 0.1 and is decayed by 0.1 at 50% and 75% of the epochs.
Table Error (%) on CIFAR-100. *** denotes that the orthogonality regularization is not applied.
Baseline
Model
Params
FLOPPs
Error
ResNet34 (baseline)
21.33M
1.17G
22.49
ResNet34
ResNec34-S32U1 (ours)
7.73M
0.78G
22.92
ResNet34-S32U1 (ours)
7.73M
0.78G
21.84
DenseNet121 (baseline)
7.05M
0.91G
21.95
DenseNet121
DenseNet121-S16U) (ours)
5.08M
0.72G
22.15
ResNeXt50 (baseline) ResNeXt50-S6444 (ours)
23.17M 19.3M
1.36G
20.71
ResNeXt50
1.19G
20.09
Table Error (%) on CIFAR-10. + denotes having shared bases in each convolution block group. 'x' denotes that the orthogonality regularization is not applied.
Baseline
Model
Params
FLOPs
Error
ResNet32 (baseline)
0.46M
0.07G
7.51
ResNet32
ResNec32-S11611 (ours)
0.24M
0.08G
6.95
ResNet56 (baseline)
0.85M
0.16G
6.97
Filter Pruning (Li al.
2017
0.77M
0.09G
6.94
ResNet56
KSE (Li et al. 2019b)
0.43M
0.06G
6.77
DR- (G et al. 2019
0.50M
0.11G
6.51
ResNee566SS1ou1 (ours)
0.27M
0.15G
7.70
ResNet566S16UJ (ours)
0.27M
0.15G
7.46
ResNee566$11611 (ours)
0.31M
0.15G
6.33
Table 2 shows the results on CIFAR-100, Networks trained with the proposed method consistently outperform their counterparts. For instance, ResNNe34-S32U1 requires only 36.2% parameters and 66.6% FLOPs of the counterpart ResNet34 while achieving lower test error (21.8 11.8%%) than much deeper ResNet50 (22.36%). ). To show the generality of our work, we apply the proposed method to DenseNet (Huang et al., 2017). and ResNeXt (Xie et al. 2017). Although the overall gain is not as pronounced as ResNets we still observe reduction resource usages in these networks. For instance, ResNeXt50-S6644
outperforms the counterpart Re50eXt50 while saving parameters and FL.OPs by 16.7% and 12. 11%%% %, respectively. In ResNeXt, the gain limited since they mainly exploit group convolutions; each group convolution is decomposed for filter basis sharing in our network.
7

---

The result on CIFAR-10 with ResNets is presented in Table 3 Unlike networks on CIFAR- 100, networks on CIFAR-I0 10 has much fewer channels (e.g. 16 channels the first residual block group) and, hence, projecting filters to such low dimensional subspace might limit the performance of the networks. For instance, in ResNet32-S8UTT filters are supposed to projected onto 9 dimensional subspace consisting of 8 shared and 1 non-shared filter basis elements. Further, for deeper networks such as ResNet56, a filter basis is supposed to be shared by many residual blocks the group, and it can damage the performance. For example, every filter basis in ResNet56 S661 is shared by 8 residual blocks, or convolution operators. Due to this excessive sharing, though ResNet56-S) $16U1 saves 41.3% parameters, its testing enror (7.46%) is higher than the counterpart ResNet56's ' (6.97%)
To remedy this problem, we introduce a variant, in which each residual block group of the networks uses 2 shared bases; one basis is shared by the first convolution operators of recursive residual blocks, and the other basis is shared by the second convolution operators. In Table 3 networks with a * mark denote this variant. Though this variant slightly increases the parameters of the networks, can prevent excessive sharing of parameters. For example, although ResNet56-S16U1 needs 0.04M more parameters for additional shared bases, it still saves 63% parameters of the counterpart ResNet56 and achieves lower testing error of 6.33%.
In Table 3 we compare our results with similar state-of-the-ar techniques. Our method achieves better performance and parameter-saving than other approaches such as filter pruning (Li et al. 2017). kernel clustering (Li et al. 2019b), and recursive parameter sharing (Guo et al. 2019).
4.3 Object Detection on MS COCO
In order to explore the generalization ability of our approach, we next use COCO 2017 dataset on object detection task using Faster R-CNN (Ren et al., 2017), Mask R-CNN (He et al., 201 7a), and RetinaNet (Lin et al. (017) as detectors. We compare ResNe150/101 and our ResNet50/101-Shi backbone networks of the detectors. These backbone networks are pre-trained on ImageNet, then are transferred to MS COCO by fine-tuning. We use MMDetection (Chen et al., 2019) toolbox and employ default settings for training and evaluation. All networks are trained on train2017 for 12 epochs using SGD with weight decay of le-4, momentum of 0.9 and mini-batch size of 8 (2 examples per GPU). The learning rate initialized to 0.01 and decays by 0.1 8-th and 11-th epochs.
Table 4: Object detection results on COCO 2017 validation set. ResNet50-Sharee uses first two convolution operators of each residual block recursively shared filter bases.
Backbone
Detector
#Params
GFLOPs
AP
APau
APts
APs
APM
AP
ResNer50 (baseline)
41.53M
207.07
36.4
58.2
39.2
21.8
40.0
46.2
-Shared (ours)
36.70M
206.87
37.2
58.1
40.2
21.6
40.8
47.9
Faster R-CNN
-Shared (ours)
34.46M
206.87
36.6
57.4
39.8
21.4
40.1
47.3
ResNet101 (baseline)
60.52M
283.14
38.7
60.6
41.9
22.7
43.2
50.4
Shared (ours)
45.67M
282.84
39.0
59.7
42.7
22.4
42.7
50.8
ResNe150 (baseline)
44.18M
275.58
37.2
58.9
40.3
22.2
40.7
48.0
-Shared (ours)
39.35M
259.94
37.9
58.4
41.4
22.4
41.2
49.2
Mask
-Shared (ours)
R-CNN
37.10M
259.94
37.3
57.6
40.7
21.5
40.3
ResNet101 (baseline)
67.17M
351.65
39.4
60.9
43.3
23.0
43.7
$1.4
-Shared (ours)
48.31M
335.91
39.8
60.3
43.6
22.7
43.5
51.9
ResNet50 (baseline)
37.74M
239.32
35.6
55.5
38.2
20.0
39.6
46.5
-Shared (ours)
32.92M
239.12
36.2
55.0
38.6
20.3
39.7
47.1
-Shared (ours)
RetinaNet
30.67M
239.12
35.6
54.1
38.2
19.8
39.2
46.9
ResNet101 (baseline)
56.74M
315.39
37.7
57.5
40.4
21.1
42.2
49.5
-Shared (ours)
41.88M
315.09
37.7
56.8
40.3
21.2
41.5
49.5
Table 5: Instance segmentation results using Mask R-CNN on COCO va12017.
Backbone
AP
APwo
APTo
APs
APM
APt
ResNet50 (baseline)
34.1
55.5
36,2
16.1
36.7
S0.0
-Shared (ours)
34.5
55.4
36,9
18.9
37.7
46.5
ResNettO1 (baseline)
35.9
57.7
38.4
16.8
39.1
53.6
-Shared (ours)
35.9
57.2
38.3
19.1
39.2
48.6
8

---

Table shows the results on va12017 containing 5000 images. Our backbone networks with shared filter bases consistently outperform baselines in detectors in terms of COCO's standard metric AP while saving up to 28. % parameters. In Table 5 our models also achieve similar performance improvement in instance segmentation using Mask R- R-CNN. This consistent performance improvement is obtained simply by replacing the backhone networks with ours and it demonstrates that the proposed recursive convolution block design and the orthogonality regularization enables learning better feature representations with a smaller amount of parameters.
4.4 Analysis: Effect of Orthogonality Regularization
To investigate the effect of the orthogonality regularization during training, track the flows of gradients while training ResNet34-S16U1. Figure showss the traces of the maximum and the mean gradients flowing in the filter bases during epoch. On every 10 iterations of batch, the maximum and the mean gradients are overlapped on top of the old plots. Therefore, the bars look darker bars are more overlapped. Jastrzebski al. (2018) and Guo al. (2019) showed that unshared batch normalization (BN) strategy mitigates the vanishing/explodee result in Figure (-bb) shows that unshared BNs following shared filter bases improve the flow of gradients. When the proposed orthogonality regularization is applied to the shared filter bases, in Figure (c), similar effect is observed on gradients. When both unshared BNs and the orthogonality regularization are applied together, in Figure (d), further stronger, but still bounded. flows of gradients are observed. This trend consistently observed during training. We conjecture that this healthy flow of gradients enables learning better feature representations during optimization process, resulting in superior performance on various tasks.
gradients problem, and our
0.0200
max- max-gradient
0.0175
Average gredients
mean-gradient
0.0150
0.0125
0.0300
3.0050
0.0925
5.0000
Non-orthonomal bases.
(b) Non-arthanomal bases. BNS after bases
Orthanormal
Orthonarmal bases. With
Without
bases
Figure 3: The flows of gradients in 4 shared filter bases of ResNec34-S16U1 the epoch. For comparison, the orthogonality regularization and the batch normalization (BN) following the filter bases are turned on and off. In (b) and (c), BNs and the orthogonality regularization, respectively, improve the flow of gradients. In (d), when both BNs and orthogonality regularization are applied together, the strongest flow of gradients is observed. This trend consistently observed during the training.
0.10
0.25
200
0.20
300
200
150
500
150
500
200
203
(a) Without orthogonality regularization
With orthogonality regularization
Figure Cosine similarities coefficients of the 2nd and the 3rd residual block groups in ResNet34. $16U1. X and Y axes are indexes to the coefficients of the residual block groups sharing filter bases. Brighter colors indicate higher similarity. In (b), when the orthogonality regularization is applied, the similarity is clearly lowered, implying less redundancy in parameters.

---

To further analyze the effect of the orthogonality regularization, in Figure we illustrate absolute cosine similarities all coefficients of the 2nd and the 3rd residual block groups of ResNet34-S16UL. The X and Y axes display the indexes to the coefficients in the residual blocks. In Figure (b), we can clearly see that coefficients manifest lower similarities when the orthogonality regularization is applied. In Figure (a), when the orthogonality regularization is not applied, an interesting grid pattern is observed in Figure (a). This repetitive grid pattern might be related to ResNets' nature of iterative feature refinement (Jastrzebski et al. 2018). However, such high cosine similarity is directly related to the higher redundancy in the networks. When our orthogonality regularization applied, such repetitive patterns are less evident, implying that the recursive convolution layers learn better feature representations with less redundancy.
Conclusions
We introduce a recursive convolution block design and effective training method for parameter- efficient CNNs. this work, a common filter basis, shared by repeating convolution layers, is learned while effectively avoiding the vanishing/cxploding gradients problem through the proposed orthogonality regularization. Experimental results on image classification and image detection show that our approach consistently outperforms over- wer-pprramttrrized counterpart models while significantly saving parameters. This consistent performance improvement demonstrates that the proposed approach enables effective learning of better feature representations while significant amount of parameters are shared. We believe that the proposed recursive convolution blocks and training method suggests important possibilities for neural architecture search (NAS) explore resource-efficient CNNs.
Acknowledgments and Disclosure of F Funding
We thank the anonymous reviewers for their constructive comments and suggestions. This work was supported by the National Research Foundation of Korea (NRF) Grants Funded by the Ministry of Science and ICT under Grant NRF-2019RIF1A 1060959.
References
Bhalgat, Y., Zhang, Y., Lin, J. M., and Porikli, F. Structured convolutions for efficient neural network design. Advances Neural Information Processing Systems, 33:5553-5564, 2020.
Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S.. Feng. W., Liu, Z., Xu, J., Zhang, Z.. Cheng, D., Zhu, C., Cheng. T., Zhao, Q.. Li, B., Lu, X., Zhu, R.. Wu, Y., Dai, J., Wang, J., Shi, J., Ouyang, W., Loy, C. C., and Lin, D. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint 1xX.::/906.07155 2019.
Denton, E. L., Zaremba, W., Bruna, J., LeCun, Y., and Fergus, R. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in neural information processing systems, pp. 1269-1277, 2014.
Eigen, D., Rolfe, J. T., Fergus, R., and LeCun, Y. Understanding deep architectures using recursive convolutional network. In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
Graves, A., Mohamed, A., and Hinton, G. Speech recognition with deep recurrent neural networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. -644, 2013.
Guo, Q., Yu, Z.,, Wu, Y., Liang, D., Qin, H., and Yan, J. Dynamic recursive neural network. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 5142-5151, 2019.
Han, S., Mao, H., and Dally, W. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
10

---

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
He, K., Gkioxari, G., Dollar, P., and Girshick, R. Mask r-cnn. 2017 IEEE International Conference Computer Vision (ICCV), Oct 2017a.
He, Y., Zhang, X., and Sun, J. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397,200
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W. Weyand, T., Andreetto, M., and Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1X04:1704.444 004611, 2017.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
loffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 456, 2015.
Jaderberg, M., Vedaldi, A., and Zisserman, A. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv: 3405.866,, 2014.
Jastrzebski, S., Arpit, D., Ballas, N., Verma, V., Che, T., and Bengio, Y. Residual connections encourage iterative inference. In International Conference on Learning Representations, 2018.
Kim, J., Kwon Lee, J., and Mu Lee, K. Deeply-recursive convolutional network for image super- resolution, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1637-1645, 2016.
LeCun, Y., Denker, J. S., and Solla, S. A. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Li, H., Kadav, A., Durdanovic, L. Samet, H., and Graf, H. P. Pruning filters for efficient convnets. In International Conference Learning Representations, 2017.
Li, Y., Gu, S., Van Gool, L., and Timofte, R. Learning filter basis for convolutional neural network compression. In Proceedings of the IEEE International Conference on Computer Vision, 2019a.
Li, Y., Lin, S., Zhang, B., Liu, J., Doermann, D., Wu. Y.. Huang. F., and Exploiting kernel sparsity and entropy for interpretable enn compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2800-2809, 2019b.
Liang, M. and Hu, X. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3367-3375, 2015.
Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, 2017.
Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318, 2013.
Polyak, A. and Wolf, L. Channel-level acceleration of deep face representations. IEEE Access, 3: 2163-2175, 2015,
Qiu, Q., Cheng, X., Sapiro, G., et al. Defnet: Deep neural network with decomposed convolutional filters. In International Conference Machine Learning, pp. 4198-4207. PMLR, 2018.
Ren, S., He, K., Girshick, R., and Sun, J. Faster r-enn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, Jun 2017.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,, Karpathy, A., Khosla, A., Bernstein, M., et Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252. 2015.

---

Savarese. P. and Maire, M. Learning implicitly recurrent CNNs through parameter sharing. In International Conference on Learning Representations. 2019.
Socher, R., Lin, C. C., Manning, C., and Ng. A. Y. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 136.136, 2011.
Vorontsov, E., Trabelsi, C., Kadoury, S., and Pal, C. On orthogonality and learning recurrent networks with long term dependencies. In Proceedings of the 34th International Conference Machine Learning, pp. 3570-3578, 2017.
Wang, Y., Xu, C., Chunjing, X., Xu, C., and Tao, D. Learning versatile filters for efficient convolu- tional neural networks. In Advances in Neural Information Processing Systems, pp. 1608-1618, 2018.
Xie, S., Girshick, R., Dollár, P., Tu, 7.,, and He, K. Aggregated residual transformations for deep neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 1492-1500, 2017.
Xingjian. S., Chen, Z.. Wang, H., Yeung, D.-Y., Wong, W.-K., and Woo, W.-. Convolutional 1stm network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems, pp. 802-810, -10, 2015.
Yang, Y., Yu, J., Jojie, N., Huan, J., and Huang, T. S. Fsnet: Compression of deep convolutional neural networks by filter summary. In International Conference on Learning Representations, 2020.
Yang. Z. Wang. Y., Liu, C., Chen, H., Xu, C., Shi, B., Xu, C., and Xu, C. LegoNet: Efficient convolutional neural networks with lego filters. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7005-7014, PMLR, 09-15 Jun 2019.
Zamir, A. R., Wu, T.-L., Sun, L., Shen, W. B., Shi, B. E., Malik, J., and Savarese, S. Feedback networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- IEEE, 2017.
Zhang, X., Zou, J., He, K., and Sun, J. Accelerating very deep convolutional networks for classi- fication and detection. IEEE transactions on paltern analysis and machine intelligence, 38(10): 1943-1955, 2015.
12


Reviews:
[{"review_id": "zRovRNTL0RV", "reviewer": "Reviewer_RBVx", "summary": "The authors introduce a new parameter sharing scheme in CNNs by learning orthogonal filter bases, and representing each convolutional filters as a linear combination of the bases. \nNotably, the authors show both empirically and theoretically that orthogonal regularization imposed on the learned filter bases improves the stability of gradients and helps achieve better overall performance.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While learning shared filter basis for parameter reduction in deep neural networks is not a new idea, the discussion on the orthogonality of the bases and its advantages on gradient flow is interesting and constructive. \nThe introduced method is well presented and the paper is easy to follow, with source code provided for reproduction. \nWide range of experiments including image classification, object detection, and instance segmentation, are appreciated. \n\nI have following concerns:\n- As discussed in L81, the authors claim gradient issues cause the performance drop in [1], any support on this?\n\n- Following the previous question, the main contribution of this paper remains a bit unclear to me. If the orthogonal regularization is indeed a principle way of stabilizing the gradient and improve the performance of parameter sharing in CNNs, can it be apply to other methods for improved performance? Or it can only improve the performance of the proposed method?\n\n- How are all the hyperparameters determined? Further ablation study is expected, e.g. the additional orthogonality regularization inevitably introduces new hyperparameter for tuning.  \n\n- Missing references and comparisons [2, 3, 4]. These are all published parameter sharing methods for networks and need to be discussed in the paper. Comparisons against [1] are also expected. \n\n- The term 'Deeply' in the proposed method is a bit confusing to me. It seems that the authors are trying to emphasize 'deep', then how is the proposed method different from the previous parameter sharing methods in terms of 'deep'? \n\n[1] Learning implicitly recurrent CNNs through parameter sharing, ICLR 2019\n[2] Legonet: Efficient convolutional neural networks with lego filters, ICML 2019\n[3] Structured convolutions for efficient neural network design, NeurIPS 2020\n[4] HyperNetworks, ICLR 2017\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "7af6jMyNzvm", "reviewer": "Reviewer_k6be", "summary": "This paper proposes a recursive convolutional block design for compact CNNs with efficient training. The convolutional filters (across all layers) are decomposed into shared basis filters and non-shared expanding coefficients, both of which are trained in an end-to-end manner. To avoid vanishing/exploding gradients, orthogonality regularization is imposed, and the efficacy of which is shown through both theoretical analysis and empirical verification. The proposed architecture consistently outperforms the corresponding baseline in terms of accuracy with a reduced model size.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "** Pros **\n1. The paper is very well-written and well-organized. It is easy to read.\n2. Imposing orthogonality constraint on the shared basis is interesting, and its effectiveness is explained both theoretically and empirically.\n3. A small portion of non-shared layer-specific basis is also used to maintain expressiveness of the model.\n4. Extensive numerical experiments have been conducted to demonstrate the effectiveness of the model.\n\n** Cons **\n1. Missing citation: the idea of filter decomposition into shared basis and non-shared coefficients is not new, e.g., it is also proposed in Qiu et al., \"DCFNet: Deep Neural Network with Decomposed Convolutional Filters\", ICML 2018.\n2. Figure 3 reports the effectiveness of regularization in preventing vanishing gradients. Is there a visual illustration demonstrating how the regularization can prevent exploding gradients?\n3. The choice of lambda is rarely mentioned in the manuscript. An ablation study on lambda should be included in section 4.4.\n\nOverall it is an interesting paper (even though the idea of filter decomposition has been widely studied and adopted), but I would love to see the feedbacks from the authors.\n\n** Review after author feedback **\nThe authors have partially addressed my concerns. I am raising my rating to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No comment.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "1SkdSPw96Q", "reviewer": "Reviewer_fjeu", "summary": "In this paper, the authors try to improve the parameter efficiency of neural networks by using recursively shared filter basis. To address the gradient vanishing/exploding problem in naive recursively parameter sharing, orthogonality regularization is proposed and shows effectiveness in ablation studies.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The motivation of this paper is very clear. Parameter efficiency is more important in resource-constrained devices like Jetson TX2.\n- The proposed parameter sharing approach shows reasonable performance on different benchmarks including ImageNet classification and COCO detection.\n- Ablations and analysis are provided to show the effectiveness of proposed orthogonality regularization.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The position of shared filter basis used in different networks is kind of confusing. It would be great if the authors can explain the reason to not share every convolutions in the blocks (as there're still some non-shared elements in shared conv), maybe also with some ablation studies.\n- The parameter reduction in compact models is not very impressive.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1RG-zUVEqr", "reviewer": "Reviewer_oyjZ", "summary": "This paper describes a method to share parameters between iterations of a residual block.  Rather than sharing all parameters or none, a filter basis W_basis and mixture coefficients alpha are defined; the basis is shared between layers, while the coefficients are unshared.  A theoretical argument shows a potential cause of training instability due to repeated application of a shared weights term; an orthonormality regularizer is applied to the weights basis, to help alleviate vanishing/exploding gradients.  Measurements are performed on ImageNet, CIFAR and COCO, applied to a variety of resnets and mobilenet.  The resulting model uses about half as many parameters (though similar number of FLOPs) as a baseline model, and achieves similar or slightly better accuracy.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The system is similar to Savarese and Maire 2019, which is described in the related work.  The differences here appear to be (a) the analysis of repeated weights that may cause large/small gradients, (b) the orthonormality regularizer, and (c) the use of small numbers of unshared weights in addition to basis-generated weights.\n\nWhile the theoretical argument seemed persuading, and the final system in showed good results in the experiments, I think additional experimental baselines and comparisons would make the benefits of the method more convincing.  In particular, there are no comparisons for using no unshared elements (U=0), or the naive approach of fully sharing weights within residual groups.  (Savarese and Maire is also not compared directly, though a configuration of this system without regularizer may be equivalent).\n\nLikewise, the effect of the orthogonality regularizer is demonstrated to create larger gradients (if I'm interpreting the plot correctly, see below additional comments), but the final effect on CIFAR-100 is about 1% accuracy gain in the best case, which is good but seems a little small compared to the motivation of unstable gradients --- might there be other effects, e.g. on the number of steps needed in the training schedule?\n\nOverall, the system improves upon similar parameter sharing schemes, with good results.  But it's unclear how much it helps relative to a few other simple baselines, described above.  Including these would strengthen the experimental case.\n\n\n\nAdditional comments and questions:\n\n\n- Steerable weights in related work:  There is some work on steerable cnns that I feel is related and might be discussed; these systems use transformed weights for enforced equivariance, e.g. https://arxiv.org/pdf/1612.08498.pdf, https://openaccess.thecvf.com/content_cvpr_2018/CameraReady/3214.pdf\n\n\n- What is the effect of orthgonality regularizer when training on ImageNet?  The difference in classification error from this term is only shown for a subset of the cifar experiments (resnet34 on cifar100 and resent56 on cifar10).  What happens if orthogonality regularizer is not used on ImageNet?  More comprehensive experiments on this would be more convincing.\n\n- What is the effect of lambda in the loss?  To me it looks it would be relatively unsensitive to the exact value, and it should be set fairly low, around 0.01 or lower.  But would be good to confirm by showing an experiment for this.  \n\n\n- Compare to naively shared weights (where successive block iters use the same weights):  I didn't see this comparison in the paper --- in addition to unshared baseline, it would be good to see param count and accuracy drop of the baseline model with weights shared in each stage, to confirm its accuracy loss.\n\n\n- Effect of U0 vs U1:  The text says that a small number of unshared weights help performacne, but I didn't see this evaluated; the smallest number of unshared basis elements is 1.  What about 0 (all shared)?\n\n\n\n- sec 4.4 Fig 3:  I don't understand how to interpret this plot, more explanation may be needed.  What does the varying lightness of the light blue bars mean --- are there multiple overlapping translucent bars, and why?  Also, what does a value of 0.0075 mean --- relative to what?  Are these per-element gradients, and if so, what is this value relative to the weight, and why is this value good?  What about the step size, incorporating the optimizer learning rate, momentum etc?  If gradients are larger or better conditioned, can the training schedule be reduced?\n\n\n- sec 4.4 Fig 4:  What are the similarity matrices for groups 1 and 4?  Showing only 2 and 3 seems incomplete to me.  Do they show the same pattern?  And if not, what might be a cause for difference?\n\n\n- eq. 5 and text:  Since the alpha coefficients don't appear here explicitly, and are part of the repeated net (with unshared weights), I think it could be useful to explain where they are here (as part of d x^j / d a^j), and relate to the paragraph on l.156 on what influences their stability\n\n\n- Appendix fig A.1:  u=0 is missing from these curves, so it's hard to conclude that having one unshared element helps, as the text says.  Also would be good to have error bars over a few different random inits, as the errors are jumping around 1% out of 22%.  This is a nice plot for the supplement, though, I thought it was interesting to see.\n\n\n\n- l.148 \"If Wbasis is orthogonal ... \":  this sentence should probably also use \"orthonormal\"\n\n- l.227: resnet50 does not appear in the table\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Authors propose to use recursively shared filter basis for compact CNNs. I agree with most reviewers that the idea of shared filter basis in CNN is not new and similar systems have been discussed before. However, ideas such as imposing orthogonality constraint on the shared basis seem interesting, and the effectiveness is well explained. Extensive experiments have been conducted to show improvement. 

3 out of 4 reviewers are inclined to accept the paper. 
Decision: Accept (Poster)