Learning Deeply Shared Filter Bases for Efficient ConvNets
Authors: Woochul Kang
Daeyeon Kim
Venue:  (2021)
Keywords: Deep learning
ConvNets
parameter sharing
model compression
convolutional neural networks
recursive networks

Abstract: Recently,  inspired by repetitive block structure of modern ConvNets,  such as ResNets, parameter-sharing among repetitive convolution layers has been proposed to reduce the size of parameters. However, naive sharing of convolution filters poses many challenges such as overfitting and vanishing/exploding gradients, resulting in worse performance than non-shared counterpart models. Furthermore, sharing parameters often increases computational complexity due to additional operations for re-parameterization. In this work, we propose an efficient parameter-sharing structure and an effective training mechanism of deeply shared parameters. In the proposed ConvNet architecture, convolution layers are decomposed into a filter basis, that can be shared recursively, and layer-specific parts. We conjecture that a shared filter basis combined with a small amount of layer-specific parameters can retain, or further enhance, the representation power of individual layers, if a proper training method is applied. We show both theoretically and empirically that potential vanishing/exploding gradients problems can be mitigated by enforcing orthogonality to the shared filter bases. Experimental results demonstrate that our scheme effectively reduces redundancy by saving up to 63.8% of parameters while consistently outperforming non-shared counterpart networks even when a filter basis is deeply shared by up to 10 repetitive convolution layers.

Under review as a conference paper at ICLR 2021
LEARNING DEEPLY SHARED FILTER BASES FOR EFFICIENT CONVNETS
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, inspired by repetitive block structure of modern ConvNets, such ResNets, parameter-sharing among repetitive convolution layers has been proposed reduce the size of parameters. However, naive sharing of convolution filters poses many challenges such as overfitting and vanishing/exploding gradients, resulting in worse performance than non-shared counterpart models. Furthermore, sharing parameters often increases computational complexity due to additional operations for re- parameterization. In this work, we propose an efficient parameter- sharing structure and an effective training mechanism for recursive ConvNets. the proposed ConvNet architecture, convolution layers are decomposed into filter basis, that can be shared recursively, and non-shared layer-specific parts. We conjecture that a shared filter basis combined with small amount of layer- specific parameters can retain, or further enhance, the representation power of individual layers, a proper training method is applied. We show both theoretically and empirically that potential vanishing/exploding gradients problems can be mitigated by enforcing orthogonality to the shared filter bases. Experimental results demonstrate that our scheme effectively reduces redundancy by saving up to 63.8% of parameters while consistently outperforming non-shared counterpart networks even when a filter basis shared by up to 10 repetitive convolution layers.
INTRODUCTION
Modern networks such as ResNets usually have massive identical convolution blocks and recent analytic studies (Jastrzebski et al. (018)) show that these blocks perform similar iterative refinement rather than learning new features. Inspired by these massive identical block structure of modern networks, recursive ConvNets sharing weights across iterative blocks have been studied as promising direction parameter-efficient ConvNets (Jastrzebski et al., 2018, Guo et al.) 2019 Savaarsse Maire 2019). However, repetitive use of parameters across many convolution layers incurs several challenges that limit the performance of such recursive networks. First of all, deep sharing of parameters might result in vanishing gradients and exploding gradients problems. which are often found in recurrent neural networks (RNNs) (Pascanu et al., 2013 [ Jastrzebski et al., 2018). Another challenge is that overall representation power of the networks might be limited by using same filters repeatedly for many convolution layers.
To address aforementioned challenges. in this paper, propose effective and efficient parameter- sharing mechanism for modern ConvNets having many repetitive convolution blocks. In our work, convolution filters are decomposed into fundamental and reusable unit, which is called filter basis, and layer-specific part, which is called coefficients. By sharing a filter basis, not whole convolution filters or layer, we can impose two desirable properties on the shared parameters: (1) resilience against vanishing/cxploding gradients, and (2) representational expressiveness of individual layers sharing parameters. We first show theoretically that shared filter basis can cause vanishing gradients and exploding gradients problems, and this problem can be controlled to a large extent by making filter bases orthogonal. To enforce the orthogonality of filter bases, we propose an orthogonality regularization to train ConvNets having deeply shared filter bases. Our experimental results show that the proposed orthogonality regularization reduces the redundancy not just in deeply shared filter bases, but also in none-shared parameters, resulting in better performance than over wer-parameteriizee counterpart networks. Next, we make convolution layers with shared parameters more expressive using hybrid approach to sharing filter bases, in which small number of layer-speeific non-shared

---

Under review as a conference paper at ICLR 2021
filter basis components are combined with shared filter basis components. With this hybrid scheme, the constructed filters can be positioned in different vector subspaces that reflect the peculiarity of individual convolution layers. We argue that these layer-specific variations contribute to increasing the representation power of the networks when a large portion of parameters is shared.
Since focus is not pushing the state-of-the-an performance, we show the validity of our work using widely-used ResNets as a base model on image classification tasks with CIFAR and ImageNet datasets. Our experimental results demonstrate that when each filter basis is shared by up to 10 convolution layers, our method consistently outperforms counterpart ConvNet models while reducing a significant amount of parameters and computational costs. For example, our method can save up to 63.8% of parameters and 33.4% of FLOPs, respectively, while achieving lower test errors than much deeper counterpart models.
Our parameter sharing structure and training mechanism can be applied to modern compact networks, such as MobileNets (Howard et al. 2017) and ShuffleNets (Zhang et al. (0188) with minor adaptations. Since these compact models already have decomposed convolution blocks, some parts of each block can be identified as a shareable filter basis and the rest of the layer-specific parts. In Experiments, we demonstrate that compact MobileNerV2 can achieve further 8-21 % parameter savings with our scheme while retaining, or improving, the performance of the original models.
2 RELATED WORK
Recursive networks and parameter sharing: Recurrent neural networks (RNNs) (Graves et al.
2013) have been well-studied for temporal and sequential data. As a generalization of RNNs, recursive variants of ConvNets are used extensively for visual tasks (Socher et al.) 2011 Liang & Hu
et al. 2015 Kim et al., 2016 Zamir et al. 2017). For instance, Eigen et al. (2014) explore recursive convo lutional architectures that share filters across multiple convolution layers. They show that recurrence with deeper layers tends to increase performance. However, their recursive architecture shows worse performance than independent convolution layers due to overfitting. In
most previous works, filters themselves are shared across layers. In contrast, we propose to share filter bases that are more fundamental and reusable building blocks to construct layer-specific filters.
More recently, Jastrzebski et al. (2018) show that iterative refinement of features in ResNets suggests that deep networks can potentially leverage intensive parameter sharing. Guo et al. (2019) introduce gate unit to determine whether to jump out of the recursive loop of convolution blocks to save computational resources. These works show that training recursive networks with naively shared blocks leads to bad performance due to the problem of gradient explosion and vanish like RNN (Pas. canu et al. 2013 Vorontsov et al. 2017). In order to mitigate the problem of gradient explosion and vanish, they suggest unshared batch normalization strategy. In our work, we propose an orthogonality regularization of shared filter bases to further address this problem.
Savarese & Maire (2199'ss 's work is also relevant to work. In their work, the parameters of recurrent layers of ConyNets are generated by a linear combination of 1-2 parameter tensors from a global bank of templates. Though similar our work, our work suggests more fine-grained filter bases as more desirable building blocks for effective parameter sharing since filter bases can be casily combined with layer-specific non-shared components for better representation power. Our result shows that these layer-specific non-shared components are critical to achieve high performance. Although they achieve about 60% parameter savings, their approach does not outperform counterpart models and incurs slight increases in computational costs due to the overheads in reparameterizing tensors from the templates.
Model compression and efficient convolution block design: Reducing storage and inference time of ConvNets has been an important research topic for both resource constrained mobile/embedddd systems and energy-hungry data centers. A number of research techniques have been developed such as filter pruning (LeCun et al. 1990, Polyak & Wolf
2015
Li et al. 2017: He et al. 2017).
low-rank factorization (Denton et al. 20.2|01444 Jaderberg et al.
2014), quantization (Han et al. 2016),
and knowledge distillation (Hinton et al. 2015 Chen et al. 2017 to name a few. These compres- sion techniques have been suggested Unfortunately, their accuracy is usually bounded by the approximated original models. By contrast, models are trained from scratch as in loannou et al. (207)'s work and our result shows that
post-processiiee
steps that applied after initial training.
2

---

Under review as a conference paper at ICLR 2021
W
R
basis
n times
α
Shared Sharael
shared
unique
Conv
W
basts
unique
S
(a) Full convolution
(b) Decomposed convolution (c) Recursive sharing of basis
Illustration of the proposed filter basis sharing method. A normal convolution layer in (a) can be decomposed into filter basis Wasis and coefficients as in (b). A filter basis generic building block that can be shared by repetitive convolution layers as in (c). Some layer-specific components (W(1) and ()) be combined for better expressiveness of recursive layers. basis
Figure I:
parameter-sharing approaches outperform the counterpart models, if proper training method is combined, while achieving significant savings in parameters.
Some compact networks such ShuffleNet (Zhang et al. 2018) and MobileNet (Howard al. 2017; Sandler et al.) 2018) show that delicately designed internal structure of convolution blocks acquire better ability with lower computational complexity. Our work can be applied these compact networks since they already exploit decomposed convolutions for their efficient convolution blocks. For instance, since MobileNet's convolution blocks have hetttiiieedep//wiis-ppiitweeee steps, the first two convolution steps can be considered a reusable filter basis and the last step can be considered layer-specift coefficients. In Experiments. we show that MobileNetV2 combined with our parameter sharrig scheme outperforms the original models while saving about 8-21% % parameters.
convolution
DEEP RECURSIVE SHARING OF FILTER BASIS
In this section, discuss how to decompose typical convolution layers into more recursive units, filter bases, and remaining layer-specific parts. We also discuss how to train ConvNets effectively when filter bases deeply shared by repetitive convolution layers.
3.1 FILTER BASES OF CONVOLUTION LAYERS
We assume that convolution layer with S input channels, T output channels, and a set of filters W = (W, E Rexkx5, E [1..T]]. Each filter W2 can be decomposed using a lower rank filter basis Whssis and coefficients
W, Σ
W of basis
where Waasis = (Wraste basis E R E []. is a filter basis, and a = {ar E R, R), t e [1.T]] is scalar coefficients. In Equation 1. R is the rank of the basis. In typical convolution layer, output feature maps V,, E Rwxkx? . E [1.T) are obtained by the convolution between input feature maps U = and filters Wot [1.27]. With Equation this convolution can be rewritten as follows:
V U W. U
0
W
basis
df Whrniss) where E [1.7].
(3)
In Equation 3 the order of the convolution operation and the linear combination of filter basis is reordered according the linearity of convolution operators. This result shows that standard convolution layer can be replaced with two successive convolution layers as shown in Figure (b)). The first decomposed convolution layer performs R convolutions between W Whesier E [1.. and

---

Under review as a conference paper at ICLR 2021
input feature maps U. and it generates intermediate feature map basis Vbasis E R The second decomposed convolution layer performs point-wise 1 x 1 convolutions that linearly combine R intermediate feature maps V Viasis generate output feature maps V. The computational complexity of the original convolution is O Owwhk2 while the decomposed operations take Owwh ttch(12 SR+ RT)). As far R < T, the decomposed convolution has lower computational complexity than the original convolution. Due to this computational efficiency, many compact networks such as MobileNets and ShuffleNets also have similar block structures of decomposed convolution layers. For instance, MobileNets have repetitive convolution blocks of pointwise deephhwise-pointwee convolutions. The filters in the first two steps can be considered reusable filter basis and the remaining 1x1 filters in the last step can be considered layer-specific coefficients.
3.2 RECURSIVE SHARING OF A FILTER BASIS
In typical ConvNets, convolution layers have different filters Ws and, hence, each decomposed convolution layer has its own filter basis Wassis and coefficients a. In contrast, our primary goal in decomposing convolution layers is to share a single filter basis (or small number of filter bases) across many recursive convolution layers. Unlike some previous works (Jastrzehski et al.] 2018) Köpüklü et al. 2019), in which convolution filters W themselves that a filter basis W Whssie is a more intrinsic and reusable building block that can be shared effectively since filter basis constitutes subspace, in which high dimensional filters across many convolution layers can be approximated.
shared recursively,
argue
Though components of a basis only need to be independent and span vector subspace, some specific bases are more convenient and appropriate for specific purposes. For the purpose of sharing a filter basis, need to find optimal filter basis W basis that can expedite the training of filters of shared convolution layers. Although this optimization can be done with a typical stochastic gradient descent (SGD), one problem is that ssploding/vanishing gradients problems might prevent efficient search of the optimization space. More formally, we consider series of N decomposed convolution layers, in which a filter basis W Wesii shared N times. Let x' be the input of the i-th convolution layer, and a'+1 be the output of the convolution of x' with the filter basis W Woasis
at a'xx (x--1)-
Wi Wasiis
In Equation 4 W basis E R is a reshaped filter basis that has basis components at its columns. We assume that input x is properly adapted (e-g.. with im2col) express convolutions using a matrix-matriy multiplication. Since Whasis is shared across N recusrive convolution layers, the gradient of W hasis for some loss function L is:
aL
N
ØL
N --1
dast1 de
da
II j=i
(5)
OW WWsis
Da
N
OW basis
where
0a++1
0a++1
0x
Ox
Dal
Ox
Dai
W
basis
Ind
(6)
If we plug W Waasis axi in Equation day gradients unstable since W Wheeis is multiplied many times. This sspoddin/vaaishing gradients can
6 into Equation 5 can see that II de the term that makes
0a
be controlled to large extent by keeping W hasis close to orthogonal (Vorontsov et al. 2017). For
instance, if W admits eigendecomposition, [Weasis N can be rewritten as follows:
[W [Wbasis]* NN [QAQ 21N QANQ
where A is diagonal matrix with the eigenvalues placed the diagonal and Q is matrix composed of the corresponding eigenvectors. If Whasis is orthogonal, [Whasis]* neither explodes nor vanishes, since all the eigenvalues of orthogonal matrix have absolute value 1. Similarly, shared basis ensures that forward signals neither explodes nor vanishes. We also need to ensure that the norm of 0x in Equation 5 is bounded (Pascanu et al. 2013) for stability during forward and backward passes. is shown that batch normalization after non- linear activation at each convolution
orthogonal
layer ensures healthy norms (Toffe & Szegedy 2015
Guo et al. 2019 Jastrzebski al. 2018).
4

---

Under review as a conference paper at ICLR 2021
For training networks, the orthogonality of shared bases can be enforeed with orthogonality regularizer. For instance, when each residual block group of ResNet shares a filter basis for its convolution layers, the objective function Ln can be defined to have orthogonality regularizer in addition to the original loss L:
=R=
L+
( T, .w(o) / basis
(8)
basis
where Whasts is shared filter basis for g-th residual block group and À is a hyperparameter. basis
3.3 A HYBRID APPROACH TO SHARING FILTER BASIS
In our filter basis sharing approach, filters of many convolution layers are constructed by the linear combination of shared filter basis as in Equation [
This implies that those high-dimensional filters are all the same low-dimensional subspace.
the rank a filter basis too low, it is very challenging to find such subspace that can express individual peculiarity of many layers' filters. Conversely. if the rank shared filter basis is too high (e.g., R 2 T), the gain in computational complexity from decomposing filters is mitigated. One way to increase the representational power of each convolution layer, while still maintaining its computational complexity low, is adding a small number of layer-specific components to the filter basis. For instance, build filter basis W Whssis not only using shared components, but also using non-shared components:
W Wbasis Wha_shared Wbs_unique>
where Wos_shared { shared E RXxxxS ,T [1..n]] are shared filter basis components, and Whe unique = (WTs anigue E R components. With this hybrid scheme, filters in different convolution layers are placed in different layer-specific subspace. One disadvantage of this hybrid scheme is that non-shared filter basis components require more parameters. The ratio of non-shared basis components be varied to control the tradeoffs. But, our results in Section 4 show that only a few per-layer non-shared basis components is enough achieve high performance.
[n+1..R]] are per-layer non-shared filter basis
EXPERIMENTS
In this section, perform series of experiments image classification tasks. Using ResNets (He) et al. 2016) as base models, we train networks with the proposed method and compare them with the baseline networks. We also analyze the effect of the orthogonality regularization and the hybrid scheme.
4.1 RESULTS ON CIFAR
4.1.1 MODEL CONFIGURATION AND TRAINING DETAILS
Throughout the experiments, we use ResNets He et al. 2016) as base networks by replacing their 3 3x3 convolution layers decomposed convolution layers sharing filter bases. Since each residual block group of ResNets have different number of channels and kernel sizes, our networks share filter basis only in the same group (Figure 5 in Appendix). In each group with n residual blocks, the first block has different stride, and, hence, it does not share a filter basis. Each residual block of the baseline ResNets has two 3 3 convolution layers, and, hence, our networks' each group has 2(n 1) decomposed convolution layers sharing filter basis. Throughout the experiments, we denote by ResNetI.-SsUu a ResNet with T. layers that a filter basis with shared components and u layer-specific non-shared components in the first residual block group. This ratio between and u maintained for all residual block groups.
The CIFAR-10/100 datasets contains 50,000 and 10,000 three-channel 32 x 32 images for training and testing, respectively. For training networks, we follow a similar training scheme in He et al. (2016). Standardized auttaaaggmenttiion and normalization are applied to input data. Networks trained for 300 epochs with SGD optimizer with a weight decay of 5c-4 and a momentum of 0.9. The learning rate is initialized 0.1 and is decayed by 0.1 at 50% and 75% of the epochs.
5

---

Under review as a conference paper at ICLR 2021
4.1.2 RESULTS
Table I shows the results on CIFAR-100. Networks trained with the proposed method consistently outperform their ResNet counterparts. For instance, ResNet34 S32U1 requires only 36.2% parameters and 66.6% FLOPs of the counterpart ResNet34 while achieving even lower test error (21.8e 21.84%) than much deeper ResNer50 (22.36%) To show the generality of our work, apply the proposed method to DenseNet (Huang et al., 2017). ResNeXt (Xie et al. 2017). and MobileNetV2 (Sandler et al., 2018 Although the overall gain is not as great as ResNets' usages in these networks. For instance, ResNeXt50-S6AUL outperforms the counterpart ResNeXt50 while saving parameters and FLOPs by 16.7% and 12.1%, respectively. In ResNeXt, the gain is limited since they mainly exploit group convolutions; each group convolution is decomposed for filter basis sharing in our network. Similarly, for DenseNet, each 3 convolution layer has relatively small number of output channels, and, hence the overall gain is not pronounced as much as ResNets'. Unlike the other networks, MobileNetV2 already has factorized block structure seattwise-depthwss--pppiiie
still observe reduction of resource
and, hence, we choose to share the pointwise-depthwise layers across successive convolution blocks with the same number of channels (Figure in Appendix). During the training. the proposed orthogonality regularization is applied to these shared filters separately. Though MobileNetV2 does not have as many repeating blocks as ResNets, we observe that our MobileNatV2-Sharee saves about 21.9% of parameters while outperforming the original MobileNetV2.
Table I: Error (%) on CIFAR-100. 'x" denotes having 2 shared bases in each group. c denotes
orthogonality regularization is not applied.
Baseline
Model
Params
FLOPs
Error
ResNet34
ResNet34 (baseline)
21.33M
2.33G
22.49
ResNec34-S3221' (ours)
7.73M
1.55G
22.92
ResNet34-S32U1 (ours)
7.73M
1.55G
21.84
DenseNet121
DenseNet121 (baseline)
7.05M
1.81G
21.95
DenseNet121-S16U1 (ours)
5.08M
1.43G
22.15
ResNeXt50
ResNeXt50 (baseline)
23.17M
2.71G
20.71
ResNeXt50-S6444 (ours)
19.3M
2.38G
20.09
MobileNatV2
MobileNetV2 (baseline)
2.43M
0.14G
27.79
MobileNatV2-Sharedd (ours)
1.90M
0.14G
27.20
Table 2: Error (%) on CIFAR-10. 'x" denotes having shared bases in each group.
denotes
orthogonality regularization is not applied.
Baseline
Model
Params
FLOPs
Error
ResNer32
ResNer32 (baseline)
0,46M
0.14G
7.51
ResNet32-S16UU1 (ours)
0.24M
0.16G
6.95
ResNet56
ResNer56 (baseline)
0,85M
0.25G
6.97
ResNet566S11U11 (ours)
0.27M
0.30G
7.70
ResNet56-S166l (ours)
0.27M
0.30G
7.46
ResNet56- " (ours)
0.31M
0.30G
6.33
Filter Pruning (Li et al.
2017
0.77M
0.18G
6.94
KSE (Li 2019
0.43M
0.12G
6.77
DR-Res (Guo et al.
2019
0.50M
0.22G
6.51
The result CIFAR-10 is presented in Table 2 Unlike networks on CIFAR-100, networks on CIFAR-I0 has much fewer channels (e.g. 16 channels in the first residual block group) and, hence, projecting filters to such low dimensional subspace might limit the performance of the networks. For instance, in ResNet32-S8Uu
filters are supposed to be projected onto 9 dimensional subspace consisting of 8 shared and layer-specific filter basis components. By increasing the rank of filter bases, the better accuracy can achieved at the cost of increased FLOPs. For deeper networks such ResNet56, filter basis is supposed be shared by many residual blocks in the group, and it can

---

Under review as a conference paper at ICLR 2021
damage the performance. For example, every filter basis in ResNet56- is shared by 8 residual blocks, or 16 convolution layers. Due to this excessive sharing. though ResNet56-S16U) saves 41.3% parameters, its testing error (7.46%) is higher than the counterpart ResNet56's (6.97%).
To remedy this problem, we introduce a variant, in which each residual block group of the networks uses 2 shared bases; one basis is shared by the first convolution layers of all residual blocks, and the other is shared by the second convolution layers of the same blocks. In Table 2 networks with a mark denote this variant. Though this variant slightly increases the parameters of the networks, it can prevent excessive sharing of parameters. For example, although ResNe156-$1614 * needs 0.04M more parameters for additional shared bases, it still saves 63% parameters of the counterpart ResNet56 and achieves lower testing error, 6.33%.
In Table 2, we compare our results with similar state-of- of-th-artr techniques. Our method achieves better performance and parameter-saving than other approaches such as filter pruning (Li et al. 2017 kernel clustering (Li al. 2019), and recursive sharing (Guo et al. 2019).
4.1.3
ANALYSIS: EFFECTS OF ORTHOGONAL BASES
0.0200
max-gradient mean-gradient
Average gradients
0.0150
0.0125
0.0100
2.0075
0.0350
0.0325
5.0000
Bases (o) Non-orth Without BNs after bases
Shared Bases
Shared Bases
Shared Bases
Non-orthogonal With BNs after bases
Orthogonal
Without BNs after bases
With GNs after bases
Figure 2: The flows of gradients in 4 shared bases of ResNet34-SS comparison, orthogonal regularization and batch normalization (BN) following the bases are turned on and off. In (b) and (c), BNs and orthogonal regularization, respectively, improve the flow of gradients. (d), when both BNs and orthogonal regularization are applied simultaneously, the strongest flow of gradients is observed. This trend is consistently observed during the training.
at the same epoch. For
To inverstigate the effect of orthogonality regularization during training, we track the flows of gradients while training ResNer34-S16U1 Figure shows the maximum and mean absolute gradients the four shared bases during the 20th epoch. Daastzzbbski et al. (2018) and Guoo et al. (2019) showed that unshared batch normalization (BN) mitigates vanishing/exploding gradients problems, and our result Figure (b) shows that unshared BNs following shared bases improves the flow of gradients. When the proposed orthogonality regularization is applied to the shared bases, similar effect on gradient flows is observed in Figure (c). When both unshared BN and orthogonality regularization applied together, in Figure (d), further stronger, but still bounded, flow of gradients are observed. This trend is consistently observed during the training. We conjecture that this healthy flow of gradients improve the optimization process during training.
To further analyze the effect of the orthogonality regularization, in Figure we illustrate absolute cosine similarities of all filter basis components and coefficients of the 2nd and the 3rd residual block groups of ResNet34-S16U4 the shared basis components first, and all non-shared basis components in respective groups next. As expected, the shared filter basis components have almost zero cosine similarities when the orthogonality regularization
In the upper low, the X and Y axes display the indexes to
Equation 8 is applied. The bottom low shows the absolute cosine similarities of coefficients of the corresponding groups. In Figure we can clearly see that coefficients manifest lower similarities when the orthogonality regularization is applied. Without the orthogonality regularization, interesting grid patterns are observed in coefficients. This repetitive grid pattern might be related to ResNets' nature of iterative refinement (Jastrzebski et al. (0188). However, since bases and coefficients are used to build filters of recursive layers, such high cosine similarity is directly related to the higher redundancy in the networks. When the orthogonality regularization is applied, such repetitive patterns are less evident, implying that recursive layers perform less repetitive tasks.
7

---

Under review as a conference paper at ICLR 2021
103
40
20
23
025
25
35-
Group 2 basis
Group 1 basis
Group bavis
Gmup basis
0.0
50
269
53
200
0.65
183
400
600
0.00
500
500
Group metficients
confficients
Group confirients
confficients
(a) Without orthogonality regularization
(b) With orthogonality regularization
Figure 3: Cosine similarities of bases and coefficients of Res.Ne334S166U (2-th and 3-th groups.) In the upper row, X and Y axis are indexes to the shared/unique components of the bases. The first 32 and 64 basis components of the 2-th and 3-th groups are shared by 6 and 10 recursive convolution layers, respectively. The others are non-shared unique basis components of those layers. Orthogonality regularization is applied only to shared components. The lower row shows corresponding coefficients in the residual block groups. Brighter color corresponds higher similarity.
+
Original ResNet
Original ResNet ResNe134-S16U
Test Error (%)
23.5
ResNet18
ResNet34-532U
Res/Ne134-SjJ1
Test Error (%))
23.5
ResNet18
ResNet34-S32Up
ResNet34-S4J1
23.0
46
23.0
Al- 16
22.5
16
32
ResNet34
22.5
#= 16
0=
y=
16
ResNet34
ResNet50
ResNet50
22.0
22.0
0-1
1
0=
em 92
21.5
21.5
5.0
7.5
10.0
12.5
15.0 17.5 20.0
22.5
0.75
1.00
1.25
1.50
1.75
2.00
2.25
2.50
Parameters (M)
FLOPs (G)
Figure 4: Testing errors vs. the number of parameters and FLOPs on CIFAR-100. The number of shared basis components (s), and non non-shared basis components (4) are varied. Using more shared basis components results in better performance. In contrast, using more non-shared components does not always improve performance.
4.1.4 ANALYSIS: EFFECTS OF RANKS OF SHARED//NSHARED BASES
Figure 4 shows test errors as parameters and FLOPs are increased by varying the number of shared/non-shuan -hanee basis components of networks. In general, the higher performance is expected with the more parameters. We observe that this presumption true for shared basis components. For instance, when the number of shared basis components s is varied from 8 to 32, the test error sharply decreases from 23.1% 1% to 21.7%. However, non-shared basis components manifest counter-intuitive results. Although a small number of non-shared basis components (e.g.. 1) are clearly beneficial to the performance, the higher u's do not always lead to the higher performance. For instance, when u 4, both ResNet34-S16Ua and ResNet34-S32U8 S32Uu show the worst performance. This result demonstrates the difficulty training networks with larger parameters. Further study is required for this problem.
8

---

Under review as a conference paper at ICLR 2021
4.2 RESULTS ON IMAGENET
We evaluate our method on the ILSVRC2012 dataset (Russakovsky et al.) 2015) that has 1000 classes. The dataset consists of 1.28M training and 50K validation images. We use ResNet34/50 and MobileNetV2 as base models. Since the block structure of ResNet50 and MobileNetV2 already resembles the decomposed convolution blocks, our models derived from ResNet50 and MobileNetV2 share the first Ixl and 3x3 convolution layers recursively without filter decomposition for repeating convolution blocks (Figure Cand 7 in Appendix). We train the ResNet-derived models for 150 epochs with SGD optimizer with mini batch size of 512, weight decay of le-4, and momentum of 0.9. The learning rate starts with 0.1 and decays by 0.1 at 60-th, 100-th, and 140-th epochs. MobileNetV2 and our BobblleeeV2-Shharee models are trained for 300 epochs with weight decay of 1e-5. Its learning rate starts with 0.1 and decays by 0.1 at 150-th, 225-th, and 285-th th epochs.
Table 3: Error (%) on ImageNet. <** denotes having 2 shared bases in each residual block group. In MabileNetV2-Shareee the first 1x1 step of each block shares parameters recursively. Latency is measured on Nvidia Jetson TX2 (GPU, batch size = 1).
Baseline
Model
Params
FLOPs
top-1
top-5
Lateney
ResNet34
ResNet34 (baseline)
21.80M
7.34G
26.70
8.58
33.6ms
BesNet34-S32U3 (ours)
8.20M
4.98G
27.83
9.42
31.6ms
RcsNer34-S48UT (ours)
11.79M
6.52G
26.67
8.54
38.6ms
Filter Pruning (Li et al.
2017
19.30M
5.52G
27.83
ResNe150
ResNe150 (baseline)
25.56M
8.22G
23.85
7.13
43.8ms
ResNet50-Shared (ours)
18.26M
8.22G
23,95
7.14
43.5ms
FSNet Yang 2020
13.9M
26,89
8.63
MobileNetV2
MobileNetV2 (haseline)
3.50M
0.66G
28.0
9.71
18.4ms
SobileNetV2-Shareee (ours)
3,24M
0.66G
27.61
9.34
17.9ms
MobileNetV2-Shhiee (ours)
2.98M
0.66G
28.21
9.85
17.8ms
FBNct-A (Wu al.
2019)
4.3M
0.49G
27.0
DR-MobileNeriV Guo et al. 2019
2.96M
0.53G
28.2
9.72
The results in Table 3 show that ResNer34-S48UI outperforms the counterpart ResNet34 while using only 54.0% parameters of the counterpart. Although ResNer34-S48U11 requires lower FLOPs than the counterpart ResNet34, it takes 14% longer latency Jetson TX2. This overhead mostly comes from performing convolution operations separately for shared and non-shared basis components.
Since our ResNet50- and MobileNerV2-derivee models do not decompose convolution blocks to define shared bases, overall parameter-saving is not as pronounced as in ResNet34. However, they still save about 28.6% and 14.9% parameters, respectively, while achieving comparable performance. For example, MobileNerV2-Sharee that even compact models such MobileNetV2 can benefit from the proposed parameter-sharing mechanism using orthogonality regularization. Since our ResNet50- and VobbileNetV2-deriieee models do not perform separate convolution for shared and non-shared basis, their theoretical FLOPs are equal to the counterpart models'. However, the latency of our parameter-efficient models is slightly lower than the counterpart models on actual devices. This because our parameter-efficien models better make use of the limited cache and memory of the embedded device.
even outperforms the counterpart model by 0.51%. This result shows
5 CONCLUSIONS AND FUTURE WORKS
In this work, we propose to share filter bases of decomposed convolution layers for efficient and effective sharing of parameters in ConvNets. We both theoretically and empirically show that gradient explosion/vanishie the proposed orthogonality regularization of the filter bases. With our approach, while a significant amount of parameters is shared, the representation power of each convolution layer is retained, or further enhanced, by reducing the redundancy effectively. Experimental results show that our approach consistently outperforms the counterpart models while significantly reducing parameters. We believe that the proposed parameter-sharing and training method suggests important architectural possibilities for future neural architecture search (NAS) for resource-efficiet networks.
problem of deeply shared filter bases can be effectively addressed by

---

Under review as a conference paper at ICLR 2021
REFERENCES
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In Advances in Neural Information Processing Systems, pp. 742-751, 2017.
Emily L Denton, Wojciech Zaremba, Joan Bruma, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances neural information processing systems, pp. 1269-1277, 2014.
David Eigen, Jason Rolfe, Robert Fergus, and Yann LeCun. Understanding deep architectures using recursive convolutional network: 2nd international conference learning representations, iclr 2014. In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
A. Graves, A. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645 6645-6649, 2013.
Q. Guo, Z. Yu, Y. Wu, D. Liang, H. Qin, and Yan. Dynamic recursive neural network. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 5142-5151, 2019.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:/503.0055 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:/704.04861 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4700 2017.
Yani Ioannou, Duncan Robertson, Roberto Cipolla, and Antonio Criminisi. Deep roots: Improving cnn efficiency with hierarchical filter groups In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1231-1240, 2017.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning, pp. 448-456.
2015.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprini arXiv: 1405.3860 2014.
Stanislaw Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In International Conference on Learning Representations, 2018.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. recursive convolutional network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1637-1645, 2016.
Okan Köpüklü, Maryam Babaee, Stefan Hörmann, and Gerhard Rigoll. Convolutional neural networks with layer reuse. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 345-349. IEFE, 2019.
10

---

Under review as a conference paper at ICLR 2021
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-698889 598-605, 1990.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference Learning Representations, 2017.
Yuchao Li. Shaohui Lin, Baochang Zhang, Jianzhuang Liu, David Doermann, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Exploiting kernel sparsity and entropy for interpretable enn compression. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2800- 2809, 2019.
Ming Liang and Xiaolin Hu. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3367-3375, 2015.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318, 2013.
Adam Polyak and Lior Wolf. Channel-level acceleration of deep face representations. IEEE Access, 2175, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chichhhhhhhhh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4510-4520, 2018.
Pedro Savarese and Michael Maire. Learning implicitly recurrent CNNs through parameter sharing. In International Conference on Learning Representatiinns, 2019.
Richard Socher, Cliff C Lin, Chris Manning, and Andrew Y Ng. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 28th international conference on machine learning (ICML-II), pp. 129-136, 2011.
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learn- ing recurrent networks with long term dependencies. In Proceedings of the 34th International Conference Machine Learning, pp. 3570-3578, 2017.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-award efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
10734-10742, 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
SHI Xingjian, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems, pp. 802-810, 2015.
Yingzhen Yang, Jiahui Yu, Nebojsa Jojic, Jun Huan, and Thomas Huang. Fsnet: Compression of deep convolutional neural networks by filter summary. In International Conference on Learning Representations, 2020.
Amir Zamir, Te-Lin Wu, Lin Sun, William Shen, Bertram E Shi, Jitendra Malik, and Silvio Savarese. Feedback networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1808-1817. IEEE, 2017.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-6856, 2018
11

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
conv3x3
cOnIV x
BN, RaLU
Conv 3x3
ReLU
conv3x3
3x3
Shared basis
BN
BN
1x
BN
ReLU
ReLU
repeating blocks
(a) Basic residual block
(b) Decomposed shared block
Figure 5: Block structure of ResNer34 and shared basis. For basic blocks, shared basis is defined by decomposing original 3x3 convolution filters.
conv 1x1 AN,
conv1x1
BN.
conv 3x3
3x3
BN. Reil"
conv1x1
Shared bases
convix1
BN
BN
ReLU
repeating blocks
ReLU
(a) Bottleneck residual block
(b) Shared bottleneck block
Figure 6: Block structure of ResNet50 and shared bases. In the shared bottleneck block, the first 1x1 convolution and the second 3x3 convolution are considered as two bases of the bottleneck block. During training, orthogonal regularization is applied to these bases separately.
conv1x1
SNJ
conv1x1
Reil
Depthwise 3x3
Depthwise 3x3
Arll
BN. ReLU
convix1
conv1x1
Shared bases
BN
repeating blocks
(a) Basic block
(b) Shared block
Figure 7: Block structure of MobileNetV2 and shared bases. In the shared block, the first Ix1 pointwise convolution and the second 3x3 depthwise convolution considered two bases of the block. During training, orthogonal regularization is applied to these bases separately.
12


Reviews:
[{"review_id": "xBx_zYJflq5", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To compress a DNN model, the proposed method improves traditional kernel sharing methods by sharing kernel bases obtained from decomposing a convolution layer. Experiments on ImageNet show that the method reduces 10.01M parameters of ResNet-34 with almost the same error rate, i.e., the method is more effective for DNN architectures without using separable convolution.\n\npros.\n1. In Equations (1)~(3), the authors clearly and formally explain one of their motivations that a convolution layer with high computing complexity can be replaced by a convolution layer with lower computing complexity and a liner layer.\n2. A classic problem, exploding/vanishing gradients, is alleviated by keeping a variable W_basis close to orthogonal. Moreover, in Section 4.1.3, experiments without orthogonality and with orthogonality are qualitatively and quantitatively compared.\n3. In Table 3, the proposed method reduces 10.01M parameters of ResNet-34 with almost the same error rate.\n\ncons.\n1. The novelty/contribution is not quite sufficient. The main novely of this paper is to simultaneously train convolution decomposing and weight sharing. The strategy of convolution decomposing (as shown in Equation 3) in this paper is similar with separable convolution used in MobileNet. For weight sharing, the authors propose to share weight across layers which is also similar with previous weight sharing papers.\n2. The experiments are not sufficient. In Section Experiments, the proposed method is only compared with classic ResNet-34 and MobileNet-V2. However, tons of traditional DNN compressiong papers work on decomposing a convolutional layer, matrix decomposing, or sharing weights. The proposed method should be compared with these traditional methods.", "extra_scores": {}, "extra_text": {}}, {"review_id": "awBMMQLPaCE", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nThis method proposes to decompose convolutional filters using a low rank filter basis where the convolutional operation in a layer consists of shareable filter basis and non-shareable layer coefficients. This is developped to save computational costs whilst maintaining performance. To regularise against vanishing/exploding gradients and promoting more useful representations, the authors seek orthogonal filter basis'. This filter basis is shared across layers in contrast to other works who look at recursive sharing.\n\n[Main Comments]\n1. I like the idea to decompose convolutions into shared weights and layer-specific components. This builds in nicely with a lot of work in multi-task learning about learning which weights to share and so I wonder about how your method might generalise to other problems - could the authors comment on this?\n2. The authors compared their method against ResNet, MobileNet and DenseNet. I would like to see comparisons against more similar techniques such as [Savarese & Maire 2019] and [Guo et al. 2019] to better appreciate the performance of the method. The results against ResNet/MobileNet are indeed promising and good to see.\n3. The presentation of the results (Table 1 and Table 2) is cumbersome. I would suggest reworking this section to make it easier to appreciate the results.\n4. The results in general seem good but due to presentation issues, it is difficult to discern what is going on. In particular, the nomenclature for ResNetL-SsUu is confusing.\n\n[Other]\n1. Could you add an extra parameter to the filter basis, to control dilation of the convolution and thus learn when it is needed to increase the receptive field in addition to sharing weights?\n2. There is a typo in in the sentence above Equation (5) ; reusrive --> recursive\n3. I would recommend a figure to display the differences between vanilla ResNet and your version\n4. Was it possible to analyse empirically the gradient magnitude to show that the orthogonality regularisation helped?\n5. For Figure 2 - I would potentially make a heatmap style figure with $u$ and $s$ as the x-y axis to help display results\n6. How does orthogonality regularisation work on a 1x1 conv?\n7. The work of Ioannou et al. (https://arxiv.org/abs/1605.06489) might be of interest as this seeks structured sparsity in ConvNets to decrease computation without loss of performance", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y8nBhfZ0FgP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of obtaining more compact CNNs by a parameter sharing method. The authors propose to represent a weight filter in a low-rank subspace (represented as a linear combination of low-rank filter basis) plus a set of non-shared low-rank filter basis (per-layer). In this way, the shared low-rank filter basis is reused across several layers, and the non-shared ones per layer are used to enhance model generalization ability. Experiments are performed on CIFAR and ImageNet datasets, using some popular CNN structures for evaluation.\n\nThis paper suffers from following issues.\n\n---Novelty issue.\n\nTo the best of my knowledge, representing a weight filter (in a specific convolutional layer) as a linear combination of a low-rank filter basis was explored in many works, such as “Efficient and Accurate Approximations of Nonlinear Convolutional Networks”, in CVPR 2015, and “Speeding up Convolutional Neural Networks with Low Rank Expansions”, in BMVC 2014. Additionally, even  the first two parts of Figure 1 are very similar to Figure 1 of CVPR 2015 paper. However, they are either missed or not discussed/compared.\n\nCompared with existing low-rank approximation related works, the main contribution of this paper is sharing low-rank filter basis across several layers (in the same group or block of a CNN), while retaining a non-shared low-rank filter basis per layer. To me, the motivation is not clear enough, e.g., such kind of recursive design may easily lead to more high computational cost; it is usually not useful to get obvious benefits. Furthermore, from the definition in the method part, sharing low-rank filter basis across layers needs weight filters have the same shape size. Such a strong constraint also limits its applications.  Additionally, how to determine shared/non-shared low-rank filter basis for different CNNs and their building blocks is not clear enough, and they are manually tuned one by one. \n\nThe authors claim that parameter sharing in this way can address “… a shared filter basis can cause vanishing gradients and exploding gradients problems”.  However, there is no convincing experiments to support this claim, e.g., how this happens in the experiments and how this is alleviated?\n\n--- Experiments issue.\n\nComparison with related works such as existing low-rank methods and recursive parameter sharing methods are completely missing.  \n\nReal wall-clock speed comparison is also missing.\n\nOn small dataset like CIFAR, experiments are run only once. Due to random effects, it may easily lead to different conclusions. For fair comparison, the authors should at least run experiments for several runs (e.g., 3/5) and report mean accuracies.\n\nAlso, on CIFAR-100 dataset, the authors use ResNet34/50/MobileNetV2. However, these CNNs are defined on ImageNet (larger image size 224x224), which have different numbers of down-sampling layers compared to those (ResNet32/56/110) for CIFAR dataset (small image size 32x32). What kind of modifications did the authors make?  Why not using more standard CNNs like  those used on CIFAR-10 to CIFAR-100?", "extra_scores": {}, "extra_text": {}}, {"review_id": "1EL4Vpq1Y0q", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a parameter-sharing method among repetitive convolution layers, where typical filters are decomposed into a set of resuable filter bases and coefficients. The experimental results show some improvement about the number of parameters and FLOPs. Generally, the paper is well written and the method is presented clearly. The authors also claimed that orthogonality regularization can reduce the potential vanishing/exploding gradients problem in weight sharing training. \nHere are my concerns:\n(1) The experiments mainly compared the proposed method with non-shared net designs. However, the comparison results or methods/algorithms with other weight sharing paper are lacking. (e.g., Jastrzebski et al.,2018; Köpüklü et al., 2019 mentioned in the paper). Therefore, it's not easy to judge the novelty of this incremental work. \n(2) Some of the non-shared baseline accuracy is not as good as reported in previous works (e.g., CIFAR100 accuracy-parameter, MobileNetV2 accuracy.). Some comparative results are not fair enough, for example, in the ImageNet experiments, MobileNetV2-Shared model is trained for 300 epochs but the baseline is only trained for 140 epochs.\n(3) The author should, I suggest, focus more on the actual performance such as memory and time overheads during network training and deployment inference, rather than theoretical MACs and FLOPs, (also discussed in ShuffleNet_v2 paper ) which can contribute more to the compact NN community.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I agree with the concerns raised by the reviewers. In particular, the issues of novelty and experimental evaluation (mentioned in the revision summary) remain the major weak points of the paper. My impression is that the changes made in the revision represent a significant experimental addition to the paper, one which might merit a full pass through peer review, and one which in any event did not alter the reviewers' scores. While I think this paper has something to contribute (and the empirical results suggest the method may outperform competitors), I think it would be improved by a rewrite (and possibly a restructure) that makes the part that is your contribution much more clear. For example, in the abstract, it's only in the sentence "We show both theoretically
and empirically that potential vanishing/exploding gradients problems can be mitigated by enforcing orthogonality to the shared filter bases" that we actually get to the part that is really novel about this contribution (the "enforcing orthogonality"): that would ideally be much earlier in the abstract.
Decision: Reject