ShardNet: One Filter Set to Rule Them All
Authors: Saumya Jetley
Tommaso Cavallari
Philip Torr
Stuart Golodetz
Venue:  (2020)
Keywords: neural network compression
filter sharing
network interpretability

Abstract: Deep CNNs have achieved state-of-the-art performance for numerous machine learning and computer vision tasks in recent years, but as they have become increasingly deep, the number of parameters they use has also increased, making them hard to deploy in memory-constrained environments and difficult to interpret. Machine learning theory implies that such networks are highly over-parameterised and that it should be possible to reduce their size without sacrificing accuracy, and indeed many recent studies have begun to highlight specific redundancies that can be exploited to achieve this. In this paper, we take a further step in this direction by proposing a filter-sharing approach to compressing deep CNNs that reduces their memory footprint by repeatedly applying a single convolutional mapping of learned filters to simulate a CNN pipeline. We show, via experiments on CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet that this allows us to reduce the parameter counts of networks based on common designs such as VGGNet and ResNet by a factor proportional to their depth, whilst leaving their accuracy largely unaffected. At a broader level, our approach also indicates how the scale-space regularities found in visual signals can be leveraged to build neural architectures that are more parsimonious and interpretable.

Under review as a conference paper at ICLR 2020
SHARDNET: ONE FILTER SET TO RULE THEM ALL
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep CNNs have achieved state-of-the-art performance on numerous machine learning and computer vision tasks in recent years, but as they have become increasingly deep, the number of parameters they use has also increased, mak- ing them hard to deploy in memory--onstraine interpret. Machine learning theory implies that such networks are highly over- parameterised and that it should be possible to reduce their size without sacri- ficing accuracy, and indeed many recent studies have begun to highlight specific redundancies that can be exploited to achieve this. In this paper, we take further step in this direction by proposing filter-sharing approach that reformulates deep, complex CNNs as an iterative application of shallower modules (a single convo- lutional mapping in the simplest case). We show, via experiments on CIFAR- 10, CIFAR-100 100, Tiny ImageNet and ImageNet that this allows to reduce the param- eter counts of networks based on common designs such as VGGNet and ResNet by a factor proportional affected. At a broader level, our approach represents a way of rethinking neural network architectures so as to leverage the scale-space regularities found in visual signals, resulting in models that are both parsimonious and casier to interpret.
environments and difficult to
their depth, whilst leaving their accuracy largely un-
INTRODUCTION
Deep CNNS have achieved state-of-the-art results on wide range of tasks, from image understand- ing (Redmon & Farhadi, 2017; Jetley et al., 2017; Kim et al., 2018; Oktay et al., 2018) to natural language processing (Oord et al., 2016; Massiceti al., 2018). However, these network architec- tures are often highly overparameterised (Zhang et al., 2016), and thus require the supervision of large number of input-output mappings and significant training time to adapt their parameters to any given task. Recent studies have discovered several different redundancies in these network archi- tectures (Garipov et al., 2016; Hubara" et al., 2018; Wu et al., 2018; Frankle & Carbin, 2019; Yang et al., 2019a;b) and certain simplicities (Pérez et al., 2018; Jetley et al., 2018) in the functions that they implement. For instance, Frankle Carbin (2019) showed that a large classification network can be distilled down to small lub-network that, owing to its lucky initialisation. is trainable in iso- lation without compromising the original classification accuracy. Jetley ct al. (2018) observed that deep classification networks learn simplistic -inearittsss for class identification, fact that might well underlie their adversarial vulnerability, whilst challenging the need for complex architectures. Attempts at knowledge distillation have regularly demonstrated that it is possible to train small stu- dent architectures to mimic larger teacher networks by using ancillary information extracted from the latter, such as their attention patterns (Zagoruyko & Komodakis, 2017), predicted soft-target distributions (Hinton et al., 2014) or other kinds of meta-data (Lopes et al., 2017). These works and others continue to expose the high level of parameter redundancy in deep CNNs, and comprise a foundational body of work towards studying and simplifying networks for safe and practical use.
Our paper experiments with yet another scheme for simplifying CNNs, in the hope that will not only shrink the effective footprint of these networks, but also open up new pathways for network un- derstanding and redesign. In particular, we propose the use a common set of convolutional filters at different levels of convolutional hierarchy to achieve class disentanglement. Mathematically, we formulate a classification CNN as an iterative function in which small set of learned convolutional mappings are applied repeatedly as different layers of a CNN pipeline (see Figure 1). In doing so, we are able to reduce the parameter count of the network by factor proportional to its depth, whilst leaving its accuracy largely unaffected. We also investigate the introduction of non-shared linear

---

Under review as a conference paper at ICLR 2020
Convt
Gone
Conoff
Conr1
Oom2
Conff
(a) Standard convolutional architecture
(b) Shared filters architecture
Figure 1: Standard CNN architectures (a) contain several convolutional layers, all of which are individually adapted using backpropagation. By contrast, we propose the use a single learned convolutional layer (b) that applied repeatedly to simulate a CNN pipeline.
10'
Acciragy (86)
90
Compression factor
70
85
Accuracy
60
oo
80
50
VOGNet
VGGNet
75-
VGGNet
10
S-VGGNNt
10c
32
64
128
256
512
baseline
64
128
256 the
320
512
bas eline
Number
convolutiona layer
Number
layer
(a) Sharing-based compression for CIFAR-I0
(b) Sharing- based compression for CIFAR-100
Figure 2: Accuracy versus compression off curves for our basic shared architecture in Fig. 1, for different widths of the shared convolutional layer, compared to the baseline VGGNet (Simonyan Zisserman, 2015), for CIFAR-10 (a) and CIFAR-100 (b). The compression factor is plotted on logarithmic scale.
layers before certain shared convolutional layers to enhance the flexibility of the model by allowing linearly combine shared filter maps for the disentanglement task.
2 RELATED WORK
This work is partly inspired by the classic literature on image processing that has long sought to characterise natural images by collating their responses, at different image scales, to small, canon- ical set hand-crafted visual operators (Mallat, 1989; Viola & Jones, 2004). Modern CNN ar- chitectures (Krizhevsky al., 2012; Simonyan & Zisserman, 2015) effectively still implement hi- erarchical feature extraction, but with the difference that there are thousands of such operators (i.e. convolutional filters) at each scale level, all of which individually adaptable and learned via back- propagation. Our work can thus be seen as an effort to reconcile the above two non- contemporaneouus approaches to image processing, in which we aim identify a common set of visual operators for all the different scales by learning them in an end-to-end manner.
Our approach bears some high-level resemblance to previous approaches (e.g. Pinheiro & Collobert (2014); Liang & Hu (2015); Liao & Poggio (2016); Savarese & Maire (2019)) that have attempted to implement, interpret and potentially improve convolutional neural networks through an iterative use of simpler modules For example, Liao & Poggio (2016) share convolutional mappings in ResNets in an attempt to approximate biological visual systems using feedback loops and recurrence, al- though their experimental analysis limited to the CIFAR dataset. By contrast, work applies the convolution-sharing paradigm to both plain feed-forward and residual constructs, and investi- gates the effectiveness of using only single shared convolutional mapping for the entire network pipeline. An additional contribution of our approach is the flexibility we add to the model by cou- pling learned linear layers with shared convolutions while still limiting the total parameter count. Experimentally, we evaluate the accuracy vs. model size tradeoff induced by our approach on realistic set of datasets that include Tiny ImageNet and ImageNet.
A steady increase in the size of datasets and the availability of computational resources has enabled neural networks to grow deeper (Simonyan & Zisserman, 2015; He al., 2016), denser (Huang et al., 2017) and wider (Zagoruyko & Komodakis, 2016). In doing so, concerns regarding their over-parameterisation have often been ignored in favour of better test set generalisation. More
"Despite previous arguments that depth gives regularization effects and width causes network to overfit, we successfully train networks with times more parameters than ResNet-1001, and outperform ResNet-1001 by significant margin." " (Excerpt from Zagoruyko & Komodakis (2016).)
2

---

Under review as a conference paper at ICLR 2020
recently, their performance some benchmarks (He et al., 2016; Oord et al., 2016) has reached near-human levels, real-world deployment of these models is being considered. This deployment has been hugely impeded by the memory requirements, latency and energy demands of their heavy computational machinery (Bianco et al., 2018).
Our approach contributes to the (extensive) literature on network compression that is focused on making these machine learning models more usable in practical scenarios. Existing compression methods can be divided into seven categories pruning, quantisation, tensorization//ensor decom- position, knowledge distillation, custom architectures, sharing-based and hybrid methods. Many of these works are beyond the scope of this paper, but for completeness, we present a brief review in $A.1 (a more exhaustive survey can be found in Cheng et al. (2017)). Our own work falls within the realm of sharing-based methods that seek to equate some of a network's weights or filters to reduce the number of independent parameters in the network. There are various ways of deciding which weights/filters to share, from somewhat arbitrary (if effective) approaches such as the hashing trick (Chen et al., 2015; Liu et al., 2018), to more principled approaches such as k-means cluster- ing (Wu et al., 2018). A few recent works have turned their attention to sharing convolutional weight matrices in more structured manner. Of these, LegoNet (Yang al., 2019b) shares filter groups across sets of channels, whilst FSNet (Yang et al., 2019a) shares filter weights across spatial loca- tions. In both cases, sharing is restricted a single layer at time. ShaResNet (Boulch, 2018 1)) reuses convolutional mappings, but within the same scale level (i.e. hetween two max- max-pooling steps). The novelty of our work lies in extending this Elter-sharing paradigm to an entire convolutional pipeline. We instantiate a single convolutional layer that is applied iteratively to mimic a deep convolutional feature extractor, and analyse the accuracy vs, memory tradeoff for different widths of this layer.
METHOD
A standard feed-forward classification CNN can be formulated
F C + F Feons = C (R,, 3 Sr. O @ fi),
(1)
where the overall function F is a composition of the convolutional feature extractor Feome followed by a fully-connected classifier C. The convolutional sub-model Feoms consists of a sequence of convolutional layers [f: < S L]. interspersed with non-linearities (ReLUs, Max-Pooling) or regularisers (dropout, BatchNorm) or some combination thereof, denoted by R, The function performed by each convolutional layer fü is completely specified by set of weights and biases that we denote using W, Crucially, the weights and biases for each different layer are independent. The number of parameters in layer Sa is then simply the size of W., calculated as
Wi n, x nuut x k2 + n.ut = U; x kē + n R 02 x k?
(2)
where neth in the number of input channels to fi. ngue is the number of output channels, v2- ni x naout is the volume of fi. and k; is the size of its (square) convolutional filters. In practice, the ni term for the biases is dominated by that for the weights, and so we disregard it in what follows. Letting W cone- = U W; denote all the parameters in Fooms (i.e. disregarding the comparatively small contributions from the con-convolutional layers), the total parameter count is given by
W comel
W
Σ
U x
(3)
Note that for many common architectures, there exists some ic such that Vi, k; = k (e.g. for VGGNet, k 3). For such architectures, Equation can then be further simplified to [Woonl| R Lxux k2 in which ē = -.-1-11 D' v; is the mean volume per network layer.
Our method proposes crude simplification to such architectures, namely to instantiate single convolutional layer f, and apply it L successive times in order to implement convolutional pipeline of equivalent depth to the original model. In particular, we enforce the following constraint:
W1 = W2 .... = W7 = W + fi = f2 = -
= ft f.
(4)
This simplifies the CNN architecture in Equation I to
f = C & Feoms C @ (Rr ② f ..... Rı 0 f).
(5)
3

---

Under review as a conference paper at ICLR 2020
Whilst our analysis focuses purely on the convolutional layers, it is interesting to note that when the R, layers are all the same, the CNN architecture simplifies further to the following iterative form:
C=C Frome = C @ (R 2 nt.
The convolutional layer f in architecture expects input tensor with predetermined number of channels, which we will call n. Meanwhile, the R, layers between the convolutional layers leave the number of channels unchanged. Thus, given the iterative application of f. the layer must also output a tensor with n channels. (In practice, f is called for the first time on the input image itself, which for colour images would normally only have 3 channels. To avoid artificially limiting n to 3, we pad the input image with empty channels to produce a tensor with n channels.) We deduce that [W], the number of parameters for f. must satisfy [W] R n² h2 = v k., where v n? is the volume of f. Furthermore, since W is shared between all T. convolutional layers, the total number of independent parameters in Fooms must also just be [W). The compression factor between the original architecture and its shared counterpart can thus be quantified as
W conee W
[ W
LX L
U
U/U
This is proportional to the depth L of the original network, and is down wwighteet by any (multiplica- tive) increase in the average per-layer volume in going from the original to the shared architecture.
We now turn to examine the convolutional operation in our architecture. Each layer f. the operation of which is completely specified by the weights and biases in W. takes input tensor X of size n x h x w. where n, h and w denote the number of channels, height and width respectively. Based on X and W, we conceptually define 2D matrices ((X) and I(W) as follows:
W11
W12
Win
X1i
$(X)=
r(W)
(8)
Wnl
Wn2
Wnn
Xmm1
X
b1
b2
by
In this, m = 1 x wo and each Xiy is a rasterisation of a l x l patch of input tensor centred at spatial location in channel j. Each Waj is a similar rasterisation of the k x k convolutional kernel that maps the input channel i E {1, 2..... n to the output channel j E (1.2.... n2,, and each 6, is the bias for output channel j. Then f can be defined concisely as f(X) (x)( x I(W)), in which T reshapes the m x n tensor (X) x I(W) back to one of size n x h x w.
In practice, this simple formulation could be seen as being too restrictive, in the sense that irre- spective of the convolutional iteration, each filter wij in I'(W only ever operates on patches from input channel (for example, the Wiy filters only ever operate on patches from channel 1). For this reason, we decided to investigate whether adding way of allowing the input channels to be reor- ganised at various points in the overall pipeline would improve performance. In principle, one way of achieving this would be to add e.g. just before each pooling operation. In practice, however, to make the operations differentiable, implement them using linear layers (i.e. 1 1 convolutions). thus implementing blending of the input channels rather than simply permuting them. The weights of these layers are separate for each instantiation and are learned as part of the end-to-end pipeline.
permutation matrices at appropriate points in the pipeline,
would be reasonable to expect this added flexibility to yield a significant increase in performance, and indeed our results in 55 show this to be the case. Nevertheless, is notable that even without this added flexibility, our shared architectures already achieve extremely good performance on the datasets on which we tested, demonstrating that our underlying approach of sharing filters between layers makes sense even in the absence of permutation/blending
ARCHITECT
4 DATASETS AND ARCHITECTURES
We evaluate our filter-shaing approach on four well-known image classification benchmarks: CIFAR-10, 10, C00, Tiny ImageNet and ImageNet. Details of these datasets can be found in $A.2. For this study, we work with two different architectures, one closely inspired by VGGNet (Si- monyan & Zisserman, 2015), and the other by ResNer (He al., 2016).
4

---

Under review as a conference paper at ICLR 2020
VGGNer-like Architectures. We base our VGGNet-like architectures VGG-16, 16, which consists of 5 convolutional blocks followed by 3 linear layers. Each block is followed by a max pooling step and contains several convolutional layers with different channel counts (in order: 2 layers with 64 channels, 2 with 128, 3 with 256, 3 with 512 and 3 layers with 512 channels). By contrast, in our case, we define single convolutional layer with fixed number of input and output channels n, and then use it repeatedly in the same arrangement as above (see Table 3 in $A.3 for more details). We define four variants of this convolutional feature extractor for our study. E-VGGNet is our equivalent of VGGNet, with n channels per layer and no sharing between the layers: we use this as a baseline. Its shared counterpart, S- S-VGGNet, has the same structure, but iteratively applies a single convolutional layer. SL-VGGNet is an extended version of S- S-VGGNer that introduces linear layers (i.e. I convolutions) before each max-pooling operation to allow the input channels to be blended at those points in the pipeline. Finally, since all the convolutional layers in SI- SLVVGGNer are the same (these exclude what we call the linear layers), we define a further variant of our architecture that simplifies the network design by setting the number of layers per block to a scalar t. We experiment with & E {2, 3). and name the corresponding networks VGGNer. Note that the predetermined number of channels is parameter of architecture: we test several variants to find the best ones. We perform experiments on CIFAR- and Tiny ImageNet. For 100 the 3 fully-connected layers that follow the feature extractor have 512, 512 and 10 output channels, respectively. For CIFAR- 100, we use the same VGGNet- like architectures as for CIFAR-10, 10, but the fully-comnected layers have 1024, 1024 and 100 output channels, respectively. For Tiny ImageNet, we use a sequence of two fully-connected layers, with 2048 and 200 output channels respectively.
ResNet-like Architectures. We base our ResNet-llie architectures on the models proposed in He et al. (2016). The simpler variants of these are built using "basic" blocks that essentially consist of two equally-size
convolutional layers and a skip connection (see Fig. 6). The deeper variants, meanwhile, are built using 'bottleneck' blocks, which similarly have a skip connection, but sandwich a single 3 convolutional layer between two x convolutional layers that decrease and then restore the number of channels to limit the number of free parameters. The network pipeline begins with a standalone convolutional layer that outputs a predetermined number of channels p. This is followed by sequence of b blocks at a number of different scale levels (generally 4, but 3 for CIFAR variants). In the original architectures, each scale level (except the first) began with a strided convolutional layer that downsampled the image and doubled the number of channels. Since we want the convolutional layers in our architectures to have the same numbers of input and output channels (to facilitate sharing), we define an equivalent architecture, E-ResNet, that instead doubles the number of channels and performs downsampling using (respectively) linear layer (i.c. 1 x 1 convolutions) and max- mnox-pooling step at the end of each scale level. Note that, as in the original ResNet, the final scale level in our architecture ends with average pooling rather than max-pooling. Despite these modifications, the predictive performances of our F-RReNett closely match those of the original architectures. The shared variant of this architecture uses channels for all scale levels and shares the weights across all the convolutional layers (excluding the linear layers). Since the architecture already contains the linear layers we were previously adding to allow blending of the input channels, we refer to as SL-ResNet.
For 1IFAR-10/100, the standalone convolutional layer uses kernel size of 3 x 3, and p of 16 and 32 for each dataset, respectively. We experiment with E {3, 5, 7} 'basic' blocks per scale level, and terminate the network with 10-way linear classifier for CIFAR-10 and a 100-way classifier for CIFAR-100. See Table 4 in 5A.3 for details. For Tiny ImageNet and ImageNet, we base our ResNet-like architectures on ResNet-34 and ResNet-50. ResNet-34 is built using 'basic" blocks, whilst ResNet-50 uses 'bottleneck' blocks. For the latter, it is clearly not possible to share filters between the layers within a block, since they are of different dimensions, so we instead use multiple shared copies of a single block. Note that the shared variants of both these models, SL-ResNet 34/50, keep the standalone convolutional layer unshared, since its kernel size is adjusted according to the dataset (3 x 3 for Tiny ImageNet and 7 x 7 for ImageNet). See Table 5 in $A.3 for details.
5 RESULTS AND DISCUSSION
Earlier, Fig. 2 showed the accuracy vs. compression trade-off for S- GGGGet, relative to the original VGGNet (Simonyan & Zisserman, 2015), for different widths n of the shared convolutional layer. Here, Fig. 3 illustrates the improvements in accuracy due to the learned linear layers (i.e. the blend-
5

---

Under review as a conference paper at ICLR 2020
Ca Comprnus pnnuuaaaaa on MAt (O
50
LE
22
64
128
256
532
buscinn
325
204
520
5iz
bendine
64
125
226
300
532 buaiin
(a)
(b)
Figure 3: Accuracy versus compression trade-off curves of the 'S' and 'SL' variants of VGGNet for different widths n of the shared convolutional layer, relative the baseline VGGNet (Simonyan & Zisserman, 2015), for CIFAR-10 (a), CIFAR-100 (b), and Tiny ImageNet (c). The compression factor plotted on a log scale.
E-VGGNe
VGGNNer I cons
VGGNe
Acc. (%)
W Wonce
Ace. (1))
[W [Wcome]
Acc. (5)
Acc. (%)
W Wonn
VGGNN
14.7M
64
32
91.2
445K
112K
85.5
73.6
9.3K
37K
12.0
12.0
87.7
74.7
53.3K
13.3K
8.3
Lego-VGGNaa-16w
(o=2,m=00.5)
93.23
3.7M
4.0
256
128
91.0
7.1M
188
90.7
500K
148K
12.2
91.3
213K
8.4
Lego-VGG VOG--GGGNe---- 16-w (o:4.m=00.55)
91.35
900K
16.0
512
94.5
28M
93.7
2.4M
11.7
94.5
3.M
SL2 VGGNGG (n=5(2)
94.5
3.4M
4.3
(a) CIFAR-10: 'E' vs. 'S' VS. 'SL' variants
SL3-YGGGNe (n=512)
3.4M
St. M.2-VGGNN (n=256)
852K
S1. VL.NVGGN (o-256)
852K
(b) CIFAR-10
Arch.
Ace.
Woose
VGGNec*
74.3
14.7M
1.0
Arch.
Tup-I (%)
Top-5 (%)
Weonel
SL2-VGGGNe (1o5512)
74.0
3.4M
4,3
S1. S1.3-VGGNet (n-512)
3.4M
ViGNNe
81.4
14.7M
1.0
SL2 V---VGGNeelt (n=256)
69.7
R52K
17.3
SL2 VOGNe (n 512)
59.4
82.8
34M
4,3
SL.-VVONNe (n=256)
51.2 (n 256)
17.3
(c) CIFAR-100
(d) Tiny ImageNet
Table I: (a): Test accuracies and parameter counts [Wamm our 'E', 'S' and 'SL' variants VGGNet, for different widths n of the convolutional layer. (b). (c), (d): Comparing the accuracies and compression factors C' of top-performing 'SL' variants of our approach, with & E 2,3 layers per convolutional block, with VGGNet (Simonyan & Zisserman, 2015) and (for CIFAR-10) variants of LegoNet (Yang et al., 2019b), another statc-of-the-art compression method. Baseline models marked with were retrained for this study.
ing layers) on CIFAR-10, CIFAR-100 and Tiny ImageNet. Observably, the use of the linear layers provides greater benefit for datasets that involve discriminating between a larger number classes, such CIFAR- 100 and Tiny ImageNet.
For CIFAR-10, CIFAR- 100 and Tiny ImageNet we compare the accuracies of the pest-perforning 'SL' variants of VGGNet with those of the baseline architccture (and competing compression meth- ods for these datasets, where available) in Table 1. For CIFAR-10 (see Table 1b). we are able to achieve comparable classification accuracy the VGGNet baseline using only for our shared convolutional layer, which yields a compression factor of R 17x. For CIFAR-100 achieve comparable
256 channels
(Table Ic), which has 10x more classes, we had use n 512 channels accuracy, but this still yields a significant compression factor of 4.3. Higher compression factors can be achieved by reducing the number of channels, in exchange for some loss in accuracy. Eval- uating our shared architecture on Tiny ImageNet (in Table 1d) evidences similar trend in the results, with SL2-VGGNet (n 512 channels) achieving an accuracy comparable the non-shared baseline, whilst using only 23% of its parameters. Detailed accuracy and memory usage numbers for E-VGGNeet S-VGGNet and SIGNGG for CIFAR-IO, are in Table Ia, while the results for CIFAR-100 100 and Tiny Imagenet can be found in the appendix (see Table 6 in $A.5)
We also evaluate our shared ResNet architecture (SL-ResNet) on Tiny ImageNet and ImageNet, with the results shown in Table 2 (the corresponding results for CIFAR-10 and CIFAR-100 can be found in the appendix, see Table 7 in $A.5). For Tiny ImageNet, our SL-ResNet34 (n = 512) variant is able achieve compression rate of 8.4 with only negligible loss in accuracy. For ImageNet, the same variant similarly achieves a compression rate 8.4 with respect to ResNet-50 and 21.6 with respect to Shared Wide ResNet (SWRN) by Savarese & Maire (2019). Whilst there is

---

Under review as a conference paper at ICLR 2020
Arch.
Tup-1 (5)
Top-5 (%)
[Wonel
SWRN S0-2
78.26
94.05
69M
1.0
ReaNet-50
77.15
93.3
26.5M
2.6
Arch.
Top-1 (86)
Top-5 (58)
|Woomel
C
ResNet-34
75.5
25.2M
ResNet-507
26.M
1.0
ShaResNes-30
75.39
92.59
205M
24
34
71.75
Legu-Res Re00:e50000222m0.55
89.7
8.IM
8.5
ML-ResNes-500 (1)-
1855
14
FSNet ResNNrt.0
64.11
85.94
4.5M
15.3
S1-RosNin-50 (1-26)
60.3
83.2
4.5M
6.0
M1-ReoNaa-54 (x15112)
62.5
83.8
3.2M
8.4
Sl.-RosNet-S0 (xx-512)
72.4
91.4
18.1M
3.8
SL.:ResNet:34 (xu-55))
56.2
50.0
794K
33.6
SL.RRssNer (xe5112)
69.7
89.3
3.2M
21.6
(a) Tiny ImageNet
(b) ImageNett
Table 2: Comparing the accuracies and compression factors C' of our shared variants of ResNet-34 and ResNet- 50 with the original models and (for ImageNet) with ShaResNet (Boulch, 2018), LegoNet (Yang al., 2019b), FSNet (Yang et al., 2019a) and Shared Wide ResNet (SWRN) (Savarese & Maire, 2019). Bascline models marked with a were retrained for this study.
308
(a) SL3-VGGNet (n = 512) trained on CIFAR-10
(b) SL7-ResNet (n = 256) trained on CIFAR-100
Figure 4: A visual depiction of the linear layers used to blend the input channels our approach. We show the layers for the two variants in the order (left to right) in which they appear in the networks. For each layer, the input channels are ordered along the x-axis, and the output channels along the y-axis. For each output channel (row), we highlight the lowest 32 weights (in terms of absolute value) in blue, and the highest 32 in red.
an accuracy trade-off, we achieve a greater compression rate than competing methods that achieve similar accuracies. Note that SWRN is able achieve state-of-the-art levels of accuracy, but does not provide savings in the number of parameters.
5.1 INTERPRETATION THROUGH VISUALISATION
Visualising the weights of the blending layers that we learn for the SL-variants of our approach reveals interesting patterns in the way in which these layers blend (or use) the input channels (see Fig. 4). For each layer, the continuous blue vertical lines signify that a subset of the input feature maps are barely used by any of the output channels, thus effectively suppressing the information they carry. (Interestingly, the location of the vertical blue lines changes from one scale to the next, thus showing that different subsets of input channels go unused at different scales.) This is significant, because it implies that the weights associated with the unused channels can be selectively pruned without affecting performance. Our next experiment with the pruning method of Han al. (2015) shows how we can exploit this observation to significantly reduce the size of our shared networks.
5.2 COMPLEMENTARITY WITH OTHER COMPRESSION SCHEMES
Our best-performing SL variants have a relatively small number of parameters in the convolutional layers, but relatively high number of parameters in the linear layers. Tables 2a and 2b show how the parameter count for these variants increases with the number of channels n and the depth (34 to 50). Notably, using bottleneck blocks, as we do for our SL-ResNet50 variants, also significantly increases the parameter count. As implied by our visualisations in the previous section, we would expect serious reductions in the number of parameters the linear layers to be possible without significantly reducing accuracy. We thus experiment with applying the magnitude-base based weight pruning approach of Han al. (2015) to the linear layers to see whether this expectation is borne out in practice. We first select proportion of the parameters to prune, then identify those weights that have the lowest absolute magnitude and set them to O. We then evaluate on the validation split of the dataset. Note that we do not retrain the network after pruning. Our results (see Figure 5) show that we can remove a significant fraction of these blending weights before starting to see a noticeable drop in the accuracy of the network.
7

---

Under review as a conference paper at ICLR 2020
100
%-age of paramed 18
Prusing
Top
Top 5 (5)
IW Woooe
80
Accuracy (%)
80
70-
20
62.7
62.7
84,5
84.5
17.1M
15.M
62.7
84.6
144M
60-
40
62.6
84.6
11.7M
Top
20
60
62.3
84,4
10.3
50
Top
616
83.8
8.M
0.0
0.1
0.2 0.3
0.4
0.5
0.6
0.7
0.8
0.9
47.3
73.5
6.2M
Lincar Mapping pruning ratio 1%))
Figure 5: Analysing the effects of pruning on Tiny ImageNct. We iteratively zero those having the smallest absolute value. The accuracy of the network stays constant even when 60% of the parameters are pruned, at which point the compression rate (in comparison to the non-shared baseline with equivalent performance) has increased from 1.4 to 2.6.
of our largest models, SL. (n 512), trained inereasing fraction the linear layer parameters, starting from
CONCLUSION
In this paper, we leverage the regularities in visual signals across different scale levels success- fully extend the filter-sharing paradigm to an entire convolutional pipeline for feature extraction. In particular, we instantiate a single convolutional layer and apply it iteratively simulate conventional VGGNet-like and ResNet-like architectures. We evaluate our shared architectures four standard benchmarks - CIFAR-IO, CIFAR-100, Tiny ImageNet and ImageNet and achieve compression rates that are higher than existing sharing-base based methods that have equivalent performance. We further show that even higher compression rates, with little additional loss in performance, can be achieved by combining our method with the magnitude- based weight pruning approach of Han et al. (2015). Study of our complementarity to more structured pruning techniques targeting complete filters and channels is reserved for future work. We conclude with two final observations. Firstly, our use of blending layers and parameter to tune the width of the shared convolutional layer to achieve a desired trade-off between compression
makes it easy adjust the architecture rate C' and accuracy. Secondly, there are interesting connections between our work and the idea of energy- based pruning explored in (Yang et al., 2017). where the authors note that significant frac- tion of the energy demands of deep network processing come from transferring weights to and from the file system. Our approach bypasses this bottleneck by using the same compact set of weights in an iterative manner. We aim to further investigate this aspect of our method in subsequent work.
REFERENCES
Alireza Aghasi, Afshin Abdi, Nam Nguyen, and Justin Romberg. Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee. In NIPS, 2017.
Arash Ardakani, Carlo Condo, and Warren Gross. Activation Pruning of Deep Convolutional Neural Networks. In JEEE Global Conference on Signal and Information Processing (GlobalSIP), 2017.
Lei Jimmy Ba and Rich Caruana. Do Deep Nets Really Need to be Deep? In NIPS, pp. 2654-2662, 2014.
S. Bianco, Cadene, L. Celona, and Napoletano. Benchmark analysis of representative deep neural network architectures. IEEE Access, 6:64270-64277 2018.
Alexandre Boulch. Reducing parameter number in residual networks by sharing weights. Pattern Recognition Letters, 103:53- 59, 2018.
Cristian Bucilă, Rich Caruana, and Alexandru Niculescu-Mizil. Model Compression. In KDD, pp- 535-541, 2006.
Wenlin Chen, James Wilson. Stephen Tyree, Kilian Q Weinberger, and Yixin Chen. Compressing Neural Networks with the Hashing Trick. In ICML, pp. 2285-2294 2015.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks, arXiv preprint arXiv: 17100002. 2017.
8

---

Under review as a conference paper at ICLR 2020
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural wwtp:/neetttttiittiiaaeea
networks. In ICLR, 2019. URL
Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry Vetrov. Ultimate tensoriza- tion: compressing convolutional and FC layers alike. urXiv: 2016.
Song Han, Jeff Pool, John Tran, and William Dally, Learning both weights and connections for efficient neural network. NIPS, pp. 1135-1143, 2015.
Song Han, Huizi Mao, and William J Dally. Deep Compression: Compressing Deep Neural Net- works with Pruning, Trained Quantization and Huffman Coding. In ICLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. In NIPS-W, 2014.
Andrew G Howard, Menglong Zhu, Bo Chen, and Dmitry Kalenichenko. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. 174.44670, 2017.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, July 2017.
Itay Hubara*, Matthieu Courbariaux * Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized Neural Networks. In NIPS, pp. 4107-4115, 2016.
Itay Hubara*, Matthieu Courbariaux * Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. JMLR, 18:1-30 2018
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5MB model size. 100:::160,.073600 2016.
Saumya Jetley, Michael Sapienza, Stuart Golodetz, and Philip H. S. Torr. Straight to shapes: Real- time detection of encoded shapes. In CVPR, July 2017.
Saumya Jetley, Nicholas Lord, and Philip Torr. With friends like these, who needs adversaries? In NewrIPS, pp. 10749-10759, 2018.
Scong Tae Kim, Jae-Hyeok Lee, Hakmin Lee, and Yong Man Ro. Visually interpretable deep network for diagnosis of breast masses on mammograms. Physics in Medicine & Biology, 63 (23):235025, 2018
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, Uni- versity of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NIPS, pp. 1097-1105. Curran Associates, Inc., 2012.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In NIPS, pp. 598-605, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H Torr. SNIP: Single-shot Network Pruning based on Connection Sensitivity. In ICLR, 2019.
Sam Leroux, Steven Bohez, Cedric De Boom, Elias De Coninck, Tim Verbelen, Bert Vankeirsbilek, Pieter Simoens, and Bart Dhoedt. Lazy Evaluation of Convolutional Filters. arXiv: 085543, 2016.
Ming Liang and Xiaolin Hu. Recurrent convolutional neural network for object recognition. In CVPR, June 2015.

---

Under review as a conference paper at ICLR 2020
Qianli Liao and Tomaso Poggio. Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex. orXiv:160.006644 2016.
Shaohui Lin, Rongrong Ji, Xiaowei Guo, Xuelong Li, et al. Towards convolutional neural networks compression via global error reconstruction. In IJCAI, pp. 1753 1759, 2016.
Zhenhua Liu, Jizheng Xu, Xiulian Peng, and Ruiqin Xiong. Frequency-domain dynamic pruning for convolutional neural networks. In NeurlPS, 2018.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-Free Knowledge Distillation for Deep Neural Networks. arXiv: 17X.::/710005335, 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression. ICCV, pp. 5058-5066, 2017.
Stephane G Mallat. A theory for multiresolution signal decomposition: the wavelet representation. TPAMI, 7:674-693 693,, 1989.
Daniela Massiceti, N. Siddharth, Puneet K. Dokania, and Philip H.S. Torr. Flipdial: A generative model for two-way visual dialogue. In CVPR, June 2018.
Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, and Dmitry Vetrov. Tensorizing Neural Networks. In NIPS, pp. 442-450, 2015.
Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. CoRR,
2018.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. CoRR, (tbb///09.03399, 2016.
Guillermo Valle Pérez, Ard A Louis, and Chico Q Camargo. Deep learning generalizes because the purameter-function map is biased towards simple functions. arXiv preprint 2018.
Pedro HO Pinheiro and Ronan Collobert. Recurrent convolutional neural networks for scene label- ing. In 31 International Conference on Machine Learning (ICML), number CONF, 2014.
Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger. In CVPR, 2017.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for Thin Deep Nets. In ICLR, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, pp. 211-252, 2015. 2015.
Bharat Bhusan Sau and Vineeth N Balasubramanian. Deep Model Compression: Distilling Knowl- edge from Noisy Teachers. drXiv:1610.096500 2016.
Pedro Savarese and Michael Maire. Learning Implicitly Recurrent CNNs Through Parameter Shar- ing. In ICLR, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Suraj Srinivas and R Venkatesh Babu. Data-free Parameter Pruning for Deep Neural Networks. In BMVC, pp. 31.1-31.12. 2015.
Paul Viola and Michael J Jones. Robust real-time face detection. IJC,, 57(2):13771154 2004.
Jiaxiang Wu, Cong Leng, Yuhang Wang. Qinghao Hu, and Jian Cheng. Quantized Convolutional Neural Networks for Mobile Devices. In CVPR, pp. 4820 2016.
10

---

Under review as a conference paper at ICLR 2020
Junru Wu, Yue Wang, Zhenyu Wu, Zhangyang Wang. Ashok Veeraraghavan, and Yingyan Lin. Deep k-Means: Re-Traine Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions. In PMLR, volume 80, pp. 5363-5372, 2018.
Tien-Ju Yang. Yu-Hsin Chen, and Vivienne Sze. Designing Energy-Efficiee Convolutional Neural Networks using Energy-Aware Pruning. In CVPR, pp. 5687-5695, 2017.
Yingzhen Yang, Nebojsa Jojie, and Jun Huan. Fsnet: Compression of deep convolutional neural networks by filter summary. CoRR, abs/1902.03226 2019a. URL wwtttttpt///aaae org/abs/ 1902. 03264.
Zhaohui Yang, Yunhe Wang, Hanting Chen, Chuanjian Liu, Boxin Shi, Chao Xu, Chunjing Xu, and Chang Xu. LegoNet: : Efficient Convolutional Neural Networks with Lego Filters. In ICML, 2019b.
Chen Yunpeng, Jin Xiaojie, Kang Bingyi, Feng Jiashi, and Yan Shuicheng. Sharing Residual Units Through Collective Tensor Factorization Deep Neural Networks. In IJCAI, pp. 635-641, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC. BMVA Press, 2016.
URL https:://// /xx doi. ort. 5244//C...2 30.87.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In ICLR, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv: 160iv:6611.00 161100330, 2016.
A APPENDIX
A.1 ADDITIONAL RELATED WORK
Pruning methods seek reduce the size of a network by removing (either physically or implicitly)
some of a network's 's weights (LeCun et al., 1990; Srinivas & Babu, 2015; Yang et al., 2017; Aghasi al., 2017; Lee et al., 2019), filters (Leroux et al., 2016; Luo et al., 2017) or neurons (Ardakani et al., 2017). Notably, reducing the computational cost (rather than just the memory usage) of network architectures that are pruned in an unstructured manner requires the use of suitable sparse inference schemes.
Quantization methods keep the number of independent parameters in a network the same, but reduce the bit-depth of the parameters and activations (Wu et al., 2016; Hubara et al., 2016; 2018) to limit the memory requirements of the network.
Tensorization/ttenso decomposition methods propose low-rank approximations to high nigh-dimensiioal neural matrices in order to downsize trained models. Early CNN architectures such as AlexNes (Krizhevsky al., 2012) and VGGNet (Simonyan & Zisserman, 2015) contained the bulk of their weights in the fully-connected layers. As a result, various rank reduction approaches exclusively targeted the matrices in these layers (Lin al., 2016; Novikov al., 2015). The beeper/wider (He et al., 2016; Zagoruyko & Komodakis, 2016) these networks have become, the more the balance of weights has shifted towards the convolutional layers, giving rise to more gener- alised tensor decomposition schemes (Garipov et al., 2016).
Knowledge distillation ( ('teacher/student) methods aim to transfer the knowledge present in a cum- bersome teacher model to a lightweight student model, without losing the teacher's 's ability to gen- eralise well. An early approach by Bucilå et al. (2006) used a heavyweight ensemble to label a large set of unlabelled data, and then used this to train compact model. Much later, Ba & Caruana (2014) proposed an alternative method that trains a shallow network to directly mimic the logits of a deep model. Subsequent methods have independently shown that training the student using temperature-seaal
softmax scores (Hinton et al., 2014) or Gaussian blurred logits (Sau & Balasub- ramanian, 2016) of the teacher can help with regularisation. Other methods in this line of work have proposed to train deep, thin neural networks using auxiliary or intermediate cues such as hidden layer outputs (Romero al., 2015) post-hoc attention maps (Zagoruyko & Komodakis, 2017).
11

---

Under review as a conference paper at ICLR 2020
4B 1x1
4n -n 1x1
BetchNorm
Batchhorr
BetchNorm
Fulu
Rulla
RoLu
ReLu
Batshherm
Reiu
ReUp
- n, 3x3
-- 3x3
Baictelurm
Butchharm
ReLu
ReLu
Rotu
(a) Standard basic
(b) Shared basic
(c) Standard bottleneck
(d) Shared bottleneck
Figure 6: The building blocks used in our ResNer-like architectures. (a) and (b) show the non-shared and shared basic blocks. Our shared variant of the residual basic block reuses the same convolutional layer within and across the different blocks. (c) and (d) show the bottleneck blocks. In this case, since the three convolutions have different sizes, cannot share single set of parameters across the whole network; instead, we consider the block as a single entity and reuse it across the network.
Custom architecture methods, rather than trying to compress or distil knowledge from existing net- works, propose entirely network architectures that smaller than existing models but still capable of providing excellent performance. Good examples include SqueezeNet (landola et al., 2016) and MobileNets (Howard al., 2017). SqueezeNet tries to reduce the parameter count, and tries to limit the number of input channels to those 3 3 filters does use. MobileNets follow similar tack and factorise traditional convolutional mappings into a depth-wise separable convolution (to process the spatial context) followed by x 1 convolu- tion (to process the channels jointly). Two adjustable hyperparameters, and p. pertaining to the intermediate feature resolution and the input spatial resolution, allow further resizing of the network.
x rather than 3 filters
Hybrid methods implement some combination of the compression schemes discussed above (Han et al., 2016; Yunpeng et al., 2017). Whilst our approach belongs to the category of filter-sharnng schemes elahorated above, we also demonstrate its complementarity and compatibility with the magnitude-based weight pruning method of Han et al. (2015).
A.2 DATASETS
CIFAR- 10 (Krizhevsky. 2009) consists of 60,000 000 32 32 colour images, each labelled as belonging one of 10 mutually exclusive classes. Each class contains 000 images, of which 5, 000 are earmarked for training, and 1, 0,00 for testing (i.e. there are 50, 000 train images and 10, 000 test images overall). CIFAR-100 consists of the same 60, 000 32 x images that are in CIFAR-10, this time they are evenly split into 100 classes, each containing 500 training images and 100 testing images. Tiny ImageNet essentially a smaller, lower-resolution variant of the ImageNet (Rus- sakovsky et al., 2015) dataset. It consists of 120, 000 64 x 64 images, evenly split into 200 classes. Each class contains 500 training images, validation images and 50 test images. ImageNet (Rus- sakovsky al., 2015) was introduced as harge-scale image classification benchmark consisting of high-resolution photographs in 1, 000 visual categories from an even larger ontology of natural con- cepts (WordNet). It consists of approximately 1M training images, divided into 1, 000 disjoint object categories. Another set of 50, 000 images, evenly split into 1, 000 classes, forms the validation set. The accuracy results we report for ImageNet were obtained this validation set.
A.3 NETWORK ARCIITECTURES
Table details the structure of VGGNer-like architectures, whilst Tables 4 and 5 show ResNet-like architectures (respectively used for CIFAR-10/100 and Tiny ImageNe//ImageNet, The notation is common to the tables and is as follows:
httpps:/tiny- inrtriyy--mmmmetttet magenet. htw.eeetttherkkupp..oo
12

---

Under review as a conference paper at ICLR 2020
convl-r A 1 convolutional layer with output feature channels. The of our SL variants. We use this layer to allow shared convolutions at different scale levels to observe different blends of the feature channels output by the previous scale. Its number of input feature channels is equal to I, except in E-ResNet-50, where we use it to increase the number of channels between scale levels, and in the first scale of SL--RsNNet-50, where we use it to increase the number of channels from x to 4r, to account for the expansion factor of the bottleneck blocks.
conv3-r A 3 convolutional layer with output feature channels. The number of input feature channels depends on the specific network variant: for the baselines is equivalent to the number of output feature channels of the previous layer (or for the very first layer), whilst for the E/S/SL-variants, it is equivalent to the number of output feature channels z. The stride is unless otherwise specified.
conv7-x A 7 convolutional layer with output feature channels. As this layer is only used as the first layer in the ResNet variants of our architectures, it always has 3 input channels. Its stride is when training ResNet-like architecture for Tiny ImageNet, and 2 when training for ImageNet.
basicblock-z
simple skip connection based block, used the ResNer-like architectures. As in He et al. (2016), it consists of two 3 x 3 convolutional layers and skip connection. In our shared architectures, the two convolutional layers share the same parameters. See Fig- 6a and 6b for details of the internal architectures of the non-shared and shared block variants.
bottleneck-z
skip connection-basd block with a bottleneck architecture, consisting of convolution (used reduce the number of feature channels), followed by 3 x 3 convo- lutional layer, and finally by another feature channels). For this reason it has 4x input and output channels. Figures 6с and 6d detail the internal architectures of the standard and shared variants of the bottleneck blocks (respectively). Crucially, as mentioned in the main paper and unlike the basicblock chitectures described above - the bottleneck block is shared as single entity, owing to the presence differently-shaped convolutions.
convolution (restoring the original number of
avgpool-o: An average pooling layer operating patches of size
maxpool-x A max-pooling layer operating on patches of size x x
FC-z A fully-connected layer with æ output channels. The number of its input channels is equal to the number of outputs of the previous layer (flattened the case the previous layer was convolutional layer).
Each spatial convolution (conv3 and conv7) is always followed by a BatchNorm layer and a Rel.u. We denote in bold the convolutional layers blocks that shared in our S and SL. architectures. The parameters of the normalisation layers are never shared, even when the corresponding convolu- tional weights shared part of S or SL architecture. Fully-connected layers (except the very last one in each architecture) are followed by a Dropout layer.
13

---

Under review as a conference paper at ICLR 2020
Input Resolution
VGGNer
E-VGGNe((n)
S-VGGNe((n)
SL-VGGNe((n)
SLA-VGGNer(1)
conv3-64 conv3-64
conv3-n conv3-ni
conv3-n conv3-n
conv3-n
CIFAR-+: 32 x 32
Exconv3-n
conv3-11 conv 1-m
TI: 56 56
convl-n
maxpool-2
conv3-128 conv3-128
conv3-n conv3-n
conv3-n
conv3-n conv3-1 conv 1-m
CIFAR-*: 16 16
exconv3-r
conv3-n
TI: 28 28
conv
maxpool-2
conv3-256 conv3-256
conv3-n conv3-n
conv3-n
conv3-n
CIFAR-+: 8
Exconv3-n
conv3-n
conv3-7 conv 1-12
TE: 14 14
convl-n
maxponl-2
conv3-512 conv3-512
conv3-n conv3-n
conv3-n
conv3-1 conv3-n conv 1-m
CIFAR- 4
Exconv3-r
conv3-n
TI: 1
conv
maxpool-2
CIFAR-+: 2 2
conv3-512 conv3-512
conv3-ni conv3-n
conv3-n conv3-n1
conv3-n conv3-7
Exconv3-n
Tl: 3 x
maxpool-2*
FC1
FC2
FC3
Table 3: The architectures for VGGNer and the VGGNer-like networks we trained as part of our experiments on the 10FAR-10/100 and Tiny ImageNet datasets. The notation is described in the main text. Note that the last max-pooling layer (marked with *) not used when training a network for Tiny ImageNet: this is in order provide longer feature vector to the first fully- connected layer (specifically of size n 3 3). The fully-conmected layer sizes differ across datasets to account for the different numbers of classes, and set as follows: (a) CIFAR-10: FC; FC- 512, FC2 = FC-512, FC3 = FC-10: (b) CIFAR- 100: FC1 = FC-1024, FC2 = FC-1024, FC3 = FC-100; (c) Tiny ImageNet: FC1 FC-2048, FC2 FC-200.
Input Resalution
Eb-ResNet(p)
SLb-ResNe((z)
conv3-p bx basicblock-p conv1-(2 * p)
conv3-n
32
basichlock-n
convl-n
maxpool-2
bx basicblock-(22 xp) convl-(4 1144 p))
bx basichlock-n
16 16
convl-1
maxpool-2
8 8
bx basicblock-(1 bx basichlock-n
avgpool-8
FC-num.
Table The architectures for the ResNer-like networks we trained as part of our experiments on the CIFAR- 10/100 datasets. The notation is described in the main text. The baselines Eb-ResNet(() use p = for training on CIFAR-10 10 (as The final fully-connected layer has its output size set to the number of classes in the dataset (i.e. num, for CIFAR-10 and num, = 100 for CIFAR-100), We experiment with different values of (3,5,7).
al. (2016)) and p = 32 training on CIFAR-100.
14

---

Under review as a conference paper at ICLR 2020
Input Resolution
E-ResNet-34
E-ResNer-50
SL-ResNNr- (4())
SL- Re-RReNN--SSS (0))
conv7-64, stride-2
conv7-64, stride-2
conv7-n, stride 2
conv7-n, stride 2 conv1-(re * 4)
224 x 224
conv1-256
112
maxpool-2
maxpool-2
maxpool-2
maxpool-2
basicblock-64
bottleneck-66 64
basicblock-n
4x botteneck-n convl-(n * 4)
56 x 56
conv1-128
conv1-512
conv1-n
maxpool-2
maxpool-2
maxpool-2
maxpool-2
4x basichlock-122 128
4x bottleneck--22
basicblock-n
4x botteneck-n convI-(r * 4)
28 x 28
c-nv1-256
conv1-1024
convI-ri
maxpool-2
maxpool-2
maxpool-2
maxpool-2
4x basicblock-256
4x bottleneck-256
4x basieblock-n
4x botteneck-n convl-(n * 4)
14 x 14
conv1-512
conv1-2048
convl-n
maxpool-2
maxpool-2
maxpool-2
maxpool-2
7 7
1x basicblock-512
1x bottleneck-512
4x basieblock-n
4x botteneek-n
avgpool-3
avgpool-3
avgpool-3
avgpool-3
FC-Rum.
FC-num.
FC-num.
FC-nam.
Table 5: The architectures for the ResNer-like networks trained part of our experiments the Tiny ImageNet and ImageNet datasets. The notation described in the main text. The final fully-comeccted layer has its output size set to the number of classes in the dataset (i.e. num, 200 for Tiny ImageNet and num, = 1000 for ImageNet). One important difference in the architectures for the two datasets is that, in the case of Tiny ImageNet, account for the smaller resolution of the images, in the first scale level we use 3 x 3 convolution without striding and suppress the first maxpool-2 layer. This has the effect of allowing feed the convolutional architecture with an input image of size 56 x 56.
15

---

Under review as a conference paper at ICLR 2020
E-VGGNer
S-VGGNet
SL- SL-VGGNer
Ace. (%)
W comv
Acc. (%)
I cono
Acc. (%)
|W cone
64
64.4
445K
41.1
37K
12.0
47.7
53.3K
8.3
128
70.8
1.8M
59.1
148K
12.2
64.1
213K
8.4
256
74.6
7.1M
67.2
590K
12.0
69.1
852K
8,3
320
75.3
11.1M
68.7
922K
12.0
71.1
1.3M
8.5
512
76.9
28M
72.5
2.4M
11.7
73.4
3.4M
8.2
(a) CIFAR-100
E-VGGNet
S-VGGNet
SL-VGGNer
Acc. (%)
I comml
Acc. (%)
I [Wooul
Acc. (%)
|W
64
50.9
445K
37.6
37K
12.0
41.5
53.3K
8.0
128
56.9
1.8M
48.7
148K
12.2
52.7
213K
8.4
256
62.3
7.1M
55.5
590K
12.0
57.6
852K
8.3
320
61.9
11.1M
56.6
922K
12.0
58.3
1.3M
8.5
512
63.0
28M
56.8
2.4M
11.7
59.7
3.4M
8.2
(b) Tiny ImageNet
Table Test accuracies and parameter counts IW cono| for our 'E', , 'S' and 'SL' variants VGGNet, for different widths of the convolutional layer. The compression factors C for the 'S' and 'SL' variants are computed relative to the corresponding E-VGGNee, which contains an equal number of channels in its convolutional layers. Note that all the models trained from state of random initialisation.
A.4 TRAINING PROTOCOL
To train networks the CIFAR datasets, perform some basic data augmentation steps: (1) we randomly decide whether or not to flip the input images horizontally, (2) we pad the 32 x 32 images with 4 pixels and then select random crop of size 32 x 32, and finally (3) normalise the RGB values to have zero mean and unit norm. During the evaluation phase, we just perform the normalisation step. We train our networks for 200 epochs, using the SGD optimiser with momentum 0.9 and weight decay 5e We use an initial learning rate of 0.05 and decrease it by a factor of 2 when the error plateaus.
To train our networks on the Tiny ImageNet and ImageNet datasets, we perform a similar data augmentation: (1) we first extract a crop random size that is then resized the input resolution of our network (56 x 56 for Tiny ImageNet and 224 x 224 for ImageNet), (2) we randomly decide whether or not to perform a horizontal flip the crop, and finally (3) we normalise the crop. During the evaluation phase, we (1) resize the image to standard resolution (64 x 64 for Tiny ImageNet and 256 x 256 for ImageNet), (2) extract ten crops (of size 56 56 for Tiny ImageNet and 224 224 for ImageNet) from the corners, the centre and their horizontally mmizzonally-mmiroredd mirrored variants (as in Krizhevsky et al. (2012)), and finally (3) normalise the crops. We train our networks for 100 epochs, using the SGD optimiser with momentum 0.9 and weight decay 54 We for the VGGNet-like architectures on Tiny ImageNet, 0.05 for the ResNet-like architectures on Tiny ImageNet, and 0.1 for the experiments ImageNet. Regardless of the initial value, we decrease by a factor of 10 when the error plateaus.
initial learning rate of 0.01
A.5 ADDITIONAL RESULTS
A.5.1 EVALUATION ON CLASSIFICATION BENCHMARKS
Table 6 presents detailed accuracy and memory usage numbers for E-VGGNet, S-VGGNer and SL- VGGNer architectures trained CIFAR- 100 and Tiny ImageNet (results for CIFAR-I0 be found in the main paper, in Table 1a in $5). Similar results for the 'E' and 'SL' variants of ResNet trained on CIFAR-10 10 and CIFAR- can be found in Table 7. Finally, an accuracy and compression rate comparison of our top-performing SL3-ResNer variant with existing baselines and competing com- pression methods for CIFAR-10 shown in Table 8.
16

---

Under review as a conference paper at ICLR 2020
Eb-ResNet
SLb-ResNet (n 64)
SL--ResNet (n 96)
SLb-ResNet (n 128)
Acc. (%)
W comvl
Acc. (%)
C
Acc. (%)
Acc. (%)
91.8
294K
89.7
6.5
92.2
2.9
93.1
1.6
92.9
488K
90.1
10.8
92.0
4.8
93.0
2.7
93.4
682K
89.6
15.2
91.9
6.7
93.2
(a) CIFAR-10
Eb-ResNet
SL (n 128)
SLb-ResNet (n 256)
Ace. (%)
Woon!
Acc. (%)
Acc. (%)
72.5
1.2M
68.1
74.0
74.1
1.9M
68.1
10.5
74.8
2.6
7
74.6
2.7M
70.1
14.9
73.9
3.7
(b) CIFAR-IUN 100
Table 7: Test accuracies and parameter counts W Wonel for our 'E' and 'SL' variants of the ResNet architecture proposed for CIFAR-10 by He et al. (2016), for different widths of the convolutional layers and different number blocks per scale level. The compression factors for the 'SL' variants computed relative to their corresponding 'E' variants, which contain equal number of blocks per scale level. Note that all the models are trained from a state random initialisation.
Arch.
Acc. (%)
I Wooee
ResNet-34
94.72
21.30M
1.0
ResNet-18
94.18
11.18M
1.9
ResNet*
93.4
682K
31.2
FSNet- ReNet-ResNNt-18 ISNec-ResNNe-33
93.93
810K
26.3
94.29
1,68M
ISNec-ResNett500
94.91
2.51M
8.5
ShaResNet- 164 (Boulch, 2018)
93.8
0,93M
23.0
SL3-ResNet (n 128)
181K
117.7
Table 8: CIFAR- 10: Comparing the accuracics and compression factors of cop-performing 'SL' variant of the ResNet architecture (He et at., 2016), for 3 blocks per scale level, with the original ResNet, other baselines ResNet-18 and ResNet-34, and state-of-the-aar compression methods. The compression factor of the proposed model with respect to the best performing ResNet-34 architec- ture is in triple digits. However, a more appropriate comparison is arguably with ResNet* from which the model has been directly compressed by virtue of sharing the convolutional layers. The compression factor is still a significant 4.0, with final weight count of only 181K. Note that the model marked with a has been retrained for this study.
17

---

Under review as a conference paper at ICLR 2020
A.5.2 INTERPRETATION THROUGH VISUALISATION
In Fig. 7, we show the linear layers for our different variants of VGGNet, trained on three different datasets CIFAR-10, CIFAR-100 and Tiny ImageNet. As highlighted by the continuous blue ver- tical lines, is notable that each layer, some the input channels barely contribute towards any of the output channels. Given this, we posit that significant proportion of the weights in the linear layers (those that apply to least important input channels) can be pruned without affecting the accuracy in any significant manner. Preliminary results, verifying this conjecture, are discussed in $5.1. Interestingly, the changing locations of these blue lines reflects the changing importance of different input channels at different scale levels.
Similar results for four different 'SL' variants of ResNet, trained on three different datasets CIFAR- 10, CIFAR-100 and Tiny ImageNet-
presented in Fig. 8. As with our visualisations for 'SL- VGGNet', the continuous blue vertical lines in Figs. 8b, 8c and 8d highlight that some input channels make only minimal contribution to any of the output channels in each layer. Once again, believe that the weights that are applied to these less-important input channels can be pruned without affecting the accuracy in any significant manner. Some indicative results that support this hypothesis can be found in $5.1. By contrast, the linear layers in Fig. 8a exhibit somewhat less regularity. From Table 7a, SL7-ResNet yields both the highest accuracy (93.2%), and the highest compression rate (3.8) for that accuracy amongst all the variants. Thus, one possible explanation for this regular distribution of linear layer weights is that the model is operating at full capacity and is using all the channels in balanced way to achieve an optimal performance.
100
100
100
100
300
300
300
300
400
500
(a) SL3- VGGNet (n 512) trained on CIFAR- 10
200
200
200
200
300
300
300
400
400
500
500
500
500
200
400
200
400
(b) 3L3NNt (n 512) trained CIFAR-100
1.00
100
100
100
200
200
200
200
300
300
400
400
400
400
200
400
200
400
200
400
200
400
(c) SL.- (n 512) trained Tiny ImageNet
Figure 7: A visual depiction of the linear layers used to blend the input channels in the 'SL' variants of VGGNer trained CIFAR-10, CIFAR-100 and Tiny ImageNet. The linear layers presented the order (left to right) in which they appear in the networks. For each layer, the input channels are ordered along the x-axis, and the output channels along the y-axis. For each output channel (row), we highlight the lowest 32 weights (in terms of absolute value) in blue, and the highest 32 in red.
18

---

Under review as a conference paper at ICLR 2020
200
53
300
180
300
(a) SL7-ResNet (n 128) trained on CIFAR-10
230
(b) SL7-ResNNt (n 256) trained CIFAR- -10
466
(c) SL-ResNet34 (n 5122) trained on Tiny ImageNet
3000
2000
1800
2003
2003
2003
SI-ResNet50 (n 512) trained Tiny ImageNet
Figure 8: A visual depiction of the linear layers used to blend the input channels in four "SL' variants of ResNet, in the order (left to right) in which they appear in the networks. For each layer, the input channels are ordered along the x-axis, and the output channels along the y-axis. For each output channel (row), we highlight the lowest 32 weights (in terms absolute value) blue, and the highest 32 in red.
19


Reviews:
[{"review_id": "rkebzlRTKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to modify a standard CNN by requiring all of its layers to share the same filter set, essentially allowing it to be expressed as an iterative (or recurrent) network.  This also has the effect of forcing the same number of feature channels to be used throughout the network.  For ResNet-like architectures with bottleneck blocks, sharing occurs at the level of the block (3 conv layers in series that are repeated).  Another variant of the sharing pattern inserts unshared 1x1 convolutional layers after shared layers or blocks; this adds some flexibility while still reducing parameters compared to standard CNNs.\n\nOn CIFAR-10, CIFAR-100, and Tiny ImageNet, experiments demonstrate the ability of the sharing scheme to reduce parameters without impacting accuracy (or more drastically reduce parameters at the cost of accuracy) (Tables 1ab, 2a).\n\nHowever, results are less compelling on ImageNet (Table 2b), where SL-ResNet-50 and SL-ResNet-34 are both less accurate than the baseline standard ResNets as well as ShaResNet [Boulch, 2018].  The accuracy gap between SL-ResNet and ResNet on ImageNet (Table 2b) is significant (approx 5% Top-1 and 2% Top-5 accuracy) and might make it difficult to justify use of the proposed method in this setting.  As ImageNet is the most challenging of the datasets used, this is cause for concern.\n\nThere is also a major concern with respect to novelty and related work.  Unfortunately, the paper appears to have completely missed the following highly related publication from ICLR 2019:\n\nLearning Implicitly Recurrent CNNs Through Parameter Sharing\nPedro Savarese, Michael Maire\nICLR 2019\n\nThis prior work proposes a network structure in which a set of L layers share a set of k parameter templates.  The templates and sharing coefficients are learned as part of the standard training procedure.  This prior work demonstrates both parameter savings and accuracy improvements when training networks in this manner.  Additionally, this prior work shows that some learned networks can be converted into explicitly recurrent forms as a post-processing step.\n\nThe paper under review appears be a special case of this prior work with the number of templates k = 1 (shared between all layers).  It is possible this is an important special case, worthy of significant attention on its own.  Notably, [Savarese and Maire, 2019] considered sharing across at most all layers within the same stage of a residual network, rather than all layers in the network.  However, arguing for the importance of this special case would require focused experimental comparison and analysis, which is not present in the current version of the paper.\n\nNovelty is clearly limited in light of this overlooked prior work.  At minimum, citation, discussion, and experimental comparison to the above ICLR 2019 paper is necessary.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJxWWgojYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose to use the *same* convolutional layer in every layer of a DNN. The network effectively is converted into repeatedly applying the same convolutional filter at multiple scales. The idea is motivated by wavelet decompositions and related work. The authors show that by repeatedly applying the same filter, the number of parameters that need to be stored for a model reduces proportionally to the depth of the network. At the same time, experimental evidence is provided that the performance of these models is not affected, when compared to the baseline (full) model. \n\nCOMMENTS:\n- the paper is well written, but overly verbose. there are several areas where the explanation can be compressed, making room to add more informative details (which are in the appendix), or increasing the size of the figures (which are too small)\n\n- the paper seems lacking a bit in experiments. If the authors can show that the same filter applied L times achieves about the same performance, why not also experiment with different L? i.e. does VGGNet actually need L layers? what if only 2 layers are used? this will help with the overparametrization problem as well. \n\n- Figures 3 and 4 are hard to read. Please increase their size.\n\n- page 5 line 2: how does padding the input with (n-3) empty channels affect performance? If you're learning the filters through backprop, will they not always be learning to fit to 0 ? or am i missing something? \n\n- along the above lines, why not have the input layer to be a different filter with 3 channels and then have a common filter for all upstream layers?\n\n- in multiple places in the text, you refer to the number of \"independent\" parameters. I dont see why the parameters need to be independent. Unless there's some orthogonalization happening at the weights, calling them independent is incorrect. \n\n- paragraph above sec4: you add separate \"linear\" layers for the 'SL' models. Can you describe how many addditional parameters you have to learn?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "ByeCD0Ritr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach to reduce the number of a neural network by sharing the convolutional weights among layers. To convert the first layer into the right number of features padding is used. The last layer I suppose is instead a normal classifier on the fully connected representation (for VGG) or on the average pooling(for ResNet). Results on different datasets and architectures show that the proposed approach can highly compress the number of needed parameters with a minimal reduction of the network test accuracy.\n\nI lean to reject this paper because, in my opinion is very similar to (\"Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex\" Qianli Liao and Tomaso Poggio), which is not mentioned in related work. This paper, published in 2016 was already proposing the idea of reducing the number of parameters of ResNet by sharing the weights of each layer and therefore consider ResNet with shared weights as a recurrent net.\nIn this paper the setting are slightly different, authors add also a variant with additional 1x1 convolutions and show also results with additional compression. However, in my opinion, the main idea is the sharing of convolutional weights, and this is not new.\n\n\nAdditional Comments:\n- This paper considers only the number of learnable parameters of a network. However, in many cases, for applications, it is more important to save memory (which is not the case as the activations should still be saved for backpropagation) and computation. In my understanding the final computation of the model is actually increased because it uses more channels at lower layers (which corresponds to high resolution features maps). Authors should comment about that.\n- In section 4, VGGNet-like Architectures and ResNet-like architectures the authors mention a baseline E-VGGNet or E-ResNet with exactly the same architecture as the shared weights network (thus same number of channels at each layer), but without sharing. However I could not find the performance of that interesting baseline in the results.\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes an interesting experiment/modification of CNNs. However, it looks like this contribution overlaps significantly with prior work (that the authors initially missed) and the comparison in the (revised) manuscript seem to not clearly delineate and acknowledge the similarities and differences.

I suggest the authors improve this aspect and try submitting this work to next venue. 
Decision: Reject