ShardNet: One Filter Set to Rule Them All

Deep CNNs have achieved state-of-the-art performance on numerous machine learning and computer vision tasks in recent years , but as they have become increasingly deep , the number of parameters they use has also increased , making them hard to deploy in memory-constrained environments and difficult to interpret . Machine learning theory implies that such networks are highly overparameterised and that it should be possible to reduce their size without sacrificing accuracy , and indeed many recent studies have begun to highlight specific redundancies that can be exploited to achieve this . In this paper , we take a further step in this direction by proposing a filter-sharing approach that reformulates deep , complex CNNs as an iterative application of shallower modules ( a single convolutional mapping in the simplest case ) . We show , via experiments on CIFAR-10 , CIFAR-100 , Tiny ImageNet and ImageNet that this allows us to reduce the parameter counts of networks based on common designs such as VGGNet and ResNet by a factor proportional to their depth , whilst leaving their accuracy largely unaffected . At a broader level , our approach represents a way of rethinking neural network architectures so as to leverage the scale-space regularities found in visual signals , resulting in models that are both parsimonious and easier to interpret . 1 INTRODUCTION . Deep CNNs have achieved state-of-the-art results on a wide range of tasks , from image understanding ( Redmon & Farhadi , 2017 ; Jetley et al. , 2017 ; Kim et al. , 2018 ; Oktay et al. , 2018 ) to natural language processing ( Oord et al. , 2016 ; Massiceti et al. , 2018 ) . However , these network architectures are often highly overparameterised ( Zhang et al. , 2016 ) , and thus require the supervision of a large number of input-output mappings and significant training time to adapt their parameters to any given task . Recent studies have discovered several different redundancies in these network architectures ( Garipov et al. , 2016 ; Hubara * et al. , 2018 ; Wu et al. , 2018 ; Frankle & Carbin , 2019 ; Yang et al. , 2019a ; b ) and certain simplicities ( Pérez et al. , 2018 ; Jetley et al. , 2018 ) in the functions that they implement . For instance , Frankle & Carbin ( 2019 ) showed that a large classification network can be distilled down to a small sub-network that , owing to its lucky initialisation , is trainable in isolation without compromising the original classification accuracy . Jetley et al . ( 2018 ) observed that deep classification networks learn simplistic non-linearities for class identification , a fact that might well underlie their adversarial vulnerability , whilst challenging the need for complex architectures . Attempts at knowledge distillation have regularly demonstrated that it is possible to train small student architectures to mimic larger teacher networks by using ancillary information extracted from the latter , such as their attention patterns ( Zagoruyko & Komodakis , 2017 ) , predicted soft-target distributions ( Hinton et al. , 2014 ) or other kinds of meta-data ( Lopes et al. , 2017 ) . These works and others continue to expose the high level of parameter redundancy in deep CNNs , and comprise a foundational body of work towards studying and simplifying networks for safe and practical use . Our paper experiments with yet another scheme for simplifying CNNs , in the hope that it will not only shrink the effective footprint of these networks , but also open up new pathways for network understanding and redesign . In particular , we propose the use of a common set of convolutional filters at different levels of a convolutional hierarchy to achieve class disentanglement . Mathematically , we formulate a classification CNN as an iterative function in which a small set of learned convolutional mappings are applied repeatedly as different layers of a CNN pipeline ( see Figure 1 ) . In doing so , we are able to reduce the parameter count of the network by a factor proportional to its depth , whilst leaving its accuracy largely unaffected . We also investigate the introduction of non-shared linear Input Classifier . layers before certain shared convolutional layers to enhance the flexibility of the model by allowing it to linearly combine shared filter maps for the disentanglement task . 2 RELATED WORK . This work is partly inspired by the classic literature on image processing that has long sought to characterise natural images by collating their responses , at different image scales , to a small , canonical set of hand-crafted visual operators ( Mallat , 1989 ; Viola & Jones , 2004 ) . Modern CNN architectures ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ) effectively still implement hierarchical feature extraction , but with the difference that there are thousands of such operators ( i.e . convolutional filters ) at each scale level , all of which are individually adaptable and learned via backpropagation . Our work can thus be seen as an effort to reconcile the above two non-contemporaneous approaches to image processing , in which we aim to identify a common set of visual operators for all the different scales by learning them in an end-to-end manner . Our approach bears some high-level resemblance to previous approaches ( e.g . Pinheiro & Collobert ( 2014 ) ; Liang & Hu ( 2015 ) ; Liao & Poggio ( 2016 ) ; Savarese & Maire ( 2019 ) ) that have attempted to implement , interpret and potentially improve convolutional neural networks through an iterative use of simpler modules For example , Liao & Poggio ( 2016 ) share convolutional mappings in ResNets in an attempt to approximate biological visual systems using feedback loops and recurrence , although their experimental analysis is limited to the CIFAR dataset . By contrast , our work applies the convolution-sharing paradigm to both plain feed-forward and residual constructs , and investigates the effectiveness of using only a single shared convolutional mapping for the entire network pipeline . An additional contribution of our approach is the flexibility we add to the model by coupling learned linear layers with shared convolutions while still limiting the total parameter count . Experimentally , we evaluate the accuracy vs. model size tradeoff induced by our approach on a realistic set of datasets that include Tiny ImageNet and ImageNet . A steady increase in the size of datasets and the availability of computational resources has enabled neural networks to grow deeper ( Simonyan & Zisserman , 2015 ; He et al. , 2016 ) , denser ( Huang et al. , 2017 ) and wider ( Zagoruyko & Komodakis , 2016 ) . In doing so , concerns regarding their over-parameterisation have often been ignored in favour of better test set generalisation.1 More 1 “ Despite previous arguments that depth gives regularization effects and width causes network to overfit , we successfully train networks with 5 times more parameters than ResNet-1001 , . . . and outperform ResNet-1001 by a significant margin. ” ( Excerpt from Zagoruyko & Komodakis ( 2016 ) . ) recently , as their performance on some benchmarks ( He et al. , 2016 ; Oord et al. , 2016 ) has reached near-human levels , real-world deployment of these models is being considered . This deployment has been hugely impeded by the memory requirements , latency and energy demands of their heavy computational machinery ( Bianco et al. , 2018 ) . Our approach contributes to the ( extensive ) literature on network compression that is focused on making these machine learning models more usable in practical scenarios . Existing compression methods can be divided into seven categories – pruning , quantisation , tensorization/tensor decomposition , knowledge distillation , custom architectures , sharing-based and hybrid methods . Many of these works are beyond the scope of this paper , but for completeness , we present a brief review in §A.1 ( a more exhaustive survey can be found in Cheng et al . ( 2017 ) ) . Our own work falls within the realm of sharing-based methods that seek to equate some of a network ’ s weights or filters to reduce the number of independent parameters in the network . There are various ways of deciding which weights/filters to share , from somewhat arbitrary ( if effective ) approaches such as the hashing trick ( Chen et al. , 2015 ; Liu et al. , 2018 ) , to more principled approaches such as k-means clustering ( Wu et al. , 2018 ) . A few recent works have turned their attention to sharing convolutional weight matrices in a more structured manner . Of these , LegoNet ( Yang et al. , 2019b ) shares filter groups across sets of channels , whilst FSNet ( Yang et al. , 2019a ) shares filter weights across spatial locations . In both cases , sharing is restricted to a single layer at a time . ShaResNet ( Boulch , 2018 ) reuses convolutional mappings , but within the same scale level ( i.e . between two max-pooling steps ) . The novelty of our work lies in extending this filter-sharing paradigm to an entire convolutional pipeline . We instantiate a single convolutional layer that is applied iteratively to mimic a deep convolutional feature extractor , and analyse the accuracy vs. memory tradeoff for different widths of this layer . 3 METHOD . A standard feed-forward classification CNN can be formulated as F = C Fconv = C ( RL fL · · · R1 f1 ) , ( 1 ) where the overall function F is a composition of the convolutional feature extractor Fconv followed by a fully-connected classifier C. The convolutional sub-model Fconv consists of a sequence of convolutional layers [ fi : 1 ≤ i ≤ L ] , interspersed with non-linearities ( ReLUs , Max-Pooling ) or regularisers ( dropout , BatchNorm ) or some combination thereof , denoted by Ri . The function performed by each convolutional layer fi is completely specified by a set of weights and biases that we denote using Wi . Crucially , the weights and biases for each different layer are independent . The number of parameters in layer fi is then simply the size of Wi , calculated as |Wi| = nini × nouti × k2i + nouti = vi × k2i + nouti ≈ vi × k2i , ( 2 ) where nini in the number of input channels to fi , n out i is the number of output channels , vi = nini ×nouti is the volume of fi , and ki is the size of its ( square ) convolutional filters . In practice , the nouti term for the biases is dominated by that for the weights , and so we disregard it in what follows . Letting Wconv = ⋃L i=1Wi denote all the parameters in Fconv ( i.e . disregarding the comparatively small contributions from the non-convolutional layers ) , the total parameter count is given by |Wconv | = L∑ i=1 |Wi| ≈ L∑ i=1 vi × k2i . ( 3 ) Note that for many common architectures , there exists some k such that ∀i , ki = k ( e.g . for VGGNet , k = 3 ) . For such architectures , Equation 3 can then be further simplified to |Wconv| ≈ L× v̄ × k2 , in which v̄ = L−1 ∑L i=1 vi is the mean volume per network layer . Our method proposes a crude simplification to such architectures , namely to instantiate a single convolutional layer f , and apply it L successive times in order to implement a convolutional pipeline of equivalent depth to the original model . In particular , we enforce the following constraint : W1 = W2 = · · · = WL = W ⇔ f1 = f2 = · · · = fL = f. ( 4 ) This simplifies the CNN architecture in Equation 1 to F̃ = C F̃conv = C ( RL f · · · R1 f ) . ( 5 ) Whilst our analysis focuses purely on the convolutional layers , it is interesting to note that when the Ri layers are all the same , the CNN architecture simplifies further to the following iterative form : F̃ = C F̃conv = C ( R f ) L. ( 6 ) The convolutional layer f in our architecture expects an input tensor with a predetermined number of channels , which we will call n. Meanwhile , theRi layers between the convolutional layers leave the number of channels unchanged . Thus , given the iterative application of f , the layer f must also output a tensor with n channels . ( In practice , f is called for the first time on the input image itself , which for colour images would normally only have 3 channels . To avoid artificially limiting n to 3 , we pad the input image with empty channels to produce a tensor with n channels . ) We deduce that |W | , the number of parameters for f , must satisfy |W | ≈ n2 × k2 = v × k2 , where v = n2 is the volume of f . Furthermore , since W is shared between all L convolutional layers , the total number of independent parameters in F̃conv must also just be |W | . The compression factor between the original architecture and its shared counterpart can thus be quantified as C = |Wconv | |W | = ∑L i=1 |Wi| |W | ≈ L× v̄ × k 2 v × k2 = L v/v̄ . ( 7 ) This is proportional to the depth L of the original network , and is down-weighted by any ( multiplicative ) increase in the average per-layer volume in going from the original to the shared architecture . We now turn to examine the convolutional operation in our architecture . Each layer f , the operation of which is completely specified by the weights and biases in W , takes an input tensor X of size n× h× w , where n , h and w denote the number of channels , height and width respectively . Based on X and W , we can conceptually define 2D matrices Φ ( X ) and Γ ( W ) as follows : Φ ( X ) = x > 11 · · · x > 1n 1 · · · · · · x > m1 · · · x > mn 1 , Γ ( W ) = w11 w12 · · · w1n · · · · · · wn1 wn2 · · · wnn b1 b2 · · · bn . ( 8 ) In this , m = h×w , and each xij is a rasterisation of a k× k patch of input tensor centred at spatial location i in channel j . Each wij is a similar rasterisation of the k × k convolutional kernel that maps the input channel i ∈ { 1 , 2 , . . . , n } to the output channel j ∈ { 1 , 2 , . . . , n } , and each bj is the bias for output channel j . Then f can be defined concisely as f ( X ) = Ψ ( Φ ( X ) ×Γ ( W ) ) , in which Ψ reshapes the m× n tensor Φ ( X ) × Γ ( W ) back to one of size n× h× w. In practice , this simple formulation could be seen as being too restrictive , in the sense that irrespective of the convolutional iteration , each filter wij in Γ ( W ) only ever operates on patches from input channel i ( for example , the w1j filters only ever operate on patches from channel 1 ) . For this reason , we decided to investigate whether adding a way of allowing the input channels to be reorganised at various points in the overall pipeline would improve performance . In principle , one way of achieving this would be to add n × n permutation matrices at appropriate points in the pipeline , e.g . just before each pooling operation . In practice , however , to make the operations differentiable , we implement them using linear layers ( i.e . 1× 1 convolutions ) , thus implementing blending of the input channels rather than simply permuting them . The weights of these layers are separate for each instantiation and are learned as part of the end-to-end pipeline . It would be reasonable to expect this added flexibility to yield a significant increase in performance , and indeed our results in §5 show this to be the case . Nevertheless , it is notable that even without this added flexibility , our shared architectures already achieve extremely good performance on the datasets on which we tested , demonstrating that our underlying approach of sharing filters between layers makes sense even in the absence of permutation/blending .

Summary: In this paper, the authors propose to use the *same* convolutional layer in every layer of a DNN. The network effectively is converted into repeatedly applying the same convolutional filter at multiple scales. The idea is motivated by wavelet decompositions and related work. The authors show that by repeatedly applying the same filter, the number of parameters that need to be stored for a model reduces proportionally to the depth of the network. At the same time, experimental evidence is provided that the performance of these models is not affected, when compared to the baseline (full) model.