A Joint Subspace View to Convolutional Neural Networks

1 INTRODUCTION . Modern developments of deep network components ( Ioffe & Szegedy , 2015 ; Xu et al. , 2015 ) , structures ( He et al. , 2016 ; Huang et al. , 2017 ) , and training techniques ( Kingma & Ba , 2014 ; Zoph & Le , 2017 ) have enabled deep convolutional neural networks to consistently grow deeper and wider , with hundreds of millions of parameters . The trend of growing network scales leads to both severe challenges and urgent needs to fast adaptations ( Finn et al. , 2017 ; Lopez-Paz & Ranzato , 2017 ; Shin et al. , 2017 ) , parameter efficiency ( Cheng et al. , 2017 ; Han et al. , 2015a ; Luo et al. , 2017 ; Savarese & Maire , 2019 ; Yang et al. , 2019 ; Wang et al. , 2018b ) , and interpretability ( Selvaraju et al. , 2017 ; Zhou et al. , 2016 ) . While recent works on efficient convolution operations ( Chollet , 2017 ; Howard et al. , 2017 ; Ma et al. , 2018 ; Zhang et al. , 2018 ) alleviate the long recognized over-parametrization problem of deep CNNs , and practical methods on modeling the shared components across CNN filters have been proposed ( Ha et al. , 2016 ; Savarese & Maire , 2019 ) , there is still a lack of a principled view to exploiting the common structure of the filters within a CNN . While preliminary studies have revealed the redundancies in deep network parameters ( Michel et al. , 2019 ; Raghu et al. , 2017 ) , and the fact that deep features across layers are highly correlated under certain linear transforms ( Kornblith et al. , 2019 ; Morcos et al. , 2018 ; Raghu et al. , 2017 ) , exploiting this potential redundancy remains challenging . We suspect that the spatial patterns of convolutional filters , which correspond to different semantics levels , are the main reason why we can not observe obvious cross-layer filter low-rankness . To validate our hypothesis , we decouple the spatial patterns and channel mixing of a convolutional filter by decomposing it over layer-specific 2D filter atoms , linearly combined using atom coefficients . Then with proper alignments to the layer outputs , we finally observe a low-rank structure among those atom coefficients across different layers . This highly non-trivial observation , which to our knowledge has never been reported before , hints the feasibility of enforcing a common block of atom coefficients across layers and suggests a joint subspace view to CNNs . Applying to a wide range of network architectures , this joint subspace view leads subsequently to a novel CNN architecture in which the majority of parameters are shared across layers as a common block of atom coefficients , with only a few hundred parameters remaining specific to each layer as filter atoms as shown in Figure 1 . This matches the intuition visualized in Figure 1 that important image regions , e.g. , the wings , beak , and tail of a bird , remain important across different layers and scales to output an informative deep representation . Atom-coefficient decomposition disentangles spatial convolution and channel mixing . Thus , channels that describe important region features will highly likely be consistently assigned with large weights across layers , in order to carry on those critical information . Thus , atom coefficients , i.e. , weights to linearly combine channels , now become shareable across layers to permit joint subspace CNN modeling . Our approach accepts many easily constructed variants , e.g. , using different filter atom numbers and sharing substructures , to allow highly flexible trade-offs between parameter reduction and model expressiveness . This simple and plug-and-play joint subspace view opens the door to better interpreting , training , adapting , and compacting deep models . We support our findings with extensive empirical evidence in this paper . By using variants of ACDC as plug-and-play replacements to the standard convolutions , we observe comparable and even better performance on challenging image classification datasets with orders of magnitude smaller models . We further demonstrate with few shot learning experiments that ACDC improves the adaptation of deep networks on novel tasks with limited supervisions . We end our paper with a model interpretability discussion . Our main contributions are summarized as follows : • We introduce a joint subspace view to CNN , which is compatible to a wide range of network architectures , to provide a structural regularization over CNN filters for better model interpretability , training , adaptation and compaction . • Highlighting the remarkable flexibility and compatibility , we introduce various model variants constructed easily by sharing coefficient within different network sub-structures . • We validate the effectiveness of our approach by plug-and-playing them into modern CNN architectures for various real-world tasks . 2 RELATED WORK . Before presenting the connections of our method with the literature , let us comment on the complementary efforts on network compression . Contrary to recent efforts on network compression and efficient architectures , we instead explore and exploit the underlying correspondence and correlations among convolutional layers within a given CNN . We decompose the convolutional kernels among layers , and enforce shared composition coefficients within sub-structures of a CNN . This plug-and-play reparametrization of convolutional kernels significantly reduces the network redundancy while maintaining performance , improving adaptation , providing additional interpretability , reducing computational cost , and adding compression ( as a by-product ) comparable to many modern heavily tailored light-weight networks . We consider directions such as neural architecture search ( Howard et al. , 2019 ) , post processing methods to deep networks ( Phan et al. , 2020 ; Stock et al. , 2020 ; Li et al. , 2019a ; Wu et al. , 2018 ; Son et al. , 2018 ) , and heavily tailored efficient networks and training ( Tan & Le , 2019 ) complementary to our efforts . CNN architectures . The tremendous success of applying convolutional neural networks ( CNNs ) on numerous tasks has stimulated rapid developments for more effective and efficient network architectures in both hand-crafted ( Chen et al. , 2017 ; He et al. , 2016 ; Howard et al. , 2017 ; Iandola et al. , 2016 ; Sandler et al. , 2018 ) and automatically discovered ( Elsken et al. , 2018 ; Liu et al. , 2018 ; Pham et al. , 2018 ; Zoph & Le , 2017 ) manners . We consider our work orthogonal to such topologybased methods , as the plug-and-play property of the proposed ACDC allows it to be added to all the aforementioned methods as a replacement to the standard convolution . Besides efforts on studying efficient network architectures , methods for network compression and pruning ( Han et al. , 2015a ; 2016 ; 2015b ; He et al. , 2017 ; Li et al. , 2016 ; Luo et al. , 2017 ) have been extensively studied for decreasing the model size by pruning the inconsequential connections and weights of a network . Methods ( Ha et al. , 2016 ; Savarese & Maire , 2019 ) align with our direction as they are also insensitive to network topologies . And as shown in the experiments , ACDC can achieve higher performance in terms of parameter reduction and classification accuracy with greater flexibility . Kernel decomposition in CNNs Convolutional kernel decomposition has been studied for various objectives . ( Sosnovik et al. , 2019 ) utilizes kernel decomposition as a tool for constructing same kernel with multiple receptive fields . DCFNet ( Qiu et al. , 2018 ) is proposed as a principle way of regularizing the convolutional filter structures by decomposing convolutional filters in CNN as a truncated expansion with pre-fixed bases . Split-wise decomposition over basis elements is introduced in ( Li et al. , 2019a ) to retrain a CNN with reduced parameters . Weight sharing in networks Exploiting sharing weight within CNNs is discussed in ( Son et al. , 2018 ; Wu et al. , 2018 ) as a post-processing step for model compression . Hypernetworks ( Ha et al. , 2016 ) adopt weight generation from a shared network ( denoted as Hypernet ) to exploit correlations among weights empirically . 3 JOINT SUBSPACE CNN MODELING . In this section , we start with a brief introduction to atom-coefficient filter decomposition , which builds the foundation of our joint subspace CNN modeling ( Section 3.1 ) . We then present the empirical support for the joint subspace view to CNN filters through a motivating experiment ( Section 3.2 ) . We show in this motivating experiment that , with atom-coefficient decomposition , a jointly low-rank structure can be clearly observed among coefficients trained independently in different layers of a CNN under certain linear transformations . This observation , together with the well recognized overparametrization problem of CNNs , leads to the idea of subspace coefficients sharing enforced across network sub-structures , which can then be constructed as plug-and-play replacements to the standard convolutions ( Section 3.3 and Section 3.4 ) . 3.1 CONVOLUTIONAL FILTER DECOMPOSITION . Previous works have shown that a convolutional filter in a CNN can be decomposed as a linear combination of pre-fixed basis ( Qiu et al. , 2018 ) . In our approach , we adopt a similar decomposition as shown in Figure 2 , in which a convolutional filter is represented as a linear combination of trainable 2D filter atoms . After decomposition , a convolution layer with c-channel output Y and c′-channel input X becomes Y = K ∗X , K = AD , ( 1 ) where ∗ denotes the convolution operation . As illustrated in Figure 2 , in ( 1 ) , a convolutional filter K ∈ Rc×c′×l×l , which can be seen as a stack of c × c′ 2D convolutional filters with the size of l × l , is reconstructed by multiplying m 2D filter atoms , denoted collectively as D ∈ Rm×l×l , with the corresponding atom coefficients A ∈ Rc×c′×m . Note that square filters are assumed here for simplicity , while all filter shapes are supported . Since both convolution and tensor multiplication are commutative , a convolutional layer can now be decomposed into two : • A atom sub-layer where each atom involves spatial-only convolution with the filter atoms , i.e. , Z ∈ Rc′m×h×w = D ∗X ; • A coefficient sub-layer that linearly combines feature channels from the atom sub-layer : Y ∈ Rc×h×w = AZ . Note that Z here denotes atom sub-layer outputs , and stride 1 and same padding are assumed for the sake of discussion . 3.2 THE MOTIVATION BEHIND l l c′ c × = … KD l l m m c c′ A Figure 2 : Atom-coefficient filter decomposition . A convolutional kernel K is decomposed over a set of m 2D filter atoms D linearly combined by atom coefficients A . Deep CNNs are long recognized to be over-parametrized . The very deep layers in modern CNN structures ( He et al. , 2016 ; Huang et al. , 2017 ; Zagoruyko & Komodakis , 2016 ) and the high-dimensional filters with little structural regularizations lead to hundreds of millions of parameters . Such over-parametrization problem is also observed in the studies of deep representations ( Raghu et al. , 2017 ) , and empirically alleviated by new network structures ( Chollet , 2017 ; Howard et al. , 2017 ) , network compression , and parameter reduction methods ( Ha et al. , 2016 ; Savarese & Maire , 2019 ) . Meanwhile , recent studies on deep representations ( Kornblith et al. , 2019 ; Morcos et al. , 2018 ; Raghu et al. , 2017 ) have shown that there exists obvious correlations in features across layers within a CNN after proper linear transformations . The correlation of features across network layers motivates us to explore and exploit correlations across filters for structural regularizations . We present here a motivating experiment on MNIST by applying CCA alignments as in ( Raghu et al. , 2017 ) to the atom sub-layer outputs and the coefficient sub-layer outputs of layer i and layer j . Note that no atom coefficient sharing is yet imposed here , and the network reports the same testing accuracy before and after filter decomposition . Formally , c , m , d , and hw denote the number of channels , number of filter atoms , test set size , and the 2D feature dimensions , respectively . The atom sub-layer outputs of the i-th and j-th layer , Zi and Zj ∈ Rcm×dhw , are firstly aligned by linear transformations Pi and Pj ∈ Rcm×cm that maximize the correlation ρz = max Pi , Pj corr ( PiZi , PjZj ) . And similarly , the coefficient sub-layer outputs of both layers , Yi and Yj ∈ Rc×dhw , are aligned by Qi and Qj ∈ Rc×c that maximize the correlation ρ = max Qi , Qj corr ( QiYi , QjYj ) . Omitting the layer indexes , the feed forwards of both layers can be rewritten as Y = QAP−1P ( D ∗X ) . ( 2 ) By merging the transform into the coefficients A by Ã = QAP−1 , we obtain ‘ aligned coefficients ’ Ãi and Ãj , that reside in a low rank structure reflected by the very similar effective ranks of Ãi and [ Ãi , Ãj ] . For example , in our MNIST experiment using a 4-layer 32-channel CNN , out of the 6 possible ( i , j ) pairs , the average effective rank of Ãi and [ Ãi , Ãj ] are 31.98 and 38.56 , respectively . Our observations agree with and further support recent studies on cross-layer feature correlations ( Kornblith et al. , 2019 ; Morcos et al. , 2018 ; Raghu et al. , 2017 ) . This observation hints that , instead of treating filters in each layers independently , it is feasible to exploit such intrinsic low-rank structure of coefficients across layers , and impose a joint subspace view to filters in a CNNs . And practically , this observation supports the feasibility of directly reducing the parameter redundancy of a CNN by enforcing shared atom coefficients across layers .

Summary: This paper proposes a scheme for partially sharing CNN parameter across layers.  Specifically, convolutional layer parameters are factored into the product of layer-specific filter atoms (which define spatial filter structure) and a set of shared coefficient that mix features across channels.  Training CNNs reparameterized in this manner yields improvements along the accuracy-parameter tradeoff curve, compared to baseline models on image classification tasks.