A Joint Subspace View to Convolutional Neural Networks

1 INTRODUCTION . Modern developments of deep network components ( Ioffe & Szegedy , 2015 ; Xu et al. , 2015 ) , structures ( He et al. , 2016 ; Huang et al. , 2017 ) , and training techniques ( Kingma & Ba , 2014 ; Zoph & Le , 2017 ) have enabled deep convolutional neural networks to consistently grow deeper and wider , with hundreds of millions of parameters . The trend of growing network scales leads to both severe challenges and urgent needs to fast adaptations ( Finn et al. , 2017 ; Lopez-Paz & Ranzato , 2017 ; Shin et al. , 2017 ) , parameter efficiency ( Cheng et al. , 2017 ; Han et al. , 2015a ; Luo et al. , 2017 ; Savarese & Maire , 2019 ; Yang et al. , 2019 ; Wang et al. , 2018b ) , and interpretability ( Selvaraju et al. , 2017 ; Zhou et al. , 2016 ) . While recent works on efficient convolution operations ( Chollet , 2017 ; Howard et al. , 2017 ; Ma et al. , 2018 ; Zhang et al. , 2018 ) alleviate the long recognized over-parametrization problem of deep CNNs , and practical methods on modeling the shared components across CNN filters have been proposed ( Ha et al. , 2016 ; Savarese & Maire , 2019 ) , there is still a lack of a principled view to exploiting the common structure of the filters within a CNN . While preliminary studies have revealed the redundancies in deep network parameters ( Michel et al. , 2019 ; Raghu et al. , 2017 ) , and the fact that deep features across layers are highly correlated under certain linear transforms ( Kornblith et al. , 2019 ; Morcos et al. , 2018 ; Raghu et al. , 2017 ) , exploiting this potential redundancy remains challenging . We suspect that the spatial patterns of convolutional filters , which correspond to different semantics levels , are the main reason why we can not observe obvious cross-layer filter low-rankness . To validate our hypothesis , we decouple the spatial patterns and channel mixing of a convolutional filter by decomposing it over layer-specific 2D filter atoms , linearly combined using atom coefficients . Then with proper alignments to the layer outputs , we finally observe a low-rank structure among those atom coefficients across different layers . This highly non-trivial observation , which to our knowledge has never been reported before , hints the feasibility of enforcing a common block of atom coefficients across layers and suggests a joint subspace view to CNNs . Applying to a wide range of network architectures , this joint subspace view leads subsequently to a novel CNN architecture in which the majority of parameters are shared across layers as a common block of atom coefficients , with only a few hundred parameters remaining specific to each layer as filter atoms as shown in Figure 1 . This matches the intuition visualized in Figure 1 that important image regions , e.g. , the wings , beak , and tail of a bird , remain important across different layers and scales to output an informative deep representation . Atom-coefficient decomposition disentangles spatial convolution and channel mixing . Thus , channels that describe important region features will highly likely be consistently assigned with large weights across layers , in order to carry on those critical information . Thus , atom coefficients , i.e. , weights to linearly combine channels , now become shareable across layers to permit joint subspace CNN modeling . Our approach accepts many easily constructed variants , e.g. , using different filter atom numbers and sharing substructures , to allow highly flexible trade-offs between parameter reduction and model expressiveness . This simple and plug-and-play joint subspace view opens the door to better interpreting , training , adapting , and compacting deep models . We support our findings with extensive empirical evidence in this paper . By using variants of ACDC as plug-and-play replacements to the standard convolutions , we observe comparable and even better performance on challenging image classification datasets with orders of magnitude smaller models . We further demonstrate with few shot learning experiments that ACDC improves the adaptation of deep networks on novel tasks with limited supervisions . We end our paper with a model interpretability discussion . Our main contributions are summarized as follows : • We introduce a joint subspace view to CNN , which is compatible to a wide range of network architectures , to provide a structural regularization over CNN filters for better model interpretability , training , adaptation and compaction . • Highlighting the remarkable flexibility and compatibility , we introduce various model variants constructed easily by sharing coefficient within different network sub-structures . • We validate the effectiveness of our approach by plug-and-playing them into modern CNN architectures for various real-world tasks . 2 RELATED WORK . Before presenting the connections of our method with the literature , let us comment on the complementary efforts on network compression . Contrary to recent efforts on network compression and efficient architectures , we instead explore and exploit the underlying correspondence and correlations among convolutional layers within a given CNN . We decompose the convolutional kernels among layers , and enforce shared composition coefficients within sub-structures of a CNN . This plug-and-play reparametrization of convolutional kernels significantly reduces the network redundancy while maintaining performance , improving adaptation , providing additional interpretability , reducing computational cost , and adding compression ( as a by-product ) comparable to many modern heavily tailored light-weight networks . We consider directions such as neural architecture search ( Howard et al. , 2019 ) , post processing methods to deep networks ( Phan et al. , 2020 ; Stock et al. , 2020 ; Li et al. , 2019a ; Wu et al. , 2018 ; Son et al. , 2018 ) , and heavily tailored efficient networks and training ( Tan & Le , 2019 ) complementary to our efforts . CNN architectures . The tremendous success of applying convolutional neural networks ( CNNs ) on numerous tasks has stimulated rapid developments for more effective and efficient network architectures in both hand-crafted ( Chen et al. , 2017 ; He et al. , 2016 ; Howard et al. , 2017 ; Iandola et al. , 2016 ; Sandler et al. , 2018 ) and automatically discovered ( Elsken et al. , 2018 ; Liu et al. , 2018 ; Pham et al. , 2018 ; Zoph & Le , 2017 ) manners . We consider our work orthogonal to such topologybased methods , as the plug-and-play property of the proposed ACDC allows it to be added to all the aforementioned methods as a replacement to the standard convolution . Besides efforts on studying efficient network architectures , methods for network compression and pruning ( Han et al. , 2015a ; 2016 ; 2015b ; He et al. , 2017 ; Li et al. , 2016 ; Luo et al. , 2017 ) have been extensively studied for decreasing the model size by pruning the inconsequential connections and weights of a network . Methods ( Ha et al. , 2016 ; Savarese & Maire , 2019 ) align with our direction as they are also insensitive to network topologies . And as shown in the experiments , ACDC can achieve higher performance in terms of parameter reduction and classification accuracy with greater flexibility . Kernel decomposition in CNNs Convolutional kernel decomposition has been studied for various objectives . ( Sosnovik et al. , 2019 ) utilizes kernel decomposition as a tool for constructing same kernel with multiple receptive fields . DCFNet ( Qiu et al. , 2018 ) is proposed as a principle way of regularizing the convolutional filter structures by decomposing convolutional filters in CNN as a truncated expansion with pre-fixed bases . Split-wise decomposition over basis elements is introduced in ( Li et al. , 2019a ) to retrain a CNN with reduced parameters . Weight sharing in networks Exploiting sharing weight within CNNs is discussed in ( Son et al. , 2018 ; Wu et al. , 2018 ) as a post-processing step for model compression . Hypernetworks ( Ha et al. , 2016 ) adopt weight generation from a shared network ( denoted as Hypernet ) to exploit correlations among weights empirically . 3 JOINT SUBSPACE CNN MODELING . In this section , we start with a brief introduction to atom-coefficient filter decomposition , which builds the foundation of our joint subspace CNN modeling ( Section 3.1 ) . We then present the empirical support for the joint subspace view to CNN filters through a motivating experiment ( Section 3.2 ) . We show in this motivating experiment that , with atom-coefficient decomposition , a jointly low-rank structure can be clearly observed among coefficients trained independently in different layers of a CNN under certain linear transformations . This observation , together with the well recognized overparametrization problem of CNNs , leads to the idea of subspace coefficients sharing enforced across network sub-structures , which can then be constructed as plug-and-play replacements to the standard convolutions ( Section 3.3 and Section 3.4 ) . 3.1 CONVOLUTIONAL FILTER DECOMPOSITION . Previous works have shown that a convolutional filter in a CNN can be decomposed as a linear combination of pre-fixed basis ( Qiu et al. , 2018 ) . In our approach , we adopt a similar decomposition as shown in Figure 2 , in which a convolutional filter is represented as a linear combination of trainable 2D filter atoms . After decomposition , a convolution layer with c-channel output Y and c′-channel input X becomes Y = K ∗X , K = AD , ( 1 ) where ∗ denotes the convolution operation . As illustrated in Figure 2 , in ( 1 ) , a convolutional filter K ∈ Rc×c′×l×l , which can be seen as a stack of c × c′ 2D convolutional filters with the size of l × l , is reconstructed by multiplying m 2D filter atoms , denoted collectively as D ∈ Rm×l×l , with the corresponding atom coefficients A ∈ Rc×c′×m . Note that square filters are assumed here for simplicity , while all filter shapes are supported . Since both convolution and tensor multiplication are commutative , a convolutional layer can now be decomposed into two : • A atom sub-layer where each atom involves spatial-only convolution with the filter atoms , i.e. , Z ∈ Rc′m×h×w = D ∗X ; • A coefficient sub-layer that linearly combines feature channels from the atom sub-layer : Y ∈ Rc×h×w = AZ . Note that Z here denotes atom sub-layer outputs , and stride 1 and same padding are assumed for the sake of discussion . 3.2 THE MOTIVATION BEHIND l l c′ c × = … KD l l m m c c′ A Figure 2 : Atom-coefficient filter decomposition . A convolutional kernel K is decomposed over a set of m 2D filter atoms D linearly combined by atom coefficients A . Deep CNNs are long recognized to be over-parametrized . The very deep layers in modern CNN structures ( He et al. , 2016 ; Huang et al. , 2017 ; Zagoruyko & Komodakis , 2016 ) and the high-dimensional filters with little structural regularizations lead to hundreds of millions of parameters . Such over-parametrization problem is also observed in the studies of deep representations ( Raghu et al. , 2017 ) , and empirically alleviated by new network structures ( Chollet , 2017 ; Howard et al. , 2017 ) , network compression , and parameter reduction methods ( Ha et al. , 2016 ; Savarese & Maire , 2019 ) . Meanwhile , recent studies on deep representations ( Kornblith et al. , 2019 ; Morcos et al. , 2018 ; Raghu et al. , 2017 ) have shown that there exists obvious correlations in features across layers within a CNN after proper linear transformations . The correlation of features across network layers motivates us to explore and exploit correlations across filters for structural regularizations . We present here a motivating experiment on MNIST by applying CCA alignments as in ( Raghu et al. , 2017 ) to the atom sub-layer outputs and the coefficient sub-layer outputs of layer i and layer j . Note that no atom coefficient sharing is yet imposed here , and the network reports the same testing accuracy before and after filter decomposition . Formally , c , m , d , and hw denote the number of channels , number of filter atoms , test set size , and the 2D feature dimensions , respectively . The atom sub-layer outputs of the i-th and j-th layer , Zi and Zj ∈ Rcm×dhw , are firstly aligned by linear transformations Pi and Pj ∈ Rcm×cm that maximize the correlation ρz = max Pi , Pj corr ( PiZi , PjZj ) . And similarly , the coefficient sub-layer outputs of both layers , Yi and Yj ∈ Rc×dhw , are aligned by Qi and Qj ∈ Rc×c that maximize the correlation ρ = max Qi , Qj corr ( QiYi , QjYj ) . Omitting the layer indexes , the feed forwards of both layers can be rewritten as Y = QAP−1P ( D ∗X ) . ( 2 ) By merging the transform into the coefficients A by Ã = QAP−1 , we obtain ‘ aligned coefficients ’ Ãi and Ãj , that reside in a low rank structure reflected by the very similar effective ranks of Ãi and [ Ãi , Ãj ] . For example , in our MNIST experiment using a 4-layer 32-channel CNN , out of the 6 possible ( i , j ) pairs , the average effective rank of Ãi and [ Ãi , Ãj ] are 31.98 and 38.56 , respectively . Our observations agree with and further support recent studies on cross-layer feature correlations ( Kornblith et al. , 2019 ; Morcos et al. , 2018 ; Raghu et al. , 2017 ) . This observation hints that , instead of treating filters in each layers independently , it is feasible to exploit such intrinsic low-rank structure of coefficients across layers , and impose a joint subspace view to filters in a CNNs . And practically , this observation supports the feasibility of directly reducing the parameter redundancy of a CNN by enforcing shared atom coefficients across layers .

Summary: This work first shows empirical observations that there exist obvious correlations in features across layers within a CNN after proper linear transformations. Accordingly, the authors propose coefficient sharing strategies across layers or across groups of filters, which aim to decompose convolution filters for reducing model sizes. The experiments are conducted on four image benchmarks using several CNN backbones, and the results show the effectiveness of the proposed ACDC on image classification and few-shot image classification tasks.