Incorporating Horizontal Connections in Convolution by Spatial Shuffling

INCORPORATING HORIZONTAL CONNECTIONS IN CONVOLUTION BY SPATIAL SHUFFLING Anonymous authors Paper under double-blind review Convolutional Neural Networks ( CNNs ) are composed of multiple convolution layers and show elegant performance in vision tasks . The design of the regular convolution is based on the Receptive Field ( RF ) where the information within a specific region is processed . In the view of the regular convolution ’ s RF , the outputs of neurons in lower layers with smaller RF are bundled to create neurons in higher layers with larger RF . As a result , the neurons in high layers are able to capture the global context even though the neurons in low layers only see the local information . However , in lower layers of the biological brain , the information outside of the RF changes the properties of neurons . In this work , we extend the regular convolution and propose spatially shuffled convolution ( ss convolution ) . In ss convolution , the regular convolution is able to use the information outside of its RF by spatial shuffling which is a simple and lightweight operation . We perform experiments on CIFAR-10 and ImageNet-1k dataset , and show that ss convolution improves the classification performance across various CNNs . 1 INTRODUCTION . Convolutional Neural Networks ( CNNs ) and their convolution layers ( Fukushima , 1980 ; Lecun et al. , 1998 ) are inspired by the finding in cat visual cortex ( Hubel & Wiesel , 1959 ) and they show the strong performance in various domains such as image recognition ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; He et al. , 2016 ) , natural language processing ( Gehring et al. , 2017 ) , and speech recognition ( Abdel-Hamid et al. , 2014 ; Zhang et al. , 2016 ) . A notable characteristic of the convolution layer is the Receptive Field ( RF ) , which is the particular input region where a convolutional output is affected by . The units ( or neurons ) in higher layers have larger RF by bundling the outputs of the units in lower layers with smaller RF . Thanks to the hierarchical architectures of CNNs , the units in high layers are able to capture the global context even though the units in low layers only see the local information . It is known that neurons in the primary visual cortex ( i.e. , V1 which is low layers ) change the selfproperties ( e.g. , the RF size ( Pettet & Gilbert , 1992 ) and the facilitation effect ( Nelson & Frost , 1985 ) ) based on the information outside of the RF ( D.Gilbert , 1992 ) . The mechanism is believed to originate from ( 1 ) feedbacks from the higher-order area ( Iacaruso et al. , 2017 ) and ( 2 ) intracortical horizontal connections ( D.Gilbert , 1992 ) . The feedbacks from the higher-order area convey broader-contextual information than the neurons in V1 , which allows the neurons in V1 to use the global context . For instance , Gilbert & Li ( 2013 ) argued that the feedback connections work as attention . Horizontal connections allow the distanced neurons in the layer to communicate with each other and are believed to play an important role in visual contour integration ( Li & Gilbert , 2002 ) and object grouping ( Schmidt et al. , 2006 ) . Though both horizontal and feedback connections are believed to be important for visual processing in the visual cortex , the regular convolution ignores the properties of these connections . In this work , we particularly focus on algorithms to introduce the function of horizontal connections for the regular convolution in CNNs . We propose spatially shuffled convolution ( ss convolution ) , where the information outside of the regular convolution ’ s RF is incorporated by spatial shuffling , which is a simple and lightweight operation . Our ss convolution is the same operation as the regular convolution except for spatial shuffling and requires no extra learnable parameters . The design of ss convolution is highly inspired by the function of horizontal connections . To test the effectiveness of the information outside of the regular convolution ’ s RF in CNNs , we perform experiments on CIFAR-10 ( Krizhevsky , 2009 ) and ImageNet 2012 dataset ( Russakovsky et al. , 2015 ) and show that ss convolution improves the classification performance across various CNNs . These results indicate that the information outside of the RF is useful when processing local information . In addition , we conduct several analyses to examine why ss convolution improves the classification performance in CNNs and show that spatial shuffling allows the regular convolution to use the information outside of its RF . 2 RELATED WORK . 2.1 VARIANTS OF CONVOLUTION LAYERS AND NEURAL MODULES . There are two types of approaches to improve the Receptive Field ( RF ) of CNNs with the regular convolution : broadening kernel of convolution layer and modulating activation values by selfattention . Broadening Kernel : The atrous convolution ( Holschneider et al. , 1989 ; Yu & Koltun , 2016 ) is the convolution with the strided kernel . The stride is not learnable and given in advance . The atrous convolution can have larger RF compared to the regular convolution with the same computational complexity and the number of learnable parameters . The deformable convolution ( Dai et al. , 2017 ) is the atrous convolution with learnable kernel stride that depends on inputs and spatial locations . The stride of the deformable convolution is changed flexibly unlike the atrous convolution , however , the deformable convolution requires extra computations to calculate strides . Both atrous and deformable convolution contribute to broadening RF , however , it is not plausible to use the pixel information at a distant location when processing local information . Let us consider the case that the information of p pixels away is useful for processing local information at layer l. In the simple case , it is known that the size of the RF grows with k √ n where k is the size of the convolution kernel and n is the number of layers ( Luo et al. , 2016 ) . In this case , the size of kernel needs to be p√ n and k is around 45 when p = 100 and l = 5 . If the kernel size is 3 × 3 , then the stride needs to be 21 across layers . Such large stride causes both the atrous and the deformable convolution to have a sparse kernel and it is not suitable for processing local information . Self-Attention : Squeeze and Excitation module ( SE module ) ( Hu et al. , 2018 ) is proposed to modulate the activation values by using the global context which is obtained by Global Average Pooling ( GAP ) ( Lin et al. , 2014 ) . SE module allows CNNs with the regular convolution to use the information outside of its RF as our ss convolution does . In our experiments , ss convolution gives the marginal improvements on SEResNet50 ( Hu et al. , 2018 ) that is ResNet50 ( He et al. , 2016 ) with SE module . This result makes us wonder why ss convolution improves the performance of SEResNet50 , thus we conduct the analyses and find that the RF of SEResNet50 is location independent and the RF of ResNet with ss convolution is the location-dependent . This result is reasonable since the spatial information of activation values is not conserved by GAP in SE module . We conclude that such a difference may be the reason why ss convolution improves the classification performance on SEResNet50 . Attention Branch Networks ( ABN ) ( Fukui et al. , 2019 ) is proposed for a top-down visual explanation by using an attention mechanism . ABN uses the output of the side branch to modulate activation values of the main branch . The outputs of the side branch have larger RF than the one of the main branch , thus the main branch is able to modulate the activation values based on the information outside of main branch ’ s RF . In our experiments , ss convolution improves the performance on ABN and we assume that this is because ABN works as like feedbacks from higher-order areas , unlike ss convolution that is inspired by the function of horizontal connections . 2.2 UTILIZATION OF SHUFFLING IN CNNS . ShuffleNet ( Zhang et al. , 2017 ) is designed for computation-efficient CNN architecture and the group convolution ( Krizhevsky et al. , 2012 ) is heavily used . They shuffle the channel to make cross-group information flow for multiple group convolution layers . The motivation of using shuffling between ShuffleNet and our ss convolution is different . On the one hand , our ss convolution uses spatial shuffling to use the information from outside of the regular convolution ’ s RF . On the other hand , the channel shuffling in ShuffleNet does not broaden RF and not contribute to use the information outside of the RF . 3 METHOD . In this section , we introduce spatially shuffled convolution ( ss convolution ) . 3.1 SPATIALLY SHUFFLED CONVOLUTION . Horizontal connections are the mechanism to use information outside of the RF . We propose ss convolution to incorporate this mechanism into the regular convolution , which consists of two components : spatial shuffling and regular convolution . The shuffling is based on a permutation matrix that is generated at the initialization . The permutation matrix is fixed while training and testing . Our ss convolution is defined as follows : yi , j = C∑ c ∑ ∆i , ∆j∈R wc , ∆i , ∆j · P ( xc , i+∆i , j+∆j ) , ( 1 ) P ( xc , i , j ) = { π ( xc , i , j ) , c ≤ bαCc , xc , i , j , otherwise . ( 2 ) R represents the offset coordination of the kernel . For examples , the case of the 3 × 3 kernel is R = { ( −1 , −1 ) , ( −1 , 0 ) , ( −1 , 1 ) , ( 0 , −1 ) , ( 0 , 0 ) , ( 0 , 1 ) , ( 1 , −1 ) , ( 1 , 0 ) , ( 1 , 1 ) } . x ∈ RC×I×J is the input and w ∈ RCw×Iw×Jw is the kernel weights of the regular convolution . In Eqn . ( 2 ) , the input x is shuffled by P and then the regular convolution is applied . Fig . 1- ( a ) is the visualization of Eqn . ( 2 ) . α ∈ [ 0 , 1 ] is the hyper-parameter to control how many channels are shuffled . If bαCc = 0 , then ss convolution is same as the regular convolution . At the initialization , we randomly generate the permutation matrix π ∈ { 0 , 1 } m×m where ∑m i=1 πi , j = 1 , ∑m j=1 πi , j = 1 and m = I · J · bαCc1 . The generated π at the initialization is fixed for training and testing . The result of CIFAR-10 across various α is shown in Fig . 2 . The biggest improvement of the classification performance is obtained when α is around 0.06 . 3.2 SPATIALLY SHUFFLED GROUP CONVOLUTION . The group convolution ( Krizhevsky et al. , 2012 ) is the variants of the regular convolution . We find that the shuffling operation of Eqn . 2 is not suitable for the group convolution . ResNeXt ( Xie et al. , 2017 ) is CNN to use heavily group convolutions and Table 1 shows the test error of ResNeXt in CIFAR-10 ( Krizhevsky , 2009 ) . As can be seen in Table 1 , the improvement of the classification performance is marginal with Eqn . 2 . Thus , we propose the spatial shuffling for the 1We implement Eqn . 2 by indexing , thus we hold m long int instead of m × m binary matrix . The implementation of ss convolution is shown in Appendix A.2 . group convolution as follows : P ( xc , i , j ) = { π ( xc , i , j ) , 0 ≡ C mod b 1αc , xc , i , j , otherwise . ( 3 ) Eqn . 3 represents that the shuffled parts are interleaved like the illustration in Fig . 1- ( b ) . As can be seen in Table 1 , ss convolution with Eqn . 3 improves the classification performance of ResNeXt .

Summary: In this paper, the authors proposed a shuffle strategy for convolution layers in convolutional neural networks (CNNs). Specifically, the authors argued that the receptive field (RF) of each convolutional filter should be not constrained in the small patch. Instead, it should also cover other locations beyond the local patch and also the single channel. Based on this motivation, the authors proposed a spatial shuffling layer which is aimed at shuffling the original feature responses. In the experimental results, the authors evaluated the proposed ss convolutional layer on CIFAR-10 and ImageNet-1k and compared with various baseline architectures. Besides, the authors further did some ablated analysis and visualizations for the proposed ss convolutional layer.