Title: INCORPORATING HORIZONTAL CONNECTIONS IN CONVOLUTION BY SPATIAL SHUFFLING

Abstract: Paper under double-blind review Convolutional Neural Networks (CNNs) are composed of multiple convolution layers and show elegant performance in vision tasks. The design of the regular convolution is based on the Receptive Field (RF) where the information within a specific region is processed. In the view of the regular convolution's RF, the outputs of neurons in lower layers with smaller RF are bundled to create neurons in higher layers with larger RF. As a result, the neurons in high layers are able to capture the global context even though the neurons in low layers only see the local information. However, in lower layers of the biological brain, the information outside of the RF changes the properties of neurons. In this work, we extend the regular convolution and propose spatially shuffled convolution (ss convolution). In ss convolution, the regular convolution is able to use the information outside of its RF by spatial shuffling which is a simple and lightweight operation. We perform experiments on CIFAR-10 and ImageNet-1k dataset, and show that ss convolution improves the classification performance across various CNNs.

1. INTRODUCTION:

Convolutional Neural Networks (CNNs) and their convolution layers (Fukushima, 1980; Lecun et al., 1998) are inspired by the finding in cat visual cortex (Hubel & Wiesel, 1959) and they show the strong performance in various domains such as image recognition (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; He et al., 2016) , natural language processing (Gehring et al., 2017) , and speech recognition (Abdel-Hamid et al., 2014; Zhang et al., 2016) . A notable characteristic of the convolution layer is the Receptive Field (RF), which is the particular input region where a convolutional output is affected by. The units (or neurons) in higher layers have larger RF by bundling the outputs of the units in lower layers with smaller RF. Thanks to the hierarchical architectures of CNNs, the units in high layers are able to capture the global context even though the units in low layers only see the local information.

It is known that neurons in the primary visual cortex (i.e., V1 which is low layers) change the selfproperties (e.g., the RF size (Pettet & Gilbert, 1992) and the facilitation effect (Nelson & Frost, 1985) ) based on the information outside of the RF (D. Gilbert, 1992) . The mechanism is believed to originate from (1) feedbacks from the higher-order area (Iacaruso et al., 2017) and (2) intracortical horizontal connections (D. Gilbert, 1992) . The feedbacks from the higher-order area convey broader-contextual information than the neurons in V1, which allows the neurons in V1 to use the global context. For instance, Gilbert & Li (2013) argued that the feedback connections work as attention. Horizontal connections allow the distanced neurons in the layer to communicate with each other and are believed to play an important role in visual contour integration (Li & Gilbert, 2002) and object grouping (Schmidt et al., 2006) .

Though both horizontal and feedback connections are believed to be important for visual processing in the visual cortex, the regular convolution ignores the properties of these connections. In this work, we particularly focus on algorithms to introduce the function of horizontal connections for the regular convolution in CNNs. We propose spatially shuffled convolution (ss convolution), where the information outside of the regular convolution's RF is incorporated by spatial shuffling, which is a simple and lightweight operation. Our ss convolution is the same operation as the regular convolution except for spatial shuffling and requires no extra learnable parameters. The design of ss convolution is highly inspired by the function of horizontal connections. To test the effectiveness of the information outside of the regular convolution's RF in CNNs, we perform experiments on CIFAR-10 (Krizhevsky, 2009) and ImageNet 2012 dataset (Russakovsky et al., 2015) and show that ss convolution improves the classification performance across various CNNs. These results indicate that the information outside of the RF is useful when processing local information. In addition, we conduct several analyses to examine why ss convolution improves the classification performance in CNNs and show that spatial shuffling allows the regular convolution to use the information outside of its RF.

2. RELATED WORK:



2.1. VARIANTS OF CONVOLUTION LAYERS AND NEURAL MODULES:

There are two types of approaches to improve the Receptive Field (RF) of CNNs with the regular convolution: broadening kernel of convolution layer and modulating activation values by selfattention.

Broadening Kernel::

The atrous convolution (Holschneider et al., 1989; Yu & Koltun, 2016) is the convolution with the strided kernel. The stride is not learnable and given in advance. The atrous convolution can have larger RF compared to the regular convolution with the same computational complexity and the number of learnable parameters.

The deformable convolution (Dai et al., 2017) is the atrous convolution with learnable kernel stride that depends on inputs and spatial locations. The stride of the deformable convolution is changed flexibly unlike the atrous convolution, however, the deformable convolution requires extra computations to calculate strides.

Both atrous and deformable convolution contribute to broadening RF, however, it is not plausible to use the pixel information at a distant location when processing local information. Let us consider the case that the information of p pixels away is useful for processing local information at layer l. In the simple case, it is known that the size of the RF grows with k √ n where k is the size of the convolution kernel and n is the number of layers (Luo et al., 2016) . In this case, the size of kernel needs to be p √ n and k is around 45 when p = 100 and l = 5. If the kernel size is 3 × 3, then the stride needs to be 21 across layers. Such large stride causes both the atrous and the deformable convolution to have a sparse kernel and it is not suitable for processing local information.

Self-Attention::

Squeeze and Excitation module (SE module) (Hu et al., 2018) is proposed to modulate the activation values by using the global context which is obtained by Global Average Pooling (GAP) (Lin et al., 2014) . SE module allows CNNs with the regular convolution to use the information outside of its RF as our ss convolution does. In our experiments, ss convolution gives the marginal improvements on SEResNet50 (Hu et al., 2018) that is ResNet50 (He et al., 2016) with SE module. This result makes us wonder why ss convolution improves the performance of SEResNet50, thus we conduct the analyses and find that the RF of SEResNet50 is location independent and the RF of ResNet with ss convolution is the location-dependent. This result is reasonable since the spatial information of activation values is not conserved by GAP in SE module. We conclude that such a difference may be the reason why ss convolution improves the classification performance on SEResNet50.

Attention Branch Networks (ABN) (Fukui et al., 2019) is proposed for a top-down visual explanation by using an attention mechanism. ABN uses the output of the side branch to modulate activation values of the main branch. The outputs of the side branch have larger RF than the one of the main branch, thus the main branch is able to modulate the activation values based on the information outside of main branch's RF. In our experiments, ss convolution improves the performance on ABN and we assume that this is because ABN works as like feedbacks from higher-order areas, unlike ss convolution that is inspired by the function of horizontal connections.

2.2. UTILIZATION OF SHUFFLING IN CNNS:

ShuffleNet (Zhang et al., 2017) is designed for computation-efficient CNN architecture and the group convolution (Krizhevsky et al., 2012) is heavily used. They shuffle the channel to make cross-group information flow for multiple group convolution layers.

The motivation of using shuffling between ShuffleNet and our ss convolution is different. On the one hand, our ss convolution uses spatial shuffling to use the information from outside of the regular 

3. METHOD:

In this section, we introduce spatially shuffled convolution (ss convolution).

3.1. SPATIALLY SHUFFLED CONVOLUTION:

Horizontal connections are the mechanism to use information outside of the RF. We propose ss convolution to incorporate this mechanism into the regular convolution, which consists of two components: spatial shuffling and regular convolution. The shuffling is based on a permutation matrix that is generated at the initialization. The permutation matrix is fixed while training and testing.

Our ss convolution is defined as follows:

EQUATION

EQUATION

R represents the offset coordination of the kernel. For examples, the case of the 3 × 3 kernel is R = {(-1, -1), (-1, 0), (-1, 1), (0, -1), (0, 0), (0, 1), (1, -1), (1, 0), (1, 1)}. x ∈ R C×I×J is the input and w ∈ R Cw×Iw×Jw is the kernel weights of the regular convolution. In Eqn.

(2), the input x is shuffled by P and then the regular convolution is applied. (2). α ∈ [0, 1] is the hyper-parameter to control how many channels are shuffled. If αC = 0, then ss convolution is same as the regular convolution. At the initialization, we randomly generate the permutation matrix π ∈ {0, 1} m×m where

m i=1 π i,j = 1, m j=1 π i,j = 1 and m = I • J • αC 1 .

The generated π at the initialization is fixed for training and testing.

The result of CIFAR-10 across various α is shown in Fig. 2 . The biggest improvement of the classification performance is obtained when α is around 0.06.

3.2. SPATIALLY SHUFFLED GROUP CONVOLUTION:

The group convolution (Krizhevsky et al., 2012) is the variants of the regular convolution. We find that the shuffling operation of Eqn. 2 is not suitable for the group convolution. ResNeXt (Xie et al., 2017) is CNN to use heavily group convolutions and Table 1 shows the test error of ResNeXt in CIFAR-10 (Krizhevsky, 2009) . As can be seen in Table 1 , the improvement of the classification performance is marginal with Eqn. 2. Thus, we propose the spatial shuffling for the 3.9

Table 1 : The result of CIFAR-10. The test error at the last epoch is used and the reported results are averaged out of 5 runs.

group convolution as follows:

P (x c,i,j )= π(x c,i,j ), 0 ≡ C mod 1 α , x c,i,j ,

otherwise.

(3)

Eqn. 3 represents that the shuffled parts are interleaved like the illustration in Fig. 1 3 .

In our experiments, we replace all regular convolutions with ss convolutions except for downsampling layers, and use single α across all layers. We conduct grid search of α ∈ {0.02, 0.04, 0.06} and α is decided according to the classification performance on validation dataset.

4.2. RESULTS:

We replace all regular convolutions with ss convolutions to investigate whether the information outside of the regular convolution's RF contributes to improving the generalization ability. The results are shown in Table 2 and 3 . As can be seen in Table 2 and 3, ss convolution contributes to improve the classification performance across various CNNs except for SEResNet50 that shows marginal improvements. The detailed analysis of the reason why ss convolution gives the marginal improvements in SEResNet50 is shonw in Sec. 5

Since α ∈ {0.02, 0.04, 0.06} is small, the small portion of the input are shuffled, thus ss convolution improves the classification performance with small amount of extra shuffling operations and without extra learnable parameters. The inference speed is shown in Table 4 and ss convolution make the inference speed 1.15 times slower in exchange for 0.5% improvements in ImageNet-1k dataset. The more efficient implementation2 may decrease the gap of the inference speed between the regular convolution and ss convolution.

5. ANALYSIS:

In this section, we demonstrate two analysis to understand why ss convolution improves the classification performance across various CNNs: the receptive field (RF) analysis and the layer ablation experiment. 

× 3 conv3 x 28 × 28 4 × 4 1 × 1, 128 3 × 3, 128 1 × 1, 512 × 4 conv4 x 14 × 14 2 × 2 1 × 1, 256 3 × 3, 256 1 × 1, 1024 × 6 conv5 x 7 × 7 1 × 1 1 × 1, 512 3 × 3, 512 1 × 1, 2048 × 3 1 × 1 average pool, 1000-d fc, softmax

Table 5 : The structure of ResNet50 (He et al., 2016) .

Receptive Field Analysis. We calculate the RF of SEResNet50, ResNet50 with ss convolution and the regular convolution. The purpose of this analysis is to examine whether ss convolution contributes to use the information outside of the regular convolution's RF.

Layer Ablation Experiment. The layer ablation experiment is conducted to know which ss convolution influences the model prediction. In the primary visual cortex, the neurons change selfproperties based on the information outside of RF, thus we would like to investigate whether spatial shuffling in low layers contribute to predictions or not.

Our analyses are based on ImageNet-1k pre-trained model and the structure of ResNet50 (i.e., the base model for analysis) is shown in Table 5 .

5.1. DOES SPATIAL SHUFFLING CONTRIBUTE TO USE THE INFORMATION FROM OUTSIDE OF RECEPTIVE FIELD?:

In our analysis, we calculate the RF to investigate whether ss convolution uses the information outside of the regular convolution's RF. The receptive field is obtained by optimization as follows:

EQUATION

x ∈ R C×I×J is input, and R ∈ R C×I×J is the RF to calculate and learnable. σ is sigmoid function, thus 0 ≤ σ(R) ≤ 1. φ l is the outpus of the trained model at the layer l.

We call the first term in Eqn. 4 as the local perceptual loss. It is similar to the perceptual loss (Johnson et al., 2016) , and the difference is the dot product of M ∈ {0, 1} C×I×J that works as masking. M is the binary mask and We use Adam optimizer (Kingma & Ba, 2015) to calculate R * . As the hyper-parameter, lr, β 1 , and β 2 are 0.1, 0.9, 0.99, respectively. The high lr is used since its slow convergence. The batch size is 32 and we stop the optimization after 10000 iterations. x is randomly selected from ImageNet-1k training images. The data augmentation and preprocessing are applied as the same procedure in Sec. 4.1. Layerwise Receptive Field. We calculate the RF for each model and the results are shown in Fig. 3 . The top row is the RF of ResNet with the regular convolution, the middle row is the one with ss convolution and the bottom row is the one of SEResNet50. The red color indicates that the value of the pixel there changes features inside the blue box, and the white color represents that features inside the blue box are invariant even if the value of the pixel there is changed.

M cij = 1 if 96 ≤ i, j ≤ 128, otherwise M cij =

In the top row of Fig. 3 , the RFs of ResNet50 with the regular convolution are shown. The size of RF becomes larger as the layer becomes deeper. This result is reasonable and obtained RFs are in the classical view. If the RF of ResNet50 with ss convolution is beyond the one with the regular convolution, it indicates that ss convolution successfully uses the information outside of the regular convolution's RF.

In the middle and bottom row of Fig. 3 are the RF of ResNet50 with ss convolution and SEResNet50, respevtively. The RFs covers the entire image unlike the RF with the regular convolution. These results indicate that both SE module and ss convolution contributes to use the information outside of the regular convolution's RF. is in the range between 0 and 1 and represents the ratio of σ(RF) that is bigger than 0.5. As can be seen in Fig. 5-(a) , the size of RFs are consistently almost 1 across layers in ResNet50 with ss convolution and SEResNet50. This result also shows that SE module and the ss convolution contributes to use the information outside of the regular convolution's RF. This may be the reason why ss convolution improves the classification performance on various CNNs. However, these results make us wonder why ss convolution improves marginally the performance of SEResNet50. Further analysis is conducted in channelwise RF analysis. These results indicate that the information outside of the regular convolution's RF is location-independent in SEResNet 50 and location-dependent in ResNet50 with ss convolution. This is reasonable since SE module uses the global average pooling and the spatial information is not conserved. This difference may be the reason why ss convolution marginally improves the classification performance on SEResNet50.

5.2. ABLATION STUDY:

We conduct layer ablation study to investigate which ss convolutions contribute to the generalization ability. The ablation is done as follows: P (x c,i,j ) = 0, c ≤ αC , x c,i,j , otherwise.

(5)

Eqn. 5 represents that the activation values of the shuffled parts become 0. The result of the ablation experiment is shown in Fig. 5 -(b). Eqn. 5 is applied to all ss convolutions in each block and the biggest drop of the classification performance happens at the ablation of conv4 4. It indicates that it is useful to use the information outside of the regular convolution's RF between the middle and high layers. The classification performance is degraded even if the ablation is applied to the first bottleneck (i.e., conv2 1). This result implies that the information outside of the regular convolution's RF is useful even at low layers.

6. CONCLUSION:

In this work, we propose spatially shuffled convolution (ss convolution) to incorporate the function of horizontal connections in the regular convolution. The spatial shuffling is simple, lightweight, and requires no extra learnable parameters. The experimental results demonstrate that ss convolution captures the information outside of the regular convolution's RF even in lower layers. The results and our analyses also suggest that using distant information (i.e., non-local) is effective for the regular convolution and improves classification performance across various CNNs. 

a) conv2 1 (b) conv2 2 (c) conv2 3 (d) conv3 1 (e) conv3 2 (f) conv3 3 (g) conv3 4 (h) conv4 1 (i) conv4 2 (j) conv4 3 (k) conv4 4 (l) conv4 5 (m) conv4 6 (n) conv5 1 (o) conv5 2 (p) conv5 3

a) conv2 1 (b) conv2 2 (c) conv2 3 (d) conv3 1 (e) conv3 2 (f) conv3 3 (g) conv3 4 (h) conv4 1 (i) conv4 2 (j) conv4 3 (k) conv4 4 (l) conv4 5 (m) conv4 6 (n) conv5 1 (o) conv5 2 (p) conv5 3

Figure 8 : The receptive field of ImageNet-1k pre-trained SEResNet50. The red color indicates that the pixel there changes features inside the blue box, and the white color represents that features are invariant even if the pixel there changes the value itself. Those images are the receptive field of all layers and the name of the layer is described in Table 5 s u p e r ( SSConv2d , s e l f ) . , i n p l a n e s , h e i g h t , w i d t h = x . s i z e ( ) 13 s e l f . s h u f f l e u n t i l h e r e = i n t ( i n p l a n e s * s e l f . a l p h a ) 14 # i f s e l f . s h u f f l e u n t i l h e r e = 0 , t h e n i t ' s e x a c t l y same a s r e g u l a r c o n v o l u t i o n 15 i f s e l f . s h u f f l e u n t i l h e r e >= 1 : 16 s e l f . r e g i s t e r b u f f e r ( ' r a n d o m i n d i c e s ' , t o r c h . r a n d p e r m ( s e l f . s h u f f l e u n t i l h e r e * h e i g h t * w i d t h ) ) 

:

We implement Eqn.

by indexing, thus we hold m long int instead of m × m binary matrix. The implementation of ss convolution is shown in Appendix A.2.

Our implementation is shown in Appendix A.2.

Figures and Tables:

Figure 1: The overview of spatial shuffling operarion.

Fig. 1-(a) is the visualization of Eqn.

Figure 2: The result of CIFAR-10 across various α. WRN 16-4 (Zagoruyko & Komodakis, 2016) and DenseNet-BC 12-100 (Huang et al., 2017) are used. The test error at the last epoch is used and the reported results are averaged out of 5 runs.

Conv w/ Eqn. (2) 4.2 SS Conv w/ Eqn. (3)

0 in our analysis. In other words, the values inside the blue box in Fig 3 are the part of M cij = 1. The local perceptual loss minimizes the distance of feature on the specific region between σ(R) • x and x. The 2nd term is the penalty to evade the trivial case such as σ(R) = 1. In layerwise and channelwise RF anlysis, we use β of 1.0 × 10 -6 and 1.0 × 10 -12 , respectively.

Figure 3: The layerwise RF of ImageNet-1k pre-trained models. The layerwise RF images in the top row are based on ResNet50 with the regular convolution, the ones in the middle row are calculated from ResNet50 with ss convolutions and the ones in the bottom row are generated from SEResNet50. The red color indicates that the pixel there changes features inside blue box, and the white color represents that features are invariant. The name of the layer is described in Table 5. The rest of RFs are shown in Appendix A.1.

shows the size of the RF across layers. The horizontal axis is the name of the layer and the vertical axis represents the size of RF that is calculated asσ(R)≥0.5 1 |R|where 1 is the L1 norm

Figure 4: The channelwise RF of ImageNet-1k pre-trained model. The RF is calculated based on the output of the channel 64 in conv2 1. (a)-(b) RF of ResNet50 with ss convolution. (c)-(d) RF of SEResNet50. The red color indicates that the pixel there changes features inside the blue box, and the white color represents that features are invariant. For clarity, we convert the values that satisfy σ(R) ≥ 0.1 into 1.0.

(a) The size of RF for each layer.(b) The result of the ablation experiment.

Figure 5: The analysis for each layer in ImageNet-1k pretrained models. (a) The horizontal axis is which block is used to calculate the layerwise RF and the vertical axis is the size of RF. (b) The horizontal axis is which block is ablated in ResNet50 with ss convolutions and the vertical axis is the top-1 error in ImageNet-1k validation dataset.

Figure6: The receptive field of ImageNet-1k pre-trained ResNet50. The red color indicates that the pixel there changes features inside the blue box, and the white color represents that features are invariant even if the pixel there changes the value itself. Those images are the receptive field of all layers and the name of the layer is described in Table5



Figure7: The receptive field of ImageNet-1k pre-trained ResNet50 with ss convolutions. The red color indicates that the pixel there changes features inside the blue box, and the white color represents that features are invariant even if the pixel there changes the value itself. Those images are the receptive field of all layers and the name of the layer is described in Table5



f . conv = t o r c h . nn . Conv2d ( i n p l a n e s , o u t p l a n e s , k e r n e l s i z e = k e r n e l s i z e , s t r i d e = s t r i d e , p a d d i n g = p a d d i n g , g r o u p s = g r o u p s , d i l a t i o n = d i l a t i o n , b i a s = b i a s ) 9 s e l f . a l p h a , s e l f . g r o u p s = a l p h a , g r o u p s 10 11 d e f c r e a t e s h u f f l e i n d i c e s ( s e l f , x ) : 12

a t i c m e t h o d 19 d e f g r o u p ( s h u f f l e d x , n o n s h u f f l e d x ) : 20 b a t c h , c h n s , h e i g h t , w i d t h = n o n s h u f f l e d x . s h a p e 21 , c h s , , = s h u f f l e d x . s h a p e 22 l e n g t h = i n t ( c h n s / c h s ) 23 r e s i d u e = c h n sl e n g t h * c u r n t o r c h . c a t ( ( s h u f f l e d x . u n s q u e e z e ( 1 ) , n o n s h u f f l e d x .view ( b a t c h , l e n g t h , c h s , h e i g h t , w i d t h ) ) , 1 ) . view ( b a t c h , c h n s + c h s , h e i g h t , w i d t h )e l s e : 28 r e t u r n t o r c h . c a t ( ( t o r c h . c a t ( ( s h u f f l e d x . u n s q u e e z e ( 1 ) , n o n s h u f f l e d x [ : , r e s i d u e : ] . view ( b a t c h , l e n g t h , c h s , h e i g h t , w i d t h ) ) , 1 ) . view ( b a t c h , c h n s + c h sr e s i d u e , h e i g h t , w i d t h ) , n o n s h u f f l e d x [ : , : r e s i d u e ] ) , 1 ) 29 30 d e f s h u f f l e ( s e l f , x ) : 31 i f s e l f . s h u f f l e u n t i l h e r e >= 1 : 32 # s s c o n v o l u t i o n 33 s h u f f l e d x , n o n s h u f f l e d x = x [ : , : s e l f . s h u f f l e u n t i l h e r e ] , x [ : , s e l f . s h u f f l e u n t i l h e r e : ] 34 b a t c h , ch , h e i g h t , w i d t h = s h u f f l e d x . s i z e ( ) 35 s h u f f l e d x = t o r c h . i n d e x s e l e c t ( s h u f f l e d x . view ( b a t c h , -1) , 1 , s e l f . r a n d o m i n d i c e s ) . view ( b a t c h , ch , h e i g h t , w i d t h ) 36 i f s e l f . g r o u p s >= 2 : 37 r e t u r n s e l f . g r o u p ( s h u f f l e d x , n o n s h u f f l e d x ) 38 e l s e : 39 r e t u r n t o r c h . c a t ( ( s h u f f l e d x , n o n s h u f f l e d x ) , 1 a s a t t r ( s e l f , ' r a n d o m i n d i c e s ' ) i s F a l s e : 46 # c r e a t e random p e r m u t a t i o n m a t r i x a t i n i t i a l i z a t i o n 47 s e l f . c r e a t e s h u f f l e i n d i c e s ( x

CIFAR-10 is the image classification dataset. There are 50000 training images and 10000 validation images with 10 classes. As data augmentation and preprocessing, translation by 4 pixels, stochastic horizontal flipping, and global contrast normalization are applied onto images with 32 × 32 pixels. We use three types of models ofWRN 16-4 (Zagoruyko & Komodakis, 2016), DenseNet-BC 12-100(Huang et al., 2017) and ResNeXt 2-64d(Xie et al., 2017). The result of CIFAR-10. The test error at the last epoch is used and the reported results are averaged out of 5 runs.

Table content: <table><tr><td>4 EXPERIMENTS</td></tr><tr><td>4.1 PREPARATIONS</td></tr><tr><td>We use CIFAR-10 (Krizhevsky, 2009) and ImageNet-1k (Russakovsky et al., 2015) for our experi-</td></tr><tr><td>ments.</td></tr><tr><td>CIFAR-10. ImageNet-1k. ImageNet-1k is the large scale dataset for the image classification. There are 1.28M</td></tr><tr><td>training images and 50k validation images with 1000 classes. As data augmentation and preprocess-</td></tr><tr><td>ing, resizing images with the scale and aspect ratio augmentation and stochastic horizontal flipping</td></tr><tr><td>are applied onto images. Then, global contrast normalization is applied to randomly cropped images</td></tr><tr><td>with 224 × 224 pixels. In this work, we use ResNet50 (He et al., 2016), DenseNet121 (Huang et al.,</td></tr><tr><td>2017), SEResNet50 (Hu et al., 2018) and ResNet50 with ABN (Fukui et al., 2019) for ImageNet-1k</td></tr><tr><td>experiments.</td></tr></table>

The inference speed per image is shown. We evaluate models on single GeForce GTX 1080 Ti cards with CUDA 10.0 environment. The implementation is based on PyTorch(Paszke et al., 2017) ver 1.0.1.post2. The results are averaged out of 1000 runs. The batch size is 256 and the size of input is 224 × 224 × 3. α of ss convolution is 0.04 that is same α reported in Table

Table content: <table/>

Peer review (iclr2020):
Summary: The authors extended the regular convolution and proposed spatially shuffled convolution to use the information outside of its RF, which is inspired by the idea that horizontal connections are believed to be important for visual processing in the visual cortex in biological brain. The authors proposed ss convolution for regular convolution and group convolution. The authors tested the proposed ss convolution on multiple CNN models and show improvement of results. Finally, detailed analysis of spatial shuffling and ablation study was conducted. Strengths: The authors proposed spatially shuffled convolution to use the information outside of its RF. The operation only requires small amount of extra shuffling operations and without extra learnable parameters. The idea is straightforward and easy to understand. I especially like the visualization analysis of the receptive field and the layer ablation study suggested that all layers can benefit from the proposed operation, though more for middle and higher layers. Weakness: The proposed method seems to be a reasonable alternative for regular convolution, but By fixing the permutation, not much insight is gained from this technique. Suggestions: It would be good to include the standard deviation for the results as the final results were average of 5 runs and can help to see if it’s consistently useful. In the current setting, the permutation matrix is generated randomly, considering the proposed method is based on shuffling, how would different permutation affect the performance? In addition, I think it could be helpful if the authors can show more utilities of the proposed method, for example, are the proposed method capable of doing tasks similar to this paper **Learning long-range spatial dependencies with horizontal gated recurrent units**, which argues that the ability of CNNs to learn long-range spatial dependencies is limited by their localized receptive fields. One minor thing, in the main paper, the abbreviation for spatial shuffled convolution (ss convolution) is mentioned multiple times.