Incorporating Horizontal Connections in Convolution by Spatial Shuffling
Authors: Ikki Kishida
Hideki Nakayama
Venue:  (2020)
Keywords: shuffle
convolution
receptive field
classification
horizontal connections

Abstract: Convolutional Neural Networks (CNNs) are composed of multiple convolution layers and show elegant performance in vision tasks.
The design of the regular convolution is based on the Receptive Field (RF) where the information within a specific region is processed.
In the view of the regular convolution's RF, the outputs of neurons in lower layers with smaller RF are bundled to create neurons in higher layers with larger RF. 
As a result, the neurons in high layers are able to capture the global context even though the neurons in low layers only see the local information.
However, in lower layers of the biological brain, the information outside of the RF changes the properties of neurons.
In this work, we extend the regular convolution and propose spatially shuffled convolution (ss convolution).
In ss convolution, the regular convolution is able to use the information outside of its RF by spatial shuffling which is a simple and lightweight operation.
We perform experiments on CIFAR-10 and ImageNet-1k dataset, and show that ss convolution improves the classification performance across various CNNs.

Under review as a conference paper at ICLR 2020
INCORPORATING HORIZONTAL CONNECTIONS CONVOLUTION BY SPATIAL SHUFFLING
IN
Anonymous authors Paper under double-blind review
Convolutional Neural Networks (CNNs) are composed of multiple convolution layers and show elegant performance in vision tasks. The design of the regular convolution is based on the Receptive Field (RF) where the information within a specific region is processed. In the view of the regular convolution's 's RF, the outputs of neurons in lower layers with smaller RF are bundled to create neurons in higher layers with larger RF. As a result, the neurons in high layers are able capture the global context even though the neurons in low layers only see the local information. However, in lower layers of the biological brain, the information outside of the RF changes the properties of neurons. In this work, we extend the regular convolution and propose spatially shuffled convolution (ss convolution). In ss convolution, the regular convolution able to use the information outside of its RF by spatial shuffling which is simple and lightweight operation. We perform experiments on CIFAR-10 and ImageNet-Ik Ik dataset, and show that ss convolution improves the classification performance across various CNNs.
INTRODUCTION
Convolutional Neural Networks (CNNs) and their convolution layers (Fukushima 1980 Lecun
ct al. [1998) are inspired by the finding in cat visual cortex (Hubel & Wiesel| 1959) and they show the strong performance in various domains such as image recognition (Krizhevsky et al. 201 Si-
monyan & Zisserman| 2015 He et al., 2016), natural language processing (Gehring et al. 2017). and speech recognition (Abdel-Hamid et al.) 2014 Zhang et al. 2016). A notable characteristic of the convolution layer is the Receptive Field (RF), which is the particular input region where a con-
volutional output is affected by. The units (or neurons) in higher layers have larger RF by bundling the outputs of the units in lower layers with smaller RF. Thanks to the hierarchical architectures of CNNs, the units in high layers are able to capture the global context even though the units in low layers only see the local information.
It is known that neurons in the primary visual cortex (i.e., VI which is low layers) change the self- properties (e.g.. the RF size (Pettet Gilbert 1992) and the facilitation effect (Nelson Frost, 1985)) based on the information outside of the RF (D.Gilbert) 1992). The mechanism is believed to originate from (1) feedbacks from the higher-order area (lacaruso et al., 2017) and (2) intra- cortical horizontal connections (D. (ilbert 1992). The feedbacks from the higher-order area convey broader-contextual information than the neurons in V1, which allows the neurons in V1 to use the global context. For instance, Gilbert & Li (2013) argued that the feedback connections work as attention. Horizontal connections allow the distanced neurons in the layer to communicate with each other and are believed to play an important role in visual contour integration (Li & Gilbert 2002) and object grouping (Schmidt et al. 2006).
Though both horizontal and feedback connections are believed to be important for visual processing in the visual cortex, the regular convolution ignores the properties of these connections. In this work, we particularly focus on algorithms to introduce the function of horizontal connections for the regular convolution in CNNs. We propose spatially shuffled convolution (ss convolution). where the information outside of the regular convolution's 's RF incorporated by spatial shuffling, which is simple and lightweight operation. Our ss convolution is the same operation as the regular convolution except for spatial shuffling and requires no extra learnable parameters. The design of convolution is highly inspired by the function of horizontal connections. To test the effectiveness of the information outside of the regular convolution's RF in CNNs, we perform experiments on CIFAR- 10 (Krizhevsky) 2009) and ImageNet 2012 dataset (Russakovsky et al. 2015) and show that ss convolution improves the classification performance across various CNNs. These results indicate that the information outside of the RF is useful when processing local information. In addition,

---

Under review as a conference paper at ICLR 2020
conduct several analyses to examine why convolution improves the classification performance in CNNs and show that spatial shuffling allows the regular convolution to use the information outside of its RF.
2 RELATED WORK
2.1 VARIANTS OF CONVOLUTION LAYERS AND NEURAL MODULES
There are two types of approaches improve the Receptive Field (RF) of CNNs with the regu- lar convolution: broadening kernel of convolution layer and modulating activation values by self- attention.
Broadening Kernel:
The atrous convolution (Holschneider al., 1989 Yu Koltun 2016 is the convolution with the strided kemel.. The stride is not learnable and given in advance. The atrous convolution can have larger RF compared to the regular convolution with the same computational complexity and the number of learnable parameters.
The deformable convolution (Dai al. 2017) is the atrous convolution with learnable kernel stride that depends inputs and spatial locations. The stride of the deformable convolution is changed flexibly unlike the atrous convolution, however, the deformable convolution requires extra compu- tations to calculate strides.
Both atrous and deformable convolution contribute to broadening RF, however, it is not plausible to the pixel information at distant location when processing local information. Let us consider the case that the information of p pixels away is useful for processing local information at layer In the simple case, it is known that the size of the RF grows with / where i is the size of the convolution kernel and n is the number of layers (Luo et al.) 2016). In this case, the size of kernel _ and k is around 45 when p 100 and I 5. If the kernel size is 3 x then needs to be /n
the stride needs to be 21 across layers. Such large stride causes both the atrous and the deformable convolution to have sparse kernel and it is not suitable for processing local information.
Self-Attention:
Squeeze and Excitation module (SE module) (Hu et al. 2018 is proposed to modulate the activation values by using the global context which is obtained by Global Average Pooling (GAP) (Lin et al, 2014). SE module allows CNNs with the regular convolution to the information outside of its RF as our ss convolution does. In our experiments, ss convolution gives the marginal improvements on SEResNet50 (Hu al. 2018) that is ResNet50 (He et al., 2016) with SE module. This result makes us wonder why ss convolution improves the performance of SEResNet50, thus we conduct the analyses and find that the RF of SEResNet50 is location independent and the RF of ResNet with ss convolution is the location-dependent. This result is reasonable since the spatial information of activation values is not conserved by GAP in SE module. We conclude that such difference may be the reason why ss convolution improves the classification performance on SEResNet50.
Attention Branch Networks (ABN) (Fukui et al. 2019 proposed for a top-down visual explanation by using an attention mechanism. ABN uses the output of the side branch to modulate activation values of the main branch. The outputs of the side branch have larger RF than the one of the main branch, thus the main branch is able to modulate the activation values based on the information outside of main branch's 's RF. our experiments, ss convolution improves the performance on ABN and we assume that this is because ABN works like feedbacks from higher-order areas, unlike ss convolution that is inspired by the function of horizontal connections.
2.2 UTILIZATION OF SHUFFLING IN CNNS
ShuffleNet (Zhang et al. 2017) designed for computation-eeffciennt CNN architecture and the group convolution (Krizhevsky et al. 2012) is heavily used. They shuffle the channel to make cross-group information flow for multiple group convolution layers.
The motivation of using shuffling between ShuffleNet and our ss convolution is different. On the one hand, our ss convolution spatial shuffling the information from outside of the regular
2

---

Under review as a conference paper at ICLR 2020
(a) Spatial Shuffling for Convolution (Eqn.
(b) Spatial Shuffling for Group Convolution (Eqn.
Figure 1: The overview of spatial shuffling operarion.
convolution's 's RF. On the other hand, the channel shuffling in ShuffleNet does not broaden RF and not contribute to use the information outside of the RF.
METHOD
In this section, we introduce spatially shuffled convolution (ss convolution).
3.1 SPATIALLY SHUFFFED CONVOLUTION
Horizontal connections are the mechanism to use information outside of the RF. We propose ss convolution to incorporate this mechanism into the regular convolution, which consists of two com- ponents: spatial shuffling and regular convolution. The shuffling is based permutation matrix that is generated at the initialization. The permutation matrix is fixed while training and testing.
Our convolution is defined follows:
Ii. = Σ E
CAALAS P(Zcitiitta))
ALAjER
P(æcAs) {{
*(Fass) c [aC), Io,d,d1
(2)
otherwise.
R represents the offset coordination of the kernel. For examples, the case of the 3 x 3 kernel is R = --) (1),(-1)))) (0, (0) 0), (0, 1), (1, -1), (1, (1, x E RCxIxX IxJ is the input and E RC xT,
the kernel weights the regular convolution. In Eqn. 2 the input shuffled by P and then the regular convolution is applied. Fig. (a) is the visualization of Eqn. to control how many channels are shuffled. If [aC] 0. then randomly generate the permutation matrix π E (0, 1] 1m where Em Fi.j = 1, Σ3 Ti. = and m = -J.J [aC] The generated the initialization fixed for training and testing.
2 a E (0, 1] the convolution is same the regular convolution. At the initialization,
The result of CIFAR-10 across various is shown in Fig. classification performance is obtained when a around 0.06.
The biggest improvement of the
3.2 SPATIALLY SHUFFLED GROUP CONVOLUTION
The group convolution (Krizhevsky et al. find that the shuffling operation of Eqn. is not suitable for the group convolution. ResNeXt (Xie et al. 2017) is CNN to use heavily group convolutions and Table shows the test error of ResNeXt CIFAR-10 (Krizhevsky) 2009). As can be seen in Table [ the improvement of the classification performance marginal with Eqn. 2 Thus, we propose the spatial shuffling for the
2012) is the variants of the regular convolution. We
WWe implement Eqn. by indexing, thus we hold long int instead of x binary matrix. The
implementation of convolution is shown in Appendix A.2

---

Under review as a conference paper at ICLR 2020
5.1
wm dense
5.0
Test Error
4.9
4.8
4.6
0.0
0.02
0.04
0.06
0.08
0.1
Figure 2: The result of CIFAR-10 across various a. WRN 16-4
Zagoruyko & Komodakis/ 2016)
and DenseNet-BC 12-100 (Huang al. (017)) are used. The test error the last epoch is used and reported results are averaged out of runs.
model
method
lest error (%)
ResNeXt-29 (2x64d)
Conv
4.3
SS Conv w/ Eqn.
4.2
SS Conv Eqn.
3.9
Table 1: The result of CIFAR-I0. The test the last epoch is used and the reported results averaged out of 5 runs.
group convolution as follows:
n ((Eei)) 0 C mod [
P(Zess))
Feckjj
otherwise.
Eqn. represents that the shuffled parts are interleaved like the illustration in Fig. [b).). As can be seen Table I convolution with Eqn. imppoves the elassification performance of ResNeXt.
EXPERIMENTS
4.1 PREPARATIONS
We use CIFAR-10 Krizhevsky 2009) and ImageNet-Il (Russakovsky et al. 2015) for our experi- ments.
CIFAR-10. CIFAR-10 is the image classification dataset. There are 50000 training images and 10000 validation images with 10 classes. As data augmentation and preprocessing, translation by 4 pixels, stochastic horizontal flipping, and global contrast normalization are applied onto images with 32 32 pixels. We use three types of models WRN 16-4 (Zagoruyko & Komodakis 2016), DenseNet-BC 12-100 (Huang et al. 2017) and ResNeXt 2-64d (Xie et al. 2017).
ImageNet-Ik. ImageNet-Il is the large scale dataset for the image classification. There are 1.28M training images and 50k validation images with 1000 classes. As data augmentation and preprocess- ing, resizing images with the scale and aspect ratio augmentation and stochastic horizontal flipping are applied onto images. Then, global contrast normalization is applied to randomly cropped images with 224 224 pixels. this work, we use ResNet50 (He et al. 2016), DenseNeti21 (Huang al.) 2017), SeResNet50 (Hu et al. 2018) and ResNet50 with ABN (uuui et al. 2019) for ImageNet- Ik experiments.
Implementation Details. As the optimizer, we use Momentum SGD with momentum of 0.9 and weight decay of 1.0 x 10 10-4 In CIFAR-10, we train models for 300 epochs with 64 batch size. In ImageNet, train models for 100 epochs with 256 batch size. In CIFAR-10 and ImageNet, the learning rate starts from 0.1 and divided by 10 at 150, 250 epochs and 30, 60, 90 epochs, respectively.

---

Under review as a conference paper at ICLR 2020
model
method
test error (96)
model
method
top-1 err (%)
WRN 16-4
Conv
5.1
ResNet50
Conv
23.5
SS Conv
4.8
SS Conv
23.0
DenseNet-BC
Conv
5.0
ResNet50 w/ ABN
Conv
22.9
SS Conv
4.6
SS Conv
22.6
ResNeXt-29 (2x64d)
Conv
4.3
SEResNet50
Conv
22.7
SS Conv
3,9
SS Conv
22.6
DenseNet121
Conv
24.5
Table 2: The result of CIFAR-10. The test error
SS Conv
24.1
the last epoch used and the reported results are averaged out of runs.
ResNeXt-50 (32x4d)
Conv
22.4
SS Conv
22.0
Table 3: Top-1 1 error on ImageNet-1k validation dataset. The top-1 error at the last epoch is used and the reported results are averaged out of 3 runs.
model
method
inference speed (ms)
ResNet50
Conv
1.44
SS Conv
1.66
SEResNet50
Conv
2.11
Table 4: The inference speed per image is shown. We evaluate models single GeForce GTX 1080 Ti cards with CUDA 10.0 environment. The implementation is based on PyTorch (Paszke al. 2017) ver 1.0.1. 1oost2. The results averaged out of 1000 runs. The batch size is 256 and the size of input is 224 x 224 x 3. a of ss convolution is 0.04 that is same a reported in Table
In our experiments, we replace all regular convolutions with ss convolutions except for downsam- pling layers, and use single a across all layers. We conduct grid search of E (0.02, 0.04, 0.06) and a is decided according to the classification performance on validation dataset.
4.2 RESULTS
We replace all regular convolutions with convolutions to investigate whether the information outside of the regular convolution's ' RF contributes to improving the generalization ability. The results are shown in Table 2 and As can be seen in Table 2 and convolution contributes to improve the classification performance across various CNNs except for SEResNet50 that shows marginal improvements. The detailed analysis of the reason why convolution gives the marginal improvements in SEResNe150 is shonw in Sec. 5
Since {0.02, 0.04, 0.06) is small, the small portion of the input shuffled, thus convolution improves the classification performance with small amount of extra shuffling operations and without extra learnable parameters. The inference speed shown in Table 4 and ss convolution make the inference speed 1.15 times slower in exchange for 0.5% improvements in ImageNet- 1k dataset. The more efficient implementation" may decrease the gap of the inference speed between the regular convolution and ss convolution.
ANALYSIS
In this section, we demonstrate two analysis understand why convolution improves the classi- fication performance across various CNNs: the receptive field (RF) analysis and the layer ablation experiment.
OOu implementation is shown in Appendix A.2
5

---

Under review as a conference paper at ICLR 2020
layer name
output size
blue box sizc
structure
input
224 x 224
32 x 32
conv1
112 x 112
16 x 16
x 7, 64, stride 2
56 x 56
8
3 max pool, 64, stride 2
1 x 1, 64
conv2 x
56 x 56
8
3 x 3,
x x
x 1, 256
[ x 1, 1287
conv3 X
28 x 28
4 4
3 x 3, 128
x
[ x 1, 512
x 1, 256
conv4_x
14 x 14
2 x 2
3 x 3, 256
6
1024
x 1, 512
conv5_x
7 x 7
1x1
3 x 3, 512
x 3
x 1, 2048
1 x
average pool, 1000-d fc, softmax
Table The structure of ResNet50) (He et al. 2016).
Receptive Field Analysis. We calculate the RF of SEResNe150, ResNet50 with ss convolution and the regular convolution. The purpose of this analysis is to examine whether convolution contributes to use the information outside of the regular convolution's RF.
Layer Ablation Experiment. The layer ablation experiment is conducted to know which volution influences the model prediction. In the primary visual cortex, the neurons change self- properties based on the information outside of RF, thus would like investigate whether spatial shuffling in low layers contribute to predictions or not.
Our analyses based on ImageNet- 1k pre-trained model and the structure of ResNet50 (i.e., the base model for analysis) is shown in Table
5.1 DOES SPATIAL SHUFFLING CONTRIBUTE TO USE THE INFORMATION FROM OUTSIDE OF RECEPTIVE FIELD?
In our analysis, we calculate the RF to investigate whether ss convolution uses the information outside of the regular convolution's RF. The receptive field is obtained by optimization as follows:
R arg min [[M (((( x) di (()())l
[CXx input, and R Cx is the RF to calculate and learnable. o is sigmoid function, thus S (R) < 1. di is the outpus of the trained model at the layer 1.
We call the first term in Eqn. as the local perceptual loss. is similar the perceptual loss Beeeeeeppeeeree (Johnson et al. 2016), and the difference is the dot product of M E {0, []xTXX that works as masking. M the binary mask and May analysis. In other words, the values inside the blue box in Fig 3 Jre the part of Maj = 1. The local perceptual loss minimizes 2nd term is the penalty to evade the trivial case such as (R) 1. In layerwise and channelwise RF anlysis, we use of 1.0 x 10 and 1.0 x 10 12, respectively.
96 < i.j < 128, otherwise Maj
in our
distance of feature on the specific region between aR) - and æ. The
We use Adam optimizer (Kingma & Ba 2015) to calculate R As the hyper-parameter, Ir, S1. and Bg are 0.1, 0.9, 0.99, respectively. The high Ir is used since its slow convergence. The batch size is 32 and stop the optimization after 10000 iterations. is randomly selected from ImageNet- 1k training images. The data augmentation and preprocessing are applied as the same
procedure in Sec. 4.

---

Under review as a conference paper at ICLR 2020
conv2_1
conv3_1
conv4_1
conv5_1
conv
conv
SE
Figure 3: The layerwise RF of ImageNet-Ik pre-trained models. The layerwise RF images in the top row are based on ResNet50 with regular convolution, the ones in the middle row are calculated from ResNet50 with convolutions and the ones in the bottom The red color indicates that the pixel there changes features inside blue box, and the white color represents that features invariant. The name of the layer is described in Table The rest of RFs are shown Appendix A.1
generated from SEResNet50.
Layerwise Receptive Field. We calculate the RF for each model and the results are shown in Fig. 3 The top row is the RF of ResNet with the regular convolution, the middle row is the one with ss convolution and the bottom row is the one of SEResNet50. The red color indicates that the value of the pixel there changes features inside the blue box, and the white color represents that features inside the blue box are invariant even if the value of the pixel there is changed.
the top row of Fig. the RFs of ResNe150 with the regular convolution are shown. The size of RF becomes larger as the layer becomes deeper. This result is reasonable and obtained RFs are in the classical view. If the RF ResNet50 with ss convolution is beyond the one with the regular convolution, it indicates that ss convolution successfully the information outside of the regular convolution's 's RF
In the middle and bottom of Fig. 3 are the RF of ResNet50 with convolution and SEResNet50, respevtively. The RFs covers the entire image unlike the RF with the regular convolution. These results indicate that both SE module and convolution contributes use the information outside of the regular convolution's 's RF.
Fig. (a) shows the size of the RF across layers. The horizontal axis is the of the layer and the vertical axis represents the size of RF that is calculated as
[|o(R)20.51l, where || is the L1 norm [R]
and is the total size of matrix. [|((k]2005]|] is in the range between 0 and I and represents the RF
ratio a (RF) that is bigger than 0.5. As can seen Fig. (a), the size of RFs are consistently almost 1 across layers in ResNet50 with ss convolution and SEResNe150. This result also shows that SE module and the ss convolution contributes to use the information outside of the regular con- volution's 's RF. This may be the reason why convolution improves the classification performance on various CNNs. However, these results make us wonder why ss convolution improves marginally the performance of SEResNet5O. Further analysis is conducted in channelwise RF analysis.

---

Under review as a conference paper at ICLR 2020
(8)
(c)
(d)
Figure 4: The channelwise RF of ImageNet- 1k pre-trained model. The RF is calculated based the output of the channel 64 in conv2 1. (a)-(b) RF of ResNet50 with ss convolution. (c)-(d) RF of SEResNet50. The red color indicates that the pixel there changes features inside the blue box, and the white color represents that features are invariant. For clarity, we convert the values that satisfy (R) 0.1 into 1.0.
1.0
37.5
ablation
0.8
35.0
w/o ablation
Size of RF
Top-1 Error
32.5
0.6
30.0
0.4
27.5
0.2
conv
25.0
se
0.0
22.5
conv2 1
conv2
conv2 3
conv3
conv3
2
conv3 3
conv3 4
conv4 1
conv4 2
convA
conva. 4
conv4 5
conve:
convo 1
conv5 2
conv5_3
conv2
conv2 2
conv2_3 3
conv3 1
conv3 2
conv3
conva
2onv3_4 4
conval 1
conv4 2
conv4 3
conv4 4
conv4 5
conv4 6
conv5
conv5 2
conv5 3
Block
Block
(a) The size of RF for each layer.
(b) The result of the ablation experiment.
Figure 5: The analysis for each layer in ImageNet-I Ik pretrained models. (a) The horizontal axis is which block is used to calculate the layerwise RF and the vertical axis is the size of RF, (b) The horizontal axis is which block is ablated ResNet50 with convolutions and the vertical axis is the top-1 error in ImageNet- 1k validation dataset.
Channelwise Receptive Field. Since layerwise RF analysis is based on the RF of the layer, the obtained results have rough directions. We calculate the channelwise RF for more fine-grained analysis. Unlike layerwise RF analysis, M becomes different and we minimize the local perceptual loss on the specific channel. The results are shown Fig.
Fig. (a) and (c) use Meij
64 < i,j < 96 and c= 64, otherwise Moij = 0. Fig. (b) and (d) use Maij 1 if 96 < i,j < 128 and c 64, otherwise Mai = O. Fig. (a)-(b) are the RF of ResNet50 with ss convolution and (c)-(d) are the RF of SEResNet50. As can be seen in Fig. B] the RFs of ResNet50 with ss convolution (i.e., Fig. (a)-(b)) are different when the blue box shifted, however, the RFs of SEResNe150 (i.e., Fig. (c)-(d) are similar even if the blue box is shifted. These results indicate that the information outside of the regular convolution' 's RF is location location-indepenne independent in SEResNet 50 and location location-dependee dependent in ResNet50 with convolution. This reasonable since SE module uses the global average pooling and the spatial information is not conserved. This difference may be the reason why convolution marginally improves the classification performance Net5sNet50.
8

---

Under review as a conference paper at ICLR 2020
5.2 ABLATION STUDY
We conduct layer ablation study to investigate which ss convolutions contribute ability. The ablation is done follows:
contrbbbu to the generalization
P(te(s)) 0,
c LaC),
(5)
Tocids otherwise.
Eqn. 5 represents that the activation values of the shuffled parts become O. The result of the ablation
experiment is shown in Fig. (b). Eqn. is applied to all ss convolutions in each block and the biggest drop of the classification performance happens the ablation of conv4 4. It indicates that is useful to use the information outside of the regular convolution' 's RF between the middle and high layers. The classification performance is degraded even the ablation applied to the first bottle- neck (i.e., 1).. This result implies that the information outside of the regular convolution's 's RF useful even at low layers.
CONCLUSION
In this work, we propose spatially shuffled convolution (ss convolution) to incorporate the function of horizontal connections in the regular convolution. The spatial shuffling is simple, lightweight, and requires no extra learnable parameters. The experimental results demonstrate that ss convolution captures the information outside of the regular convolution's ' RF even in lower layers. The results and our analyses also suggest that using distant information (i.e., non-local) is effective for the regular convolution and improves classification performance across various CNNs.
REFERENCES
Ossama Abdel-Hamid, Ahdel-Rahman Mohamed, Hui Jiang, Li Deng, Gerald Penn, and Dong Yu. Convolutional neural networks for speech recognition. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2014.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. ICCV, 2017.
Charles D.Gilbert. Horizontal integration and cortical dynamics. Neuron, 9:1-13, 1992.
Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Attention branch network: Learning of attention mechanism for visual explanation. CVPR, 2019.
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 1980.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence sequence learning. PMLR, 2017.
Charles D. Gilbert and Wu Shyong Li. Top-down influences visual processing. Nature Reviews Neuroscience, 14:350-363, 2013. 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
M. Holschneider, R. signal analysis with the help of the wavelet transform. Wavelets: Time Frequeryyy Methods and Phase Space, pp. 289-297, 1989.
J. Morlet, and Ph. Tchamitchian. A real-time algorithm for
Jie Hu, Li Shen, and Gang Sun. Squeeze-ane Squeeze-and-ex and---citatiiionn networks. CVPR, 2018.
Gao Huang, Zhuang Liu, Laurens der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. CVPR, 2017.
David H. Hubel and Torsten N. Wiesel. Receptive fields of single neurons the cat's 'ss striate cortex. Journal of Physiology, 1959.

---

Under review as a conference paper at ICLR 2020
Maria lacaruso, Ioana Gasler, and Sonja Hofer. Synaptic organization of visual space in primary visual cortex. Nature, 547, 2017.
Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. ECCV, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, Imagenet classification with deep convo- lutional neural networks. NIPS, 2012.
Yann Lecun, Lon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
Wu Li and Charles Gilbert. Global contour saliency and local colinear interactions. Journal of neurophysiology. 88:2846-56, 2002.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. ICLR, 2014.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. NIPS, 2016.
J. I. Nelson and B. J. Frost. Intracortical facilitation among co-oriented, co-axially aligned simple cells cat striate cortex. Experimental Brain Research, 61(1):54-61 1985.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. NIPS Workshop, 2017.
M Pettet and Charles Gilbert. Dynamic changes in receptive-field size in cat primary visual cortex. Proceedings of the National Academy of Sciences of the United States of America, 89:8366-70, 1992.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.
Kerstin Schmidt, Rainer Goebel, Siegrid Lwel, and Wolf Singer. The perceptual grouping criterion of colinearity is reflected by anisotropies of connections in the primary visual cortex. European Journal of Neuroscience, 9:1083 1089, 2006.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Saining Xie, Ross Girshick, Piotr Dollr, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. CVPR, 2017.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. ICLR, 2016.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. BMVC, 2016.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. CVPR, 2017.
Ying Zhang, Mohammad Pezeshki, Philemon Brakel, Saizheng Zhang, César Laurent, Yoshua Ben- gio, and Aaron Courville. Towards end-to-end speech recognition with deep convolutional neural networks. Interspeech, 2016.
A APPENDIX
A. RESULTS OF LAYERWISE RECEPTIVE FIELD
The receptive fields of all layers shown in Fig. and
10

---

Under review as a conference paper at ICLR 2020
(a) conv2_1
(h) conv2_2
(c) conv2_3
(d) conv3_1
(e) conv 3
(f) conv3_3
(g) conv3_4 4
(h) conv4_ __
(i) conv4
(i) conv4. 3
(k) conv4 4
(I) conv4 5
(m) conv4_6
conv5_1
(0) conv5.2
(p) conv5
Figure 6: The receptive field of ImageNet- 1k pre-trained ResNet50. The red color indicates that the pixel there changes features inside the blue box, and the white color represents that features invariant even if the pixel there changes the value itself. Those images are the receptive field of all layers and the name of the layer described Table
11

---

Under review as a conference paper at ICLR 2020
(a) conv2_1
(b) conv2.2
(c) conv2_3
(d) conv3_1
(e) conv3.2
(f) conv3_3
(g) conv3_4 4
(h) conv4_
(i) conv4
(i) conv4 3
(k) conv4 4
(I) conv4 5
(m) conv4_6
conv5_1
(0) conv5
(p) conv5 3
Figure 7: The receptive field of ImageNet-Ik pre-trained ResNer50 with ss convolutions. The red color indicates that the pixel there changes features inside the blue box, and the white color repre- sents that features are invariant even if the pixel there changes the value itself. Those images are the receptive field of all layers and the name of the layer is described in Table
12

---

Under review as a conference paper at ICLR 2020
(a) conv2_1
(h) conv2_2
(c) conv2_3
(d) conv3_1
(e) conv
(f) conv3_3
(g) conv3_4 4
(h) conv4_
(i) conv4
(j) conv4.3 3
(k) conv4 4
(I) conv4 5
(m) conv4_6
conv5_1
(0) conv5
(p) conv5_3
Figure 8: The receptive field of ImageNet-Ik pre-trained SEResNet50. The red color indicates that the pixel there changes features inside the blue box, and the white color represents that features invariant even if the pixel there changes the value itself. Those images are the receptive field of all layers and the name of the layer described Table
13

---

Under review as a conference paper at ICLR 2020
A.2 EXAMPLE OF CODE
The code of spatially shuffled convolution is shown in Listing 1. It is written in python with Pytorch (Paszke al. 2017) of 1.0.1.post2 version. The training and model codes will be available online after the review.
import torch
class SSConv2d(torch Module):
def ..innt.. -niit--(self. in-planes. aut-planes, sixes,, stride=l, padding=l, bias=None, groups=l, iillation=,, alpha=0.04):
Super(SSConv2., relf)...init...0
self.com torch. Conv2d(inn planes. planes. kernel size=kernel size. stride= stride padding=padding groups=groups. dilation=dilation bias=bias)
self.alpha. self groups alpha. groups
10
def create shuffle indices x):
12
in-planes, height, width x. self. shuffle until here int(in int(eee x self alpha) A self shuffle until_bere =, then it's exactly regular convolution self, wwwffle setiflleuuntl_hhe Se
13
15
relf.reggitte.bbffeie "random.indices'. torch. randpera height width))
17
18
@staticmethod
19
der group shufffld x, non shuffled x): batch, ch.ns, height. width non_shuffled.. shape shuffled_x.shap
21
ch.s, length inh chis) residue = ch ns length ch chis # shuffled.x interleaved if residue
22
23
24
25
26
return torch. cal((shuffled. unsqueeze (1). non.shuffled.a.. viww (batch. length. ch.s height, width)), I). (bbtcch,,,, ch_os ch.s, height, width) clse:
27
return catorch catch,.at(((a) ((shufffeee unsqueere non-shuffled.x residue :] view view(hateh (batch. length ch.s, height. width)), I). viewwwhh,,, ch + ch residue, height, width). , non.shuffledxxI:. :residue
29
30
def shuffle x):
self. shuffle_until_p
32
ss convolution
shuffled shufflc.until. until_heree
shuffled xl:. sseff shuffle until
self.
34
batch. ch. height, width = shuffled size
35
shuffled.x torch. index.selectsshuffie
view batch. -1), 1. self. sandom.indion
). vieewhaich ch, height, width)
if self groups return self egf.ugg (shufflpss non_shuffled.x)
37
else:
39
return torch. (athuffffll. non.shufflld.xx, 1)
40
clse:
regular convalution return
42
44
def forward x):
hasatir(sell "random indices ") False:
46
create random permutation matrix initialization
47
self.
spatial shuffling
49
x self self. shuffle
regular convolution
51
x self conv.conv(x)
Listing 1: Implementation Spatially Shuffled Convolution
14


Reviews:
[{"review_id": "rJeDcCUb9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a simple “spatial shuffling” operation for modifying CNNs \nbased on a permutation matrix created at initialization time.  \n\nThe approach is motivate by a very high-level discussion of biological brains.\nImprovements are claimed on Cifar10 but results are not near the state of the art.\nThe results do seem to improve incrementally over the previous vanilla results \non the particular architecures on which they are applied.\n\nThe authors dedicate some space to a qualitative analysis of why \nthe models improve although it is at best intuition-y.\n\nIn the end I’m left with inconclusive results, a weakly motivated story, \nand a paper that despite exceeding the page limit by a page lacks \ninformation density. \n\n\nMinor:\n\nConvolutional neural networks achieve … “elegant performance in computer vision tasks”\n>>> \twrong adjective.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJelH-XZcH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Overview]\n\nIn this paper, the authors proposed a shuffle strategy for convolution layers in convolutional neural networks (CNNs). Specifically, the authors argued that the receptive field (RF) of each convolutional filter should be not constrained in the small patch. Instead, it should also cover other locations beyond the local patch and also the single channel. Based on this motivation, the authors proposed a spatial shuffling layer which is aimed at shuffling the original feature responses. In the experimental results, the authors evaluated the proposed ss convolutional layer on CIFAR-10 and ImageNet-1k and compared with various baseline architectures. Besides, the authors further did some ablated analysis and visualizations for the proposed ss convolutional layer.\n\n[Pros]\n\n1. The authors proposed a new strategy for convolutional layers. The idea is borrowed from the biological domain, and then transformed to a spatial shuffling layer which can shuffle the feature response at each convolutional layer.\n\n2. The authors performed experiments on both small-scale dataset (CIFAR-10) and large-scale data (CIFAR-100) for evaluations.\n\n3. The authors further added some ablated analysis on the proposed model. Specifically, the authors visualize the receptive field which can be used in ss layer compared with the original convolutional layer, which indicates that ss layer can incorporate the global context at the very beginning.\n\n[Cons]\n\n1. The motivation behind the proposed ss layer is not explained very well. Though the authors mentioned that it is biologically inspired, I would not buy that since it is still a unclear phenomenon, and even it is true, using a randomized shuffling seems not align with the observations to some extent.\n\n2. The paper is poorly written in general. The motivation behind the proposed method, and the presentation of method section part are cluttered much. In the model analysis in experiment section, the presentation and explanations are also vague and not clear to me.\n\n3. The proposed model seems increase the baseline models' performance very marginally on all architectures. It is hard to say that it is because the shuffling layer enable the neurons to incorporate the global context information. Instead, it might just because the randomization which would increase the generalization ability of the trained model.\n\n4. Finally, the comparison with previous models, such as SENet, ShuffleNet, etc are not systematically. I would like to see a more comprehensive summarization of the differences between the proposed ss layer and other architectures, because all of them are trying to incorporate more contextual information from other channels or locations.\n\n[Summary]\n\nOverall, I think the proposed ss layer is still a reasonable way to incorporate the contextual information in CNNs. However, the poor presentation and the weak experimental results and analysis make the paper overall a one under the bar of the venue. I would suggest the authors revise the paper with more well-motivated formula and more solid experiments and analysis in the next submission.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1l-8NElcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors extended the regular convolution and proposed spatially shuffled convolution to use the information outside of its RF, which is inspired by the idea that horizontal connections are believed to be important for visual processing in the visual cortex in biological brain. The authors proposed ss convolution for regular convolution and group convolution. The authors tested the proposed ss convolution on multiple CNN models and show improvement of results. Finally, detailed analysis of spatial shuffling and ablation study was conducted.\nStrengths: The authors proposed spatially shuffled convolution to use the information outside of its RF. The operation only requires small amount of extra shuffling operations and without extra learnable parameters. The idea is straightforward and easy to understand. I especially like the visualization analysis of the receptive field and the layer ablation study suggested that all layers can benefit from the proposed operation, though more for middle and higher layers.\nWeakness: The proposed method seems to be a reasonable alternative for regular convolution, but  By fixing the permutation, not much insight is gained from this technique. \nSuggestions: It would be good to include the standard deviation for the results as the final results were average of 5 runs and can help to see if it’s consistently useful. In the current setting, the permutation matrix is generated randomly, considering the proposed method is based on shuffling, how would different permutation affect the performance? In addition, I think it could be helpful if the authors can show more utilities of the proposed method, for example, are the proposed method capable of doing tasks similar to this paper **Learning long-range spatial dependencies with horizontal gated recurrent units**, which argues that the ability of CNNs to learn long-range spatial dependencies is limited by their localized receptive fields. One minor thing, in the main paper, the abbreviation for spatial shuffled convolution (ss convolution) is mentioned multiple times.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is well-motivated by neuroscience that our brains use information from outside the receptive field of convolutive processes through top-down mechanisms. However, reviewers feel that the results are not near the state of the art and the paper needs further experiments and need to scale to larger datasets. 
Decision: Reject