SwGridNet: A Deep Convolutional Neural Network based on Grid Topology
for Image Classification
Atsushi Takeda
Tohoku Gakuin University
2-1-1, Tenjinzawa, Izumi-ku, Sendai, Japan

arXiv:1709.07646v3 [] 3 Oct 2017

takeda@cs.tohoku-gakuin.ac.jp

Abstract

ResNeXt [24] are constructed as a multipath network which
has many processing paths between input and output. In a
multipath network, each processing path performs the calculation using individual parameters. The results are integrated as output data. A multipath network mitigates overfitting and improves generalization abilities because a multipath network has the same effect as ensemble learning.

Deep convolutional neural networks (CNNs) achieve remarkable performance on image classification tasks. Recent studies, however, have demonstrated that generalization abilities are more important than the depth of neural
networks for improving performance on image classification tasks. Herein, a new neural network called SwGridNet is proposed. A SwGridNet includes many convolutional processing units which connect mutually as a grid
network where many processing paths exist between input
and output. A SwGridNet has high generalization capability because the multipath architecture has the same effect
of ensemble learning. As described in this paper, details of
the SwGridNet network architecture are presented. Experimentally obtained results presented in this paper show that
SwGridNets respectively achieve test error rates of 2.95%
and 15.67% in a CIFAR-10 and CIFAR-100 classification
tasks. The results indicate that the SwGridNet performance
approximates that of state-of-the-art deep CNNs.

As described in this paper, a new deep convolutional neural network called Sandwiched Grid convolutional neural
Network (SwGridNet) is proposed. A SwGridNet is constructed as a chain of grid blocks which include many convolutional processing units. The convolutional processing
unit mutually connects as a grid network with many processing paths. Because a SwGridNet contains many processing paths between input and output, the SwGridNet
gains high generalization capabilities. This paper presents
an explanation of SwGridNet architecture and describes
that a SwGridNet has many processing paths between input and output. In addition, experimentally obtained results
of image classification tasks of CIFAR-10 and CIFAR-100
dataset [15] are presented. The experiment results show that
SwGridNets yield test error rates of 2.95% and 15.67%,
respectively, for CIFAR-10 and CIFAR100 classification
tasks.

1. Introduction
A deep convolutional neural network AlexNet [16] won
the ImageNet Large Scale Visual Recognition Competition
(ILSVRC) [19] in 2012. This result demonstrated that deep
convolutional neural networks (CNNs) are suitable for image recognition tasks. Therefore, in recent years, many researchers have been investigating deep CNNs for use in image recognition. Although it is not easy to train deep neural
networks, deep neural networks generally have higher capabilities than shallow neural networks. Therefore, many
techniques have been proposed for training deep CNNs.
Deep CNNs achieve high accuracy rates for image classification tasks [10, 25, 2, 8, 24]. Deep CNNs, however,
tend to memorize all training data. For that reason, a mechanism against overfitting is necessary for deep CNNs [26].
A lot of successful deep CNNs such as Xception [2] or

The organization of this paper is the following. Section 2 introduces existing deep CNNs, which have been proposed for image classification tasks, and presents explanation importance of generalization abilities of deep CNNs.
Section 3 presents a proposal of a new deep convolutional
neural network called SwGridNet. This section explains
details of a SwGridNet architecture, and explains that a
SwGridNet contains many processing paths between input
and output. Section 4 presents experimentally obtained results of SwGridNets in CIFAR-10 and CIFAR100 classification tasks. The experimentally obtained results show that
SwGridNets achieve low error rates which are close to other
state-of-the-art deep CNNs. Finally, the study is concluded
in section 5.
1

A multipath neural network has the same effect as ensemble
learning [27] which improves the capability of a neural network for generalization. Therefore, a multipath architecture
is used in many state-of-the-art deep CNNs.
Inception [22] and Xception [2] networks are constructed as a chain of Inception modules that contain more
than one convolutional layer connected in parallel. Therefore, Inception and Xception have many processing paths
between input and output. A FractalNet [17] is also a multipath neural network. A FractalNet contains not only shallow processing paths but also deep processing paths. A
Residual Network (ResNet) [10] is constructed as a chain
of Residual Blocks which are the same as multipath neural
networks as shown in Figure 1. Therefore, a ResNet has numerous processing paths between input and output. Wide
Residual Network [25], Pyramidal Residual Network [8]
and ResNeXt [24] are extended versions of a ResNet. These
networks are also multipath neural networks. Because the
multipath neural networks show remarkable performances
in image classification tasks, A multipath architecture is an
important technique for use with deep CNNs.
As described herein, a new multipath convolutional neural network is proposed Sandwiched Grid convolutional
neural Network (SwGridNet). A SwGridNet has many convolutional processing units which connect mutually as a
grid network. A grid network contains many processing
paths between input and output. Therefore, a SwGridNet
is a multipath neural network which has a high generalization ability. An existing convolutional neural network GridNet [4] is also constructed using grid network topology. A
GridNet, however, uses a grid topology for the integration
of different resolution feature maps. However, a SwGridNet uses a grid topology for improvement of generalization
capability.

conv 1a
conv 1b

conv 1a

conv 2a

conv 1a

conv 1b

conv 2b

conv 1b

conv 2a

conv 2a

conv 2b

conv 2b

(a) Residual blocks

(b) Unraveled network of residual blocks

Figure 1. Residual networks that include two residual blocks (four
convolutional layers): networks presented in (a) and (b) are the
same convolutional neural networks.

2. Related Works
2.1. Deep CNN for Image Classification
A deep convolutional neural network (CNN) is the most
successful method for image recognition tasks. For that
reason, many types of deep CNNs for image classification
tasks have been proposed. AlexNet [16] and VGGNet [20]
respectively achieved outstanding results in ILSVRC 2012
and 2014. Results showed that deep CNNs work well in image classification tasks. Deep CNNs generally have higher
capabilities than shallow CNNs, but training a deep CNN is
not easy because of vanishing gradient problems [1, 6].
To avoid vanishing gradient problems, state-of-the-art
deep CNNs contain not only convolutional layers but also
shortcuts. A ResNet is constructed as a chain of Residual
Blocks which include both convolutional layers and a shortcut [10]. The shortcut leads to an error signal directly from
output to input. For that reason, it is possible to train a
very deep ResNet which includes more than 1,000 convolutional layers. Other state-of-the-art deep CNNs such as
Xception [2], FractalNet [17] and ResNeXt [24] also have
shortcuts between input and output.

3. SwGridNet
3.1. Architecture of SwGridNet

2.2. Generalization Ability of Deep CNN

A SwGridNet is constructed as a chain of grid blocks.
A grid block includes a residual architecture to lead error
signals directly from output to input. A function of a grid
block is defined as

Deep CNNs have numerous parameters, so deep CNNs
can memorize much information. In fact, AlexNet and
Inception can perfectly memorize CIFAR-10 training images that are labeled randomly [26]. Therefore, deep CNNs
easily overfit the training data. To improve test accuracy
rates, generalization abilities are required for deep CNNs.
In recent years, many generalization techniques such as
Dropout [21] and Shakeout [14] have been proposed. In
addition, many state-of-the-art deep CNNs are constructed
as a multipath neural network that contains more than one
processing path to improve the generalization ability. In a
multipath neural network, each processing path performs
the calculation using each parameter. The network integrates the results from all processing paths as output data.

y = F(x) + x.

(1)

Here, x and y respectively denote an input matrix and an
output matrix of the grid block. The function F(·) represents a nonlinear function of the grid block. A grid block
includes a split layer, a grid layer, and a join layer, as shown
in Figure 2. The function F(·) is separated to three functions as
F(x) := G join (G grid (G split (x))).
2

(2)

split layer other units

split layer
grid
layer

unit

unit

mean

unit

unit

BN
conv 1x1

join layer

(a) 1-dimensional grid block

BN

BN

ReLU

ReLU

conv 1x1

conv 3x3

conv 1x1

BN

BN

BN

units

split layer

(a) split layer

unit

join layer other units
(b) unit

output
(c) join layer

Figure 3. Calculations in a grid block: (a) depicts a chain of calculations in a split layer, (b) depicts a chain of calculations in a
convolutional processing unit, and (c) depicts a chain of calculations in a join layer. In these chains, ‘conv’, ‘BN’, and ‘ReLU’
respectively represent a convolutional layer, a batch normalization
layer and a ReLU function. In addition, ‘mean’ is an element-wise
mean calculation.

grid
layer
unit

units

input

unit

join layer

(b) 2-dimensional grid block

convolutional processing units, as shown in Figure 2. I define a function performed by an convolutional processing
unit which is located at (p0 , · · · , pN −1 ) in a grid layer as

Figure 2. Architectures of grid blocks in a SwGridNet.

Here, G split (·), G grid (·) and G join (·) respectively denote nonlinear functions of a split layer, a grid layer, and a join layer.
A grid layer contains convolutional processing units
which connect mutually as a multi-dimensional grid network. Consequently, the number of the units in a grid layer
is LD , where D denotes the dimension of the grid layer and
L denotes the side length of the grid layer. As described in
this paper, a location of the unit is represented as a coordinate (p0 , p1 , · · · , pN −1 ). Each unit includes convolutional
layers, which have individual parameters.
A split layer creates matrices input to all convolutional
processing units at first. The calculation of a split layer is
defined as
{ sp0 ,··· ,pN −1 | 0 ≤ pi < L } = G split (x).

=G

grid

({ sp0 ,··· ,pN −1 | 0 ≤ pi < L }).

(5)

vp0 ,··· ,pN −1 = { up0 ,··· ,pm −1,··· ,pN −1 | pm > 0 } .

(6)

Here, vp0 ,··· ,pN −1 is a set of output matrices of neighbor
units; H(·) represents a nonlinear function of the unit. Each
unit in a grid layer receives not only a matrix sp0 ,··· ,pN −1
from a split layer but also a set of matrices vp0 ,··· ,pN −1 from
neighbor units. To perform the function H(·), each convolutional processing unit in a grid layer is also constructed
as a neural network that contains convolution layers, batch
normalization layers and a ReLU [7] function as shown in
Figure 3 (b).
A set of matrix output by a grid layer is sent to a join
layer as input matrices. Calculation of a join layer is defined
as

(3)

Here, sp0 ,··· ,pN −1 denotes an output matrix that is sent to
a unit located at (p0 , · · · , pN −1 ). To perform the function
G split , a split layer is constructed as a neural network that
contains a convolution layer and batch normalization layers [13] as shown in Figure 3 (a).
The calculation of a grid layer is defined as
{ up0 ,··· ,pN −1 | 0 ≤ pi < L }

up0 ,··· ,pN −1 = H(sp0 ,··· ,pN −1 , vp0 ,··· ,pN −1 ),

y = G join ({ up0 ,··· ,pN −1 | 0 ≤ pi < L }).

(7)

To perform the function G join , a join layer is constructed as
a neural network that contains a convolution layer, a batch
normalization layer and a ReLU function, as portrayed in
Figure 3 (c).

3.2. Processing Paths in a Grid Block
(4)

A grid block contains many convolutional processing
units, which are mutually connected as a grid network.
More than one communication path connects a source node
to a destination node in a grid network. In the same way as

Here, up0 ,··· ,pN −1 is a set of output matrices of a grid layer.
To perform the function G grid , a grid layer contains many
3

split layer
unit

unit

unit

unit

unit

join layer

unit

unit

unit

split layer
unit

join layer

(c) 3-depth paths

unit

unit

unit

N =1, L=16
16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
136

depth=1
depth=2
depth=3
depth=4
depth=5
depth=6
depth=7
depth=8
depth=9
depth=10
depth=11
depth=12
depth=13
depth=14
depth=15
depth=15
total

(b) 2-depth paths

split layer
unit

unit

join layer

(a) 1-depth paths

unit

Table 1. Number of processing paths in a grid block. Here, D and
L respectively denote the dimension and the side length of the grid
block.

split layer

unit

join layer

(d) 4-depth path

Figure 4. Processing paths in a one-dimensional grid block with
four convolutional processing units.

a grid network, a grid block also has more than one processing path between input and output. Figure 4 shows processing paths in a one-dimensional grid block that consists of
four convolutional processing units. This grid block shown
in Figure 4 has four depth processing paths. Here, the processing path depth represents the number of units passed
by the processing path. In the case of a one-dimensional
grid block with four convolutional processing units, the grid
block arranges four one-depth processing paths, as shown
in Figure 4 (a). In the same manner, the grid block also arranges three two-depth processing paths, two three-depth
processing paths, and one four-depth path. The number
of processing paths depends on dimension N and the side
length L of the grid block. The deepest processing path in
the grid block is an N (L − 1)-depth path. Each processing
path in a grid block performs calculations using individual
parameters, so the multipath architecture of a grid block has
the same effect as ensemble learning.
Table 1 presents the number of processing paths in a grid
block that consists of 16 convolutional processing units. A
four-dimensional grid block (N =4, L=2) has more shallow
processing paths than a one-dimensional grid block (N =1,
L=16). Therefore, the effect of ensemble learning in a fourdimensional grid block is better than a one-dimensional
grid block because the effect of ensemble learning is generally improved along with the increased number of processing paths. However, a four-dimensional grid block
has no deeper processing path than five-depth, but a onedimensional grid block has a 16-depth processing path.
Therefore, a one-dimensional grid block can support more
complex calculation than a four-dimensional grid block. A
report of a study of deep CNNs [23] indicates that ensemble learning by shallow processing paths is more effective
than calculations of a deep processing path in image classi-

N =2, L=4
16
24
34
44
48
40
20
0
0
0
0
0
0
0
0
0
226

(4, 0)

N =4, L=2
16
32
48
48
24
0
0
0
0
0
0
0
0
0
0
0
168
(4, 4)

22 25

25 27

27 29

29 32

20 22

22 25

25 27

27 29

18 20

20 22

22 25

25 27

16 18

18 20

20 22

22 25

(0, 0)

(0, 4)

Figure 5. Number of channels of convolutional processing units in
a two-dimensional grid layer (N =2, L=4), where the number of
unit channels is [16, 32]. Each box depicts a unit in a grid layer,
and a left number and a right number in the box are, respectively,
the number of input channels and the number of output channels.

fication tasks. In addition, many experiment results support
the report [25]. Therefore, a four-dimensional grid block is
expected to be more suitable than a one-dimensional grid
block for image classification tasks.

3.3. Unit Width in a Grid Block
A grid layer contains numerous processing paths between input and output. To improve the generalization abilities of the grid layer, a feature map calculated using each
processing path must be different from feature maps calculated using other paths. For this purpose, in a SwGridNet, the number of channels of a convolutional processing unit differs from other units in the grid layer. Figure 5
presents an example of the number of unit channels in a
two-dimensional grid layer. The number of input channels
and the number of output channels of a convolutional pro4

Table 2. Test error rates of SwGridNets in a CIFAR-10 classification task. The test error rates are median values of five runs.

conv 3x3
# of output channels = 64

settings
N =1, L=16, k=16
N =2, L=4, k=16
N =4, L=2, k=16

BN
grid block
# of output channels = 128
# of unit channels = [k, k × 2]

# of params
3.7M
3.7M
3.8M

error (%)
4.56
4.39
4.36

average pooling 2x2

4. Implementation and Experiment Results

grid block
# of output channels = 256
# of unit channels = [k × 2, k × 4]

4.1. Implementation
The author implemented SwGridNets1 for CIFAR-10
and CIFAR-100 classification tasks using Chainer2 , which
is a framework for neural networks. The SwGridNets are
implemented as a chain of grid blocks, as presented in Figure 6. The SwGridNets increase the number of channels
at each grid block and decrease the resolution of feature
maps after each grid block. Unlike existing state-of-the-art
deep CNNs [10, 8, 24], the SwGridNets use average pooling to decrease the resolution. The basic concept of the
SwGridNet implementations, however, is the same as the
other state-of-the-art deep CNNs.
CIFAR-10 and CIFAR-100 are datasets used for image
classification tasks [15]. CIFAR-10 consists of 60,000 color
images of 32 × 32 pixels, which are categorized into 10
classes. Actually, CIFAR-100 includes the images, but the
images are categorized into 100 classes. For this study,
50,000 images were used in the datasets for training; 10,000
images were used in the datasets for validation. For the
training, parameters of the SwGridNets are initialized using MSRA [9]. The parameters are updated using Momentum SGD (momentum of 0.9, weight decay of 1e-4). The
SwGridNets for 630 epochs were trained with batch size
of 128. The learning rate is initially set to 0.2. Then it is
changed according to Cosine Annealing with Warm Restart
(T0 =10, Tmult =2) [18]. To improve the performance, I apply a standard data augmentation (random clipping and flip)
to the images for training just like existing state-of-the-art
deep CNNs [10, 8, 24]. Images for training, however, are
not normalized using the mean or standard deviation.

average pooling 2x2
grid block
# of output channels = 512
# of unit channels = [k × 4, k × 8]
ReLU
global average pooling
full
soHmax

Figure 6. Network architecture of a SwGridNet for CIFAR-10 and
CIFAR-100 classification tasks.

cessing unit located at (p0 , · · · , pN −1 ) is defined as

cinput
p0 ,··· ,pN −1
coutput
p0 ,··· ,pN −1

P

pi
c
1 + N (L − 1)
P
1 + pi
c
= bcmin + (cmax − cmin )
1 + N (L − 1)
min

= bc

+ (c

max

−c

min

)

(8)
(9)

Here, cinput
p0 ,··· ,pN −1 is the number of input channels of a unit
at (p0 , · · · , pN −1 ); coutput
p0 ,··· ,pN −1 is also the number of output channels of the unit. Parameters cmin and cmax respectively denote the minimum number of unit channels and the
maximum number of unit channels. The number of input
channels of a unit equals the number of output channels of
the neighbor units. Therefore, channel modification such as
zero-padding is not necessary for a SwGridNet.

4.2. Dimensions of Grid Blocks
Table 2 presents the performance of SwGridNets for a
CIFAR-10 classification task. Here, N and L respectively
denote the grid block and the side length dimensions of a
of a grid block, as described in 3.1. In addition, k denotes
the minimum number of unit channels in a first grid block
as presented in Figure 6. The dimensions of grid blocks in
these SwGridNets differ, but these SwGridNets include the
same number of convolutional processing units.
The performance of a four-dimensional SwGridNet

The number of output channels of a split layer equals the
sum of input channels of the convolutional processing units
because all convolutional processing units receive their input matrices from the split layer. Therefore,
P input the number
of output channels of a split layer is
cp0 ,··· ,pN −1 . In the
same manner, the number of input channels of a join layer
also equals the sum of output channels of the convolutional
processing units because all convolutional processing units
forward their output matrices to thePjoin layer. The number
of input channels of a join layer is coutput
p0 ,··· ,pN −1

1 The
2 the

5

repository is https://github.com/takedarts/swgridnet.
web site is https://chainer.org.

Table 3. Test error rates of SwGridNets and state-of-the-art deep CNNs in CIFAR-10/100 classification tasks. The test error rates of
SwGridNets are median values of five runs.

model
ResNet-110 [10]
FractalNet-20 [17]
FractalNet-40
WideResNet-28 [25]
DenseNet-100 [11]
PyramidNet-110 [8]
PyramidNet-164
ResNeXt-29 [24]
Shake-Shake-26 [5]
Shake-Even-29
Shake-Shake-26 + cutout [3]
Shake-Even-29 + cutout
SwGridNet (proposal)

settings
with drop-path and dropout
k=10, with dropout
k=24
k=40, bottleneck and compression
α=84
α=270
α=270, bottleneck
8×64d
16×64d
2×32
2×96
2×4×64
2×96
2×4×64
N =2, L=4, k=16
N =2, L=5, k=32
N =2, L=5, k=32, ensemble of 4 models

# of params
1.7M
38.6M
22.9M
36.5M
27.2M
25.6M
3.8M
28.3M
27.0M
34.4M
68.1M
2.9M
26.2M
34.4M
26.2M
34.4M
3.7M
18.1M
18.1M×4

CIFAR-10
6.43%
4.59%
5.21%
3.89%
3.74%
3.46%
4.26%
3.73%
3.48%
3.65%
3.58%
3.55%
2.86%

CIFAR-100
23.36%
21.49%
18.85%
19.25%
17.18%
20.66%
18.25%
17.01%
17.77%
17.31%

15.85%
2.56%
15.20%
4.39%
3.55%
2.95%

17.77%
15.67%

5. Conclusion

(N =4, L=2) is better than that of a one-dimensional
SwGridNet (N =1, L=16). As described in 3.2, a fourdimensional SwGridNet includes more processing paths
than a one-dimensional SwGridNet. Therefore, a fourdimensional SwGridNet has a higher generalization ability
than a one-dimensional SwGridNet. Because these processing paths have the same effect as ensemble learning, a fourdimensional SwGridNet obtains a lower error rate in the image classification task than a one-dimensional SwGridNet.

A SwGridNet described in this paper is a neural network
in which many convolutional processing units are mutually
connected as a grid network. The grid network of convolutional processing units has many processing paths between
input and output, which perform calculations using individual parameters. A SwGridNet has a high generalization
capability because the multipath network architecture has
the same effect as ensemble learning. Experimental results
presented in this paper show that SwGridNets respectively
achieve test error rates of 2.95% and 15.67% in a CIFAR10 and CIFAR-100 classification tasks. The experimentally obtained results demonstrate that the performance of
SwGridNets is close to state-of-the-art deep CNNs.

4.3. Comparison with State-of-the-art Deep CNNs
Table 3 presents the respective performance parameters
of SwGridNets and state-of-the-art deep CNNs in CIFAR10 and CIFAR-100 classification tasks. In this experiment,
five models of SwGridNets are trained. Four models in
the trained models are used to build ensemble models of
SwGridNets. The ensemble models of SwGridNets achieve
a test error rate of 2.95% in the CIFAR-10 classification
task and a test error rate of 15.67% in the CIFAR-100 classification task. The performance results of the SwGridNets are nearly equal to the performance results obtained
for state-of-the-art deep CNNs, even though the SwGridNets do not use popular generalization techniques such as
Dropout [21], Stochastic Depth [12], Shake Network [5] or
Cutout [3]. Because it is possible to apply the generalization
techniques to a SwGridNet, A SwGridNet with generalization techniques is expected to achieve better performance
than current results presented in Table 3.

References
[1] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term
dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166, 1994.
[2] F. Chollet. Xception: Deep learning with depthwise separable convolutions. arXiv preprint arXiv:1610.02357, 2016.
[3] T. DeVries and G. W. Taylor. Improved regularization of
convolutional neural networks with cutout. arXiv preprint
arXiv:1708.04552, 2017.
[4] D. Fourure, R. Emonet, E. Fromont, D. Muselet,
A. Tremeau, and C. Wolf. Residual conv-deconv grid
network for semantic segmentation.
arXiv preprint
arXiv:1707.07958, 2017.

6

[5] X. Gastaldi. Shake-shake regularization. arXiv preprint
arXiv:1705.07485, 2017.
[6] X. Glorot and Y. Bengio. Understanding the difficulty of
training deep feedforward neural networks. In Proceedings
of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249–256, 2010.
[7] X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics,
pages 315–323, 2011.
[8] D. Han, J. Kim, and J. Kim. Deep pyramidal residual networks. arXiv preprint arXiv:1610.02915, 2016.
[9] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into
rectifiers: Surpassing human-level performance on imagenet
classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 1026–1034, 2015.
[10] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
770–778, 2016.
[11] G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten.
Densely connected convolutional networks. arXiv preprint
arXiv:1608.06993, 2016.
[12] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger.
Deep networks with stochastic depth. In European Conference on Computer Vision, pages 646–661. Springer, 2016.
[13] S. Ioffe and C. Szegedy. Batch normalization: Accelerating
deep network training by reducing internal covariate shift. In
International Conference on Machine Learning, pages 448–
456, 2015.
[14] G. Kang, J. Li, and D. Tao. Shakeout: A new approach to
regularized deep neural network training. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 2017.
[15] A. Krizhevsky. Learning multiple layers of features from
tiny images. Tech Report, 2009.
[16] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
Advances in Neural Information Processing Systems, pages
1097–1105, 2012.
[17] G. Larsson, M. Maire, and G. Shakhnarovich. Fractalnet:
Ultra-deep neural networks without residuals. In Proceedings of the Fifth International Conference on Learning Representations, 2017.
[18] I. Loshchilov and F. Hutter. Sgdr: stochastic gradient descent with restarts. In Proceedings of the Fifth International
Conference on Learning Representations, 2016.
[19] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
et al. Imagenet large scale visual recognition challenge.
International Journal of Computer Vision, 115(3):211–252,
2015.
[20] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. In Proceedings
of the Third International Conference on Learning Representations, 2015.
[21] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and
R. Salakhutdinov. Dropout: a simple way to prevent neu-

[22]

[23]

[24]

[25]
[26]

[27]

7

ral networks from overfitting. Journal of Machine Learning
Research, 15(1):1929–1958, 2014.
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.
Rethinking the inception architecture for computer vision.
In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, pages 2818–2826, 2016.
A. Veit, M. J. Wilber, and S. Belongie. Residual networks
behave like ensembles of relatively shallow networks. In
Advances in Neural Information Processing Systems, pages
550–558, 2016.
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated
residual transformations for deep neural networks. arXiv
preprint arXiv:1611.05431, 2016.
S. Zagoruyko and N. Komodakis. Wide residual networks.
arXiv preprint arXiv:1605.07146, 2016.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals.
Understanding deep learning requires rethinking generalization. In Proceedings of the Fifth International Conference on
Learning Representations, 2017.
Z.-H. Zhou, J. Wu, and W. Tang. Ensembling neural networks: many could be better than all. Artificial Intelligence,
137(1-2):239–263, 2002.

