Title: No title found

Abstract: This paper proposes a powerful regularization method named ShakeDrop regularization. ShakeDrop is inspired by Shake-Shake regularization that decreases error rates by disturbing learning. While Shake-Shake can be applied to only ResNeXt which has multiple branches, ShakeDrop can be applied to not only ResNeXt but also ResNet, Wide ResNet and PyramidNet in a memory efficient way. Important and interesting feature of ShakeDrop is that it strongly disturbs learning by multiplying even a negative factor to the output of a convolutional layer in the forward training pass. The effectiveness of ShakeDrop is confirmed by experiments on CIFAR-10/100 and Tiny ImageNet datasets.

1. INTRODUCTION:

Recent advances in generic object recognition have been brought by deep neural networks. After ResNet (He et al., 2016) opened the door to very deep CNNs of over a hundred layers by introducing the residual block, its improvements such as Wide ResNet (Zagoruyko & Komodakis, 2016) , PyramdNet (Han et al., 2017a; b) and ResNeXt (Xie et al., 2017) have broken the records of lowest error rates. On the other hand, in learning, they often suffer from problems such as vanishing gradients. Hence, regularization methods help to learn and boost the performance of such base network architectures. Stochastic Depth (ResDrop) (Huang et al., 2016) and Shake-Shake (Gastaldi, 2017) are known to be effective regularization methods for ResNet and its improvements. Among them, Shake-Shake applied to ResNeXt is the one achieving the lowest error rates on CIFAR-10/100 datasets (Gastaldi, 2017) . Shake-Shake, however, has following two drawbacks. (1) Shake-Shake can be applied to only multibranch architectures (i.e., ResNeXt). (2) Shake-Shake is not memory efficient. Both drawbacks come from the same root. That is, Shake-Shake requires two branches of residual blocks to apply. If it is true, it is not difficult to conceive its solution: a similar disturbance to Shake-Shake on a single residual block. It is, however, not trivial to realize it.

The current paper addresses the problem of realizing a similar disturbance to Shake-Shake on a single residual block, and proposes a powerful regularization method, named ShakeDrop regularization. While the proposed ShakeDrop is inspired by Shake-Shake, the mechanism of disturbing learning is completely different. ShakeDrop disturbs learning more strongly by multiplying even a negative factor to the output of a convolutional layer in the forward training pass. In addition, a different factor from the forward pass is multiplied in the backward training pass. As a byproduct, however, learning process gets unstable. Our solution to this problem is to stabilize the learning process by employing ResDrop in a different usage from the usual. Based on experiments using various base network architectures, we reveal the condition that the proposed ShakeDrop successfully works.

2. EXISTING METHODS REQUIRED TO INTRODUCE THE PROPOSED METHOD:



2.1. DEEP NETWORK ARCHITECTURES:

ResNet (He et al., 2016) opened the door to very deep CNNs of over a hundred layers by introducing the residual block, given as 2016) experimentally showed that some residual blocks whose channels greatly increase cause high error rates.

EQUATION

PyramidNet (Han et al., 2017a; b) 

EQUATION

where F 1 (x) and F 2 (x) are the outputs of the residual branches as shown in Figure 1 (a). The number of residual branches is not limited to 2, and the number is the most important factor to control the result. Depth (Huang et al., 2016) is a regularization method which overcame problems of ResNet such as vanishing gradients. It makes the network apparently shallow in learning by dropping residual blocks stochastically selected. On the l th residual block from input layer, the Stochastic Depth process is given as

2.3. REGULARIZATION METHODS:



Stochastic:

EQUATION

where b l ∈ {0, 1} is a Bernoulli random variable with the probability of p l . Huang et al. ( 2016) tested two rules for determining p l : uniform rule and linear decay rule. The uniform rule using a constant for all p l did not work well. The linear decay rule which defines p l as

EQUATION

where L is the number of all layers and p L is the initial parameter, worked well.

It was originally proposed for improving ResNet. It was also applied to PyramidNet; the combination of PyramidNet and Stochastic Depth was named PyramidDrop (Yamada et al., 2016) .

Shake-Shake (Gastaldi, 2017 ) is a powerful regularization method for improving ResNeXt architectures. It is given as

EQUATION

where α is a random coefficient given as α ∈ [0, 1]. As shown in Figure 1 (b), calculation of the backward pass is disturbed by another random coefficient β ∈ [0, 1] instead of α. As a result, Shake-Shake decreased error rates than ResNeXt.

3. PROPOSED METHOD:



3.1. INTERPRETATION OF SHAKE-SHAKE REGULARIZATION AND ITS DRAWBACKS:

We give an intuitive interpretation of the forward pass of Shake-Shake regularization. To the best of our knowledge, it has not been given yet, while the phenomenon in the backward pass is experimentally investigated by Gastaldi (2017) . As shown in Eqn. (5) (and also in Fig. 1(b )), in the forward pass, Shake-Shake interpolates the outputs of two residual branches (i.e., F 1 (x) and F 2 (x)) with a random variable α that controls the degree of interpolation. As DeVries & Taylor (2017a) demonstrated that interpolation of two data in the feature space can synthesize reasonable augmented data, the interpolation of two residual blocks of Shake-Shake in the forward pass can be interpreted as synthesizing data. Use of a random variable α generates many different augmented data. On the other hand, in the backward pass, a different random variable β is used to disturb learning to make the network learnable long time. Gastaldi (2017) demonstrated how the difference between α and β affects.

The regularization mechanism of Shake-Shake relies on two or more residual branches, so that it can be applied only to 2-branch networks architectures. In addition, 2-branch network architectures consume more memory than 1-branch network architectures. One may think the number of learnable parameters of ResNeXt can be kept in 1-branch and 2-branch network architectures by controlling its cardinality and the number of channels (filters). For example, a 1-branch network (e.g., ResNeXt 1-64d) and its corresponding 2-branch network (e.g., ResNeXt 2-40d) have almost same number of learnable parameters. However, even so, it increases memory consumption due to the overhead to keep the inputs of residual blocks and so on. By comparing ResNeXt 1-64d and 2-40d, the latter requires more memory than the former by 8% in theory (for one layer) and by 11% in measured values (for 152 layers).

3.2. SIMILAR REGULARIZATION TO SHAKE-SHAKE ON 1-BRANCH NETWORK:



ARCHITECTURES:

In order to realize a similar regularization to Shake-Shake on 1-branch network architectures, in the forward pass, we need a mechanism, different from interpolation, to synthesize augmented data in the feature space. Actually, DeVries & Taylor (2017a) demonstrated not only interpolation but also noise addition in the feature space works well. Hence, following Shake-Shake, we apply random perturbation, using α, to the output of a residual branch (i.e., F (x) of Eqn. ( 1)). In the backward pass, we can use the same way as Shake-Shake for 1-branch network architectures.

We call the regularization method mentioned above 1-branch Shake. It is given as

EQUATION

where α is a coefficient that disturbs learning in the forward pass. As shown in Figure 1 (c), β is a coefficient similar to the one of Shake-Shake and used in the backward pass. The 1-branch Shake is expected to realize powerful generalization like Shake-Shake regularization. However, by applying it to 110-layer PyramidNet with α ∈ [0, 1] and β ∈ [0, 1] following Shake-Shake, the result on the CIFAR-100 dataset was hopelessly bad (i.e., an error rate of 77.99%).

3.3. STABILIZING LEARNING WITH INTRODUCTION OF MECHANISM OF RESDROP:

Failure of 1-branch Shake is caused by too strong perturbation. However, weakening the perturbation would also weaken the effect of regularization. Thus, we need a trick to promote learning under strong perturbation.

Our idea is to use the mechanism of ResDrop for solving the issue. As is written in Sec.2.3, in the original form, ResDrop promotes learning by dropping some residual blocks. In other words, it makes a network apparently shallow in learning. In our situation, however, the original usage of ResDrop does not contribute because a shallower network to which 1-branch Shake is applied would also suffer from strong perturbation. Thus, we use the mechanism of ResDrop as a probabilistic switch of two network architectures: the original network (e.g., PyramidNet) and the one to which 1-branch Shake is applied (e.g., PyramidNet + 1-branch Shake). By mixing up two networks, the following effects are expected.

1. When the original network (e.g., PyramidNet) is selected, learning is correctly promoted. 2. When the network with strong perturbation (e.g., PyramidNet + 1-branch Shake) is selected, learning is disturbed.

To achieve a good performance, two networks should be in a good balance.

The proposed ShakeDrop is given as

EQUATION

where b l is a Bernoulli random variable following the linear decay rule. We call the new method "ShakeDrop." According to the value of b l , Eqn. ( 7) is deformed as

EQUATION

In the case that the base network is PyramidNet, ShakeDrop is equivalent to PyramidNet if b l = 1, and it is equivalent to "PyramidNet + 1-branch Shake" if b l = 0. We found that the linear decay rule of ResDrop (Eqn. ( 4)) works well, while we also tested the uniform rule which did not work well. As shown in Figure 1 (d), similar to Shake-Shake, β is used in the backward pass instead of α. It is noteworthy that regardless of the value of β, the weights of the network are updated. For example, let us consider the case of β = 0. In such a case, the weights of the residual blocks selected to perturbate (i.e., the layers with b l = 0) are not updated. However, the inputs to their succeeding residual blocks are perturbated and the weights of the residual blocks are updated reflecting the perturbation.

4. EXPERIMENTS:

See Sec. A for implementation details.

4.1. PRELIMINARY EXPERIMENTS: SEARCH FOR PARAMETER SETTINGS OF THE PROPOSED:



METHOD:

The best parameter settings of the proposed method, ShapeDrop, are searched when it is applied to PyramidNet. They include parameter ranges of α and β, and the timing these parameters are updated. In the preliminary experiments, the CIFAR-100 dataset was used. PyramidNet had 110 layers, which consisted of a convolution layer, 54 additive pyramidal residual blocks and a fully connected layer. The number of channels at the last residual block was 286. Table 1 shows representative parameter ranges of α and β we tested and their results. From the results, we can see how the combinations of α and β affect. First of all, except PyramidDrop (α = 0, β = 0), only case F (α ∈ [-1, 1], β = 0) and case H (α ∈ [-1, 1], β ∈ [0, 1]) were better than the original PyramidNet (case A: α = 1, β = 1). Among them, case H was the best. It is interesting to observe three cases E (α ∈ [-1, 1], β = 1), F and H all of which take the same range of α. Since cases F and H were better than PyramidNet, we confirm strong perturbation is effective. However, case E was even worse than PyramidNet. This implies that if strong perturbation is applied, the residual blocks selected to perturbate strongly suffer from the perturbation, so that the weights of the residual blocks should not be fully updated (i.e., β = 1 should be avoided). On the other hand, the range of β of the best case H was in-between cases E and F. This implies that the values of β should be well balanced.

[0, 1] [-1, 1] 20.61 D [0, 1] [0, 1] 18.27 E [-1, 1] 1 18.68 F [-1, 1] 0 17.28 G [-1, 1] [-1, 1] 18.26 H [-1, 1] [0, 1] 16.22

The best strategy to update the parameters (referred as scaling coefficient) among "Batch," "Image," "Channel" and "Pixel" are explored. "Batch" means that the same scaling coefficients are used for all the images in the mini-batch for each residual block. "Image" means that the same scaling coefficients are used for each image for each residual block. "Channel" means that the same scaling coefficients are used for the each channel for each residual block. "Pixel" means that the same scaling coefficients are used for each element for each residual block. ShakeDrop was trained using the best parameters found in the experiment above, i.e, ranges α ∈ [-1, 1], β ∈ [0, 1]. Table 2 shows that "Pixel" level was the best. However, it required a lot of memory. Hence, we selected "Image" level, which was the second best, considering memory efficiency.

Through the preliminary experiments, we found that the most effective parameters for α and β are α ∈ [-1, 1] and β ∈ [0, 1], and the effective and memory efficient level is "Image" level. We conduct the following experiments using these parameters.

4.2. COMPARISON WITH REGULARIZATION METHODS:

The proposed ShakeDrop is compared with vanilla (without regularization), ResDrop and 1-branch Shake in different network architectures including ResNet, PyramidNet, Wide ResNet and ResNeXt. Tables 3 and 4 show experimental results on CIFAR-100 dataset (Krizhevsky, 2009) and Tiny Ima-geNet dataset1 , respectively.

Table 3 shows that the proposed ShakeDrop can be applied to not only PyramidNet but also ResNet, Wide ResNet and ResNeXt. However, to successfully apply it, we found that the residual blocks have to end with batch normalization (BN) (Ioffe & Szegedy, 2015) . Since without BN, the outputs of residual blocks are not in a certain range, sometimes factors α and β could be too big. This seems to cause diverge in learning. With this regard, EraseReLU (Dong et al., 2017) which elimi- nates the last ReLU is a very convenient for us because in the EraseReLU versions of ResNet and ResNeXt, the residual blocks ends with BN. On the other hand, in the case of Wide ResNet, we need to intentionally add BN in the end of the residual blocks. In ResNeXt, we examined two ways, referred as "Type A" and "Type B," to apply ResDrop and ShakeDrop. They mean the regularization module is inserted before and after "add" module, respectively. As far as we examined, Type B was Table 4 : Top-1 errors (%) at the final (300th) epoch of ResNet and its improvements to which different regularization methods are applied on Tiny ImageNet dataset. Method names are followed by components of their residual blocks. For ResNet and ResNeXt, in addition to the original form, ones without ReLU unit in the end of residual blocks following EraseReLU (Dong et al., 2017) are also examined. For Wide ResNet, ones with bath normalization added in the end of residual blocks, referred as "w/ BN," are also examined. "Type A" and "Type B" of ResDrop and ShakeDrop mean that regularization unit is inserted before and after "add" unit for residual branches, respectively. "×" means learning did not converge. * indicates the result is quoted from the literature.

( always better than Type A. We also confirmed that ShakeDrop Type B outperformed Shake-Shake in RexNeXt. As a conclusion, the table shows that as long as the residual blocks end with BN, the proposed ShakeDrop achieved the lowest error rates.

Table 4 shows the results on Tiny ImageNet. To conduct the experiments, it is better to use networks tuned for ImageNet. However, except PyramidNet, we could apply the same networks tuned for CIFAR-100 dataset were used due to time constraint. As a result, in ResNet, the lowest error rates were achieved by vanilla EraseReLU networks. This could be caused by use of inappropriate base networks for the task since learning ended with relatively high training loss and no regularization methods correctly worked. However, in PyramidNet and Wide ResNet, the proposed ShakeDrop achieved the lowest error rates. This confirms that the proposed ShakeDrop works on Tiny ImageNet dataset.

4.3. COMPARISON WITH STATE-OF-THE-ART METHODS:

The proposed method was compared with state-of-the-arts on the CIFAR-10/100 datasets. Stateof-the-art methods introduced some techniques that can be applied to many methods in the learning process. One is longer learning. While most of methods related to ResNet use 300-epoch scheduling for learning like the preliminary experiment in Sec. 4.1, Shake-Shake use 1800-epoch cosine annealing, on which the initial learning rate is annealed using a cosine function without restart (Gastaldi, 2017) . Another one is image preprocessing. DeVries & Taylor (2017b) and Zhong et al. (2017) showed that accuracy is improved by data augmentation which randomly fills a part of learning images. For fair comparison with these methods, we also applied them to the proposed method.

PyramidNet (including PyramidDrop and "PyramidDrop + ShakeDrop") had 272 layers, which consisted of a convolution layer, 90 additive pyramidal "bottleneck" blocks (He et al., 2016) and a fully connected layer. The number of channels at the last pyramidal residual block was 864. Table 5 shows the error rates. The proposed method, "PyramidNet + ShakeDrop," without longer learning and image preprocessing, was 3.41% on the CIFAR-10 dataset and 14.90% on the CIFAR-100 dataset. We make following comparisons with existing methods. Note that the proposed method used less numbers of parameters than rival methods in all conditions.

(1) Fair comparison with Shake-Shake (ResNeXt + Shake-Shake) in the same condition "PyramidNet + ShakeDrop" combined with longer training achieved error rates of 2.67% (better by 0.19%) on the CIFAR-10 dataset and 13.99% (better by 1.86%) on the CIFAR-100 dataset.

(2) Fair comparison with Cutout (ResNeXt + Shake-Shake + Cutout) in the same condition "PyramidNet + ShakeDrop" combined with both longer learning and image preprocessing achieved error rates of 2.31% (better by 0.25%) and 12.19% (better by 3.01%) on the CIFAR-10/100 datasets, respectively.

(3) Comparison with the state-of-the-arts, Cutout on CIFAR-10 and Coupled Ensemble on CIFAR-100 "PyramidNet + ShakeDrop" combined with both longer learning and image preprocessing achieved error rates of 2.31% (better by 0.25% than Cutout) and 12.19% (better by 2.85% than Coupled Ensemble) on the CIFAR-10/100 datasets, respectively.

5. CONCLUSION:

We proposed a new stochastic regularization method ShakeDrop which can be successfully applied to ResNet and its improvements as long as the residual blocks ends with BN. Its effectiveness was confirmed through the experiments on CIFAR-10/100 and Tiny ImageNet datasets. ShakeDrop achieved the state-of-the-art performance in the CIFAR-10/100 datasets, and the best results among existing regularization methods on Tiny ImageNet dataset except some cases using inappropriate networks. 

:

https://tiny-imagenet.herokuapp.com/

https://github.com/facebook/fb.resnet.torch

Please be aware that the implementation in sgd.lua of Torch7 with nesterov mode is an approximation of the original Nesterov accelerated gradient and momentum method. See https://github.com/torch/ optim/issues/27.

Figures and Tables:

Figure 1: Network architectures.

Figure 3: The average of the gradients on each iteration in PyramidNet.

Figure 4: The variance of the gradients on each iteration in PyramidNet.

Average Top-1 errors (%) of "PyramidNet + ShakeDrop" with several ranges of parameters of 4 runs at the final (300th) epoch on CIFAR-100 dataset in the "Batch" level. In some settings, it is equivalent to PyramidNet and PyramidDrop.

Table content: <table><tr><td/><td>α</td><td>β</td><td>Error (%)</td><td>Note</td></tr><tr><td>A</td><td>1</td><td>1</td><td>18.01</td><td>Equivalent to PyramidNet</td></tr><tr><td>B</td><td>0</td><td>0</td><td>17.74</td><td>Equivalent to PyramidDrop</td></tr><tr><td>C</td><td/><td/><td/><td/></tr></table>



Table content: <table><tr><td colspan="4">: Average Top-1 errors (%) of "PyramidNet + ShakeDrop" with different levels of 4 runs at</td></tr><tr><td colspan="2">the final (300th) epoch on CIFAR-100 dataset.</td><td/><td/></tr><tr><td>α</td><td>β</td><td>Level</td><td>Error (%)</td></tr><tr><td/><td/><td>Batch</td><td>16.22</td></tr><tr><td colspan="2">[-1, 1] [0, 1]</td><td>Image Channel</td><td>16.04 16.12</td></tr><tr><td/><td/><td>Pixel</td><td>15.78</td></tr></table>

Top-1 errors (%) at the final (300th) epoch of ResNet and its improvements to which different regularization methods are applied on CIFAR-100 dataset. Method names are followed by components of their residual blocks. For ResNet and ResNeXt, in addition to the original form, ones without ReLU unit in the end of residual blocks following EraseReLU(Dong et al., 2017) are also examined. For Wide ResNet, ones with bath normalization added in the end of residual blocks, referred as "w/ BN," are also examined. "Type A" and "Type B" of ResDrop and ShakeDrop mean that regularization unit is inserted before and after "add" unit for residual branches, respectively. "×" means learning did not converge. * indicates the result is quoted from the literature.

Table content: <table><tr><td colspan="5">(a) 1-residual-branch network architectures (ResNet, ResNeXt and PyramidNet)</td></tr><tr><td>Methods</td><td colspan="2">Regularization</td><td colspan="2">Original (%) EraseReLU (%)</td></tr><tr><td>ResNet-110 &lt;Conv-BN-ReLU-Conv-BN-add-(ReLU)&gt;</td><td colspan="2">Vanilla ResDrop 1-branch Shake ShakeDrop</td><td/><td>28.51 24.09 24.18 ×</td><td>24.93 22.88 23.80 22.68</td></tr><tr><td>ResNet-164 Bottleneck &lt;Conv-BN-ReLU-Conv-BN-ReLU-Conv-BN-add-(ReLU)&gt;</td><td colspan="2">Vanilla ResDrop 1-branch Shake ShakeDrop</td><td/><td>22.00 21.96 22.20 ×</td><td>21.96 20.35 21.60 19.89</td></tr><tr><td>ResNeXt-29 8-64d Bottleneck &lt;Conv-BN-ReLU-Conv-BN-ReLU-Conv-BN-add-(ReLU)&gt;</td><td colspan="2">Vanilla ResDrop 1-branch Shake ShakeDrop</td><td/><td>20.90 20.66 22.70 ×</td><td>20.25 20.28 24.00 19.90</td></tr><tr><td>PyramidNet-272 α200 Bottleneck &lt;BN-Conv-BN-ReLU-Conv-BN-ReLU-Conv-BN-add&gt;</td><td colspan="2">Vanilla ResDrop 1-branch Shake ShakeDrop</td><td/><td>*  16.35 15.94 71.51 14.90</td><td>N/A</td></tr><tr><td colspan="5">(b) 1-residual-branch network architectures (Wide ResNet)</td></tr><tr><td>Methods</td><td/><td colspan="2">Regularization</td><td>Original (%) w/ BN (%)</td></tr><tr><td/><td/><td>Vanilla</td><td/><td>26.49</td><td>24.24</td></tr><tr><td>Wide-ResNet-28-10k</td><td/><td>ResDrop</td><td/><td>34.19</td><td>26.64</td></tr><tr><td colspan="2">&lt;BN-ReLU-Conv-BN-ReLU-Conv-(BN)-add&gt;</td><td colspan="2">1-branch Shake</td><td>90.73</td><td>58.89</td></tr><tr><td/><td/><td colspan="2">ShakeDrop</td><td>76.87</td><td>19.12</td></tr><tr><td colspan="4">(c) 2-residual-branch network architectures</td></tr><tr><td>Methods</td><td colspan="2">Regularization</td><td colspan="2">Original (%) EraseReLU (%)</td></tr><tr><td/><td/><td>Vanilla</td><td/><td>23.82</td><td>21.75</td></tr><tr><td>ResNeXt-164 2-1-40d Bottleneck &lt;Conv-BN-ReLU-Conv-BN-ReLU-Conv-BN-add-(ReLU)&gt;</td><td colspan="2">ResDrop Type-A ResDrop Type-B Shake-Shake ShakeDrop Type-A</td><td/><td>21.38 21.34 22.35 ×</td><td>20.44 20.21 22.51 19.98</td></tr><tr><td/><td colspan="2">ShakeDrop Type-B</td><td/><td>×</td><td>19.83</td></tr><tr><td/><td/><td>Vanilla</td><td/><td>21.19</td><td>×</td></tr><tr><td>ResNeXt-29 2-4-64d Bottleneck &lt;Conv-BN-ReLU-Conv-BN-ReLU-Conv-BN-add-(ReLU)&gt;</td><td colspan="2">ResDrop Type-A ResDrop Type-B Shake-Shake ShakeDrop Type-A</td><td/><td>21.12 19.27 19.16 ×</td><td>20.13 19.01 18.82 20.07</td></tr><tr><td/><td colspan="2">ShakeDrop Type-B</td><td/><td>×</td><td>18.17</td></tr></table>

Peer review (iclr2018):
 This paper proposes a regularization technique for deep residual networks. It is inspired by regularization techniques which disturb the training by applying multiplicative factors to the convolutional layer outputs e.g Shake-Shake (Gastaldi *17) and PyramidDrop (Yamada *16). The proposed approach samples a Bernoulli variable randomly to either follow the standard variant of Pyramid net, or applies a variant of shake-shake to pyramid net. + Experimental results on CIFAR-10 and CIFAR-100 well-exceed exceed the existing "vanilla" techniques + regularizers. - Clarity: some statements are not clear / not substantiated e.g. how does the proposed method overcome the memory problem that shake-shake has? There are some minor issues wrt presentation, e.g. grammatical correctness of sentences, consistent usage of references, which can be fixed with more careful proofreading. - Quality: even though the experimental results are compelling, the paper lacks thorough analysis in understanding the effects of the regularizer. The two experiments looks at (1) the training error, which the paper openly states does not explain why the proposed regularization works and (2) variance of the gradients throughout learning; the larger variance of gradients is speculated to be the cause, but this is almost expected, given that the method is designed to allow larger fluctuations and perturbations during training.