RotationOut as a Regularization Method for Neural Network
Authors: Kai Hu
Barnabas Poczos
Venue:  (2020)
Keywords: Neural Network
Regularization

Abstract: In this paper, we propose a novel regularization method, RotationOut, for neural networks. 
Different from Dropout that handles each neuron/channel independently, RotationOut regards its input layer as an entire vector and introduces regularization by randomly rotating the vector. 
RotationOut can also be used in convolutional layers and recurrent layers with a small modification.
We further use a noise analysis method to interpret the difference between RotationOut and Dropout in co-adaptation reduction. 
Using this method, we also show how to use RotationOut/Dropout together with Batch Normalization. 
Extensive experiments in vision and language tasks are conducted to show the effectiveness of the proposed method. 
Codes will be available.

Under review as a conference paper at ICLR 2020
ROTATIONOUT AS A REGULARIZATION METHOD FOR NEURAL NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, propose novel regularization method, RotationOut, for neu- ral networks. Different from Dropout that handles each neuron/channel inde- pendently, RotationOut regards its input layer as an entire vector and introduces regularization by randomly rotating the vector. RotationOut can also be used in convolutional layers and recurrent layers with small modifications. We fur- ther use a noise analysis method to interpret the difference between RotationOut and Dropout in co-adaptation reduction. Using this method, we also show how to use RotationOOu//rropout together with Batch Normalization. Extensive ex- periments in vision and language tasks are conducted to show the effectiveness of the proposed method. Codes are available at https: /ttt:://gitttie com/ Rotationout/Rota:
INTRODUCTION
Dropout (Srivastava et al. 2014) has proven be effective for preventing overfitting over many deep learning areas, such as image classification (Shrivastava et al. (017). natural language processing (Hu al. 2016) and speech recognition (Amodei et al., 2016). In the years since, a wide range of variants have been proposed for wider scenarios, and most related work focus on the improvement of Dropout structures, i.e., how to drop. For example, drop connect (Wan al., 2013) drops the weights instead of neurons, evolutional dropout (Li et al. (016) computes the adaptive dropping probabilities on-the-fly, max max-pooling dropout (Wu & Gul 2015) drops neurons in the max-poolle max-pooling kernel so smaller feature values have some probabilities to to affect the activations.
These Dropout- methods process each neuron/channel in one layer independently and introduce randomness by dropping. These architectures are certainly simple and effective. However, randomly dropping independently is not the only method to introduce randomness. Hinton et al. (2012) argues that overfitting can be reduced by preventing co-adaptation between feature detectors. Thus it is helpful to consider other neurons' information when adding noise to one neuron. For example, lateral inhibition noise could be more effective than independent noise.
In this paper, we propose RotationOut as a regularization method for neural networks. RotationOut regards the neurons in one layer vector and introduces noise by randomly rotating the vector. Specifically, consider a fully-connected layer with n neurons: x E TR.". If applying RotationOut to this layer, the output is Ræ where R E x is random rotation matrix. It rotates the input with random angles and directions, bringing noise to the input. The noise added to neuron comes not only from itself, but also from other neurons. Dropout-like like methods. We further show that RotationOut uses the activations of the other neurons as the noise one neuron so that the co-adaptation between neurons can be reduced.
the major difference between RotationOut and
RotationOut uses random rotation matrices instead of unrestricted matrices because the directions of feature vectors are important. Random rotation provides noise to the directions directly. Most neural networks use dot product between the feature vector and weight vector as the output. The network actually learns the direction of the weights, especially when there is a normalization layer (e.g. Batch Normalization (Toffe & Szegedy 2015) Weight Normalization (Salimans & Kingma 2016)) after the weight layer. Random rotation feature vecoters introduces noise into the angle between the feature and the weight, making the learning of weights directions more stable. Sabour jet al. (2017) also uses the orientation of feature vectors to represent the instantiation parameters in capsules. Another motivation for rotating feature vectors comes from network dissection. Bau et al.

---

Under review as a conference paper at ICLR 2020
2017) finds that random rotations of learned representation destroy the interpretability which is axis- uxis-aligned. Thus random rotating the feature during training makes the network more robust. Even small rotations can be strong regularization.
We study how RotationOut helps prevent neural networks from overfitting. Hinton et al. (2012) in- troduces co-adaptation to interpret Dropout but few literature give clear concept of co-adaptation. In this paper,we we provide a metric to approximate co-adaptations and derive a general formula for noise analysis. Using the formula, we prove that RotationOut can reduce co-adaptations more effec- tively than Dropout and show how to combine Dropout and Batch Normalization together.
In our experiments, RotationOut can achieve results on par with better than Dropout and Dropout- like methods among several deep learning tasks. Applying RotationOut after convolutional layers and fully connected layers improves image classification accuracy of ConvNet on CIFARI00 and ImageNet datasets. On COCO datasets, RotationOut also improves the generalization of object detection models. For LSTM models, RotationOut can achieve competitive results with existing RNN dropout method for speech recognition task on Wall Street Journal (WSJ) corpus.
The main contributions of this paper are as follows: We propose RotationOut as a regularization method for neural networks which is different from existing Dropout-like methods that operate on each neuron independently. RotationOut randomly rotates the feature vector and introduces noise to one neuron with other neurons' information. We present a theoretical analysis method for general formula of noise. Using the method, we answer two questions: 1) how noise-based regularization methods reduce co-adaptions and 2) how to combine noise-based regularization methods with Batch Normalization. Experiments in vision and language tasks are conducted to show the effectiveness of the proposed RotationOut method.
Related Work Dropout is effective for fully connected layers. When applied convolution layers, is less effective. GGiaas et al. (2018) argues that information about the input can still be sent to the next layer even with dropout, which causes the networks overfit (Ghiasi al. 2018). SpatialDropout (Tompson et al. 2013) drops the entire channel from the feature map. Shake-shake
regularization (Gastaldi] 2017) drops the residual branches. Cutout Vries & Taylor 2017 and Dropblock (Ghiasi et al. 2018) drop a continuois square region from the inputs/feature maps.
Applying standard dropout recurrent layers also results in poor performance (Zaremba et al.) 2014 Labach et al. 2019), since the noise caused by dropout at each time step prevents the network from retaining long-term memory. [Ga Ghahramani (2016); Moon et al. (2015): Merity et al.) 2017) generate dropout mask for each input sequence, and keep it the same at every time step so that memory can be retained.
Batch Normalization (BN) (loffe & Szegedy] 2015) accelerates deep network training. It is also regularization to the network, and discourage the strength of dropout to prevent overfitting (Toffe & Szegedy, 2015). Many modern ConvNet architectures such as ResNet (He et al. 2016) and DenseNet (Huang et al., 2017) do not apply dropout in convolutions. Li et al. (2019) is the first to argue that is caused by the variance shift. In this paper, use the noise analysis method to further explore this problem.
There is a lot of work studying rotations in networks. Rotations the images (Lenc & Vedaldi 2015 Simard et al. (003) are important data augmentation methods. There are also studies about rotation equivalence. Worrall et al. (2017) uses an enriched feature map explicitly capturing the underlying orientations. [arcos et al. (2017) applies multiple rotated versions of each filter to the input to solve problems requiring different responses with respect to the inputs' rotation. The motivations of these work are different from ours. The most related work is network dissection (Bau al. 2017). They discuss the impact on the interpretability of random rotations of learned features, showing that rotation in training can be strong regularization.
ROTATIONOUT
In this section, we first introduce the formulation of RotationOut. Next, we use linear models to demonstrate how RotationOut helps for regularization. In the last part, we discuss the implementa- tion of RotationOut in neural networks.
2

---

Under review as a conference paper at ICLR 2020
2.1 RANDOM ROTATION MATRIX
rotation in D dimension is represented by the product between a rotation matrix R E DD and the feature vector IR The complexity for random rotation matrix generation and the matrix multiplication are both O(D2), which would be less efficient than Dropout with O(D) complexity. We consider a special case that uses Givens rotations (Anderson 2000) to construct random rotation matrices to reduce the complexity.
Let D 2d be an even number, and P [mg, 12,, Ro,.... T22d be a permutation of (1,2. , A rotation matrix can be generated by function (6, P) {riz} DDDD:
cos
if i j if i P..j = P(+d
sin 0
Tij = sin 0
(1)
if i = Pitd.J = P.
otherwise.
Here Pt represents the 1th element of P where 1 I d. See Appendix A.1 for some examples of such rotation matrices. Suppose we sample the angle 0 from zero-centered distributions, e.g., truncated Gaussian distribution or uniform distribution and sample the permutation /' from P,the of all permutations of 21,2.... ,D. with equal probability. The RotationOut operator R can be generated using the function (P,
P P~ Un),(- R 1ARR-1MMMP MP,00).
cos
Here 1/ cos 0 is normalization term and R. is not a rotation matrix strictly speaking. The random operator generated from Equation 2 have some good properties. 1) The noise zero centered: ER [Ræ] x. 2) For any vector æ and any random permutation P, the angle between æ and Ræ is determined by angle 6: (xx Rx)
3) For fixed angel 0, there exists D1/ /!! different rotations. 4) The complexity for random rotation matrix generation and the matrix multiplication are both O(D).
Permutation draws the rotation direction and angel draws the rotation angle. As an analogy, permutation P is similar to the dropout mask widely used in RNN dropout. There exists 2D dif- ferent dropout mask (2D < D!/d! for > 8), thus the diversity of random rotation Equation D is sufficient for network training. Angle 0 is similar to the percentage of dropped neurons in Dropout, and the distribution of 0 controls the regularization strength. (Srivastava al., 2014) used the multiplier's variance to compare Bernoulli dropout and Gaussian dropout. Following this set- ting. RotationOut is equivalent Bernoulli Dropout with the keeping rate and Gaussian dropout with variance 2 if (1 p)/p p- o² Eg tan? 8.
Reviewing the formulation of the random rotation matrix, it arranges all D dimensions of the input into d pairs randomly, and rotates the dimension vectors with angle 0 in each pair. Suppose and v are two dimensions/neurrns one pair, the outputs of u and v after RotationOut are
u' o'
tan 0
u =
u v tan
tan O
t
The noise of w' comes from e and the noise of o' comes from u since 0 is random. Note that the pairs randomly arranged, thus RotationOut one dimension/neuron of the feature vector. With RotationOut, the neurons are trained to work more independently since one neuron has to regard the activation of other neurons noise. Thus the co-adaptations are reduced.
all other dimensionnsneeuross
the noise for
Consider Gaussian dropout, the outputs are u' ue, v' u- where Ee 0, Ee² Eg tan 0. The difference between Gaussian dropout and RotationOut is the source of noise, i.c., the Gaussian dropout noise for one neuron comes from itself while the RotationOut noise comes from other neurons.
2.2 ROTATIONOUT IN LINEAR MODELS
First we consider a simple case of applying RotationOut to the classical problem linear regression. Let ((x;, y1) J be the dataset where a; IRD yi R. Linear regression tries to find the weight

---

Under review as a conference paper at ICLR 2020
N Σ
w E RDD that minimizes
(i w w²xx)²² When applied RotationOut to each U;, we generate R,
from Equation
2 for each mi- The objective function becomes:
min ER
U w " )
(4)
Denote = 18, ... ynl" E RR,, X 01 22. ... xnl E ERNN NxD. To compare RotationOut
with Dropout with keep rate p. suppose Eg tan 0 = (1 p)/p X. Equation 4 reduces to:
Xw|l2 2 + Aw trace X X X wt
min ly
(5)
D-1
Details see Appendix A.2
Solutions Equation
(LR with Rotation) and the mirror problem with
dropout (Srivastava et al.
2014) are:
X + trace XIX)I- - X
W8o
X
D-1
(6)
X + Adiag X xx]
XTy
Drop
Therefore, linear regression with RotationOut and Dropout are equivalent to ridge regression with different regularization terms. Set = 1 (Dropout rate = 0.5) for simplicity. LR with Dropout
doubles the diagonal elements of X X to make the problem numerical stable. LR with RotationOut more close to ridge regression:
x trace XXXX- X
D D-1
x trace trace x)) D
D-1
The condition number of Equation Jnd and the LR with RotationOut problem is up bounded by D 1. For the Dropout case, if some data dimensions have extremely small variances, both X X and diag(X' (X X) are ill-conditioned. LR with Dropout problem has unbounded condition number.
Next we consider an m-way classification model of logistic regression. The input is E RD and the weights are W [w1, W2. wm] E R, The probability that the input belongs the category is:
x Pk Σ Exxp(we (w;+)
exp( e
x cos cx)
(8)
> exp(1 wu [ cos 0.)
In Equation 0g denotes the angel between and w; Assume that the length of each weights w; are very close, the input æ belongs to the èo category if æ is most close to wy in angle.
Consider a hard sample case that 0 f are the two smallest weight-data angles. But 6, and 8y are very close: 8; z 0,, i.e., the data are close to the decision boundary. The model should classify the data correctly but could make mistakes if there some noise. Applying RotationOut, the angle between the data and the weights can be changed, and the new angles can be 0, > 0,. To classify the data correctly, there should be gap between 6, and 0y. In other words, the decision boundary changed from 0, < 02 to 0g < 0j- θ where θ a positive constant that depends on the regularization. Thus RotationOut can be regarded
margin-based hard sample mining.
Here we provide an intuitive understanding of how Dropout with low keep rates leads to lower performance. Randomly zeroing units, Dropout method also rotates the feature vector. A lower
keep rate results in bigger rotation angle: cos? 0 Σ. "f E. E hidden layer in neural networks, it is similar to logistic regression on the features. If one feature æ is most close to wx. it belongs to the Jtt A lower keep rate Dropout would rotate the feature with a bigger angle, and the Dropout output can be most close to another weight with higher probability, which may hurts the training.
(t)))
(Ep;)² = p. Consider the last Ep.
2.3 ROTATIONOUT IN NEURAL NETWORKS
Consider a neural network with L hidden layers. Let x', y¹, and W denote the vector of inputs, the vector of output before activation, and the weights for the layer 1. Let R. be generated from
4

---

Under review as a conference paper at ICLR 2020
Equation 2 and a be the activation function, for example Rectified Linear Unit (ReLU). The MLP feed-forward operation with RotationOut training time can be:
T = R(xx E[=')) - -[zz), y' w'=', x(t1=a(y)
We rotate the zero-centered features and then add the expectation back. The reasons will ex- plained later. Here we give an intuitive understanding. If features are not zero-centered, we do not know the exact regularization strength. Suppose all features clements are in one interval, say 1 < < 2. The angle between any two feature vectors is sharp angle. In this case rotation angle of πT 7/4 would be too big. is the same for Dropout. The regularization strength influenced by the mean value of features which may not know. At test time, the RotationOut operation is removed.
Consider 2D case for example, the input for 2D convolutional layers are three dimensional: number of channels C, width H and height W:
X (a Thw E R ,1 S H,15 W
(10)
We regard each The as feature vector with semantic information for each position (h, w). and apply rotation to each position. As As|Giiasi et al. (2018) argued, the convolutional feature maps are spatially correlated, information can still flow through convolutional layers if features are dropped out randomly. Similarly, we rotate feature vectors in different positions with random directions, random directions offset each other and result in rotation. So rotate all feature vectors with the same directions but different angles. The operation on convolutional feature maps can be:
P~ P, P11.-.. HHH Unif(0,
Vh, w Riw = MMP, 0nw),
(11)
Fhw Rhw (Itw Ex + Ea
The operation for general convolutional networks very similar. Also note that RotationOut can combined with DropBlock (Ghiasi er al.] 2018) easily: only rotating features in a continuous block. get extra performance gain, As mentioned in Section
Experiments show that the combination 3.1, the rotation directions defined by P is similar to the dropout mask in RNN drops. RotationOut can also be used in recurrent networks following Equation I1
2.4 NOISE ANALYSIS
In this section, we first study the general formula of adding noise. Using the formula, we show how introducing randomness/noise helps reduce co-adaptations and why RotationOut is more efficient than the vanilla dropout. Strietly speaking, the co-adaptations describe the dependence between neu- rons. The mutual information between two neurons may be the best metric to define co-adaptations. To compute mutual information, need the exact distributions of neurons, which generally unknown. So we consider the correlation coefficient to evaluate co-adaptations, which only need the first and second moment. Moreover, if we assume the distributions of neurons are Gaussian, correlation coefficient and mutual information are equivalent in co-adaptations evaluation.
Suppose R is the activations of one hidden layer. Let E[æx] c RD,, Var(a:) Σ. E DDD The ideal situation is that V = diag2.. i.e., the neurons are mutually independent. We define the co-adaptations the distance between 2 and Σ diag(2)..
Σ diag( (E)l1 diag Σ)]l
| diag E)
co(
(12)
trace(E)
Here trace(E)) is normalization term that defines the regularization strength. Let be the of x with arbitrary noise (e.g. Dropout or RotationOut). We assume that the noise should follow two
assumptions: 1) zero-center: E [[ [z] æ;; non-trivial: Var( [x] O (avoid that always equals
to x). Consider the law of total variance, we have:
Var [
E [Var[ā]<]]] Var [B[ E [Var[ā]=]]] + Var[zr]
(13)
Let Drop be the out of after Dropout with drop rate p. and TRoo be the of after RotationOut with Eg tan? (1 p)/p, we have Lemma 1 (proof Appendix A.3
A.33
Lemma 1. Var [x]
--P diag(xx>
Var[
wb-1 (x VI xx²)
(x
5

---

Under review as a conference paper at ICLR 2020
Note that E[æx") = E cc,, ,E(x" trace(E) + c we have:
Var[@Drop)
p diag(2 +c²))
(14)
trace E ++ cc.
VarlT
D-1
We can compute the co-adaptations of æ (Assume = 0):
Drop
diag (Drop) || || trace (E)]
co (Duup)
= co(x)
trace(
trace
(15)
p(D-1) ||Σ: - (2))l
1-
co (a))))
(p
D D-1 co(x))
trace(E) + 1-p Dirace Σ) ())
D-1
Under zero-center assumption, Dropout with keep rate p reduces co-adaptation by p times, and the equivalent RotationOut reduces co-adaptation by p カー1 times.
take a close look at the correlation coefficient to see what makes the difference. Let æ; be the 7th element of æ. Recall Equation 13 we have:
cov(2x, xr, + E ccov [][=]]]
cor (2, T5y]
(16)
((V [V.r(..]
[ ar [xy]] + aar [=]=
For Dropout-and other dropout-like methods, they add noise different neurons independently. so cov(ă, 22 = O. The only term to reduce correlation coefficients in Equation I6 is E ar [ē, [=]] Under out non-trivial noise assumption, Var(ā, (ar[z,laz always positive. Thus non-trivial noise can always reduce co-adaptations. For RotationOut, there is another term to reduce correlation coeffi- cients: E [cov(a,, [z)] = PT cov .by) and typically <
p(D-T) < 1. In addition to increasing the uncertainty of each neuron Dropout does, RotationOut can also reduce the corre- lation between two neurons. In other words, inhibition noise.
Here we explain why we need a zero-center assumption and rotate the zero-centered features in Section 2.3 Equation [4 and [6 show that the non-zero mean value can further reduce the co- adaptations. If we do not know the exact mean value. do not know the exact regularization strength. Suppose the neurons x ~ (O, 1) follow a normal distribution, and we apply Dropout on the ReLU activations y = ReLU(x:). With keep rate 0.9, Dropout reduces the co-adaptations by 0.86 times, while Dropout reduces the co-adaptations by 0.61 times with keep rate 0.7, which is non-linear mapping and influenced by the mean value. We rotate/drop the zero-centered features so that the regularization strength is independent with the mean value.
3 EXPERIMENTS
In this section, we evaluate the performance RotationOut for image classification, object detec- tion, and speech recognition. First, we conduct detailed ablation studies with CIFAR100 100 dataset. Next, we compare RotationOut with other regularization techniques using more data and higher resolution. We test on two tasks: image classification on ILSVRC dataset and object detection on COCO dataset.
3.1 ABLATION STUDY ON CIFAR100
The CIFARI00 dataset consists of 60,000 colour images of size 32 32 pixels and 100 classes. The official version of the dataset is split into training with 50,000 images and test set with 10,000 images. We conduct image classification experiments on the dataset.
Our focus is the regularization abilities, the experiment settings for different regularization techniques are the same. We follow the setting from He et al. (2016). The network inputs are 32 x 32 and normalized using per-channel mean and standard deviation. The data augmentation methods are as follows: first zero-pad the images with 4 pixels on each side to obtain a 40 40 pixel image, then randomly crop 32 x 32 pixel image, and finally mirror the images horizontally with

---

Under review as a conference paper at ICLR 2020
50% probability. For all of these experiments, with batches of 128 images using SGD, momentum 0.9, and weight decay of le-5. We start with a learning rate of 0.1, divide it by 10 at 32k and 48k iterations, and terminate training at 64k iterations. For each run, we record the best validation accuracy and the avergae validation accuracy of the last 10 epochs. Each experiment is repeated 5 times and we report the top (best and avergae) validation accuracy as "mean + standard deviation" of the runs.
the same optimizer: training for 64k iterations
We compare the regularization abilities of RotationOut and Dropout on two classical architectures: ResNetl 10 from He et al. (2016) and WideResNer28-10 from Zagoruyko & Komodakis (2016). ResNetl 10 is a deep but not so wide architecture using 18 3 BasicBlocks (Zagoruyko & Ko- modakis 2016) in three residual stages. The feature map sizes are {32, 16, 8} respectively and the numbers of filters are {16, 32, 64) respectively. WedeResNet28-10 is a wide but not so deep ar- chitecture using 4 x 3 BasicBlocks in three residual stages. The feature map sizes are (32, 16, 8) respectively and the numbers of filters are {160, 320, 640) respectively. For ResNetl 10, we only apply RotationOut or Dropout (with the same rate) to all convolutional layers in the third residual stages. FOr WideResNet28-I0 10, we apply RotationOut or Dropout (with the same keep rate) to all convolutional layers in the second and third residual stages since WideResNet28- 10 has much more parameters.
As mentioned calier, is controlled by E tan 0 = 1/p- 1. We compare RotationOut with the corresponding Dropout. We tried different distributions and found that the performance difference is very small. We report the results of Gaussian distributions here.
different distributions generate 8. and the regularization strength
Table Top accuracy of Dropout and corresponding RotationOut on CIFAR 100
(a) ResNetl 10: Standard Dropout
(b) ResNetl 10: tan 0 /V000 x²)
keep rate
Avg lop-1(%) 1(%)
Best top- 1(%)
Avg (op-1(%)
Best top- ((%)
71.93 + 0.16
72.12 + 0.18
71.98 + 0.16
72.12 0.18
0.9
73.13 + 0.30
73.32 + 0.28
0.333
74.23 + 0.25
74.41 + 0.16
0.8
73.33 + 0.27
73.59 + 0.28
0.500
74.14 = 0.11
74.35 + 0.13
0.7
72.42 1 0.23
72.71 0.14
0.655
73.131
73.45 0.11
0.6
71.79 1 0.20
72.10 1 0.21
0.816
71.83 1 0.35
72.17 0.33
(c) WideResNet28: Standard Dropout
(d) WideResNer28: tan 0 ~N((0,
keep rate
Avg top-1(%) I(%)
Best ((%)
o
Avg ((%)
Best ((%)
0
78.05 + 0.23
78.20 + 0.21
0
78.05 + 0..3
78.20 + 0.21
0.9
78.61 : 0.09
78.78 1.18
0.333
78.94 + 0.22
79.09 0.22
0.8
78.77 1 0.20
78.91 0.19
0.500
79.47 1 0.14
79.60 0.12
0.7
78.75
+ 0.15
78.87 0.13
0.655
79.69
1 0.11
78.80 - 0.15
0.6
0.07 78.55
78.75 1 0.18
0.816
79.76 = 0.32
79.93 1 0.33
Table 1 shows the results CIFAR100 dataset with two architectures. Table Ia and lb the results for ResNetl 10. Table Ic and id are the results for WideRessNe28-101 10. Results in the same row compare the regularization abilities of Dropout and the equivalent keep rate RotationOut. We can find dropping too many neurons is less effective and may hurt training. Since WideRessNe128-10 has much more parameters, the best performance is from heavier regularization.
3.2 EXPERIMENTS WITH MORE DATA AND HIGHER RESOLUTION
ImageNet Classification. The ILSVRC 2012 classification dataset contains 1.2 million training images and 50,000 validation images with 1,000 categories. We following the training and test schema as in (Szegedy et al. 2015 He et al. rate is decayed by the factor of 0.1 at 120, 190 and 230 epochs. We apply RotationOut with with normal distribution of tangent E tan 0 1/4 to convolutional layers in Res3 and Res4 as well as the last fully connected layer. As mentioned earlier. RotationOut is easily combined with DropBlock idea. We rotate features in continuous block size of 7 x 7 in Res3 and 3 x 3 in Res4.
2016) but train the model for 240 epochs. The learning
7

---

Under review as a conference paper at ICLR 2020
Table 2 shows the results of some state of the methods and results. Our results average over runs. Results of other methods are from Ghiasi et al. (2018), and also regularize on Res3 and Res4. Our result is significantly better than Dropout and SpatialDropout. By using the DropBlock idea, RotationOut can get competitive results compared with state of the art methods and get a 2.07% improvement compared with the baseline.
Table 2: Comparison with state of the art: Top accuacy of ResNet50 on ImageNet Validation
Model
top- (op-
top-5(%))
ResNet-50 (He et al. 2016)
76.51 + 0.07
93.20 + 0.05
ResNet-50 (pppppoutkkp)0)) (Srivastava et al. 2014)
76.80 0.04
93.41 = 0.04
ResNet-50 DropPath(kp=O..... Larson et al. 2016)
77.10 0.08
93.50 + 0.05
ResNet- 50 SpatialDropout(pp-D)) Tompson et al.
2015
77.41 = 0.04
93.74 = 0.02
ResNet-50 Cutout (DeVries & Taylor]201
76.52 - 0.07
93.21 = 0.04
ResNet-50 (ropBlocc((p==0.9) (Ghiasi et al. 2018)
78.13 + 0.05
94.02 + 0.02
ResNet-50 RotationOut
77.87 0.34
93.94 1 0.17
ResNet-50 RotationOut (Block)
78.58 1 0.48
94.27 + 0.24
COCO Object Detection. Our proposed method can also be used in other vision tasks, for ex- ample Object Detection on MS COCO (Lin et al.] 2014). this task, we use RetinaNet (Lin al. 2017) as the detection method and apply RotationOut to the ResNet backbone. We use the same hy- perparameters as in ImageNet classification. We follow the implementation details in (Ghiasi al.] 2018): resize images between scales [512, 768) and then crop the image to max dimension 640. The model are initialized with ImageNet pretraining and trained for 35 epochs with learning decay at 20 and 28 epochs. We set 0.25 and 1.5 for focal loss, weight decay of 0,0001, momentum of 0.9 and batch size of 64. The model is trained on COCO train2017 and evaluated on COCO val2017. We compare result with DropBlock (Ghiasi al. 2018 table 3 shows.
Table 3: Object detection COCO using RetinaNet and ResNet-50 FPN backbone
Model
Initialization
AP
AP50
AP75
RetinaNet
ImageNNet
36.5
55.0
39.1
RetinaNet, no DropBlock RetinaNet, Dropout, keep-prob 0.9
Random
36.8
54.6
39.4
Random
37.9
56.1
40.6
RetinaNet. keep-prob block
Random
38.4
56.4
41.2
RetinaNet, RotationOut
ImageNet
38.2
56.2
41.0
RetinaNet, RotationOut (Block)
ImageNet
38.7
56.6
41.4
Due to limited computing resources, finetune the model from Py PoTcch library's pretraining Im- ageNet classification models while DropBlock method trained the model from scratch. We think it fair compare DropBlock method since the initialization does help increase the results as showed in the first two rows. Our RotationOut can still have additional 0.3 AP based on the DropBlock result.
4 CONCLUSION
In this work, we introduce RotationOut an alternative for dropout for neural network. Rota- tionOut adds continuous noise to data/features and keep the semantics. We further establish an analysis of noise to show how co-adaptations more effective than dropout. Our experiments show that applying RotationOut helps training and increase the accuracy. Possible direction for further work is the theoretical anal- ysis of co-adaptations. As discussed earlier, the proposed correlation analysis is not optimal. cannot explain the difference between standard Dropout and Gaussian dropout. Also it can not
reduced in neural network and why dropout is neural network
8

---

Under review as a conference paper at ICLR 2020
plain some methods such Shake-shake regularization. Further work co-adaptation analysis can help better understand noise- based regularization methods.
REFERENCES
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, al. Deep speech 2: End-to- end speech recognition in english and mandarin. In International conferere on machine learning,
pp. 173-182, 2016.
Edward Anderson. Discontinuous plane rotations and the symmetric eigenvalue problem. 2000.
David Bau, Bolci Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 6541-6549, 2017.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: /00.. 045552, 2017.
Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. In Advances in neural information processing systems, pp- 1019-1027, 2016.
Xavier Gastaldi. Shake-shake regularization. arXiv preprint arXiv:1705.074855 2017.
Golnaz Ghiasi, Tsung-Yi Lin, and Quoe V Le. Dropblock: A regularization method for convo- lutional networks. In Advances in Neural Information Processing Systems, pp. 10727-10737, 2018.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Pro- ceedings of the 23rd international conference on Machine learning, pp. 369-376. ACM, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Improving neural networks by preventing oo-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
Ronghang Hu, Huazhe Xu, Mareus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4555-4564, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Alex Labach, Hojjat Salehinejad, and Shahrokh Valaee. Survey of dropout methods for deep neural networks. arXiv preprint arXiv: 13310, 2019.
Gustay Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural net- works without residuals. arXiv preprint arXiv: 2016.
Karel Lene and Andrea Vedaldi. Understanding image representations by measuring their equiv- ariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 991-999, 2015.
Xiang Li, Shuo Chen, Xiaolin Hu, and Jian Yang. Understanding the disharmony between dropout and batch normalization by variance shift. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2682-2690, 2019.

---

Under review as a conference paper at ICLR 2020
Zhe Li, Boqing Gong, and Tianbao Yang. Improved dropout for shallow and deep learning. In Advances in Neural Information Processing Systems, pp. 223-2231,,000
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr
Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017.
Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. In arXiv preprint 1.Xivv/809.00000 pp- 17, 2018.
Diego Marcos, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In Proceedings of the IEFE International Conference on Computer Vision, pp. 5048- 5057, 2017.
2017.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing Istm lan- guage models. arXiv preprint arXiv: 021008 2017.
Taesup Moon, Heeyoul Choi, Hoshik Lee, and Inchul Song. Rnndrop: A novel dropout for mnns in asr. In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 65-70. IEEE, 2015.
Douglas Paul and Janet M Baker. The design for the wall street journal-based esr corpus. In Proceedings of the workshop on Speech and Natural Language, pp. 357-362. Association for Computational Linguistics, 1992.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Advances in neural information processing systems, pp. 3856-3866, 2017.
Dynamic routing between capsules. In
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterizatioon to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp- 901-909, 2016.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2107-2116, 2017.
Patrice Y Simard, David Steinkraus, John C Platt, et al. Best practices for convolutional neural networks applied to visual document analysis. Icdar, volume 3, 2003.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. In The journal of machine learning research, pp. 1929-1958, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. Efficient object localization using convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 648-656. 2015.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference machine learning, pp. 1058-1066, 2013.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5028-5037, 2017.
10

---

Under review as a conference paper at ICLR 2020
Haibing Wu and Xiaodong Gu. Towards dropout training for convolutional neural networks. Neural Networks, 71:1-10, 2015.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146 2016.
Wide residual networks.
arXiv preprint
Wojeiech Zaremba, Ily Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv: 1409.2.999 2014.
A APPENDIX
A.1 RANDOM ROTATION MATRIX
One example of such matrix that rotates the (1,3) dimensions and (2, 4) dimensions can be:
cos 0
sin 0
0
0
cos 0
0
sin 0 0
(0, 1,,2
(17)
sin 0
cos
sin
0
cos
In Section 2 we mentioned the complexity of RotationOut O(D). is because we can avoid matrix multiplications to get R.x, For example, let the R be the operator generated by Equation 17 we have:
0
0
-1
0
0
Rx x + tan
1
0
x.
(18)
0
-1
The sparse matrix in Equation 18 similar to combine of permutation matrix, and do not need matrix multiplications to get the output. The output can be get by slicing and an elementwise multiplication: æ33, 4, 1, 2) 1,1,1,1,,-1
A.2
MARGINALIZING LINEAR REGRESSION
Recall that Ex = I, the marginalizing linear regression expression:
ER
(v: w )
i=1
R
(We wh
TuTTr m. u0 T0=(I R.) R=))
N
[y w T 221)
+ Σ (ys- w x: ER 10 - -Ro)are
+ +w ER (I - R, ax) (I-R))"
w
1m1
N
= yly XwIl2
w
Var [[I R.)a::w
=1ly
Xw
w
Vara [R
(19)
From Lemma one, have Varg RRæ,,]
1-p (x.x;II xTx;).
Write the second term of Equa-
ph tion 19 in the matrix form, get Equation

---

Under review as a conference paper at ICLR 2020
A.3 PROOF OF LEMMA 1
The Dropout form is trivial. We consider the RotationOut equation. Denote x² as the 7th term of æ.
The probability distribution of each element of TRot is:
ji i = x² + tan Ox:") ) = P( = x. tan Oxx1)
(20)
2(D - 1)
The joint distribution of each two elements of TR is:
vm # i, n + j. m n :a = + tan 9m. Rot = xx + tan fxx
= =P(TRot = x² + lan Ox." m Th x.
lan 0x."
P Rox x² tan Oz." m ToRo x tan 0x:")
= PP = x² tan Ox To x¹ + tan fx")
1
(21)
4(D - 1)(D -
Vi + j IP (T Rot - x² + tan Ox TRoa x² - tann Ox
P [ = x² tan fx T x tan Oxx)
2(D 1)
So we have:
Ep tan?
Egg tan 2
EvRo 0, 1. (((RRx))*
Σ (x')) (x²
D-1 20' 'x
(22)
A.4 DROPOUT BEFORE BATCH NORMALIZATION
Dropout changes the variance of a specific neuron when transferring the network from training to inference. However, BN requires a consistent statistical variance. The variance inconsistency (variance shift) in training and inference leads to unstable numerical behaviors and more erroneous predictions when applying Dropout-before BN.
We can easily understand this using Equation 13 If Dropout layer is applied right before BN layer. In training time, the BN layer records the diagonal element of Var[z] as the running variance and uses them in inference. However, the actul variance in inference should be the diagonal element of Var [æ) which is small than the recorded running variance (train variance). Li et al. (2019) argues:
Instead of using Dropout, more variance-stabe stable form Uout can be used mitigate the problem::;; = xz; (1 + r;) where ~ Unnf[-3,,3]
P2 Instead of applying Dropout-a (Figure [ applying Dropout-b can mitigate the problem.
P3 In Dropout-b, let r the ratio between train variance and test variance. Expanding the input dimension of weight layer D can mitigate the problem: D - Dc, +
Wx
Wx
WF
(a)
ReLU
Weight
Dropout
BN
ReLU
(b)
ReLU
Dropout
Weight
BN
ReLU
Figure 1: Two types of Dropout. The weight layer can be convolutional or fully connected layer.
We revisit these propositions and discuss how to mitigate the problem. For Proposition 1, Uout unlikely mitigate the problem. The Uout noise to different neurons independent, so the variance shift is the only term to reduce co-adaptations Equation 16 Though Uout variance- stable, it provides less regularization, which is equivalent to Dropout with higher keep rate.
Proposition 2 and 3 discuss the positions to insert Dropout. Let æ be the output from ReLU layer with E[ær] e, Var[ar] Σ and y be the input of BN layer. The weight layer in Dropout-a and b
12

---

Under review as a conference paper at ICLR 2020
the same with weight W E RuxD. During test time, the imputs to BN layers in Dropout-a and b are the same y = Wx x with variance Var[yl = WEWT. During training time, the inputs are different. In Dropout-a, the formulation is yo Wa where [
Wx. In Dropout-b, the formulation is Ybb = W where B[a [ = æ. So the training variance for the two types are different. Recall Lemma I, we have:
Var Øo] = Var[ W + Var[Wa]] WEWT + 1- P
diag ( +cC
P
(23)
1- W (E |ar [| + Var[ = WEWT + Wdiagg2 [ + cc")
Var yb
p
Let w be 7th row of of W and assume w; is uniformly distributed the unit ball. Since the length of w expands the training and testing variance with the same proportion, it does not affect the ratio between training and testing variance, and we can assume the length of w is fixed. The 2t element of actual testing variance is wEw". For Dropout-a, the 7th element of running variance (i.e., the training variance) is Var ww]: = wEw" + w + cc cc")w" "wo" For Dropout-b, the 7th element of running variance is Var Wi ww] = wEw" + - wddiagg(> ( + cc")w", Dropout-a and b have the same expected variance shift:
1 1-P p
Ew Varl w|:
Var(y//w]] Ew [Var gb /w]] Varly/ w]l]]
(trace (E) cccc (24)
Though the expected variance shift is the same, the variance of the shift is different. Let r(w) be the ratio between the training variance and the testing variance: r(w) Var y (il/::///rrly /arlyy/ We have the following observation:
Observation. I >> O which is the case that the activation function is ReLU. The ratio in Dropout-b
more centered: Varw [ur)] < Var (ra(()) (w)] ] = of () Sample n weights to make the weight layer W, the maximum ratio in Dropout-a is bigger than the maximum ratio in Dropout-b with high probability: max ro(ww)) < mmxx (w.).
ISkgn
15k.n
According to this observation, Proposition 2 and 3 are basically right but might not be precise. Dropout-b does help mitigate the problem but there might be other reasons. The expected variance shift is the same in Dropout-a and b: D - 00, + 1. Dropout-b has more stable variance shift among different dimensions. Dropout-a more likely to have very big training/testing variance ratio, leading to more serious unstable numerical behaviour.
Consider zero-centered Dropout-a in Equation 23 Var[ūu] = WEWT + 1-P "dag( WEWT). The ratio is fixed to be 1/p for any weights i.c. Varm [a(w]] behaviour since there is no extreme variance shift ratio, and we can modify BN layer's validation mode (reduce the running variance by 1/p times). Zero-centered Dropout-a can be solution to mitigate the variance shift problem.
p O. It leads to fewer unstable numerical
We verified this claim on the CIFARI00 dataset using ResNetl We apply Dropout between the convolutions of all residual blocks in the third residual stage (18 dropout layers are added). We test three types of Dropout with keep rate of 0.5: 1) Dropout-a-centeret Following (Li et al. 2019). the experiments are conducted by fol-
2) Dropout-b)
and 3) cropout-b-centered lowing three steps: 1) Calculate the running variance of all BN layers in training mode. It is the the training variance. 2) Calculate the running variance of all BN layers in testing mode. It is the the testing variance. Data augmentation and the dataloader are also kept to ensure that every possible detail for calculating neural variances remains exactly the same with training. 3) Obtain r max{ tain / /Var Var / /rin (Note that Var and Var are 64 dimentional vectors). For dropout-a-centert we reduce the running variance by 1/p times (We also tried this for the other two dropout, but the results are not better). The obtained ratio measures the variance shift between training and testing mode. A smaller ratio r is better. The results are averaged over 3 runs and shown in Figure
A.5 EXPERIMENT IN SPEECH RECOGNITION
We show that our RotationOut can also help train LSTMs. We conduct an Auto2Text experiment on the WSJ (Wall Street Journal) dataset (Paul & Baker 1992). The dataset is database with 80 hours
13

---

Under review as a conference paper at ICLR 2020
1.6
dropout-b-center
dropout-b
dropout-a-center
Variance ratio
1.4
1.3
1.2
2
8
10
12
14
16
BN layer index
Figure The variance shif ratio for differet dropout variants in the 18 residual blocks from ResNet1 10, the third residual stage. Lower values are better.
of transcribed speech. The inputs are variable length speech X E RTxx where T is the length and the feature dimension for one time step. The labels are character-based words. We use four- layer bidirectional LSTT network to design CTC (Connectioniss temporal classification) Graves) ctt al. (2006) model. The input dimension, hidden dimension and output dimension of the four-layer bidirectional LSTM network are 40, 512, 137 respectively. We use Adam optimizer with learning rate 1e-3, weight decay le-5 and batch size 32, and train the model for 80 epochs and reduce the learning rate by 5x epoch 40. We report the edit distance between prediction and ground truth on the "eval92" test set. Table shows the performance of different regularization methods.
Table 4: Auto2Text experiment on the WSJ
Mothod
Distance
No regularization
9.1
Standard Dropout(kp=0.9
8.6
Weight Drop(kp=0.8)
7.8
Variational Weight Drop(kp=0.8)
7.5
Locked Drop(kp=0.7)
7.3
Locked Drop(kp=O.8)+Wart) Variatioonl Weight Drop(kp=0.8)
6.7
RotationOut
6.8
RotationOut +Variational Weight Drop(kp=0.9)
6.4
A.6 RETHINKING SMALL BATCHSIZE BATCHNORMALLIZATION
BN also introduces noise to the neurons by using the batch mean and variance. The noise to different neurons/channels are independent, so the effect of BN's noise is similar to Dropout. believed that the noise causes BN performance to decrease with small batch size (Wu & He 2018 Luo et al. 2018). However, Dropout usually decrease the performance when the keep rate is very low. We study the effect of BN's noise and argue that BN is not linear operation. The nonlinearity increases when the batch size decreases, which is also one reason for the small batch size BN's performance drop.
widely
14

---

Under review as a conference paper at ICLR 2020
Let {a:)/=1 be one dimension of the data where D is the dataset size. During aani-batch training, one batch of B data (Tb, |k= is sampled and the BN operation can be formulate as:
B
Tbe
B
k=1
JB
B k=1
(2bb MB)²
(25)
The 14B
Tbe V +
Yb. 7 Tde + 3.
The batch normalization operation records a running mean and running variance o to be used in testing:
E[a] EB/BB
B
V Var [x] B E
(26)
Tb
E|r]
Tbe
/Varlo [2] +
We want to check whether the test mode formula can be good estimation of the training mode formula. Suppose we have a batch of data ( B Denote:
B
B
B
B
μBB B
Σ
Tke
OB
Σ
nB)
18-1 =
Tk. o B-1
118-1)2
B
B
B
k=1
k=1
k=2
k=
We have:
01
1B
B
31 148-1
(27)
dB
B
o
+ 1 (I)
WB-1)
Note that pa-1 and _ are independent from T1. So the expected output of any æ is:
E [ormalizze())] =
x - 1481
(28)
B
1 (x 1B-1)
Let the function in 28 be f(x B). Easy to know that it is not a linear function (but BN assumes it should be y x!!) Suppose the data follows normal distribution, we can plot f(x B) by Monte Carlo sampling: Figure 3 shows that BN is not linear operation. The nonlinearity increases when the batch size decreases. is another important reason for the small batch size BN's performance drop. To validate our conlusion, we propose cross normalization:
B
P1
Tba
B
B
0
[hi. p4)
(29)
A
Th Tb. 14i
Vo
06 To. + B.
For each data in the batch, cross normalization comute its normalization mean and variance. In this case, the expectation of operation on any data is stricity linear in expection, but uses less data.
the sample and variance except itself to
We do not intend to propose a better alternative for BN but want to check whether the nonlinearity an important issue for BN when batch size is small. If cross normalization outperform BN
15

---

Under review as a conference paper at ICLR 2020
3
8
16
32
y=x
fox)
-1
22
-3
-2
0
2
X
Figure 3: Left: values of f(B) against different values of B. Right: the error rates of ImageNet classification against different values of batch size.
in batch size case, then the nonlinearity is definately important issue. On CIFAR100 dataset, following the settings in our ablation study, ResNet50 with cross normalization has lower test loss when the batch size is 8 and 16. But the test accuracy is almost the same in terms of 95% confidence interval since cross normalization leads to higher variance.
16


Reviews:
[{"review_id": "rkeEbaapFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a new type of DropOut layer to regularize neural network training. The basic idea is to rotate the features using a random rotation matrix. The authors use Givens rotations to do this in linear time. The authors analyze their DropOut formulation for linear models and contrast it to Bernoulli DropOut. The further provide a probabilistic analysis of the effect on the co-adaption of their DropOut approach. Experimental results are shown on standard classification datasets, object detection, and speech recognition.\n\nPros: \n+ Interesting analysis that supports the idea. Both the analysis of linear models and the co-adaptation analysis help in building intuition about the method.\n+ Extensive experiments: The presented experiments cover different applications and are large-scale (e.g. ImageNet). The experiments indicate that there is a small, but consistent, improvement over the baselines.\n\nCons:\n- Parts of the paper are unclear:\n\n1) Equation 2: This equation mentions that the operator is normalized by the cosine of the rotation angle. While the text briefly mentions this and acknowledges that the resulting operator is not a rotation matrix anymore, this is not further mentioned in the text. It is not clear why this normalization was chosen and what would happen if one chooses a proper rotation. When looking at the actual operator that includes this normalization it seems that the term \"rotation\" is rather misleading. The proposed approach effectively applies a signed and uniformly scaled permutation matrix to the features and adds the results back onto the features.\n\n2) The analysis of the co-adaption is not completely clear to me. Can you elaborate more on the sentence \"We use L_1 distance but not L_2...\"? Why is it significant here that the variance is a second moment? Equation (15) states that RotationOut reduces co-adaption by a factor of p - (1-p)/(D-1). If we take the extreme case of only two neurons we have that this factor is negative when p < 0.5. More generally, the factor is negative whenever D < 1/p. What does negative co-adaption mean? What does this tell us about this analysis?\n\n3) Section 3.2 is completely disconnected from the rest of the paper. It examines the interdependence between dropout and batch norm, but I don't see how this specifically connects to the contribution at hand.\n\n- The improvement with respect to other DropOut variants is small. RotationOut incurs a higher computational overhead (even if the \"rotation\" can be done in linear time) in practice according to the authors.\n\nSummary: The authors introduce their idea together with interesting analysis, however, there are some problems with clarity. The practical gains that are afforded by the approach are small, and DropOut, in general, is less and less used (at least in image processing architectures) thus the impact is questionable.\n\n=== Post rebuttal update ===\nI'd like to thank the authors for addressing some of my comment. Similar to the other reviewer, I still believe that the improvement even beyond CIFAR is too small. I thus maintain my rating.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gfyUK2KS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary: This paper proposes a novel regularization method for training neural networks. The high-level motivation is to add noise (and thus regularize) neurons in an inter-dependent fashion, unlike existing methods such as DropOut where each neuron is treated independently. The authors evaluate their approach on image and speech classification, and object recognition benchmarks. They also discuss how different regularization schemes might help reduce neuron co-adaptation.\n\nHigh-level comments: I find the proposed method interesting, and I think the paper is well-written. In particular, I like the exposition of their approach, as well as the common framing of different regularization schemes (in Section 3). \n\nMy chief concerns with the paper are:\n\n1. The improvement over state-of-the-art, across tasks, seem marginal and largely is within the reported statistical error margins. From a practical standpoint, I think that alternatives to popular techniques like DropOut should either offer significant benefits empirically, or be computationally more feasible/simpler to implement. RotationOut does not seem to offer major benefits along either of these axes.\n\n2. Even though I find the theoretical comparison of co-adaptation reduction in different regularization methods interesting, its significance is unclear. Based on my understanding, there is no concrete evidence that co-adaptation hampers training or generalization. Moreover, the authors also do not demonstrate that RotationOut actually decreases co-adaptation in a meaningful way---they propose a specific definition for co-adaptation but do not look at this quantity experimentally. There are a number of other proxies for mutual information in the community (such as CCA from arXiv:1806.05759) that the authors should also evaluate. \n\nGiven that improvements over prior art are small, I think the paper would have more significance if the authors demonstrate that reducing co-adaptation is important---from the perspective of training/generalization/interpretability and that RotationOut significantly helps with this compared to state-of-the-art approaches.\n\nOther comments:\n\ni. The authors mention that they fix rotate all feature vectors with the same direction but different angles: is the performance actually worse if the directions are also randomized?\n\nii. In general, I feel the authors should substantiate the various claims they make in Section 3 with experimental evidence. For instance, the authors mention that zero-centered Dropout-a might be more compatible with BatchNorm, but provide no experimental evidence to support this claim. \n\niii. How did the authors pick the hyperparameters and learning rate schedules for their experiments? These numbers do not seem standard (for instance the learning rate schedule)---did the authors grid over hyperparameters for each of the approaches? I think this is extremely important in papers which are proposing a new approach to establish a rigorous comparison to baselines.\n\niv. For several of the experiments, the authors report that they use RotationOut only for a few residual blocks. Do the remaining residual blocks have DropOut, or is no regularization applied to these? In the comparisons between different regularization schemes, are the regularizers applied to the same layers? For instance, in the comparison in Table 2, are all the methods applied only to Res3 and Res4?\n\nv. For the results on COCO, is the same regularization method used even for ImageNet pre-training? For instance, does the “RetinaNet, RotationOut + ImageNet” row apply RotationOut even during the ImageNet pre-training? It would be nice to see the results of using “RetinaNet, keep prob = 0.9, block size = 5 + ImageNet” given that “RetinaNet, keep prob = 0.9, block size = 5” performs the best among prior approaches.\n\nOverall, I find the idea of the paper interesting, I am not convinced of its significance. In particular, I think that the improvement over state-of-the-art is marginal and it is not clear that this method actually reduces co-adaptation between neurons in practice, or more broadly, that co-adaptation is a relevant quantity in deep network training. Thus, I am recommending rejection.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BkgZO0DjtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new regularization method to mitigate the overfitting issue of deep neural networks. Specifically, unlike the regular dropout which randomly zeros out some neurons in each layer, the proposed RotationOut rotates the features with a random rotation matrix. The authors argue that the rotational operation can reduces the co-adaptation of features. The experiments have shown some improvement over existing methods. \n\nI have some concerns about the proposed method as follows:\n\n1. In this paper, the authors use the correlation to measure the co-adaptation of features and prove RotationOut can reduce the correlation. However, it is well known that the rotation matrix is an orthogonal matrix, which cannot decorrelate two random variables. So, how can the rotational operation reduce the correlation? It looks a paradox. \n\n2.  Compared with the regular dropout, RotationOut involves more computational overhead. It's better to show the running time of these methods. \n\n3. In Eq.(15), D is the dimension of features. For a large D, such as 224x224 in the imagenet, 1/D is very small. So, the improvement of co_{Rot} over co_{Drop} is very small. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All of the reviewers agree the paper has an interesting idea (using rotations of the representation as regularization). However, the reviewers also agree the empirical gains are too insignificant. While the paper shows results on CIFAR, the reviewers mentioned a few other ways to improve performance, such as more complex and unconstrained datasets. These additional experiments would make the effectiveness of proposed approach more convincing.
Decision: Reject