Multi-Sample Dropout for Accelerated Training and Better Generalization
Authors: Hiroshi Inoue
Venue:  (2020)
Keywords: dropout
regularization
convolutional neural networks

Abstract: Dropout is a simple but efficient regularization technique for achieving better generalization of deep neural networks (DNNs); hence it is widely used in tasks based on DNNs. During training, dropout randomly discards a portion of the neurons to avoid overfitting. This paper presents an enhanced dropout technique, which we call multi-sample dropout, for both accelerating training and improving generalization over the original dropout. The original dropout creates a randomly selected subset (called a dropout sample) from the input in each training iteration while the multi-sample dropout creates multiple dropout samples. The loss is calculated for each sample, and then the sample losses are averaged to obtain the final loss. This technique can be easily implemented without implementing a new operator by duplicating a part of the network after the dropout layer while sharing the weights among the duplicated fully connected layers. Experimental results showed that multi-sample dropout significantly accelerates training by reducing the number of iterations until convergence for image classification tasks using the ImageNet, CIFAR-10, CIFAR-100, and SVHN datasets. Multi-sample dropout does not significantly increase computation cost per iteration for deep convolutional networks because most of the computation time is consumed in the convolution layers before the dropout layer, which are not duplicated. Experiments also showed that networks trained using multi-sample dropout achieved lower error rates and losses for both the training set and validation set.


Under review as a conference paper at ICLR 2020
MULTI-SAMPLE DROPOUT FOR ACCELERATED TRAINING AND BETTER GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Dropout is simple but efficient regularization technique for achieving better gen- eralization of deep neural networks (DNNs); hence it is widely used in tasks based on DNNs. During training, dropout randomly discards portion of the neurons avoid overfitting. This paper presents an enhanced dropout technique, which we call multi-sample dropout, for both accelerating training and improving generaliza- tion over the original dropout. The original dropout creates randomly selected subset (called a dropout sample) from the input in each training iteration while the multi-sample dropout creates multiple dropout samples. The loss is calculated for each sample, and then the sample losses are averaged to obtain the final loss. This technique can be easily implemented without implementing new operator by duplicating a part of the network after the dropout layer while sharing the weights among the duplicated fully connected layers. Experimental results showed that multi-sample dropout significantly accelerates training by reducing the number of iterations until convergence for image classification tasks using the ILSVRC 2012 (ImageNet). CIFAR-10. 10, CIFAR- 100, and SVHN datasets. Multi-sample dropout does not significantly increase computation cost per iteration for deep convolutional networks because most of the computation time is consumed in the convolution layers before the dropout layer, which are not duplicated. Experiments also showed that networks trained using multi-sample dropout achieved lower error rates and losses for both the training set and validation set.
INTRODUCTION
Dropout (Hinton et al. (2012) is one of the key regularization techniques for improving the general- ization of deep neural networks (DNNs). Because of its simplicity and efficiency, the original dropout and various similar techniques are widely used to train neural networks for various tasks. The use of dropout prevents the trained network from overfitting to the training data by randomly discarding (i.e., "dropping") 50% of the neurons each training iteration. As a result, the neurons cannot depend on each other, and the trained network achieves better generalization. During inference, neurons are not discarded, all information is preserved; instead, each outgoing value multiplied by 0.5 make the average value consistent with the training time. The network used for inference can be viewed as an ensemble of many sub-networks randomly created during training. The success of dropout inspired the development of many techniques using various ways for selecting information to discard. For example, DropConnect (Wan et al. (2013) discards a portion of the connections between neurons randomly selected during training instead of randomly discarding neurons.
This paper reports multi-sampy sample dropout, dropout technique extended different way especially for deep convolutional neural networks (CNNs). The original dropout creates a randomly selected subset (a dropout sample) from the input during training. The proposed multi-sample dropout creates multiple dropout samples. The loss is calculated for each sample, and then the sample losses are averaged to obtain the final loss used for back propagation. By calculating losses for A dropout samples and ensembling them, network parameters are updated to achieve smaller loss with any of these samples. This is similar to performing M training repetitions for each input image in the same minibatch. Therefore, it significantly reduces the number of iterations needed for training. We observed that multi-sample dropout also improved accuracy of the trained network; it can reduce the gradient noises caused by dropout. We do not discard neurons in the inference, as with the original

---

Under review as a conference paper at ICLR 2020
original dropout
our multi-sample dropout (with two dropout samples)
convolution relu
convolution relu
pooling
pooling
dropout
dropout
d dffforont masks
dropout
fully connected
fully connected
shared weights
full connected
softmax loss func
softmax loss
softmax loss func
average
loss
Figure 1: Overview of original dropout and our multi multi-sample dropout.
dropout. Experiments demonstrated that it achieves smaller losses and errors for both the training set and validation set on image classification tasks.
similar approach to accelerate training. In CNNs, dropout is typically applied to layers near the end of the network. VGG16 (Simonyan & Zisserman )2014), ) for example, uses dropout for 2 fully connected layers following 13 convolution layers. Because the execution time for the fully connected layers is much shorter than that for the convolution layers, duplicating the fully connected layers for each of multiple dropout samples does not significantly increase the total execution time per iteration. Experiments using the ImageNet, CIFAR-10, CIFAR-100, and Street View House Numbers (SVHN) datasets showed that, with an increasing number of dropout samples created at each iteration, the improvements obtained (reduced number of iterations needed for training and higher accuracy) became more significant at the expense of a longer execution time per iteration and greater memory consumption. Consideration of the reduced number of iterations along with the increased time per iteration revealed that the total training time was the shortest with moderate number of dropout samples, such as 8.
As far the authors know, other dropout technique
Multi-sample dropout can be easily implemented on deep learning frameworks without adding a new operator by duplicating part of the network after the dropout layer while sharing the weights among the fully connected layers duplicated for each dropout sample.
The main contribution of this paper is multi-sample dropout, new regularization technique for accelerating the training of deep neural networks compared to the original dropout. Evaluation of multi-sample dropout image classification tasks demonstrated that it increases accuracy for both the training and validation sets as well as accelerating the training.
2
MULTI-SAMPLE DROPOUT
2.1 OVERVIEW
This section describes the multi-sample dropout technique. The basic idea is quite simple: create multiple dropout samples instead of only one. Figure depicts an easy way to implement multi- sample dropout (with two dropout samples) using an existing deep learning framework with only common operators. The dropout layer and several layers after the dropout are duplicated for each dropout sample; in the figure, the "dropout," " "fully connected," and "softmax + loss func" layers are duplicated. Different masks are used for each dropout sample in the dropout layer so that a different subset of neurons is used for each dropout sample. In contrast, the parameters (i.e., connection weights) are shared between the duplicated fully connected layers. The loss is computed for each dropout sample using the same loss function, e.g., cross entropy, and the final loss value obtained by averaging the loss values for all dropout samples. This final loss value is used the objective function for optimization during training. We select the class label as the prediction based on the average of outputs from the last fully connected layer. Although a configuration with two dropout
2

---

Under review as a conference paper at ICLR 2020
samples is shown in Figure 1, multi-sample dropout can be configured to any number of dropout samples. The original dropout can be seen as a special case of multi-san multi-sample dropout where the number of samples is set one.
During inference, Neurons are not discarded as done in the original dropout. The loss can be calculated using only dropout sample because the dropout samples become identical at the inference time if we do not drop any neurons. Hence, We always use only one dropout sample at inference regardless of the training method.
When dropout is applied to a layer near the end of the network, the additional execution time due to the duplicated operations is not significant: this characteristic makes multi-sample dropout more suitable for deep CNNs. We can apply multi-sample dropout for shallow networks, such as the multilayer perceptron. As we show an example Appendix, multi-sample dropout reduces the number of iterations for training. but the costs of the increased execution time per iteration may surpass the benefits.
If the network includes multiple dropout layers, we can apply multi-sample dropout at any of these dropout layers. Multi sampling at an earlier dropout layer may increase diversity among dropout samples and increase the benefits in trade for the higher additional costs due to more duplicated layers. For example, WideResNet (Zagoruyko employs residual blocks which include dropout layers. However, we do not need to duplicate the entire network for applying multi-sample dropout to WideResNet; we can apply the multi sampling at the last residual block for example.
Komodakis (20T6) repeatedly executes dropout since
2.2 WHY MULTI-SAMPLE DROPOUT ACCELERATES TRAINING
Intuitively, the effect of nulti-sample dropout with M dropout samples is similar to that of enlarging the size of a minibatch M times by duplicating each sample in the minibatch M times. For example, if minibatch consists of two data samples (A, B), training a network by using multi-sample dropout with two dropout samples closely corresponds to training a network by using the original dropout and minibatch of (A, A, B, B). Using a larger minibatch size with duplicated samples may not make sense because it increases the computation time M times. In contrast, multi-sample dropout can enjoy similar gains without huge increase in computation cost for deep CNNs because it duplicates only the operations after dropout. Because of the non-linearity of the activation functions, the original dropout with duplicated samples and multi-sample dropout do not give exactly the same results. However, similar acceleration was observed in the training in terms of the number of iterations, as shown by the experimental results.
2.3
OTHER SOURCES OF DIVERSITY AMONG SAMPLES
The key to faster training with multi-sample dropout is the diversity among dropout samples; if there is no diversity, the multi-sampling technique gives no gain and simply wastes computation resources. Although dropout is one of the best sources of diversity, the multi-sampling technique can be used with other sources of diversity. For example, variants of dropout, such as DropConnect, can be enhanced by using the multi-sampling technique.
To demonstrate that benefits can be obtained from other sources of diversity, two additional diversity- creation techniques employed in this paper: 1) horizontal flipping and zero padding pooling layer. As shown in Appendix, improvements due to these additional sources of diversity are visible but much less significant than the improvements from dropout. Such results show that dropout is an ideal source of diversity to use in our multi-samplil technique. In this paper, these additional sources of diversity are used only the training phase, but not in the inference.
EXPERIMENTAL RESULTS
3.1 IMPLEMENTATION
This section describes the effects of using the multi-sample dropout for various image classification tasks using the ImageNet, CIFAR-10, 10, CIFAR- 100, and SVHN datasets. For the ImageNet dataset, as well as for the full dataset with 1,000 classes, reduced dataset with only the first 100 classes
3

---

Under review as a conference paper at ICLR 2020
eight the number of dropout samples, which generally gives good tradcoff between benefits and additional cost. For the CIFAR- 10. CIFAR-100, and SVHN datasets, an 8- 8-layer network with six convolutional layers and batch normalization (loffe & Szegedy 2 followed by two fully connected layers with dropout was used (see appendix for detail). This network executes dropout twice with dropout ratios of 40% and 30%, which are tuned for the original dropout but are used here for all cases unless otherwise specified. The same network architecture except for the number of neurons in the output layer was used for the CIFAR- 10, SVHN (10 output neurons), and CIFAR- 1000 (100 output neurons) datasets. The network was trained using the Adam optimizer (Kingma & Ba (2014) with a batch size of 100. These tasks were run on a NVIDIA K20m GPU with CUDA 9.0 using the Chainer v4.0 (Tokui et al. (2015) as the framework. For the ImageNet datasets, VGG16 (Simonyan & Zisserman (2014)) was used as the network architecture, and the network was trained using stochastic gradient descent with momentum as the optimization method with a batch size of 100 samples. The initial learning rate of 0.01 was exponentially decayed by multiplying it by 0.95 at each epoch. Weight decay regularization was used with a decay rate of 5 10 by following the original paper. In the VGG16 architecture, dropout was applied for the first two fully connected layers with 50% as dropout ratio. A NVIDIA V100 GPU with CUDA 10.0 was used for the training with ImageNet datasets. For all datasets, data augmentation was used by extracting patch from random position of the input image and by performing random horizontal flipping during training (Krizhevsky et al. (2012)). For the validation set, the patch from the center position was extracted and fed into the classifier without horizontal flipping.
was tested (ImageNet-100) 100).. For most of the experiments,
3.2 IMPROVEMENTS BY MULTI-SAMPLE DROPOUT
Figure 2 plots the trends in training losses, training errors and validation errors against training time for three configurations: trained with the original dropout, with multi-sample dropout, and without dropout. For multi-sample dropout, the losses for eight dropout samples were averaged. How the number of dropout samples affects performance is discussed in the next section. The figure shows that multi-sample dropout achieved faster convergence than the original dropout for all datasets (including SVHN, which is not shown here due to space limitation), i.e. both losses and errors became smaller with the same training time for both training sets and validation sets, As is common in regularization techniques, dropout achieves better generalization (i.e., lower validation error rates) compared with the "no dropout" case at the expense of slower training. Multi salple dropout alleviates this slowdown while still achieving better generalization.
Table summarizes the final training losses, training error rates, and validation error rates. For the CIFAR- 10, CIFAR-100, and SVHN datasets, the results for 200 epochs were averaged after training the network for 1,800 epochs. For ImageNet- 100, we trained VGG16 for 16 million images. For ImageNet, we trained VGG16 for 96 million images. After training, the networks trained with multi-sample dropout were observed to have reduced losses and error rates for all datasets compared with those of the original dropout. The original dropout increased the training losses and error rates by avoiding overfitting for all datasets compared with the no dropout case. Multi-samp sample dropout achieved lower training losses than no dropout for some datasets while avoiding overfitting.
3.3 EFFECTS OF PARAMETERS ON PERFORMANCE
Number of dropout samples: Figure 3(a) and 3(b) compare the training losses and validation errors for different numbers of dropout samples for CIFAR-100 against the number of training epochs. Using larger number of dropout samples made convergence faster. With eight dropout samples, for example, the validation accuracy reached 95% of the final accuracy after the 25th epoch while it was after the 42nd epoch with the original dropout. A clear relationship evident between the number of dropout samples and the speedup of convergence in terms of the number of iterations (epochs) with up to 64 dropout samples. For the validation error shown in Figure 3(b), the benefit of using more than eight dropout samples
not significant.
The accelerated training in terms of the number of iterations (epochs) due to using more dropout samples came at cost: increased execution time per iteration. The execution time per iteration relative to that of original dropout is shown in Table for different numbers of dropout samples. The VGG16 network architecture was used for the ImageNet dataset and smaller 8-layer CNN was used for
4

---

Under review as a conference paper at ICLR 2020
training loss
training error
validation error
62%
origina dropoual samplee
eamppe semples!
45%
-oggiaal dropout II comple lo)
-muut comple dispout 18 samples)
-ma li-oe drspou (8 samplers
orginll dropout (1 campk)
whhoot
42%
wthour dropout
witheud sloppot
35%
CIFAR-10
caakigg am
85%
00%
15%
10%
0.1
naiing (ind yreuuss)
16
traning Lin (Tenzo))
15
5.
Fakiing geireas))
10
15
20
80%
-onggeal droposs seemple)
-nub--aappe drupous (8 sumplees
90%
sample) saaple samples)
70%
--tu mosill-saameee drepous sumpheas wiltout dropcct
original dropout cample)
without crosout
80%
60%
50%
CIFAR-100
600
40%
40%
20%
200%
20%
10%
0.1
0%
200
30
113
13
20
tistting Sime (Tibuute)
rtarting
Inairing Site (1t112001)
100%
100%
90%
-mull sunple dropoul samples) Whi drosout
80%
diopout samples) weehce dhopouu
original dropout sampo ()
80%
80%
70%
ImageNet -100
makiig 1004
80%
60%
50%
60%
40%
30%
30%
20%
-uuli-o momut-sammle dropout samples)
diopout
10%
10%
0%
0%
woiring time
2000
200
2011
300
10
100%
-crginal
-mmllle sumples?
90%
-crg nal cropout IT somple)
-mubiserple dropout sumplee))
90%
orgia cropout samp withuut diopoct
crspout
50%
dropout
80%
70%
ImageNet
50%
50%
40%
4t%
30%
300%
20%
0%
Yaiting Sne (7ouin)
10
15
25
30
|ilning Sin (hoLi)))
30
Figure 2: Trends in training losses, training errors and validation errors against training time for original dropout and multi-sample dropout. Multi-sample dropout achieved faster convergence than the original dropout. Note that training losses and errors for multi-sample dropout include the effects of the inherent ensemble mechanism, but no ensemble was used while evaluating validation sets.
Table 1: Training losses, training error rates, and validation error rates with original dropout and multi-sample dropout (after trained for 96 million images for ImageNet, 16 million images for ImageNet-100 100, and 1,800 epochs for others). Multi-sample dropout reduced losses and error rates compared with original dropout.
training loss
training error rate
validation error rate
dataset
original dropout
multi sample dropout
no dropout
original dropout
multi sample dropout
no dropout
original dropout
multi- sample dropout
no dropout
CI
0.0170
0.0045
0.0074
0.39%
0.01%
0.11%
8.05%
7.46%
9.13%
CIFAR-100
0.1530
0.0537
0.0325
4.49%
0.04%
0.40%
30.0%
29.3%
36.6%
SVHN
0.0188
0.0046
0.0062
0.45%
0.01%
0.09%
4.46%
4.21%
4.82%
ImageNet-100
0.0472
0.0344
0.0346
1.41%
0.89%
0.92%
27.1%
25.9%
31.4%
ImageNet
0.4922
0.3808
0.1777
14.4%
8.61%
3.88%
30.5%
30.1%
30.8%
5

---

Under review as a conference paper at ICLR 2020
(4] training loss against number of epochs
orggr desport NAYSMi
[c] training loss against wall-clock time
exmple orppddt lang
wit legin what
degic sampled
-Tul en depact
validation error
against number of epochs
120
20%
20
100
120
2010
Figure 3: Training losses and validation errors during training for different numbers of dropout samples. Using more dropout samples makes convergence faster in terms of number of iterations at the cost of increased execution time per iteration.
Table 2: Execution time per iteration relative to that of original dropout for different numbers of dropout samples. Increasing the number of dropout samples lengthened the computation time per iteration. We failed to execute VGGI6 with 16 dropout samples due to the out-of-memory error.
original dropout
multi-sample dropout
dataset
network
sample
samples
4 samples
8 samples
16 samples
32 samples
64 samples
CIFAR-10
1.00
1.08
1.18
2.16
CIFAR-100
8-layer CNN
1.00
1.04
1.09
1.19
1.44
2.22
3.55
SVHN
1.00
1.04
1.08
1.19
1.47
2.07
3.62
ImageNet
VGG16
1.00
1.05
1.09
the other datasets, as mentioned above. Because larger network tends to spend more time in deep convolutional layers than in the fully connected layers, which are duplicated our multi-sample dropout technique, the overhead in execution time compared with that of the original dropout is more significant for the smaller network than it is with the VGGI6 network architecture, Multi-sample dropout with eight dropout samples increased the execution time per iteration by about 9% for the VGG16 architecture and by about 19% for the small network. For even smaller networks, the increases in the execution time per iteration became significant and hence using multi-sample dropout may not make sense as discussed in Appendix using multilayer perceptron as an example.
Consideration of the increased execution time per iteration along with the reduced number of iterations revealed that multi-sample dropout achieves the largest speed up in training time when a moderate number of dropout samples, such 8, is used, shown in Figure 3(c). Using an excessive number of dropout samples may actually slow down the training.
The final values of the training losses, training error rates, and validation error rates are shown in Figure 4. The average losses and error rates between the 1,800th and 2,000th epoch were used. Multi-sample dropout achieved lower losses and error rates as the number of dropout samples was increased. The gains were not significant when the number was increased above eight.
From these observations, it was determined that eight reasonable value for the number of dropout samples, and it is used in other experiments.
Dropout ratio: Another important parameter is the dropout ratio. In the 8-layer CNN used for CIFAR datasets, 40% and 30% were used as the ratios in the two dropout layers. These values were tuned for the original dropout. Here is shown how multi-sample dropout works for the CIFAR-10 10 dataset with various dropout ratios: (10%, 10% ), (25%. 25% ). (40%, 30%} (default), (50%, 50%), (75%, 75%), ), (90%, 90%].
Figure 5(a) shows the final validation error rates. Regardless of the dropout ratio setting, multi-sample dropout outperformed the original dropout. Figure 5(b) compares the trend in convergence of training losses for dropout ratios of 25% and 75%. For both configurations, the acceleration in training over the original dropout was similar to that for the default case (Figure 2). For the original dropout with 75% dropout ratio, there were spikes in the training loss history, which were not observed with

---

Under review as a conference paper at ICLR 2020
0.50
0.42
10%
3390
CIFAR-10 10 training loss
0:47
CIFAR-10 training error
1%
CIFAR-10 validation error
115
0.379
2230
0.0%
4%
376
3.86
0.19
04
3.18
CIFAR-100 training loss
4.5%
CIFAR- 100 training error
s0n.
CIFAR-10 validation error
4.0%
30h-
317
3.5%
2.%%
3.64
200
15%
40%
332
15
04
00%
15
32
original dropout
stuli-samppl drapcut
original
mn.li-eample
original dropout
rtull-samppee drapout
number of compout samples
number of dropous samples
number of crrpost samples
Figure 4: Losses and error rates after the training for different numbers of dropout samples, Multi- sample dropout achieved lower losses and error rates with more dropout samples.
Validation emrrr rates
Training loss with and without multi-sample dropout with 25% and 75% dropout ratio
12%
oorignal digpout
Dreasat sumpln deipest somplos)
-rrginal srapound sample -utt sample cropon namplen
10%
.r7.BB-osmple deppott camphel
8%
0.01
25% dropout rBtiO 122
75% dropout ratio
0%
25%
35%
75%
20
60
50
340
300
01220.11
Figure 5: (a) Validation error rates and (b) trend in training losses with original and multi-sample dropout for various drop ratios. Here, 35% dropout ratio means two dropout layers use 40% and 30% respectively. salti- dropout works regardless of the dropout ratio.
smaller dropout ratios. Multi-sample dropout did not cause such spikes and exhibited a smoother history and stable training by reducing gradient noise.
These results show that multi-sample dropout does not depend on a specific dropout ratio to achieve improvements and that can be used with wide range of dropout ratio settings.
3.4 WHY MULTI-SAMPLE DROPOUT EFFICIENT
As discussed in Section 2.2, the effect of multi-sample dropout with M dropout samples is similar that of enlarging the size of minibatch M times by duplicating each sample in the minibatch M times. This is the primary reason for the accelerated training with multi-sample dropout. This is illustrated Figure the training losses with multi-sample dropout match well with those of the original dropout using duplicated data in minibatch. We also observed that the original dropout with duplicated data achieved comparable validation error rates to multi-sample sample dropout. If the same sample is included in minibatch multiple times, the results from the multiple samples are ensembled when the parameters are updated, even there no explicit ensembling the network. Duplicating sample in input and ensembling them the parameter updates seems to have quite similar effect on training to that multi-sample dropout, which duplicates a sample at the dropout and ensembles them at the end of the forward pass. However, duplicating data M times make the execution time per iteration (and hence the total training time) M times longer than without duplication. Malpi dropout achieves similar speedup much smaller computation cost.
7

---

Under review as a conference paper at ICLR 2020
-ongenal enapoea (1 semple) multt ce-m-nmmlee samples]
-uti sample diopout samples)
erppout with duplicated minbtatch -mul-asmple dropout sempies) orignnl cropout duplicated minibatch
onggiall with 2x dup icated minbatth
-mult -sample dropout (4 sumples)
original dropout icated miniatch
Batic
0.1
CIFAR 10 training loGS
CIFAR-108 10BR-1000aaaaae training loss
100
150
200
200
300
350
400
50
100
150
200
850
300
350
400
Figure 6: Comparison of original dropout with data duplication in minibatch and multi-sample dropout. X-axis shows number of epochs. For fair comparison, we do not use additional diversity due to horizontal flipping and zero padding for multi-sample dropout in this experiment.
RELATED WORK
The multi-sample dropout regularization technique presented in this paper can achieve better gen- eralization and faster training than the original dropout. Dropout is one of the most widely used regularization techniques, but a wide variety of other regularization techniques for better general- ization have been reported. They include, for example, weight decay (Krogh & Hertz (1991) ), data augmentation (Cubuk et al. (2018), Chawla al. (2002), Zhang et al. (2017), Inoue (2018), label smoothing (Szegedy et al. (2016)), and batch normalization (loffe & Szegedy (2015)). Although batch normalization is aimed at accelerating training, it also improves generalization. Many of these techniques are network independent while others, such as Shake-Shake (Gastaldi (2017)) and Drop-Path (Larsson et al. (017)). ) are specialized for a specific network architecture.
The success of dropout led to the development of many variations that extend the basic idea of dropout (e.g. Ghiasi et al. (2018), |Hang et al. (2016). Tompson et al. (2014). [Derries & Taylor (2017)). The techniques reported use variety of ways randomly drop information in the network. For example, DropConnect (Wan et al. (2013) discards randomly selected connections between neurons. DropBlock (Ghiasi et al. (2018)) randomly discards
in convolution layers while dropout is typically used in fully connected layers after the convolution layers. Stochastic Depth (Huang et al. (2016) randomly skip layers in very deep network. However, none of these techniques the approach used our multi-sample dropout. Also many of them can be used with multi sampling technique to make the divergence among dropout samples.
Multi-sample dropout calculates the final prediction and loss by averaging the results from multiple loss functions. Several network architectures have multiple exits with loss functions. For example, GoogLeNet (Szegedy et al.] (2015) has two early exits in addition to the main exit, and the final prediction made using a weighted average the outputs from these three loss functions. Unlike multi-sample dropout, GoogLeNet creates the two additional exits at carlier positions in the network. Multi-sample dropout creates multiple uniform exits, each with loss function, by duplicating a part of the network.
5 CONCLUSION
In this paper, we described multi-sample dropout, a regularization technique for accelerating training and improving generalization. The key is creating multiple dropout samples the dropout layer while the original dropout creates only one sample. Multi-sample dropout can be easily implemented using existing deep learning frameworks by duplicating a part of the network after the dropout layer. Experimental results using image classification tasks demonstrated that multi-sample dropout reduces training time and improves accuracy. Because of its simplicity, the basic idea of the multi- malliisammping technique can be used in wide range of neural network applications and tasks.
REFERENCES
Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. Smote: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16(1):321-357,
8

---

Under review as a conference paper at ICLR 2020
2002.
Ekin D. Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoe V. Le. Autoaugment:
Learning augmentation policies from data. arXiv:/805.09501, 2018. htttts:/://aaxx
brg/abs/1805. 09501
Terrance DeVries and Graham W. Taylor. Improved regularizain of convolutional neural networks with cutout. arXiv:1708.04552. 2017. htt.atttss//xxx..crr////////////
Xavier Gastaldi. Shake-shake regularization. arXiv:1705.07485, 2017. URL https://arxiv
org/abs 07485
Golnaz Ghiasi, Ys Lin, and Quoc Le. Dropblock: regularization method for convolutional
networks. In Anmual Conference Neural Information Processing Systems (NIPS), pp. 10727- 10737, 2018.
Geoffrey Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Improving neural networks by preventing co-adaptation feature detectors. arXiv:1207.05800 2012.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. arXiv:/603.09382, 2016.
Hiroshi Inoue. Data augmentation by pairing samples for images classification. arXiv:1801.02922, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv: 2015.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980. 2014.
Alex Krizhevsky, llya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. Annual Conference on Neural Information Processing Systems (NIPS), pp. 1106-1114, 14, 2012.
Anders Krogh and John A. Hertz. simple weight decay can improve generalization. Annual Conference on Neural Information Processing Systems (NIPS), pp. 950-957, 1991.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In International Conference on Learning Representation (ICLR), 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv:1409. 1556. 2014. URL BBR/aappse//aaxttorr////////// 1556
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Conference Computer Vision and Pattern Recognition (CVPR), 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Conference Computer Vision and Puttern Recognition (CVPR), 2016.
Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. Chainer: next-generation open source framework for deep learning. In Workshop on Machine Learning Systems (LearningSys), 2015.
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christopher Bregler. Efficient object localization using convolutional networks. 1:xi::4111.421 2014. URL httpss///xxxxi
sxg/abs/. 1411.1111 4280
Li Wan, Matthew Zeiler, Sixin Zhang, Yann LeCun, and Rob Fergus. Regularization of neural networks using dropconnect. In International Conference on Machine Learning (ICML), pp. I1-1058-1I1-166, 2013.

---

Under review as a conference paper at ICLR 2020
Sergey Zagonuyko and Nikos Komodakis. Wide residual networks. arXiv: 2016. URL
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lapez-Paz mixup: empircaal risk minimization. arXiv:1710.09412, 2017. Ott.tto.///axx.....///////701010
10

---

Under review as a conference paper at ICLR 2020
2.
CIFAR-10 training loss against number of epochs
CIFAR-10 training
2.0
against wall-clock time
1.8
1.8
18
1.2
088
08
0.6
-orginal dropoLt samp
-originl diopout (1 sompic)
64
-rrts--ommll deopoue samplesi
-mi miooril dropece (4 samples)
10
20
25
30
90
100
150
200
Figure 7: Comparison of training loss using the multilayer perceptron with and without multi-sample dropout for CIFAR-10.
A APPENDIX
A.1 APPLYING MULTI-SAMPLE DROPOUT FOR SHALLOW NETWORKS
As discussed in Section 2. multi-sample dropout is mainly targeting deep convolutional neural networks in which most of the computation time is consumed in the convolution layers before the dropout. Here, we show the effect of multi-same -smple dropout on shallow networks using a multilayer perceptron as an example. We use a network consists of four fully connected layers each has 2,000 neurons. We apply dropout for each fully connected layer. For multi-sam salppig,, we created four dropout samples at the last fully connected layer; i.e. only one layer is duplicated. Figure 7 shows the training loss for CIFAR- 10 dataset with and without multi-sample dropout. Multi-sample dropout yields smaller training loss compared to the original dropout after training the same number of epochs. However, due to the increase in the computation time per iteration, multi-sample dropout actually degraded the training speed in terms of the training time; the execution time per iteration increased by more than 50% even we created only four dropout samples. Hence, to make multi-sample dropout effective, it is important that dropout is typilly applied to layers near the end of the network to limit the number of operations duplicated for multiple dropout samples.
A.2 IMPROVEMENTS FROM OTHER SOURCES OF DIVERSITY AMONG DROPOUT SAMPLES
As discussed in Section 2.3, zero padding at pooling layer and horizontal flipping can be used as sources of diversity among dropout samples in addition to the different masks in dropout.
In the current implementation, as detailed in the next section using pseudocode, the horizontal flipping is applied immediately before the first fully connected layer; half of the dropout samples are horizontal flipped deterministically. When pooling an image with a size that is not a multiple of the window size, e.g., when applying 2x2 pooling to a 7x7 image, the zero padding can be added on the left or right and at the top or bottom. Here, zero padding is added on the right (and bottom) for half of the dropout samples and on the left (and bottom) for the other half. The location of the zero padding is controlled by using horizontal flipping: i.e., "flip, pool with zero padding on right, and then flip" is equivalent "pool with zero padding on left."
Figure& 8 shows how these additional sources contribute to training acceleration. Although their contributions to the training speed are not negligible, they are much smaller than those of dropout. Nevertheless, these results show that the multi-sampling technique can work with not only dropout but also with other sources of divergence among samples while the dropout is an ideal source of divergence among multiple samples.
A.3 DETAIL OF THE USED NETWORK ARCHITECTURE
The network design of the 8-layer CNN used for CIFAR and SVHN datasets are shown. We also present an overview of the implementation using Chainer for more detail.
11

---

Under review as a conference paper at ICLR 2020
original dropout cample)
onginall dispout (1 sample)
-wih honzonta flpping and padding samples)
wwth honzontal fippiin and padding (2 samples)
0.0
without horizontal floping and padding samples)
witoout horizental fippiin and podiing samples)
-wit honzontal Aipping and paokiig samples)
-wth honzontal flippin and padeing (8 eamplee)
0.7
without horizontal llpping and padding (8 samples)
withou horizontal lipping and paddiing samples)
0.6
0.5
0.4
CIFAR-10 10 training loss
CIFAR-100 training loss
spocto
spocts
Figure 8: Comparison of training loss with and without horizontal flipping to create additional diversity among dropout samples in multi-sample dropout. X-axis shows number of epochs.
tensor size
Batch Normalization convolution 3x3 Rell)
28x28x3 28x28x64
Batch Cermalization
Convolution 3x3 RellJ
28x28x96
Max Pooling 2x2 Batch Normalization
14x1 4x
Convolution 3x3 ReLJ
14x114996
Batch Nermalization Convolution 3x3 ReLJ
14x14x12
Max Fooling 2x2 Batch Normalization Convolution 3x3 ReLJ Batch Normalization Convolution 3x3 ReIJ for 1 0 to bumSamples/2-1:
7x7x128
7x7x128
7x7x192
it (i & 1) != 0: HorisontalFlip
Max Pool ing 2x2
4x4x192
Batch Normalization
for j 0 to 1:
Dropout 408 dropout ratio
if j == 1: HorizontalFlip
Fully Connected ReLU Dropout 308 dropout ratic Fully Connected softmax CrossEntropy Average all losses
512
10 or 100
Figure 9: Design of classification network (with eight weight layers) with multi-sample dropout
used for CIFAR and SVHN. Refer Section 2.3 for details of horizontal flipping used to create more diversity among dropout samples in addition to dropout.
12

---

Under review as a conference paper at ICLR 2020
clase che
init
suger (tt.tt/smmmlllooopssttee tnit)
11 I. Ratehtlora pettttoaallaatttaa on13)
C.CovolucionnD 66, pad-1),
15 Canov utfoonn//oore, 95, 3, padell,
18
19
pad-1l,
112
Conoollation22 (None, 128, 3, pad-1l,
115 aatchhormalii22:2nn1222
convolation200oont 128,
pad-l),
BetchMormaliaaannn(222
118
C.oovolttioon00ceenn (None, 192, 3, pad-l),
122
L.ninerr niabelst,
def _ne11 (salf,
with
sels.treia):
h1 ccb)
batch nomellzatioon
82
self.12 (b11
conviuttionll
85 selr.15 convolutional
86 (h2)
F.nax_peoling_2006
(27)
bazch normelization
(188
convolutional
h10 F.zelu (1:9)
solf.111
nommalization convolutional
colt-112
h13 (.12)
814 E_nax_rooiing_2d 2)
batch nommelization
self.116 convolutional
817 (b16)
115 swif..18 (h177
bauch normalixation
seif.119 convolutional
(.19)
12: Sinn_poollim_2ilae 21
h22
self.122 (h21) batch normalization
P.dropoui (h22, sacio 0.40;
h24a
se1f,124 (h23a) fully connacted
825a
P.relu (h24a]
self.
fully connected
10
sampic
F.drapout (h22, ratio 0.43)
5245
swif.124 (1.124/F..1... (F.11ip axis=3)) fully connected afeer herizon:al fllp P.relo (h24b) 45))
5255
5265
F.dropout (i25b,
swif.127 (hi2611
Jully connected
drugout sample
b21e t.max_pooling_24F r11p(123, axia-3), pooling after horisontal
5220 felf..12 batch normalize:io
5236
P.dropout (122s, satie 0.46)
drupout sample
3d F.dropput (h22e, ratio 0.40)
594d swif. (...f.24/...... (F. (h223, axis=5)] fully connected after horisens flip
felf = softsttt ent (127,, t) ... / 8.0 b2/b hile /
seltvy gain
naking prediction
self.y fo (h27a) 7.aoftmex (1276) 1...t /
else:
felr. loss Prroftnex cross (h27a, h274
return sel Flvar loas
Figure 10: Example of an implementation of CNN with multi-sample dropout shown in Figure on Chainer using eight dropout samples
13


Reviews:
[{"review_id": "rJxuXvkhtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose an ensemble of dropout: it applies multiple copies of a neural net with different dropout configurations (i.e., dropout masks) to the same mini-batch, and the training loss is computed as the sum of losses incurred on the multiple copies. They claim that such ensembling can improve training performance over the original dropout without increasing too much computation. Experiments on several datasets show that the proposed method can achieve slightly better validation accuracies than the original dropout.\n\nMain comments and questions:\n\n1) The contribution of this paper is very limited: it simply applies the ensemble method to dropout (ensemble method can be applied to any existing models). Moreover, this contribution might not be very necessary: dropout is itself an ensemble method, so an ensemble of dropout does not make much sense.\n\n2) The paper keeps claiming that the computation will not dramatically increase because only the computations after dropout neede to be re-computed if applying multiple dropout masks to the same data(batch). This is only true for the case when dropout is used in the last few layers. For a rich class of modern neural nets such as WideResNet and many Language models, this is not true since each block contains dropout and the model is a stack of many blocks. Hence, the advantage claimed here can be only applied to very limited cases. In addition, only the layers after dropout can get the benefits of dropout training since only they get different inputs for different dropout masks but need to produce similar final outputs.\n\nIn Sec 2.2, the authors said their method \"can enjoy similar gains without a huge increase in computation cost because it duplicates only the operations after dropout\". So the computational advantage does not hold if we apply dropout in earlier/shallow layers. For example, if you apply dropout in the first layer, the method needs to execute all the operations after the first layer for each dropout sample (dropout changes everything after it), hence it increases the computation time nearly M times (for M dropout-samples) and loses the computational advantage. This might be the reason why the experiments only consider DNNs with dropout in the last fully-connected layers. \n\n3) The experiments did not evaluate the method on any modern neural nets and only tried small neural nets with dropout only applied to the last one or two fully-connected layers. It is not clear how it performs when used to train the modern models, which might prefer dropout above the last few layers.\n\n4) The dropout with 1-sample in experiments is not a standard baseline in any previous works. The two standard ways to do inference on dropout trained model are 1) one-time inference without any dropout, or 2) ensemble the inference results of multiple dropout samples. One dropout sample suffers from a higher variance. Hence, the comparison is unfair.\n\n5) It is not professional to report only training loss and validation error for some experiments, but report training loss/error and validation error for some others. In fact, it does not make sense to compare training loss/error of different dropout methods, since dropout always reduces training loss but improve generalization.\n\n6) Except for the vanilla dropout, no other baseline is compared. The paper simply ignores many successful variants of dropout proposed in the past several years.\n\n----------------\n\nUpdate:\n\nThe rebuttal does not answer my questions. So I will keep my rating unchanged.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Sygq8TiaKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new variation of Dropout -- Multi-Sample Dropout. The method is said to 1) accelerate training, and 2) decrease validation error. To achieve that, the authors propose to average a loss function over several dropout samples per object during training. This leads to faster convergence in terms of iterations, though at the cost of multiple forward-passes. In order to decrease the computational time of one iteration, the authors propose to evaluate the part of a network before the first dropout layer only ones and duplicate the remaining part with different dropout masks and shared weights. This decreases the run time of the naive approach and leads to faster convergence in terms of time in comparison to the original dropout. The authors also show that the model trained with the proposed method achieves lower validation error.\n\nThe only confusing part of the method for me is the prediction. In the second paragraph in section 2.1, the authors state that during inference neurons are not discarded, and only one dropout sample is used for the prediction. This is confusing because using a dropout sample implies the dropping of neurons. Could you please elaborate on this? Moreover, in the first paragraph of section 2.1, the authors state the prediction is based on the average of outputs from the last fully-connected layer, which is different from the inference procedure. Do you use different prediction methods to evaluate the performance of the train and test sets?\n\nThere are several concerns regarding empirical evaluation and baselines:\n(1) The improvements in validation error shown in Tab.1 seem to be insignificant, and confidence intervals are needed for justification.\n(2) To the best of knowledge, there is at least one more paper that proposes a technique for accelerating dropout training --- Fast Dropout [http://proceedings.mlr.press/v28/wang13a.html]. The method seems to be a direct competitor and should be considered as a baseline and be included in the relevant work section with further discussion.\n\nOverall, the proposed approach is heuristic, and the novelty is very limited. Along with empirical evaluation issues, I would suggest improving the work and rejecting the current version.\n\nAdditional comments:\n1. I think it would be beneficial to discuss other works that similarly interpret dropout as an ensembling technique: Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning [https://arxiv.org/abs/1506.02142] and Variational Dropout and the Local Reparameterization Trick [https://arxiv.org/abs/1506.02557].", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Bkee3UlGcB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "PAPER SUMMARY: The paper proposes a new and efficient implementation of dropout in which multiple dropout samples are obtained from a single input during training. The authors claim that this enhanced dropout technique (i) accelerates training and (ii) improves generalization by achieving lower error rates than standard dropout in training and validation sets. Experiments in image classification tasks for 4 different datasets (CIFAR-10, CIFAR-100, IMAGENET and SVHN) are presented and the effect of the number of dropout samples and dropout ratio are investigated.\n\nREVIEW SUMMARY: Despite the clear computational advantages of the proposed implementation, this paper should be rejected because  (1) the authors claim multi-sample dropout to be a “new regularization technique”, but the conceptual differences between standard dropout and their proposed method are not clearly stated, (2) one of the main claims regarding “better generalization” is not well supported by their experiments, and (3) the experiments do not consider the stochasticity of the methods they are evaluating.\n\nDETAILED COMMENTS: The multiple-sample dropout proposed by the authors leverages a parallel structure to provide significant acceleration in training, which is supported by experimental results. However, their claims regarding the introduction of a “new regularization technique” and “better generalization” are not convincingly proven in the paper. As mentioned in the paper, the proposed technique of multiple-sample dropout is analogous to the original implementation of dropout, but with larger batch size and duplicated samples. Therefore, the advantage of the proposed method is simply computational efficiency, but not a new “regularization technique”. In addition, while the paper claims the method provides better generalization, the results shown in Figure 6, in which the training losses for their multi-sample dropout and original dropout with duplicated samples are not significantly different, suggest that their validation error might not be significantly different either. The experiments shown in Figure 2, from which they conclude that their method generalizes better, are not conclusive since the validation error is evaluated at a stage in which the original dropout might have not converged, given the slower training. Therefore, the observed differences in validation error might come from the differences in training speed, and not from a better generalization ability of the model. Moreover, in the experiments the authors present single realizations of each training scenario, disregarding the stochastic nature of dropout-based techniques. In particular, the validation error curves observed in Figure 3 (where they analyze the effect of the number of dropout samples) seem very noisy, and the differences observed might simply come from the stochasticity of the training process. The authors should have executed multiple training runs with the same set of hyperparameters, and reported the statistics of their performance (same for results presented in Figure 2 and Table 1).  Overall, the advantages regarding training acceleration and computational efficiency are clear and well supported by the experiments. However, the novelty of the proposed technique and their claim regarding better generalization is not well supported theoretically or experimentally.\n\nFor the experiments, the authors should consider:\n1)\tReport validation errors after convergence of both methods\n2)\tRun the training scenarios multiple times and report performance statistics instead of single run performance\n3)\tReport validation curves for experiments in section 3.4 (Figure 6).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a multi-sample variant of dropout, claiming that it accelerates training and improves generalization. CIFAR10/100, ImageNet and SVHN results are presented, along with a few ablations.

Reviewers were in agreement that the novelty of the contribution appears to be very limited, the evidence for the claims is not strong, and that the applicability of the method for achieving efficiency gains is limited to architectures that only apply dropout very late in processing, precluding applicability to models that employ dropout throughout. Importantly, comparisons to Fast Dropout (Wang 2013) seem highly relevant and are missing.

While the reviewers acknowledged some of the criticisms, virtually no arguments were offered to rebut them and no updates were made to address them. I therefore recommend rejection.
Decision: Reject