Revisiting Structured Dropout
Authors: Yiren Zhao
Oluwatomisin Dada
Xitong Gao
Robert D. Mullins
Venue: Submitted to ICLR 2023 (2023)

Abstract: Large neural networks are often overparameterised and prone to overfitting, Dropout is a widely used regularization technique to combat overfitting and improve model generalization. However, unstructured Dropout is not always effective for specific network architectures and this has led to the formation of multiple structured Dropout approaches to improve model performance and, sometimes, reduce the computational resources required for inferencing. In this work we revisit structured Dropout comparing different Dropout approaches on natural language processing and computer vision tasks for multiple state-of-the-art networks. Additionally, we devise an approach to structured Dropout we call \textbf{\emph{ProbDropBlock}} which drops contiguous blocks from feature maps with a probability given by the normalized feature salience values. We find that with a simple scheduling strategy the proposed approach to structured Dropout consistently improved model performance compared to baselines and other Dropout approaches on a diverse range of tasks and models. In particular, we show \textbf{\emph{ProbDropBlock}} improves RoBERTa finetuning on MNLI by $0.22\%$, and training of ResNet50 on ImageNet by $0.28\%$. 

Under review as a conference paper at ICLR 2023
REVISITING STRUCTURED DROPOUT
Anonymous authors
Paper under double-blind review
ABSTRACT
Large neural networks are often overparameterised and prone to overfitting. Dropout is widely used regularization technique combat overfitting and im- prove model generalization. However, unstructured Dropout not always effec- tive for specific network architectures and this has led to the formation of multiple structured Dropout approaches to improve model performance and, sometimes, re- duce the computational resources required for inference. In this work, we revisit structured Dropout comparing different Dropout approaches to natural language processing and computer vision tasks for multiple state-of-the-ard networks. Ad- ditionally, we devise an approach to structured Dropout we call ProbDropBlock which drops contiguous blocks from feature maps with probability given by the normalized feature salience values. We find that with simple scheduling strategy the proposed approach to structured Dropout consistently improved model perfor- mance compared to baselines and other Dropout approaches on diverse range of tasks and models. In particular, we show ProhDropBlock improves RoBERTa finetuning MNLI by 0.22%, and training of ResNet50 on ImageNet by 0. 0.28%.
INTRODUCTION
In our modern society, Deep Neural Networks have become increasingly ubiquitous, having achieved significant success in many tasks including visual recognition and natural language pro- cessing Heaton (2020); Jumper et al. (202I); Schrittwieser al. (2020). These networks now play a larger role in our lives and our devices, however, despite their successes they still have notable weaknesses. Deep Neural Networks are often found to be highly overparameterized, and as a result, require excessive memory and significant computational resources. Additionally, due to overparam- cterization, these networks are prone to overfit their training data.
There are several approaches to mitigate overfitting including reducing model size or complexity, early stopping Caruana et al. (2000), data augmentation (De & Taylor 2017) and regularisation (Loshchllov & Hutter 2017). In this paper, focus on Dropout which is widely used form of regularisation proposed by Srivastava et al. (2014b). Standard Unstructured Dropout involves randomly deactivating subset of neurons in the network for each training iteration and training this subnetwork, at inference time the full model could then be treated as an approximation of an ensemble of these subnetworks.
Unstructured Dropout was efficient and effective and this led to it being widely adopted, however, when applied Convolutional Neural Networks (CNNs), unstructured Dropout struggled to achieve notable improvements He et al. (016); Huang et al. (2017) and this led to the development of sev- eral structured Dropout approaches Ghiasi al. (2018) Dai Dai et al. (2019) Cai et al. (2019) including DropBlock and DropChannel. DropBlock considers the spatial correlations between nearby entries feature map of a CNN and attempts to stop that information flow by deactivating larger contigu- ous areas/blocks, while DropChannel considers the correlation of information within particular channel and performs Dropout at the channel level. However, since the development of these struc- tured approaches, there have been further strides in network architecture design, with rising spread and interest in Transformer- based models.
Given the success achieved by block-wise structured Dropout on CNNs, it is only natural to ask the question, do these approaches apply to Transformer-bee based models? Structured Dropout approaches for transformers seem foeus reducing the model size and inference time, these works place

---

Under review as a conference paper at ICLR 2023
Original image and its RGB channels
(b) Dropout
(c) BatchDropBlock
(d) DropBlock
(e) Adaptive DropBlock
Figure 1: An illustration of applying different Dropouts to an image.
more emphasis pruning reducing computational resources Xin al. (2020) Fan et al. (2019)
than combating overfitting which is the focus of this paper.
In this paper, revisit the idea of structured Dropout for current state-of toe-the-rd art models lan- guage and vision tasks. Additionally, we devised our own form of adaptive structured Dropout ProbDropBlock and compare preexisting approaches to structured and unstructured Dropout.
In Figure I we illustrate the effects of select structured and unstructured Dropout approaches on an image of cat. As be seen in Figure Taj the original image consists of three channels (RGB) which are aggregated to form the image. Different approaches to Dropout may treat channels dif- ferently. In Figure Tb illustrate the effect of unstructured Dropout this image, the many small black squares represent deactivatee/dropped weights at pixel level and we also see different pixels have been deactivated in each channel. In Figure Ic we see fewer but larger black squares, and that the locations of dropped pixels are consistent between channels, however, this is not the case in Figure Te and Figure [d this work, we say that BatchDropBBloc is channel consistent i.e. chan- nels do not deactivate blocks independently rather the deactivated blocks consistent between channels.
In Figure lb Figure Ic Figure ld for single channel there is uniform probability of any pixel or block (depending on the approach) to be dropped and so deactivated pixels may not contain any of the key information required to identify this image pixel/block belonging to the cat is the same as that one belonging to the background). This is not the case for Figure Tel in our adaptive DropBlock approach the probability of block being dropped is dependent on the value of the center pixel in the block. It can be seen that this approach is not channel consistent and deactivated pixels are concentrated on the cat.
cat (i.e. the probability of deactivating
Figure is illustrative to give one an intuitive understanding of these techniques, as in practice these techniques are applied feature maps which the output activations of preceding layer of the network. The contributions of this paper include:
The testing of preexisting unstructured and structured Dropout approaches on current state- of-the-art models including transformer- based models natural language inference and vision tasks. We reveal that structured Dropouts are generally better than unstructured ones both vision and language tasks.
2

---

Under review as a conference paper at ICLR 2023
The proposal of new approach to structured dropout named ProbDropBlock, which im- proved model performance on both vision and language tasks. ProbDropBlock is adap- tive and the blocks dropped are dependent on the relative per-pixel values. It improves RoBERTa finetuning on MNLI by 0.22% and ResNet50 on ImageNet by 0.28%
Further observation of the benefits of simple linear scheduling observed Ghiasi et al. (2018) for both structured and unstructured Dropout on range of vision and language models.
RELATED WORK
In this section, we briefly review related works in the areas of structured and unstructured Dropouts used as both regularization technique to improve model performance and as an approach pruning to reduce the model size and computational requirements. We briefly detail unstructured Dropout and the various structured Dropouts devised for other network architectures.
2.1 UNSTRUCTURED DROPOUT
To help address the problem of overfitting neural networks, Srivastava et al. (2014a) proposed Dropout as simple way of limiting the co-adaptation of the activation of units in the network. By randomly deactivating units during training they sample from an exponential number of different thinned networks and at test time an ensemble of these thinned networks is approximated by single full network with smaller weights. Dropout led to improvements in the performance of neural networks on various tasks and has become widely adopted. This form of Dropout in this work we refer to as unstructured Dropout as any combination of units in the network may randomly droppedd/deactivated. In the following subsection, we consider forms of structured Dropout which extend this idea further for other network architectures and tasks.
2.2 DROPBLOCK AND OTHER STRUCTURED DROPOUTS
Ghiasi et al. 2018 proposed DropBlock as a way to perform Structured Dropout for Convolutional Neural Nets (CNNs). They suggest that unstructured Dropout is less effective for convolutional layers than fully connected layers because activation units in convolutional layers are spatially cor- related so information can still flow through convolutional networks despite Dropout and so they devised DropBlock which drops units in a contiguous area of the feature map collectively. This ap- proach inspired by Devries & Taylor (2017)'s Cutout, data augmentation method where parts of the input examples are zeroed out. DrocBlock generalized Cutout by applying Cutout at every feature map in convolutional networks. Ghiasi al. (2018) also found that scheduling scheme of linearly increasing DropBlock's ' zero-out ratio performed better than a fixed ratio.
Dai et al. (2019) extended DropBlock to Batch DropBlock. Their network consists of two branches; a global branch and a feature-dropping branch. In their feature dropping branch they randomly zero the same contiguous area from each feature map in batch involved in computing loss function. They suggest zeroing out the same block in each batch allows the network to learn a more comprehensive and spatially distributed feature representation.
Larsson et al. (2016) proposed DropPath in their work on FractalNets. Just as Dropout prevents the co-adaptation of activations, DropPath prevents the co-adaptation of parallel paths in networks such as FractalNets by randomly dropping operands of the join layers. DropPath provides at least one such path while sampling subnetwork with many other paths disabled. DropPath during training alternates between global sampling strategy which returns only single path and a local sam- pling strategy in which a join drops each input with fixed probability, but with guarantee, at least one survives. This encourages the development of individual columns performant stand-alone subnetworks.
Cai et al. (2019) proposed DropConv2d they suggest the failure of standard dropout is due to conflict between the stochasticity of unstructured dropout and the following Batch Normalization (BN) step. They propose placing dropout operations right before the convolutional operation instead of BN or replacing BN with Group Normalization (GN) to reduce this conflict. Additionally, they devised DropConv2d which draws inspiration from DropPath and DropChannel, they treat each
3

---

Under review as a conference paper at ICLR 2023
channel connection path between input and output channels and perform dropout replicates of each of these paths.
DropBlock, Bloocck DropPath and DropConv2d are forms of structured Dropout designed with specific architecture in mind. However, as seen by Cai et al. 2019) DropConv2d an approach to structured Dropout designed for given network can still be useful to novel network architec- ture. Aside from being used to improve generalization, structured Dropout has also been used as an approach to pruning and reducing computational resource requirements inference time.
Fan et al. (019)) proposed LayerDrop as a means of regularization for transformers during train- ing and efficient pruning inference time reducing the large amount of computation these models require. This approach is form of structured Dropout where instead of deactivating weights inde- pendently throughout the network, weights that collectively form a single structure in the network are deactivated. Attention heads in a transformer are typically computed in parallel, as a result in the paper the structure they focused on deactivating where fully connected layers. Using this ap- proach they found that they were able to select sub-networks of any depth from one large network that without finetuning achieved similar performance.
Other forms of run-time structured pruning include (Xin et al. 2020, Gao et al. 2018 Wu et al., 2018), Xinn et al. (2020) proposed DeeBERT which accelerates inferencing in BERT models by allowing samples to exit earlier without passing through the entire model under certain conditions as they believe that, for BERT, features provided by the intermediate transformer layers may suffice to classify some input samples. Gao et al. (2018) preserve the full network structure of CNNs and accelerates convolution by dynamically skipping unimportant input and output channels determined by a saliency criterion exploiting the fact that the importance of features computed by convolutional layers is highly input-dependent. [Wu et al. (0188) propose BlockDrop where an RL agent learns which blocks a ResNet select dynamically for given novel input image.
In this work, we are interested in dropping structured patterns in the spatial dimension and would like understand how this style of structured Dropouts can affect current mainstream models such as Transformers and Vision Transformers. We mainly compare our proposed approach with Drop- Block and revised version of BatchDropBlock, since these two methods are dropping at the same granularity as us.
3 METHOD
In this section we detail the approach to structured dropout we employed in this paper. Our approach a form of adaptive DropBlock we call ProbDropBlock, it is inspired by DropBlock domly removes larger blocks from each feature map, but rather than assigning a uniform probability to each element of being the center point of the block it assigns higher probability to elements with higher activation values.
also ran-
As such the block removed depends on the model's learned representation of the feature map and we believe this encourages the model to learn more balanced and diverse feature map. We also employ a simple linear schedule where we linearly increase the base probability a of dropping a block. Ghiasi et al. (2018) made an observation in their paper that this approach can significantly improve performance and is more robust.
In Algorithm 1 we detail our adaptive DropBlock method, the algorithm takes as input the output activations from a layer A, a block size B, base drop probability a and the mode of the network.
When not in inference mode the algorithm computes a drop ratio "i.j for each entry in the feature map. This is a normalized drop ratio as illustrated in Line of Algorithm [ this drop ratio is equal to ratio of the absolute value of the entry abs(As.,) and the average of absolute values of all entries
||o
in the feature map
||4
Each drop ratio yr is multiplied with the base drop probability and constrained to range [o, 1] to give the drop probabilities qr, for each entry in the feature map illustrated in Line 7 of Algorithm [ As result, entries with values that are higher than the mean absolute entry value for the feature map have a higher probability of being dropped.
4

---

Under review as a conference paper at ICLR 2023
Algorithm Adaptive DropBlock
1:
procedure PROBDROPBLOCKE B, a,a.mode mode)
2: Inputs Layer Output Activations A, Block Size B, Base Drop Probability a, mode
3: if mode Inference then
4:
return
5: else
||Aloo x (bs,))
6:
N.
||A||
Compute drop ratio "i for each element in A
7:
Qt.j = min((a x 1.g)) (.1)).))
D Compute drop probabilities ds,j Randomly sample mask M
8:
M Mas, Bernoulli(1 qi.s)
9:
for Mi, in M do
M.,j 0 then
11:
store((i, j))
D. Store mask indices with zero entries
B
12:
Bib = floor
Compute lower bound buffer zone for mask
2
BB-1
13:
Bub = round
> Compute upper bound buffer zone for mask
2
for (i.j) in store do
15:
Mi MMi-Brahee Bubb- Booss---Botiii -oot
Set values in square centered at Mi.j to O Apply mask M to A
A M
sum (M)
17:
A = A
Normalize
| llo
18:
return
Using drop probabilities Pi we sample mask M, modify the mask by constructing a block of size B around each zero entry in the mask and setting values in the box to zero to create a larger contiguous block of zeros. We handle even block sizes by effectively shifting the block's center point by half an entry right.
After modification this mask M applied to the feature map by element wise matrix multiplica- tion. Finally, we re-normalize A by the ratio of non-zero entries in M to the total number of entries M and return it, this the same re-normalization technique used in DropBlock (Ghiasi et al.] 2018). The linear scheduling is implemented by gradually increasing the base drop probability over the training cycle. The details and effect of the linear scheduling are evaluated in details in Section 4.2
4 EVALUATION
In the following section, we lay out the experimental setup for this paper and briefly discuss the datasets and models considered in Section
We then address the importance of probability scheduling for both structured and unstructured demonstrate that structured Dropouts gener-
Dropout methods in Section 4.2. In Section ally outperform their unstructured counterpart for both vision and language tasks.
To properly assess our approach other structured Dropouts we on both vision and language tasks for state-of-the-art models in Section 4.4 and Section 4.5) We compare the performance of models trained with the proposed ProbDropBlock approach those trained with other forms of Dropout and baseline models trained without any forms of Dropout.
4.1 EXPERIMENT SETUP
In this work, we considered datasets, NLP (natural language processing) datasets and 3 CV (computer vision) datasets.

---

Under review as a conference paper at ICLR 2023
The NLP datasets considered are all part of GLUE the multitask benchmark and analysis platform for natural language understanding (Wang et al. 2018), we consider MNLI, QNLI and RTE in our evaluation.
For the CV datasets, we consider CIFARIO, CIFARI00 (if and ImageNet (Deng et al. 2009)
classification. More details about these datasets can be found in Appendix B
For finetuning the RoBERTa model, we use the hyperparameter setup in Liu et al. 2019 For training the ResNet family models, train all models using the Adam optimizer Kingma & Ba (2014) and pick the best learning rate from {le-5, - 5e- 1e- le- (PVT-V2) model, We use the standard setup described in Wang et al. (2022). We slightly changed the augmentations in the original PVT-V2 setup, removing Mixup and Random Erasing, so that it is more closely aligned with the ResNet training setup for a better comparison. In the ResNet family and PVT-V2 models, we insert the Dropout mechanism after each residual block. For RoBERTa, we add Dropout, DropBlock or ProbDropblock to the end of each encoder layer.
For the pyramid vision transformer
We run each data point 3 times with different random seeds, and report both the average and standard deviations, the details of our hardware system setup can be found Appendix [D] We picked the dropping Block Size (B) to be B 4, and show an ablation of this parameter in Appendix F
4.2 PROBABILITY SCHEDULINGG
Table Structured (BatchDropBlock) and non-structured Dropouts on CIFAR-10. with and without the linear scheduling the dropping probability. BDB is BatchDropBlock. A shows the difference between the accuracy compared to baseline, the baseline accuracy on this task is 94.37%.
Method
Dropout
Dropout- Schedule
BDB
BDB BDB-Schedule
Resnet50 Acc 1 A1
94.50 + 0.04
94.70 = 0.14
93.11 + 0.22
94.77 = 0.29
+0.13
+0.33
-1.26
+0.41
Ghiasi et al.] mentioned in their experiments that DropBlock with a linear dropping scheme that decreases the value of keep probability from to 1 a can significantly improve the performance. We test a E 10.1, 0.2, 0.3, 0.5). and pick the best performing a (a = 0.2 this case). A detailed explanation of how we pick the keep probability
Appendix
In Table we apply this linear dropping scheme to both the standard Dropout (Srivastava et al., 2014b) and BatchDropBlock (Dai al. 2019). We consider a modified version of Blockk,,, where all channels of single imput are dropped consistently, but datapoints in a batch can drop independently. One major observation from Table [ that an appropriate probability scheduling improves the performance of both structured and unstructured Dropout methods. The scheduling shows a greater impact on structured Dropout.
Intuitively, Dropout servers as regularization method, and applying it at the start of the training interferes with the optimization; this type of regularization method should be introduced at a later stage of training when the training accuracy starts to become larger than the validation accuracy, or in other words when overfitting starts to arise.
In general, we observed that:
Structured Dropout (eg. DropBlock) with linear dropping scheme of decreasing the value of keep probability can significantly improve the performance, this aligns with the obser- vation made by Ghiasi et al.)
Non-structured Dropout also benefits from the linear dropping scheme, although it is less significant improvement than the structured Dropout.
4.3 STRUCTURED AND CON-STRUCTURED DROPOUTS
In this section, we compare the performance of the standard Dropout, various structured Dropout schemes (BatchDropBlock and DropBlock) and ProbDropBlock. All methods have a linear drop- ping scheme of decreasing the value of the keep probability from We experimented also

---

Under review as a conference paper at ICLR 2023
E {0.1, 0.2, 0.3, 0.5), and used 0.2 for ResNet and 0.1 for RoBERTa models, ablation study of different dropping probabilities can be found in Appendix E
Table 2: A comparison of the performance of Dropout and various Structured Dropout schemes. BDB is BatchiDroBllok, DropBlock is not channel consistent (blocks deactivated are not consistent between channels), ProbDropBlock additionally has dropping probabilities correlated to pixel-wise saliency. RoBERTa is evaluated MNLI with a bascline accuracy of 87.60%, and ResNet50 is evaluated on CIFARIO 10 with a baseline accuracy of 94.37%. is the difference between the current accuracy and baseline.
Method Resnet50 Acc f
Dropout
BDB
DropBlock
ProbDropBlock
94.70 = 0.14
94.77 = 0.29
95.05 = 0.21
94.73 1 0.19
At
+0.33
+0.41
+0.68
+0.35
RoBERTa Acc f
87.51 + 0.08
87.39 = 0.29
87.71 0.22
87.83 0.15
A
-..0
-0.21
+0.11
+0.22
Language models such as BERT (Devlin et al. 2018 and RoBERTa (Liu et al. 2019) are based on the multi-head attention mechanism (Vaswami et al. 2017). Prior work has not studied how to apply coarse-grained Dropout techniques transformer- based architectures.
We apply these structured Dropouts in a head-wise manner, this means for BatchDropBlock, we drop the same pattern across heads in multi-head attention. DropBlock, in contrast, then drops each head independently.
The original RoBERTa used Dropout with a 0.1, and replaced all of these Dropouts with the regularization strategies shown in T Notice, in this case, our baseline considered a standard RoBERTa without any Dropouts. Our RoBERTa baseline on MNLI achieves 87.60% The baseline accuracy for ResNet50 on CIFARIO is 94.37%.
The ReNet50 model is evaluated on CIFARIO. The striding of the network is adjusted fit into this smaller image size of CIFARIO. The details of this network architecture are summarized in Appendix C Table 2 confirmed with the observation made by Ghiasi al. (2018) that structured Dropouts are generally better than standard, unstructured Dropouts on vision tasks. In addition, our results in Table also suggest that structured Dropout better on MNL.I
Another interesting observation is that unstructured Dropout does not provide any performance gains for language models (-0.09 on RoBERTa). the meantime, we see that BDB, although works rea- sonably well on ResNet50, has a detrimental impact the performance of RoBERTa. This means that applying structured Dropout methods to each Transformer head independently is important for these methods improve the performance of language models. We will investigate this phenomenon in greater detail in Section 4.4
In general, we observed that:
In addition to what was originally shown by Ghiasi et al.) we observed structured Dropout techniques generally better not only vision tasks but also language tasks.
Structured Dropout techniques are more advantageous on language tasks compared to vi- sion tasks.
4.4 LANGUAGE TASKS
Table 3 demonstrates the performance of RoBERTa finetuned three GLUE tasks (MNLI, QNLI, RTE) using different structured Dropout techniques. To our best knowledge, we are the first to investigate the effect of block-wise structured Dropout methods on bruseformer-bused models.
We observe that both DropBlock and ProbDropBlock consistently outperform BatchDropBlock (BDB) and that ProbDropBlock is the best performing method. Both DropBlock and ProbDrop- Block can significantly help RoBERTa achieve better performance, while BDB has a negative impact on its accuracy.
The major difference with BDB is that each head drops blocks with the same pattern across heads, observe this is having negative effect on the performance of RoBERTa.
7

---

Under review as a conference paper at ICLR 2023
The proposed strategy, ProbDropBlock, is able to achieve the best performance all tasks and is able to outperform the original RoBERTa model by a significant margin. This is a clear indication that dropping structured patterns based the saliency values of the attention maps is advantageous.
Table 3: Different Structured Dropout schemes. DropBlock is channel independent (channels are not dropped independently), ProbDropBlock additionally has dropping probabilities correlated to element-wise saliency. The RoBERTa model is first pretrained on large unlabeled text corpus and subsequently finetuned these tasks, which is the same setup in Liu et al.
Method
Metric
MNLI
QNLI
RTE
Baseline
Accuracy 1
87.60 + 0.04
92.75 + 0.03
73,28 = 0.02
Accuracy t
87.39 0.29
92.70 + 0.06
70.64
0.07
BatchDropBlock
A1
-0,21
-0.05
-2.64
Accuracy 1
87.71 + 0.24
92.81 + 0.11
72.51 + 0.08
DropBlock
A 1
+0.11
+0.06
-0.77
Accuracy f A
87.83 4 0.15 0.15
92.90 + 0.12
74.25 = 0.03
ProbDropBlock
+00.22
+0.15
+0.97
In general, we made the following observations from Table
BatchDropBlocc generally has negative impact the performance of language models.
Structured Dropout applied identically per head (BotchDropBlock) does not improve model performance on language tasks, both DropBlock and ProbDropBlock outperform BDB by significant margin. Dropping heads independently is critical for better performance on language models.
The multi- attention modules in the transformer benefit the most from ProbDropBlock.
4.5
VISION TASKS
Table 4: Different Structured Dropout schemes. DropBlock is channel independent (channels not dropped independently), ProbDropBlock additionally has dropping probabilities correlated to element-wise saliency. ResNet50 and WidcResNet28 are from the ResNet family with adjusted striding to match the CIFAR image size. PVTv2-B1 is the pyramid vision transformer.
CIFARTO
CIFARIO0
Method
Metric
ResNet50
PVTv2-B1
WideResNet28
PVTv2-B1
Baseline
Accuracy
94.37 + 0.32
95.59 = 1.00
7 0.08
82.38 = 0.19
Accuracy f
94.77 + 0.29
95.99 = 0.15
74.97 + 0.26
82.22 = 0.34
BatchDropBlock
A
+0.41
+0.40
+0.25
-0.16
Accuracy f
95.05 + 0.21
95.89 1 0.09
74.99 + 0.08
82.26 = 0.35
DropBlock
+0.68
+0.30
+0.27
-0.12
Accuracy t At
94.73 + 0.19
96.15 + 0.01
75.13 = 0.27
82.44 = 0.16
ProbDropBlock
+0.35
-0.56
+0.41
+0.06
Table ProbDropBlock and baseline performance for ImageNet classification.
ImageNet
Method
Metric
ResNet50
PVTv2-B1
Baseline
Accuracy f
74.22 + 0.06
78.27 + 0.04
Accuracy 1
74.50 + 0.17
78.88 + 0.22
ProbDropBlock
A
+0.28
+0.61
Table and Table 5 demonstrate the results of applying different structured Dropout methods on CIFAR10, 0, CIFARI00 and ImageNet. Vision Transformers (ViTs) recently have demonstrated great capabilities on major vision benchmarks (Liu et al. 2021] Wang et al.) (022). so we consider both the
ResNet family (He al. 2016 Zagoruyko & Komodakis 2016) and Pyramid Vision Transformer
8

---

Under review as a conference paper at ICLR 2023
Wang et al. (2022) in our experiment. Prior research has hardly systematically studied the effect of structured Dropouts on Vision Transformer models.
We observe that in general ProbDropBlock shows the best performance all dataset network com- binations, except for one outlier which is ResNet50 on CIFARIO. Interestingly, we observe a phe- nomenon that BDB (BatchDropBlock) in general improves the performance of both CNNs and ViTs according to T This is very different from the phenomenon observed in Section 4.4 that BDB generally decreases the performance of language models.
We also notice that only ProoDropBlock can slightly increase the accuracy of PVTv2-B1 on CI- FARI00. We then realised that PVTv2-B1 has a validation accuracy that not greatly larger than its training accuracy, meaning that this model does not overfit the dataset by significant margin. For instance, ResNet50 on CIFARIO overfits heavily on the CIFARIO task and thus it benefits the most from regularization methods. The PVTv2-BI model architecture used for both CIFARIO and CIFAR100 is significantly smaller than the original model, the details of this model architecture dif- ference are explained in Appendix C In addition, CIFAR 100 is a harder task than CIFARIC 10, we see PVTv2-B1 benefits less from regularization methods such as structured Dropouts when the model is not overfitting.
Table generally demonstrate that ProbDropBlock the best regularization method on three out of the four model-dataset combinations. When train PVTv2-B1 on CIFARI00, both BDB and DropBlock fail to improve the model performance since the model does not overfit greatly to the dataset; but, ProbDropBlock still makes positive impact model performance. We further tested the effect ProbDropBlock on ImageNet and j our results. We demonstrate that ProbDropBlock is an effective regularization method, it provides us +0.28% and +0.61% accuracy gains for ResNet50 and PVTv2-B1 respectively.
Structured Dropout techniques in general can help vision models based observations on Table 4 and Table We make the following observations:
BrochDro has a positive impact on computer vision models, this is different from what we have observed in Section[.44
Models that are not overfitting benefit less from structured Dropouts.
Structured Dropout methods generally help vision models to learn better, the proposed BrooDropBlock is effective on both CNNs and ViTs.
CONCLUSION
In this paper, we revisit the ideas of structured Dropout for current state-of-the-art models, we devise our form of adaptive structured Dropout- - ProbDropBlock and compare preexisting structured and unstructured Dropout approaches ours on vision and language tasks. We demonstrated the utility of a simple linear dropping schedule for both structured and unstructured Dropouts supporting similar observation made by Ghiasi et (2018).
Our approach, ProbDropBlock was able to achieve improvements in performance for all networks and task combinations and outperformed other forms of Dropout considered for the majority of combinations we evaluated. In particular, ProbDropBlock improved RoBERTa finetuning on MNLI by 0.22%, and training of ResNet50 on ImageNet by 0.28%.
This work demonstrates the utility of structured Dropout approaches not just on residual networks and CNNs, but on language and vision transformers. However, there is limit to the gains achievable through Dropout alone, as demonstrated by the results of the PVTv2-B1 model on the CIFAR-100 dataset, when there is minimal overfitting in the model regularization only provides minimal gain.
9

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
Language and vision models typically require a significant amount of data for training and test- ing. Our work only uses datasets that are widely used within the ML community, however, there are some ethical concerns about the collection methodologies and entries in some of these datasets Asano et al. (02T]. We make use of these datasets because they are established and recognized benchmarks while we acknowledge possible ethical issues with these datasets. In this work, we focus on improving generalization by combating overfitting through adaptive structured Dropout. Generally, improved model generalization is positíve outcome but that assumes these trained mod- els are not applied to nefarious purposes, however, we can not ensure this.
REPRODUCIBILITY STATEMENT
We discuss how we setup our experiments in Section 4.1 We explained the learning rate and op- timizer setup in that section. At the beginning of Section 4.2 Section 4.4 and Section 4.5 explained the choice of our probability scheduling and the picked a value. In Appendix and Appendix F explained how picked the hyperparameters and B based these ablation studies. Each of our experiments repeated for times, and both the arithmetic mean and standard deviation are reported.
Appendix B details the setup of each dataset and Appendix details the models we have used and included any modifications have made these models. Our hardware system used to perform experiments is reported Appendix D
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Cifar-10 and cifar-100 datasets.
Accessed: 2022-09-22.
Yuki Markus Asano, Christian Rupprecht, Andrew Zisserman, and Andrea Vedaldi. PASS: an ima- pretaacnnn without humans. CoRR, abs/2109.132288 2021.
genet replacement for self-supervised pretraining
URL httpp:ttaaxa.i.....tta httpavvv.rggb///////aa 13228
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, 2015.
Shaofeng Cai, Yao Shu, Gang Chen, Beng Chin Ooi, Wei Wang, and Meihui Zhang. Effective and efficient dropout for deep convolutional neural networks. arXiv preprint arXiv: #rXhv:990..0332 2019.
Rich Caruana, Steve Lawrence, and C Giles. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. Advances in neural information processing systems, 13, 2000.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
Zuozhuo Dai, Mingqiang Chen, Xiaodong Gu, Siyu Zhu, and Ping Tan. Batch dropblock network for person re-identification and beyond. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Jacob Devlin, Ming-Wes Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800. 2018.
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional come:ooioiioocoo neural networks with cutout. CoRR, hts/1708..45522 2017. URL hBBl://p..//aaxx...r///////////1 04552]
Terrance DeVeies and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint 1708.v7700.044 2017.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. arXiv preprint arXiv: 15556, 2019.
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng-zhong Xu. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint drXiv:1810.05331, 2018.
Golnaz Ghiasi, Tsung Yi Lin, and Quoe V Le. Dropblock: A regularization method for convolu- tional networks. Advances in neural information processing systems, 31, 2018.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal rec- ognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entuilment and paraphrasing, pp. 1-9, 2007.
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume 7, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Jeff Heaton. Applications of deep neural networks. arXiv preprint arXivv2009.056673 2020.
11

---

Under review as a conference paper at ICLR 2023
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(78733:583-588 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /4126694,, 2014.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural net- works without residuals. arXiv preprint arXiv:1605.07644 2016.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 11007.11692.. 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:171.05101,, 2017.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 1606000000.05500 2016.
Julian Schritwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, (88(7889))604 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56)1929-11988 2014a. URL http://t fmm wtw.m.rrrr.............m..ooo.re httrg//aprr//vs//
srivastaval4a. .htll
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199) 1958, 2014b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs//804.072 2018 URL http: /ttp///rxvvvv/v wttp://wwwvvr/a/////a/eaaa/e
Wenhai Wang. Enze Xie, Xiang Li, Deng- Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415 424, 2022.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 11221122. Association for Computational Linguistics, 2018. URL http:// //ac. web. org. antho ogy/Nl8 8
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8817-8826, 2018.
12

---

Under review as a conference paper at ICLR 2023
Ji Xin, Raphael Tang. Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. arXiv preprint arXiv:2004. 12993,, 2020.
Sergey Zagoruyko and Nikos Komodakis. 1rXiv:1605.07146, 2016.
Wide residual networks.
arXiv preprint
13

---

Under review as a conference paper at ICLR 2023
A MOTIVATION FROM A THEORETICAL LENS
A.1 MARGINALIZING PROBDROPBLOCK
In considering the effect of our adaptive structured approach to Dropout we present a modified scenario of marginalising out Dropout in simple linear regression case with block size of 1. This setup was used in the original Dropout paper Srivastava et al. (201bb) to illustrate how Dropout acts regulariser when marginalised out.
In our scenario we take as input the output activations R N of a layer of a network, a weight vector E R.M and target vector of y E R.N.
As we all know the aim of linear regression to minimise lly - Awll. ProbDropBlock randomly drops entries in the matrix A with entries being retained with probability Pt.j where Oi gives the adaptive ratio for dropping feature entries based of relative magnitude, Pi.j may be rewritten as p higj where p constant keep probability and hij is a sealar that modifies p to give the keep probabilities for matrix entry i.j.
(q x ogg
After ProbDropBlock the input A can be expressed as A where E {0, 1JM is a random matrix and R Bernoulli(pi.. (pi.) and represents element wise matrix multiplication. So marginal- ising the effect of dropout out of the objective function becomes
E BR Bernonlii)))))
[[ly (R A)
This becomes
yTy y TE (Rx A)" y y A)] w + wT (R* A)" (R* A)
(2)
It is clear that the E [ 1)21 = =[ A) and E [[R A)] P where is matrix of the keep probabilities for each entry (i,j). So E [R * A)] may be rewritten as Pd where = A, constant and H is re-scaling matrix of entries hu that modify the constant p needed.
E [R A)" (R+ A)] is a little more involved but works out to be p² AT À (p (a0ppp)dic dii Ar A'AA
And so the full expectation works out to be
yf y - pw A - py" Aw + ppw A" Aw + (p p²)w²" "diag (A"
(3)
Which can be rewritten
lly p/w/|2 2 (p- |diag A A)
w|1
(4)
The second term is never negative as p 2 p² because p S and so the second term acts as a regu- lariser when we try to minimise this expression with respect to as is the case in linear regression. fact this equation takes the same form as that seen in the original Dropout paper (Srivastava et al. 2014b), with the exception of an Á instead on an A.
Effectively the modifier matrix H is re-scaling the input features scaling down entries with higher than average values and scaling up entries with lower than average values. While in a neural network dynamics are more complex and the activation outputs are evolving as training progresses we believe this adaptive structured dropout approach encourages the network to learn a more robust feature representation.
A.2 APPROXIMATING ENSEMBLES USING DROPOUT
The initial Dropout paper Srivastava et al. (211b) also states consider unstructured Dropout as a way of training exponentially many sparse networks with extensive weight sharing and then at test time approximately average this ensemble of networks by rescaling the weights.
14

---

Under review as a conference paper at ICLR 2023
ProbDropBlock is adaptive and structured approach to Dropout. In ProbDropBlock instead of uniformly sampling sparse networks from the original network during training we sample sparse networks with lower activations more often. If assume activations relevant information the sampled sparse networks are faced with harder problems are they are miss- ing likely relevant information and So by sampling these 'worse' performing sparse networks more often during training we provide additional training for these 'worse' sparse networks and improve the overall performance of the approximated average ensemble at test time.
proxy for containing
likely perform 'worse" the same stage of training.
B DATASETS
B.1 LANGUAGE: TASKS
MNLI (Multi-Genre Natural Language Inference) corpus (Williams al. 2018 Wang et al. 2018) is collection of sentence pairs with textual entailment annotations gathered via crowd sourcing. The sentences are paired as premise and hypothesis and the task is to predict if the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction) or neither (neutral), The corpus is modeled on the SNLI (Stanford Natural Lanfuage Inference) corpus (Bowman et al. 2015). but differs in that covers range of genres of spoken and written text, and supports distinctive cross-genre generalization evaluation (Williams al. 20188) Wang et al.) 2018). The premises are gathered from ten different sources, including fiction, government reports and transcribed speeches. consists of 393k train samples and 20k test samples.
QNLI Questtonn-answering Natural Language Inference) corpus is a dataset automatically derived from SQuAD (Stanford Question Answering Dataset) (Rajpurkar et al. 2016 Wang et al.) 2018). SQuAD is question-answerte tence in the paragraph contains the answer to the corresponding question. QNLI is constructed by converting the task into sentence pair classification by forming a pair between each question and each sentence in the corresponding context. It consists of 105k training samples and 5.4k testing samples.
dataset which consists of parrggrpp pairs, where sen-
RTE RTE The Recognizing Textual Entailment (RTE) datasets combination of several datasets which came from a series of annual textual entailment challenges Wang et al. (2018): Dagan et al.) 2205); Haim et al. (2006); Giampiccolo al. (2007); Bentivogli et al. (2009). It consists 2.5k training samples and 3k testing samples.
B.2 VISION TASKS
CIFAR-10 & CIFAR-100 The CIFAR-10 dataset contains 60000 32x32 colour images divided into 10 classes, each with 6000 images. There are 50,000 training and 10,000 test images.
The CIFAR-100 dataset is just like the CIFAR-10, except it has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class. The 100 classes in the CIFAR- 100 are grouped into 20 superclasses. Each image comes with "fine" label (the class which belongs) and "coarse" label (the superclass to which belongs).
The ILSVRC 2012 image classification dataset contains 1.2 million images for training and 50,000 for validation from 1000 classes. The input image sizes are 224 x 224 center crop to images at test time. The results are reported on the validation set.
C NETWORKS
C.I ROBERTA
The RoBERTa model is kept the same as its original form proposed by Liu et al. (2019). our experiment, consider the RoBERTa-base model only. The base model contains 12 layers, with a hidden size 768, an FFN inner hidden size of 3072 and 12 attention heads. The original model uses Dropout and replace all of the original Dropouts structured Dropout methods.
15

---

Under review as a conference paper at ICLR 2023
C.1.1 RESNET
We consider both the original ResNet (He et al,, 2016 and its wider alternative (WideResNet) (Zagoryyko & Komodakis 2016). These networks normally have convolutional layer (named stem) and four other residual blocks. For the CIFARIO and CIFAR100 classification, we change the striding of the first convolution network to operate with the 32 x 32 image size on CIFAR datasets
and deleted the first max pooling. These adaptions help the
C.1.2 PVT-V2
Table demonstrates the detailed setup the PVT-V2 structure used for CIFAR and ImageNet tasks. The rest of the setup parameters the same as Wang et al. (2022). and the setup is the same as the PVTV2-B1 model.
Table 6: PVT-V2 setup for vision datasets, e is the embedding dimension and is the striding used for the overlapping patch embedding.
Layer name
CIFARIO/CIFARIOM
ImageNet
Stage
16,8 16,8=4
6 64,
Stage
32, =2
128,
Stage 3
= 64, 8
256, 2
Stage 2
e 128, =
512, 2
C.1.3 ADDITIONAL RESULTS ON MACHINE TRANSLATION
To illustrate that the proposed dropout scheme also works on Sequence to Sequence tasks, per- formed an evaluation of it on IWSLT" 14 English (EN) to German (DE) and German (DE) to English (EN).
Table 7: Machine Translation Task (IWSLT'14) using RoBERTa model, results are reported as BLEU scores.
EN - DE
DE EN
Baseline
27.24
33.21
ProbDropBlock
27.84 (+0.60)
33.44 (+0.23)
D HARDWARE SYSTEM
We used variety of hardware systems, our initial testing and CIFAR10 results are generated on a hardware system with NVIDIA GeForee RTX 2080 GPUs. The ImageNet training and RoBERTa training performed on 4 x Nvidía A100 SXM4 80GB GPUs. The total amount of GPU training cost for all the expriments in this paper is around 20 GPU-days.
E PICKING THE DROPPING PROBABILITY
Table 8: Different Dropping probabilities for ProbDropBlock.
Probability
ResNet50 on CIFAR10
PVT-V2 on CIFAR100
RoBERTa MNLI
0.0
94.37 + 0.32
82.38 + 0.19
87,60 0.04
0.1
94.70 0.14
82.44 0.16
$7.83
0.2
94.73 + 0.19
82.21 + 0.13
87.32 0.11
0.3
94.20 0.30
82.10 0.16
69.45 - 0.16
16

---

Under review as a conference paper at ICLR 2023
F PICKING THE BLOCK SIZE
The block size for DropBlock is a hyperparameter that needs to be tuned. We test E
{2, 4, 6, 8, 10), and pick the best performing B (B 4 in this case).
94.2
Accuracy
93.8
93.4
2
3
Block Size
Figure The effect of block size on the performance of DropBlock.
17


Reviews:
[{"review_id": "hgalpS4Yfk_", "reviewer": "Reviewer_Fidd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes application of structured dropout methods (e.g. drop block) to transformers.  It also proposes an enhancement to drop block which utilizes activation values to assign dropout probability to nodes in feature maps as opposed to random selection of blocks.  Empirical results on language and vision tasks demonstrate that structured dropout is advantageous for transformer models and the proposed modification to drop block methods leads to an improvement in accuracy for language and vision tasks.\n", "summary_of_the_review": "see comments above", "strength_and_weaknesses": "Pros:\n* Exploration of structured dropout for transformers\n* An enhancement to drop block that shows better empirical performance as compared to the baseline approach\n* Combination with linear dropping schedule gives further gains, reconfirming the value of linear dropout schedule\n\nCons:\nN/A", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gccwMTRHLMT", "reviewer": "Reviewer_osqf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Structured dropout are methods that determine probability of deactivation based on certain inductive priors such as neighborhood or location, which are important in architectures like CNN.  \nThe author surveyed existing structure dropout method and developed improved version based on block dropout called PROBDROPBLOCK. which assign different deactivation probability based on activation levels. It is interesting to see these method work well in transformers. ", "summary_of_the_review": "Although lacking novelty in method, due to the importance and popularity of dropout in machine learning and the well designed experiments, I think this could bring values to the community  ", "strength_and_weaknesses": "strength: clear motivation ,simple method, wide impact \n\nweakness: this manuscript serve well as a systematic review article , but lack of novel in method as a original research article by ICLR standard. The motivation or theoretical analysis of why choosing to conduction stochastic dropout in PROBDROPBLOCK is unclear. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RL-LwFVlxM", "reviewer": "Reviewer_zLHu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an adaptive structured dropout method, ProbDropBlock, which drops contiguous blocks from feature maps with a probability given by the normalized feature salience values. The authors evaluate ProbDropBlock on both vision and language tasks.", "summary_of_the_review": "The paper proposed an adaptive structured dropout method that is beneficial for both vision and language tasks. It indicates the potential of using structured dropout in Transformer-based models. However, the experiment results are not sufficient to demonstrate the effectiveness of structured dropout and the proposed adaptive idea, especially in Transformers. ", "strength_and_weaknesses": "Strengths:\n\n1. The paper proposed an adaptive structured dropout method that is beneficial for both vision and language tasks. It indicates the potential of using structured dropout in Transformer-based models.\n2. The paper provided an thorough investigation of existing structured dropout methods.\n\nWeakness:\n\n1. The experimental settings and the performance gains are insufficient to demonstrate the effectiveness of the method. In Table 2, I am not quite convinced by the RoBERTa results, as unstructured dropout can usually achieve some gains for Transformers/ViTs. The authors may present results on more models/tasks to show that structured dropout works better in Transformers/ViTs. For a fair comparison, the authors may compare the proposed adaptive structured dropout to adaptive unstructured dropout. In Table 3, the gains are marginal (typo: the gain on QNLI should be $0.15$). \n\n2. The motivation of assigning higher drop rate to activations with higher magnitude is unclear. More ablations and analysis are needed to understand its influence to the training. It seems to impose a strong regularization but may also prevent the model from seeing important features. \n\n3. Expanding blocks based on the sampled $A_{i,j}$s may be problematic as the surrounding activations may not have high magnitude on average.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-f33Upka7OY", "reviewer": "Reviewer_EuKt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes ProbDropBlock, an adaptive version of DropBlock in which the drop probability of the mask is defined to be negatively correlated to the absolute value of the centroid entry. Experiments are conducted on 3 NLP datasets (MNLI, QNLI, RTE) and 3 CV datasets (Cifar-10, Cifar-100, Imagenet).", "summary_of_the_review": "In general, this paper is well motivated and focuses on an important problem which will be beneficial for future studies in both NLP and CV communities. However, both theoretical and experimental analysis are not thorough and do not support the claims well.", "strength_and_weaknesses": "Strength:\n\n1. The paper is easy to follow\n\n2. The idea of investigating both unstructured and structured dropout methods on SOTA models w.r.t. both NLP and CV tasks is well motivated and beneficial for advance of these two research tasks.\n\nWeakness:\n\n1. Experiment results of baseline method are not SOTA. For example, the (top-1) accuracy of Resnet50 on Imagenet reported in Table 5 is lower than that in Ghiasi et al. (74.22 vs 76.51). Other examples include the accuracy of WideResNet28 on CIFAR100 (which is lower than that reported in the original WRN paper). This weakens the claim that made in the introduction section about testing dropout methods on current SOTA models. Some SOTA results can be found in https://paperswithcode.com/sota/image-classification-on-cifar-100\n\n2. Lack of explanation and analysis about the adaptive dropout ratio, which is negatively correlated to the absolute value of layer output activations. It is hard to understand why the proposed method performs better than existing methods. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject