Not All Attention Is All You Need
Authors: Hongqiu Wu
hai zhao
Min Zhang
Venue: NeurIPS 2021 Submitted (2021)
Keywords: dropout
pre-trained language model
self-attention

Abstract: Beyond the success story of pre-trained language models (PrLMs) in recent natural language processing, they are susceptible to over-fitting due to unusual large model size. To this end, dropout serves as a therapy. However, existing methods like random-based, knowledge-based and search-based dropout are more general but less effective onto self-attention based models, which are broadly chosen as the fundamental architecture of PrLMs. In this paper, we propose a novel dropout method named AttendOut to let self-attention empowered PrLMs capable of more robust task-specific tuning. We demonstrate that state-of-the-art models with elaborate training design may achieve much stronger results. We verify the universality of our approach on extensive natural language processing tasks.

Not All Attention Is All You Need
Anonymous Author(s) Affiliation
Address email
Abstract
Beyond the success story of pre-trained language models (PrLMs) in recent natu- ral language processing, they are susceptible to over-fitting due to unusual large model size. To this end, dropout serves therapy. However, existing methods like random-based, trowledge-based and search-based dropout are more general but less effective onto self-attention based models, which are broadly chosen as the fundamental architecture of PrL.Ms. In this paper, we propose novel dropout method named AttendOut to let self-attention empowered PrLMs capable of more robust task-specific tuning. We demonstrate that tfate-o--th-aat models with elab- orate training design may achieve much stronger results. We verify the universal- ity of our approach on extensive natural language processing tasks.
5
7
Introduction
11
12 Self-attention network (SAN) empowered models like Transformer [0]] have achieved remarkable
13 success in recent natural language processing, which have been broadly chosen as basic architec- ture in series successful pre-trained language models (PrLMs) such BERT [ RoBERTa [3],
15
ALBERT [M]. ELECTRA [S]] DeBERTa [6]] and GPT [Z]].
SAN has drawn great deal of curiosity on its conceptually simple but powerful attention mecha- 17 nism. However, SAN still remains a black box and more and more works attempt to unveil its inner principle, where the biggest mystery lies in its attention matrix. Our work is inspired by several 19 recent discoveries which turn our views up and down. [8, 9] show that fixed Gaussian or even ran- dom alignment attention matrix may rival standard SAN, while
recently, [TT. [T]] prove that SAN may encounter a rank collapse with deepening of layers. A more concrete explanation is in- formation diffusion [02], which states that the input vectors are progressively assimilating through continuously making self-attention. We attribute these problems to the sever co-adaption [03] be- tween attention elements, form of over-fitting onto SAN. As result, self-attention empowered PrLMs hardly bring into their full play, especially for the fine-tuning stage, where task-specific data always with limited capacity.
21
22
23
24
25
26
27 Dropout [ serves as a therapy to deal with the problem, by randomly shutting down a set of units during training stage. When specified self-attention, dropout is equivalent to adding attention mask to the attention matrix. However, random-bass dropout methods like vanilla Dropout [[S)] or DropConnect [[44] all subject pre-defined distribution like Bernoulli Gaussian, longing for exhaustive grid search for an optimal probability. Thereby a variety of works attempt to utilize man- informative attention matrix [[55, [G]], whereas all these methods require prior knowledge on model or data, which could be costly or unavailable. More recently, the
29
30
31
32
ual attention mask to obtain
33
34 rise of Neural Architecture Search [IZ] [X] gives birth search-based dropout [LL which automat-
35 ically chooses an optimal dropout pattern based on additional validation performances. However,
38 the huge search space brings heavy consumption and more importantly, the obtained dropout pattern
37 is still fixed with a pre-defined probability, which is static and sample-independen ignoring the
38 dynamics within different samples. this paper, we focus on task-specific tuning of self-attention
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

Prienty
Conrolier
Reward
Validution
Reward
Attenion Layer
Samples
39 empowered PrLMs and propose novel dropout method named AttendOut onto attention layers, which leverages self-attention to dynamically generate dropout patterns for each attention layer as well as each sample through an end-to-end manner. demonstrate that the previous state-of-the-ar models with elaborate training design may achieve much stronger results. We verify the universality SAN Block
SAN Block
SAN Block
SAN Block
Random-based
Knowleege-based
Search-base
AttendOut
Figure I:
A diagram of different dropout methods, where p refers the dropout probabilities while R refers to the reward in reinforcement learning.
41
43 of our approach on extensive natural language processing tasks. Guided by AttendOut, we pro- pose another two attention regularizers to enable simple but effective performance boost with no additional cost.
45
2 Related Work
47
Dropout is proposed to alleviate over-fitting problem in DNNs. Apart from vanilla Dropout [
48
and DropConnect [D=] which randomly shut down a subset of activations or hidden weights, there
are variety of dropout methods proposed, e.g. Alpha Dropout [20], Variational Dropout [211, 22). Adversarial Dropout [23]], Energy-based Dropout [24]]. However, random- based dropout encounters slow experiment cycle due to inevitable grid search. Inspired by Neural Architecture Search [[ZZ. [С]
50
52
[T proposes AutoDropout to automate the process of designing dropout patterns. A similar line of work is dynamic tuning of dropout, which further allows adaptive dropout probabilities under differ- ent training moments. [ proposes Concrete Dropout with continuous relaxation under Concrete distribution, [2G]] proposes Learnable Bernoulli Dropout under discrete Bernoulli distribution using estimator [27]], while [28] proposes Context Dropout by optimizing
54
56
the evidence lower bound.
58 With self-attention network continuously stands out, dropout is being explored onto self-attention based models. LayerDrop [2_] randomly removes entire SAN blocks, while DropHead [30], Head- 80 Mask [3]]] randomly remove certain attention heads. UniDrop [32] unifies these dropout methods, which facilitates text classification and machine translation tasks. Additionally, prior knowledge is shown highly effective for guiding attention dropout as in SG-Net [TS] and SIT [T6]], which inten- tionally discard syntax-unre untellttd attention units with the help of structural clues.
82
Preliminaries
64
In this section, we provide the preliminaries for the proposed approach. We first review the details s6 of self-attention proposed in [0]]. Based on the specific architecture, we elaborate the concerned
65
67
attention dropout.
88
3.1 Self-Attention
89
Generally, a standard SAN block mainly composed an attention layer and several feed-forward layers (actually there are residual connection, layer normalization, etc. as well). The input of it is sentence batch of sentences of length n, which first embedded through an embedding layer. The embedded input E may go through three linear projections Wo. WK and Wy referring query, key
71

---

73 and value layers respectively, and then obtain three matrices Q. K and V referring to the query, key
74 and value components of self-attention. Subsequently, dot-product of Q and K taken and then normalized using Sof Imar function to obtain the attention matrix A. Then another dot-produce of and follows. The mentioned calculation can be formalized as follow:
75
76
Attention K, V) Softmar (2 (vid (2.K*)
(1)
77 where v dlg is a scaling factor. Finally, the self-attention layer ends up with a linear projection Wo to output.
79 During the aforementioned process, we highlight a key phases, that the attention matrix A, which
80 is a ddo-pproduct of n x n from two separate linear projections Wo and WK. A is viewed as a feature
81 map which stores the node-to-node significance different scores. Various works show that there hides implicit but highly needed semantic clues.
83
3.2 Dropout on Self-Attention
Our dropout will apply the attention matrix of the concerned attention layer. We first define two specific dropouts onto Eq. MI where both implementations are just as simple as standard dropout mask matrix M.
85
87
Weights Dropout. Weights dropout is applied to the attention matrix after Softmar function by default, which is formulated as:
88
Attention(,,K,W ((Sof Sof Q-KT (2.
M) M).
89
where M is a binary matrix with elements in (0, 1) and @ refers to element-wiss multiplication.
Scores Dropout Different from weights dropout, scores dropout applied before Softra func- tion, which is formulated as:
81
Attention K, V) = Softmar
ddz +
(3)
92 Since the outer Softmar, we conduct an addition instead of multiplication, where elements in M set 0 for kept units and -inf for removed ones. Note that the Softmaz takes similar function as the scaling factor of 1/p in vanilla Dropout [L)] which balances the expectation of the
34
95
network.
96 Weights dropout is commonly used in self-attention based models, while scores dropout is less explored, which is our focus in this paper. For scores dropout, need to pay attention to special case, when all attentions are shut down, that is, all elements in M equal to -in at the same time. Such case can be formulated follow:
96
Attention Q K, V) Softmar (M) -V
100
Note that Softmar (M) obtains to constant matrix, where each unit equals to 1/n. In this case, the attention matrix fixed and consequently the Wo. WK and dot-product in between are skipped.
101
4
Methodology
102
103
In this paper, we propose Attention differentiable dropOul (AttendOut), which contributes technique novelty in the following way: (1) dynamic and task-specific tuned; (2) end-to-end trained; (3) gra- dient optimized dropout method onto self-attention empowered PrLMs. We elaborate our approach with two parts, which the first is composition, while the second is training algorithm.
105
108
107
4.1
Elements of AttendOut
108
Our training architecture is composed of three modules, A-Net (Attacker), D-Net (Defender) and G-Net (Generator). D-Net and A-Net are two identical models and trained simultaneously through standard gradient descent, while G-Net is a learnable dropout maker and trained through policy gradient. Now we elaborate each of them.
110
3

---

112
Defender Attacker As suggested, defender and attacker are two competitors playing game with each other on specific criteria, e.g. training accuracy, training loss. Specifically, D-Net and A-
113
114 Net are two identical self-attention empowered Prs.Ms, e.g. BERT, RoBERTa. However, they follow different dropout strategies. D-Net receives regular dropout as default specific models, while A- Net receives additional dropout decision from G-Net onto its corresponding attention layers.
115
116
Generator
G-Net acts dropout maker through generating mask matrix for each attention layer during training stage. As aforementioned, the common dropout strategies rely on randomness, which intends shut down the co-adaption but not powerful enough. However,
118
dropout maker an agent which is able to intelligently choose and learn dropout patterns for each sample. Specif- ically, after training for fixed number of steps, we conduct evaluation for both A-Net and D-Net. When A-Net obtains a higher score than D-Net, which means attacker wins the game, G-Net will be rewarded positively. When defender wins, G-Net will be punished with negative reward. In con- sequence, G-Net learns appropriate dropout patterns through the game between D-Net and A-Net, while assisting A-Net to win the game. On the other hand, A-Net needs be stronger when training under such powerful dropout, which makes it much more robust from over-fitting. Compared to search-based dropout, G-Net triggered by the difference between two model derivatives with and without dropout, instead of the final feedback on validation set, which makes it siddttoendd-ossibbee and sample-dependent.
120
122
123
124
125
126
127
128
129
130
The design of G-Net is the most delicate part, which is also a self-attention based model with iden- tical number of layers with D-Net and A-Net. However, we make several improvements. 1) G-Net only exports the attention scores from attention layers with no extra output layers, from which we apply Gumbel [35] [34] sample the actions obtain the dropout mask. 2) G-Net only makes one-head attention and share one group of parameters for all attention layers. 3) G-Net is excluded
132
133
134
135
of feed-forward layers, which may obscure the impact of self-attention [ [0]
136
4.2
Training with AttendOut
137
The core of training with AttendOut to find a way to optimize G-Net, which receives signals from the difference between D-Net and A-Net. Supposing there is a list of dropout actions by G-Net:
138
41:7 = {a1, 02, a3 ... .r
where T refers to the number of samples, for each action ae, G-Net may achieve reward ra. The optimization objective is to maximize the overall rewards of list a1,T, denoted as Rł that is:
140
J(Oc) [R)
where R = =T= Te. Since R is non-differentiable, use policy gradient update 0g as in [
T
Ve.. J(e))
E E (Voc log ((aylayt-1)al) rc)
The above equation could be approximated as:
1
m
Vag e log P(a, ((-1));)))))
m
For model with attention layers, each dropout decision is composed of n inner decisions of each layer. Additionally, each attention layer contains an attention matrix of x I, namely I? elements dropped or kept. Thus, denote dropout unit as d where refers to the 8th layer while refers to the jth element of the attention matrix.
144
146
However, nl2 dropout units bring huge space, which makes it impossible to calculate the joint prob- ability. To this end, we introduce the independence assumption that each dropout unit is independent with each other. Under the relaxation, make the following probability likelihood:
148

---

log ((a))((-n1)))))) 1
nl? Σ log (d (t-1)a fG)
150 where the summation Ex=1 Σj=1 is briefly denoted as Σij'
Thus, the final gradient could be formalized as:
V (
Voc log ld
(5)
ml
E
152
where is a baseline function moving average [<S Note that we do not apply additional regular- izers like LO and LI penalty, which impose unnecessary bias.
154
Algorithm summarizes the overall procedure of training PrLMs with AttendOut. We first initialize all three networks. Note that D-Net and A-Net should be kept identical at the beginning of each training step. A straightforward strategy is choose the better one to cover the other. To add randomness, we sample from D-Net and A-Net based on their evaluation performances, with higher probability for the better one. Then for each step. D-Net and A-Net are fed with the same data and updated via standard gradient descent, meanwhile each batch will be cached. After training for T steps, which we denote as a dropout step, both D-Net and A-Net are evaluated on additional validation samples, which could be development set data, noisy training data small split of train- ing data. In this paper, we simply use development set. For efficiency, we make random sampling on retrieve T samples for evaluation. Based the evaluation scores, G-Net is rewarded with (ri. r2, T3. ... Fr} and updated via Eq. H. At the end of each dropout step, the cached samples will be released and D-Net and A-Net will be ee-initialized.
155
156
158
160
162
184
Algorithm AttendOut
Output
Input: Attacker A, Defender D, Generator G, dropout step
1: initialize 0p. A fg. where 0D = 0
2: for each training step do
Output
3: @D + 8''
186
4: dropout A with G via Eq.
5: 0 < 8'
6: for each T steps do
7: evaluate D and A and reward
Query
Key
Value
8: Og 1- 0'3 via Eq. 5
9: initialize 0p. 0 for next step
Embedding
end for
11: end for
Figure 2: Architecture of G-Net.
187
Resource Usage We notice that training PrLMs with AttendOut may sacrifice time and memory cost. The detailed resource usage is shown in Appendix. Taking RoBERTa
example, the algorithm requires two RoBERTa models as well as a smaller self-attention based generator, which 1/3 of RoBERTa size. Considering cached samples, roughly speaking, AttendOut requires twice graphic memory as well as twice training time compared to a single model, which is a middle speed line between random-based dropout and neural architecture search (Dropout
189
170
172
[TC]] AttendOut << AutoDropout [[T]]. However, AttendOut contributes to remarkable performance gain compared to other attention dropout methods.
173
174
175
Our approach both feasible for both fine-tuning and pre-training stage of PrLMs but expensive for the latter. However, we serve for the most delicate part of concerned issue, since pre-training is generally done on large-scale data with modest training epochs, which makes less susceptible from over-fitting.
Pre-training
177
5

---

Table 1: Results (test / dev) of GLUE sub-tasks.
SST-2 Acc
MRPC F1
QNLI Acc
MNLI-mm Acc
CoLA Mec
Model
BERT
92.9 / 92.2
86.6 / 86.3
89.7 / 88.9
83.3/ 84.0
51.2 58.8
AtendOut
93.6 / 93.8
$8.1
90.2 / 91.1
84.2 / 84.6
57.4 / 60.9
RoBERTa
95.4 94.4 4
90.5 90.2
92.9 /2.00
86.1 / 86.6
61.3 / 62.5
AteenOOut
96.2 / 95.1
91.2/90.9
93.3 / 93.0
87.3/ 87.8
63.0 / 63,8
Table 2: Results of IMDB, CoNLLO3, PTB and SWAG respectively.
IMDB Acc
CoNLLO3 F1
PTB F1
SWAG Acc
Model
BERT
92.2
94.1
95.4
81.1
AttendOut
92.9
94.7
96.5
81.6
RoBERTa
93.6
94.5
96.6
83.8
AttendOut
94.2
95.2
97.3
84.1
Experimental Setup
179
180
We demonstrate the universal effectiveness of AttendOut on extensive natural language processing tasks. For all mentioned tasks, we apply our method on BERT [2] and its stronger variant RoBERTa [3] Our implementations are based on PyTorch using transformers [66]. For further training details, please refer Appendix.
181
185
184
Our experiments include: (1) natural language understanding: General Language Understanding Evaluation (GLUE) benchmark [37]] a collection of nine natural language understanding tasks (here we experiment on five of them, SST-2, MRPC, QNLI, MNLI-mm and CoLA; (2) document clas- sification: IMDB [[8], sentiment analysis dataset where about 15% of the documents are longer than 512 word- pieces; (3) named entity recognition: CoNLL2003 [[9];; (4) part-of-speech tag- ging: English Penn Treebank (PTB) [ (5) multiple choices question answering: SWAG [[]]]
185
186
187
188
189
190
We report both test and development results for GLUE sub-tasks since the large bias between them,
191
while development results only for the other tasks.
192
Note that we only adjust the dropout steps and keep all other parameters the same for strict fair comparison. For example, the parameters training with AttendOut including both D-Net and A-Net.
in RoBERTa identical with what use in
194
195
6
Results
196
6.1 Significance Analysis
197
Pictorially in Table RoBERTa is strong enough as it outperforms BERT by big margin, while AttendOut empowered RoBERTa still outperforms it on all five GLUE sub-tasks. For small-scale datasets, which are more likely to over-fit, AttendOut helps unfold remarkable performance gain (12.1% / 3.5% over BERT CoLA, 1.7% / 1.4% over BERT MRPC). However, for large- scale one like MNLI, which tends to be more stable, AttendOut still produces considerable boost, (1.4% / 1.4% over RoBERTa, 1.1% 0.7% over BERT).
199
201
203
Furthermore, AttendOut shown universally effective as Table For POS Tagging, BERT and RoBERTa have achieved very strong baselines, while AttendOut empowered ones are even stronger, (1.1 % over BERT on PTB). Similar results are seen on document classification and NER.
205
For SWAG, however, AttendOut seems weakly effective (0.6% % over BERT, 0.4% over RoBERTa).

---

1.0
1.0
0.6
0.6
0.6-
0.0
0.0
0.0
100
200
300
400
500
100
200
300
400
500
600
100
200
300
400
500
Convergence Figure depicts the accuracy trends of RoBERTa on SST-2, QNLI, MNLI respec- tively. Due to a stronger dropout module, the one with AttendOut tends to fall behind (SST-2, MNLI) at the beginning of training. However, model becomes stronger since the second epoch (SST-2, QNLI). Especially on MNLI, RoBERTa obtains better results in the first two epochs and it (a) SST-2
(b) QNLI
(c) CoLA
Figure Dropout probabilities on specific attention layers over training steps.
93.0
95.0
84.2
92.8
94.8
92.6
4-
92.4
85.6
85.4
94.2
RoBERTa RoR+And.O
RoBERT
83.
RoBERTs
94.0
ReB-ArdO
83.0
RoB+AidO.
91.6
(a) SST-2
(b) QNLI
(c) MNLI
Figure 4: Convergence over training epochs.
207
6.2
Visual Analysis
208
Dropout Patterns Another concerned issue the dropout proportions by AttendOut. Figure
209
depicts the patterns on several datasets. We may find several interesting phenomenons. First, the overall patterns largely differ from datasets, which fair since AttendOut is sample-dependent.
210
However, we may observe something in common. Overall, the lower layers take higher dropout probabilities. For example on QNLI, the first layer almost remains steady with the probability of 0.55 during the training process, while the fourth one continuously decays in a higher rate. Intu- itively, the first three layers undertake a similar trend in each dataset, while there might be an up and down for the fourth one as in SST-2 and CoL.A. Especially for CoLAA, we see unusual high dropout probabilities in the final period (around 0.9), which are close to complete dropout. We notice that CoLA is small set with 8500 training samples, on which SAN model is more inclined to suffer from over-iitting. Therefore, PrLM on CoLA encounters more intensive dropout through AttendOut.
212
213
214
216
218
219
220
221
222
223
drops the last one, while with AttendOut, the performance steadily rising for all three epochs.
224
7
Ablation Study
225
In this section, we conduct further experiments to demonstrate the effectiveness AttendOut. Due to space limitation, conduct corresponding experiments development sets only.
227
7.1
Attention Dropout
Vanilla Dropout We conduct comparison with vanilla Dropout [03]] in which we dropout the
229
attention matrix for all layers with Bernoulli distribution of p. Here, we choose the dropout proba- bilities in (0.1, 0.2).
7

---

Table 3: Comparison of AttendOut, vanilla Dropout and LryerDrop.
Model
CoLA
QNLI
MNLI-mm
RoBERTa
62.5
92.0
86.6
+ Vanilla
61.3
92.2
86.9
+ AttendOut
63.8
93.1
87.8
+ LayerDrop
62.1
92.6
87.1
+ Layer.ayerrrrpp
64.2
92.7
87.3
Table 4: Comparison of AttendOut and scheduled Bernoulli dropout.
Model
CoLA
QNLI
SWAG
RoBERTa
62.5
92.0
83.8
Scheduler
63.3
92.6
83.6
AttendOut
63.8
93.1
84.1
We also compare with LayerDrop [2%], which focuses on skipping the entire encoder blocks, Inspired it, we design another strategy which randomly skips attention layers via Eq. For fair enough comparison, set the dropout probabilities to 0.2 for both methods, following the settings in [29]].
LayerDrop
232
234
Intuitively in Table BL vanilla Dropout with fixed probability does not produce noticeable gain (1.9% bellow RoBERTa on CoLA).. However, AttendOut shows powerful advantage (4. %,, 1.0% and 1.0% over vanilla Dropout CoLA, QNLI and MNLI), which stresses the necessity of dynamic dropout patterns rather than fixed static one. On the other hand, both layer-level regularizers are effective, while attention LayerDrop performs stronger and more stable on all the three. Especially CoLA, it outperforms RoBERTa by 1.7 points, while LayerDrop meets a performance drop, which demon- effective regularizer than removing the entire
236
238
240
strates that removing the attention layers act
242
SAN block as for self-attention based models.
Pattern Approximation
243
Guided by AttendOut, design dropout scheduler, in which utilize piece-wiss linearity to approximate the real curves as depicted Figure Taking QNLI as an example, we initialize the dropout probabilities to 0.6 for all attention layers and set specific slope for each of them. Note that here the corresponding mask matrices are randomly--ennerted and subject to Bernoulli distribution. In AttendOut, however, the distribution learned dynamically through self-attention of G-Net.
245
247
249
As shown in Table EL RoBERTa with scheduled Bernoulli dropout works surprisingly well on both CoLA and QNLI, which outperforms RoBERTa by 0.8 and 0.6 points respectively, closer to At- tendOut, even if the strategy here is random-based and much looser. The guided scheduled dropout helps unfold the correctness the dynamic dropout patterns learned by AttendOut as well as self-attention based dropout maker.
251
253
8
Conclusion
256
This paper focuses on the co-adaption problem of deep self-attention networks, and presents a novel dropout method onto self-attention empowered pre-trained language models. Extensive experiments on multiple natural language processing tasks demonstrate that our proposed approach is universal and qualified to enable more robust task-specific tuning, which contributes to much stronger state- of-the-arts. We probe into the learned dropout patterns on different tasks, which empirically guide us to the very needed dynamic attention dropout design.
257
258
260

---

References
262
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gareett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
264
265
286
267
268
[2] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidi- rectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, NAACL-HLLT 2019, Minneupolis, MN, USA, June 2-7, 2019, Volume (Long and Short Pupers), pages 4171-4186. Association for Computational Linguis- tics, 2019.
269
270
272
273
274
[3] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeutlemoer. and Veselin Stoyanov. Roberta: robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. 2019.
275
276
277
[4] Zhenzhong Lan, Mingda Chen. Sebastian Goodman, Kevin Gimpel. Piyush Sharma, and Radu Soricut. ALBERT A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICI. 2020, Addis Ababa, Ethiopia, April 26-30, 2020. Open- Review.net, 2020.
278
279
281
[5] Kevin Clark, Minh- Thang Luong, Quoe V. Le, and Christopher D. Manning. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Rep- resentations, ICIR 2020, Addis Ahaba, Ethiopio, April 26-30, 2020. OpenReview.net, 2020.
282
284
[6] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. [DEBERTA]: (DECODING)- (enhanced) [bert] [with) {disentangled) [attention]. In International Conference on Learning Repre- sentations, 2021.
285
287
7 Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
288
289
[8]
Weiqiu You, Simeng Sun, and Mohit lyyer. Hard-coded gaussian attention for neural machine translation. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7689-7700. Association for Computational Linguistics, 2020.
290
292
293
[9] Yi Tay. Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models. CoRR, abs/2005.00743,, 2020.
294
295
[10] Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. CoRR, abs/2006.04768,, 2020.
296
297
[II]
Yihe Dong, Jcan-Baptiste Cordonnier, and Andreas Loukas, Attention is not all you need: Pure attention loses rank doubly exponentially with depth. CoRR, abs/2103.03404, 2021.
298
299
12]
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabhar- wal, and Ashish Verma. Power-bert: Accelcrating BERT inference via progressive word-vector climi- nation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3690-3699. PMLR, 2020.
300
301
302
304
[13] Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929- 1958, 2014.
305
306
307
[14] Li Wan, Matthew D. Zeiler, Sixin Zhang, Yann LeCun, and Rob Fergus. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings. pages 1058-1066. JMg..org, 2013.
309
310
[15] Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. Sg-net: Syntax- guided machine reading comprehension. In The Thirty-Fourd Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secome
312
313
Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pages 9636 964669963.444 AAAI Press, 2020.
314

---

316
[16] Hongqiu Wu, Hai Zhao, and Min Zhang. Code summarization with structureiiducced transformer. arXiv preprini (tti::2002..177111 2020.
[17] Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations ICLR 2017. Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
319
320
321
[18] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: diffcrentiable architecture search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
322
324
[19] Hieu Pham and Quoe V. Le. Autodropout: Learning dropout patterns to regularize deep networks. CoRR, abs/2101.01 11,161111666.11 2021.
325
[00]
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: An- mual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pages 971-980, 2017.
327
328
329
[21] Avrim Blum, Nika Haghtalab, and Ariel D. Procaccia. Variational dropout and the local reparameteriza- tion trick. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Gareet,, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Infor- mation Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2575-2583, 2015.
332
335
334
335
336
[22]
Dmitry Molchanov, Arsenii Ashukha, and Dmitry P. Vetrov. Variational dropout sparsifies deep neural networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia. 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1498-2503 PMLR, 2017.
338
339
[23] Sungrac Park, 340
Park, Su-Jin Shin, and Il-Chul Moon. Adversarial dropout for supervised and semi semi-supervised learning. In Sheila A. Mcllraith and Kilian Q Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 3917-3924. AAAI Press, 2018.
341
342
344
345
346
[24] Hojjat Salehinejad and Shahrokh Valaee. Edropout: Energy-based dropout and pruning of deep neural networks. CoRR, abs/2000.00427.. 2020.
347
348
[25] Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3581-3590, 2017.
350
351
352
[26]
Shahin Boluki, Randy Ardywibowo, Siamak Zamani Dadaneh. Mingyuan Zhou, and Xiaoning Qian. Learnable bernoulli dropout for bayesian deep learning. In Silvia Chiappa and Roberto Calandra, edi- tors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Pulermo, Sicily, Italy]. volume 108 of Proceedings of Machine Learning Research, pages 3905 3916. PMLR, 2020.
353
354
356
357
[27] Mingzhang Yin and Mingyuan Zhou. ARM: nngment-reinforce-merg gradient for stochastic binary networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
358
359
360
[28] Xinjie Fan, Shujian Zhang, Korawat Tanwisuth, Xiaoning Qian, and Mingyuan Zhou. Contextual dropout: An efficient sample-dependent dropout module. CoRR, abs/2103.04181, 2021.
362
[29] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with struc- tured dropout. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiupia. April 26-30, 2020. OpenReview.net, 2020.
384
385
[30] Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou, and Ke Xu. Scheduled drophead: regularization method for transformer models. In Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16- 20 November 2020, pages 1971-1980 Association for Computational Linguistics, 2020.
386
387
10

---

389
[31] Zewei Sun, Shujian Huang, Xinyu Dai, and Jiajun Chen. Alleviating the inequality of attention heads for neural machine translation. CoRR, 0bs/2209.009672 2020.
[32]
Zhen Wu, Lijun Wu, Meng Qi. Yingee Xia, Shufang Xie, Tao Qin, Xinyu Dai, and Tie- Yan Liu. Unidrop: A simple yet effective technique to improve transformer without extra cost. In Proceedings of the The 2021 Conference of the North American Chapter of the Association for Computational Linguistics -Human Language Technologies, Volume I (Long Papers), 2021.
372
373
374
375
[33] Chris J. Maddison, Daniel Tarlow, and Tom Minka. A sampling. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3086-3094, 2014.
376
378
379
[34] Xinwei Geng, Longyue Wang, Xing Wang, Bing Qin, Ting Liu, and Zhaopeng Tu. How does selective mechanism improve self-attention networks? In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguis- tics, ACL 2020, Online, July 5-10, 2020, pages 2986-2995. Association for Computational Linguisties. 2020.
380
381
382
[35] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Leam.. 8:229-256, 1992.
385
[36] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the the- art natural language processing. In Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 45, Online, October 2020. Association for Computational Linguistics.
387
388
389
391
382
[37]
Alex Wang, Amanprect Singh, Julian Michacl, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. 7th International Conference on Learning Reprexentations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net, 2019.
393
394
385
396
[38] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors, The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA, pages 142-150. The Association for Computer Linguistics, 2011.
398
389
400
401
[39] Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language- independent named entity recognition. In Walter Daelemans and Miles Osborne, editors, Proceedings of the Seventh Conference Natural Language Learning, CoNLL 2003, Held in cooperation with HLT- NAACL 2003, Edmonton, Canada, May 31 June 1, 2003, pages 142 147. ACL, 2003.
402
404
405
[40]
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewiez. Building a large annotated corpus of english: The penn treebank. Comput. Linguistics, 19())313-330, 1993.
406
407
[41] Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: large-scale adversarial dataset for grounded commonsense inference. In Ellen Riloff, David Chiang, Julia Hockenmaier. and sn'ichi Tsujii editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing, Brussels, Belgium, October 31 November 4, 2018, pages 93-104. Association for Computational Linguisties, 2018
408
409
410
Checklist
412
413
1. For all authors...
414
(a) Do the main claims made the abstract and introduction accurately reflect the paper's contri- butions and scope? [Yes]
415
416
(b) Did you describe the limitations of your work? [Yes] See Section
417
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms them?
419
[Yes]
2. If you including theoretical results...

---

(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
422
(b) Did you include complete proofs of all theoretical results? [No]
423
3. If you ran experiments...
424
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes] See supplemental material.
428
(b) Did you specify all the training details (e.g., data splits, hyperparameters. how they were cho- scn)? [Yes] See Section 8.2 and appendix.
(c) Did you report error hars (e.g., with respect to the random seed after running experiments mul- tiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs,
internal eluster, cloud provider)? [Yes] See Section EM and appendix.
432
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
433
(a) your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
436
(d) Did you discuss whether and how consent was obtained from people whose data you're rou'r us-
ing/eurating? [N/A]
438
(e) Did you discuss whether the data you are using/curating contains personally identifiable infor- mation offensive content? [No]
440
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable?
442
[N/A]
(h) Did you descrihe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
444
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
446
12


Reviews:
[{"review_id": "kDnz1WBzgSy", "reviewer": "Reviewer_xozH", "summary": "This work proposes a new adaptive dropout technique for large self-attention pre-trained language models. Their method, called AttendOut has three main elements, an Attacker, a Defender and a Generator. The Generator is trained through policy gradient to (adaptively) generate the dropout which is used in the Attacker network. The Generator network will get positive rewards when the Attacker network performs better than the Defender network in an evaluation phase. The authors test their method on a subset of tasks from GLUE, document classification, named entity recognition, POS tagging and multiple choice question answering.\nTheir method, which could be applied universally, achieves better results (compared to vanilla models) on the aforementioned tasks when applied on BERT and RoBERTa models.   \n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors have done a good job explaining different components in their method. The algorithm explains their procedure of applying adaptive and learned dropout clearly. \n\nThe method evaluates Defender and Attacker networks every $T$ steps. One part which seems to be missing from the paper is a discussion of the relationship between $T$ and the reward that the Generator network receives. How do different values of $T$ affect how the Generator is trained?\n\nHow much training is required for the Generator to provide the Attacker with better than vanilla Dropout?\n\nWere the rest of the tasks in GLUE tested on?\n\nIt would also be great if the authors reported error bars for different seeds to be able to realize the significance of the proposed dropout method.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A few are discussed in the main review section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "JzBJB24JiBq", "reviewer": "Reviewer_fND3", "summary": "This paper points out that the self-attention network can easily suffer from the co-adaption problem. To address the co-adaption problem, this paper introduces a novel dropout method, named Attention differentiable dropOut, for robust task-specific tuning. Experimental results also demonstrate the effectiveness of the proposed methods. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper introduces an attacker-defender-generator architecture for attention differentiable dropout. The idea is interesting and experimental results verify the effectiveness of the proposed method.\n\n====\n\nWeaknesses:\n1. The motivation of this paper is quite unclear. In the introduction, this paper first argues that self-attention remains a black box problem, and then introduces some works for self-attention by using attention masks, and some other works for linear self-attention. I cannot understand the connection between these works. And then, authors attribute these problems to the co-adaptation. However, these related works are not for addressing this problem, and the authors also do not provide the reason or relationship between these papers and co-adaption, let alone black box problem as mentioned by the authors. I think authors should re-organize the introduction to highlight the motivation.  \n2. Just as aforementioned, authors have attributed these problems to co-adaptation, some theoretical proof should be provided to prove the connection between co-adaption and self-attention, rather than only algorithm designing.\n\n===\n\nQuestions:\n\n1. Why not report all tasks of GLUE datasets? I think it is necessary to report all GLUE tasks to validate the robustness of the proposed method. \n2. In my understanding, the figure 3 has shown the dropout probabilities of {1,2,3,4}-attention layer. Why not report some high layers? (For example, you can uniformly choose 1, 4, 7, 10 layer).\n3. Although authors have demonstrated that the proposed method is feasible for pre-training and fine-tuning stage of pre-trained language model, some papers [2] have validated the dropout is unnecessary for large-scale pre-training and close dropout may be better for pre-training. So what is your practical value under this scenario?\n4. It is unfair to compare your method with the NAS-based methods in terms of resource usage, since the searching procedure of NAS is always time-comsuming. The design of attacker and defender brings me a feeling of ensemble. Therefore, it is necessray to compare AttendOut with the vanilla dropout under the same computation. \n\n\n===\n\nComments:\n1. Line 232, \",\" => \".\"\n\n\n[1] Big Bird: Transformers for Longer Sequences\n[2] ALBERT: A Lite BERT for Self-supervised Learning of Language Representations", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "DtSoJSUyeC2", "reviewer": "Reviewer_b59b", "summary": "This paper proposes a novel dropout strategy for self-attention, namely AttendOut. AttendOut trains 3 independent networks. \n- Network G generates a $nxn$ dropout mask for the self-attention matrix for each layer conditioned on the input data.  \n- Network D is trained by applying the dropout mask generated by G\n- Network A is trained by applying the random dropout mask over self-attention. \nNetwork D and A are trained with the task-specific objective. Importantly, the network G is trained with a reward defined by the advantage of D over A and optimized by the policy gradient. \n\nEmpirically, the proposed shows clear gains under language pretraining and fine-tuning.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed idea of learning to perform input-dependent dropout is interesting. The empirical performance of AttendOut is reasonably good across a wide variety of tasks. \n\nFrom the current version, there are some unclear points.\n- Is the AttendOut used during pretraining or only used in fine-tuning (or vice versa)?\n\nAs acknowledged by the authors, the main concern of the proposed method is the additional computational cost.\n- If it's also used for pretraining, a naive but strong baseline is to use the 2x computation consumed by AttendOut to pre-train a model without AttendOut.\n- If not, the main concern would be the 2x memory issue involved in joint training may also restrict the empirical application of this method.\n\nAnother concern I have is the role of AttendOut during fine-tuning. Specifically, the G-Net may contribute to the final performance as a complement attention network (though the output is binarized). Hence, the question would be, how much does the G-Net really serve the purpose of being a regularization versus an additional network.\n\nIn addition, does the proposed model have a high variance in terms of both training stability and final performances given that it involved RL optimization?\n\nFinally, the baseline performance reported in this work is a bit different from the original paper. For example, if we use the numbers from RoBERTa paper, the comparison will be like:\n\nModel\t\tMNLI\tQNLI\tSST-2\tMRPC\tCoLA\n\nroberta.base\t87.6 \t92.8 \t94.8 \t90.2 \t63.6\n\nattention-out\t87.8 \t93.0 \t95.1 \t90.9 \t63.8\n\nThen, the gain looks very limited. \n\nIn summary, the work presents an adaptive dropout method that shows promising results on downstream tasks. There are some unclear points in the current version that clouds the judgment. Also, the method does add some computational complexity to the model training.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation is well discussed and I don't see any concern w.r.t. negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "2zl9K4AlLFr", "reviewer": "Reviewer_Sz77", "summary": "The paper proposes adaptive dropout learned through reinforcement learning applied to attention in Transformer models.  Improved performance in finetuning is demonstrated on several NLP tasks.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Summary\n\nThe paper proposes adaptive dropout learned through reinforcement learning applied to attention in Transformer models.  Improved performance in finetuning is demonstrated on several NLP tasks.\n\n\n\n1) Originality \n\nI find the combination of attention dropout learned by RL novel and interesting.\n\n\n2) Quality\n\n2a) Strong points\n\n* I find the general idea of controlling dropout through RL an interesting approach that potentially can save a lot of time when training new models.\n\n* I like per layer dropout rate analysis from Sec 6.2.\n\n* I also like the step made in Sec 7.2 where simpler dropout formulation is reverse engineered based on behavior of the RL policy.\n\n\n2b) Weak points\n\n* It would be nice to have confidence intervals for results in Tab 1 - 4. Finetuning usually doesn't require that much resources. I think that doing e.g. three runs should be feasible and it would greatly improve the empirical message of the paper.\n\n* In my opinion comparing validation set accuracy obtained with AttendOut to other metrics isn't completely fair since AttendOut is using it as its reward. Therefore it is easier for AttendOut to overfit even on a valid set than for e.g. standard dropout with early stopping on valid. I still think that the method is beneficial since this argument doesn't hold for the test set. However the paper should mention this methodological issue.\n\n* Line 192 says that dropout steps were adjusted for AttendOut. How many values were tried for each dataset? ​This leads me to the most important question. In high level this paper proposes to replace one parameter (dropout rate) by a learned adaptive module that: 1) makes training 2x slower (that might be fine) and 2) has another set of hyperparams. The question is, how much care has to be given to tuning dropout steps (the only hyperparam of AttendOut)? How stable is the training? In other words, with exactly the same compute budget would it be better to use AttendOut or some other dropout variant? I feel that the message of the paper would be greatly improved if it would be clear that for practitioners this method is clearly better than vanila dropout. At the moment this isn't the case because: A) GLUE numbers for official BERT and RoBERTa are better and it isn't obvious where the difference is comming from and B) it isn't clear how difficult it is to tune AttendOut. However both of these points can be hopefully easily adressed in the text and it might improve rating of the paper.\n\n3) Clarity\n\n* Line 81 --- Various works show that there hides implicit but highly needed semantic clues. --- If there are various works some citations would be nice.\n\n* Line  141-2 --- a_{(t−1):1} should be a_{1:(t−1)} ?\n\n* Alg 1 --- I believe that various theta' weren't defined in the text before. I assume that those are params after grad update but it isn't explicitly stated for theta_D and theta_A.\n\n* I can't match differences from lines 200 - 202 to Tab 1. The same for line 206 and Tab 2 and for paragraph starting in line 235. Has the tables changed after writing the text?\n\n* Fig 3 and 4: label axes of each figure.\n\n\n\n4) Significance\n\nThe proposed adaptive dropout can be potentionaly significant for practitioners if it proves to work consistently on other datasets and it will reduce need for tuning dropout rates by hand/hyperparam optimization. In that case even ~2x slowdown of the method would be completely acceptable since hyperparam tuning often needs even more resources. \n\n\nQuestions:\n\n* Why is the paper using a subset of GLUE tasks and not the full set?\n\n* Fig 4 --- is that for dev? Why only three epochs of accuracy for QNLI and MNLI when accuracy is still improving at that point? Has it decreased in the fourth epoch? \n\n* I am wondering whether the 0.1 and 0.2 dropout rates used in sec 7.1 for vanilla dropout are enough. E.g. Tab 3 shows that for QNLI and CoLA higher dropout rates are probably better.\n\n* For both BERT and RoBERTa result on the official GLUE leaderboard are higher than those reported in Tab 1 (including AttendOut). Where is this difference coming from? Is it due to different implementation or was the finetuning protocol different?\n\n\n\nMinor language issues\n\n* Line 14 ---  a series successful pre-trained -> a series OF successful pre-trained\n\n* Line 175 --- Our approach is both feasible for both -> approach is feasible for both\n\n* Line 232 --- Inspired of it -> Inspired BY it?\n\n\n\n**************************************************\nPost rebutal update\n**************************************************\n\nSince many of my questions were resolved I am increasing my final score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One limitation of the paper that wasn't discussed in depth is stability of the training.\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns regarding this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission introduces an approach to preventing overfitting when fine-tuning large pre-trained NLP models. A network is trained to predict a dropout mask to improve validation performance, optimizing the network with policy gradients. Reviewers agree that the approach is interesting, and appears to be effective. However, they raised a number of concerns, including the lack of confidence intervals on results, the number of new hyperparameters introduced, only reporting results on a subset of the GLUE tasks, and several issues with the presentation of the work. I think that the author responses do promise significant improvements here, but the current submission does not meet the bar for NeurIPS.
Decision: Reject