Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a novel dropout strategy for self-attention, namely AttendOut. AttendOut trains 3 independent networks. 
- Network G generates a $nxn$ dropout mask for the self-attention matrix for each layer conditioned on the input data.  
- Network D is trained by applying the dropout mask generated by G
- Network A is trained by applying the random dropout mask over self-attention. 
Network D and A are trained with the task-specific objective. Importantly, the network G is trained with a reward defined by the advantage of D over A and optimized by the policy gradient. 

Empirically, the proposed shows clear gains under language pretraining and fine-tuning.
main_review: The proposed idea of learning to perform input-dependent dropout is interesting. The empirical performance of AttendOut is reasonably good across a wide variety of tasks. 

From the current version, there are some unclear points.
- Is the AttendOut used during pretraining or only used in fine-tuning (or vice versa)?

As acknowledged by the authors, the main concern of the proposed method is the additional computational cost.
- If it's also used for pretraining, a naive but strong baseline is to use the 2x computation consumed by AttendOut to pre-train a model without AttendOut.
- If not, the main concern would be the 2x memory issue involved in joint training may also restrict the empirical application of this method.

Another concern I have is the role of AttendOut during fine-tuning. Specifically, the G-Net may contribute to the final performance as a complement attention network (though the output is binarized). Hence, the question would be, how much does the G-Net really serve the purpose of being a regularization versus an additional network.

In addition, does the proposed model have a high variance in terms of both training stability and final performances given that it involved RL optimization?

Finally, the baseline performance reported in this work is a bit different from the original paper. For example, if we use the numbers from RoBERTa paper, the comparison will be like:

Model		MNLI	QNLI	SST-2	MRPC	CoLA

roberta.base	87.6 	92.8 	94.8 	90.2 	63.6

attention-out	87.8 	93.0 	95.1 	90.9 	63.8

Then, the gain looks very limited. 

In summary, the work presents an adaptive dropout method that shows promising results on downstream tasks. There are some unclear points in the current version that clouds the judgment. Also, the method does add some computational complexity to the model training.
limitations_and_societal_impact: The limitation is well discussed and I don't see any concern w.r.t. negative societal impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper points out that the self-attention network can easily suffer from the co-adaption problem. To address the co-adaption problem, this paper introduces a novel dropout method, named Attention differentiable dropOut, for robust task-specific tuning. Experimental results also demonstrate the effectiveness of the proposed methods. 
main_review: Strengths:
1. This paper introduces an attacker-defender-generator architecture for attention differentiable dropout. The idea is interesting and experimental results verify the effectiveness of the proposed method.

====

Weaknesses:
1. The motivation of this paper is quite unclear. In the introduction, this paper first argues that self-attention remains a black box problem, and then introduces some works for self-attention by using attention masks, and some other works for linear self-attention. I cannot understand the connection between these works. And then, authors attribute these problems to the co-adaptation. However, these related works are not for addressing this problem, and the authors also do not provide the reason or relationship between these papers and co-adaption, let alone black box problem as mentioned by the authors. I think authors should re-organize the introduction to highlight the motivation.  
2. Just as aforementioned, authors have attributed these problems to co-adaptation, some theoretical proof should be provided to prove the connection between co-adaption and self-attention, rather than only algorithm designing.

===

Questions:

1. Why not report all tasks of GLUE datasets? I think it is necessary to report all GLUE tasks to validate the robustness of the proposed method. 
2. In my understanding, the figure 3 has shown the dropout probabilities of {1,2,3,4}-attention layer. Why not report some high layers? (For example, you can uniformly choose 1, 4, 7, 10 layer).
3. Although authors have demonstrated that the proposed method is feasible for pre-training and fine-tuning stage of pre-trained language model, some papers [2] have validated the dropout is unnecessary for large-scale pre-training and close dropout may be better for pre-training. So what is your practical value under this scenario?
4. It is unfair to compare your method with the NAS-based methods in terms of resource usage, since the searching procedure of NAS is always time-comsuming. The design of attacker and defender brings me a feeling of ensemble. Therefore, it is necessray to compare AttendOut with the vanilla dropout under the same computation. 


===

Comments:
1. Line 232, "," => "."


[1] Big Bird: Transformers for Longer Sequences
[2] ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
limitations_and_societal_impact: yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work proposes a new adaptive dropout technique for large self-attention pre-trained language models. Their method, called AttendOut has three main elements, an Attacker, a Defender and a Generator. The Generator is trained through policy gradient to (adaptively) generate the dropout which is used in the Attacker network. The Generator network will get positive rewards when the Attacker network performs better than the Defender network in an evaluation phase. The authors test their method on a subset of tasks from GLUE, document classification, named entity recognition, POS tagging and multiple choice question answering.
Their method, which could be applied universally, achieves better results (compared to vanilla models) on the aforementioned tasks when applied on BERT and RoBERTa models.   

main_review: The authors have done a good job explaining different components in their method. The algorithm explains their procedure of applying adaptive and learned dropout clearly. 

The method evaluates Defender and Attacker networks every $T$ steps. One part which seems to be missing from the paper is a discussion of the relationship between $T$ and the reward that the Generator network receives. How do different values of $T$ affect how the Generator is trained?

How much training is required for the Generator to provide the Attacker with better than vanilla Dropout?

Were the rest of the tasks in GLUE tested on?

It would also be great if the authors reported error bars for different seeds to be able to realize the significance of the proposed dropout method.

limitations_and_societal_impact: A few are discussed in the main review section.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes adaptive dropout learned through reinforcement learning applied to attention in Transformer models.  Improved performance in finetuning is demonstrated on several NLP tasks.


main_review: Summary

The paper proposes adaptive dropout learned through reinforcement learning applied to attention in Transformer models.  Improved performance in finetuning is demonstrated on several NLP tasks.



1) Originality 

I find the combination of attention dropout learned by RL novel and interesting.


2) Quality

2a) Strong points

* I find the general idea of controlling dropout through RL an interesting approach that potentially can save a lot of time when training new models.

* I like per layer dropout rate analysis from Sec 6.2.

* I also like the step made in Sec 7.2 where simpler dropout formulation is reverse engineered based on behavior of the RL policy.


2b) Weak points

* It would be nice to have confidence intervals for results in Tab 1 - 4. Finetuning usually doesn't require that much resources. I think that doing e.g. three runs should be feasible and it would greatly improve the empirical message of the paper.

* In my opinion comparing validation set accuracy obtained with AttendOut to other metrics isn't completely fair since AttendOut is using it as its reward. Therefore it is easier for AttendOut to overfit even on a valid set than for e.g. standard dropout with early stopping on valid. I still think that the method is beneficial since this argument doesn't hold for the test set. However the paper should mention this methodological issue.

* Line 192 says that dropout steps were adjusted for AttendOut. How many values were tried for each dataset? ​This leads me to the most important question. In high level this paper proposes to replace one parameter (dropout rate) by a learned adaptive module that: 1) makes training 2x slower (that might be fine) and 2) has another set of hyperparams. The question is, how much care has to be given to tuning dropout steps (the only hyperparam of AttendOut)? How stable is the training? In other words, with exactly the same compute budget would it be better to use AttendOut or some other dropout variant? I feel that the message of the paper would be greatly improved if it would be clear that for practitioners this method is clearly better than vanila dropout. At the moment this isn't the case because: A) GLUE numbers for official BERT and RoBERTa are better and it isn't obvious where the difference is comming from and B) it isn't clear how difficult it is to tune AttendOut. However both of these points can be hopefully easily adressed in the text and it might improve rating of the paper.

3) Clarity

* Line 81 --- Various works show that there hides implicit but highly needed semantic clues. --- If there are various works some citations would be nice.

* Line  141-2 --- a_{(t−1):1} should be a_{1:(t−1)} ?

* Alg 1 --- I believe that various theta' weren't defined in the text before. I assume that those are params after grad update but it isn't explicitly stated for theta_D and theta_A.

* I can't match differences from lines 200 - 202 to Tab 1. The same for line 206 and Tab 2 and for paragraph starting in line 235. Has the tables changed after writing the text?

* Fig 3 and 4: label axes of each figure.



4) Significance

The proposed adaptive dropout can be potentionaly significant for practitioners if it proves to work consistently on other datasets and it will reduce need for tuning dropout rates by hand/hyperparam optimization. In that case even ~2x slowdown of the method would be completely acceptable since hyperparam tuning often needs even more resources. 


Questions:

* Why is the paper using a subset of GLUE tasks and not the full set?

* Fig 4 --- is that for dev? Why only three epochs of accuracy for QNLI and MNLI when accuracy is still improving at that point? Has it decreased in the fourth epoch? 

* I am wondering whether the 0.1 and 0.2 dropout rates used in sec 7.1 for vanilla dropout are enough. E.g. Tab 3 shows that for QNLI and CoLA higher dropout rates are probably better.

* For both BERT and RoBERTa result on the official GLUE leaderboard are higher than those reported in Tab 1 (including AttendOut). Where is this difference coming from? Is it due to different implementation or was the finetuning protocol different?



Minor language issues

* Line 14 ---  a series successful pre-trained -> a series OF successful pre-trained

* Line 175 --- Our approach is both feasible for both -> approach is feasible for both

* Line 232 --- Inspired of it -> Inspired BY it?



**************************************************
Post rebutal update
**************************************************

Since many of my questions were resolved I am increasing my final score.
limitations_and_societal_impact: One limitation of the paper that wasn't discussed in depth is stability of the training.
review_title: null

Meta-review: The submission introduces an approach to preventing overfitting when fine-tuning large pre-trained NLP models. A network is trained to predict a dropout mask to improve validation performance, optimizing the network with policy gradients. Reviewers agree that the approach is interesting, and appears to be effective. However, they raised a number of concerns, including the lack of confidence intervals on results, the number of new hyperparameters introduced, only reporting results on a subset of the GLUE tasks, and several issues with the presentation of the work. I think that the author responses do promise significant improvements here, but the current submission does not meet the bar for NeurIPS.
Decision: Reject