Not All Attention Is All You Need

Beyond the success story of pre-trained language models ( PrLMs ) in recent natu-1 ral language processing , they are susceptible to over-fitting due to unusual large2 model size . To this end , dropout serves as a therapy . However , existing methods3 like random-based , knowledge-based and search-based dropout are more general4 but less effective onto self-attention based models , which are broadly chosen as5 the fundamental architecture of PrLMs . In this paper , we propose a novel dropout6 method named AttendOut to let self-attention empowered PrLMs capable of more7 robust task-specific tuning . We demonstrate that state-of-the-art models with elab-8 orate training design may achieve much stronger results . We verify the universal-9 ity of our approach on extensive natural language processing tasks.10 1 Introduction11 Self-attention network ( SAN ) empowered models like Transformer [ 1 ] have achieved remarkable12 success in recent natural language processing , which have been broadly chosen as basic architec-13 ture in a series successful pre-trained language models ( PrLMs ) such as BERT [ 2 ] , RoBERTa [ 3 ] ,14 ALBERT [ 4 ] , ELECTRA [ 5 ] , DeBERTa [ 6 ] and GPT [ 7 ] .15 SAN has drawn a great deal of curiosity on its conceptually simple but powerful attention mecha-16 nism . However , SAN still remains a black box and more and more works attempt to unveil its inner17 principle , where the biggest mystery lies in its attention matrix . Our work is inspired by several18 recent discoveries which turn our views up and down . [ 8 , 9 ] show that fixed Gaussian or even ran-19 dom alignment attention matrix may rival standard SAN , while more recently , [ 10 , 11 ] prove that20 SAN may encounter a rank collapse with deepening of layers . A more concrete explanation is in-21 formation diffusion [ 12 ] , which states that the input vectors are progressively assimilating through22 continuously making self-attention . We attribute these problems to the sever co-adaption [ 13 ] be-23 tween attention elements , a form of over-fitting onto SAN . As a result , self-attention empowered24 PrLMs hardly bring into their full play , especially for the fine-tuning stage , where task-specific data25 is always with limited capacity.26 Dropout [ 13 ] serves as a therapy to deal with the problem , by randomly shutting down a set of units27 during training stage . When specified on self-attention , dropout is equivalent to adding attention28 mask to the attention matrix . However , random-based dropout methods like vanilla Dropout [ 13 ] or29 DropConnect [ 14 ] are all subject to a pre-defined distribution like Bernoulli or Gaussian , longing for30 exhaustive grid search for an optimal probability . Thereby a variety of works attempt to utilize man-31 ual attention mask to obtain a more informative attention matrix [ 15 , 16 ] , whereas all these methods32 require prior knowledge on model or data , which could be costly or unavailable . More recently , the33 rise of Neural Architecture Search [ 17 , 18 ] gives birth to search-based dropout [ 19 ] , which automat-34 ically chooses an optimal dropout pattern based on additional validation performances . However,35 the huge search space brings heavy consumption and more importantly , the obtained dropout pattern36 is still fixed with a pre-defined probability , which is static and sample-independent , ignoring the37 dynamics within different samples . In this paper , we focus on task-specific tuning of self-attention38 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . empowered PrLMs and propose a novel dropout method named AttendOut onto attention layers,39 which leverages self-attention to dynamically generate dropout patterns for each attention layer as40 well as each sample through an end-to-end manner . We demonstrate that the previous state-of-the-art41 models with elaborate training design may achieve much stronger results . We verify the universality42 of our approach on extensive natural language processing tasks . Guided by AttendOut , we pro-43 pose another two attention regularizers to enable simple but effective performance boost with no44 additional cost.45 2 Related Work46 Dropout is proposed to alleviate over-fitting problem in DNNs . Apart from vanilla Dropout [ 13 ] 47 and DropConnect [ 14 ] which randomly shut down a subset of activations or hidden weights , there48 are a variety of dropout methods proposed , e.g . Alpha Dropout [ 20 ] , Variational Dropout [ 21 , 22 ] ,49 Adversarial Dropout [ 23 ] , Energy-based Dropout [ 24 ] . However , random-based dropout encounters50 slow experiment cycle due to inevitable grid search . Inspired by Neural Architecture Search [ 17 , 18 ] ,51 [ 19 ] proposes AutoDropout to automate the process of designing dropout patterns . A similar line of52 work is dynamic tuning of dropout , which further allows adaptive dropout probabilities under differ-53 ent training moments . [ 25 ] proposes Concrete Dropout with continuous relaxation under Concrete54 distribution , [ 26 ] proposes Learnable Bernoulli Dropout under discrete Bernoulli distribution using55 Augment-REINFORCE-Merge estimator [ 27 ] , while [ 28 ] proposes Context Dropout by optimizing56 the evidence lower bound.57 With self-attention network continuously stands out , dropout is being explored onto self-attention58 based models . LayerDrop [ 29 ] randomly removes entire SAN blocks , while DropHead [ 30 ] , Head-59 Mask [ 31 ] randomly remove certain attention heads . UniDrop [ 32 ] unifies these dropout methods,60 which facilitates text classification and machine translation tasks . Additionally , prior knowledge is61 shown highly effective for guiding attention dropout as in SG-Net [ 15 ] and SIT [ 16 ] , which inten-62 tionally discard syntax-unrelated attention units with the help of structural clues.63 3 Preliminaries64 In this section , we provide the preliminaries for the proposed approach . We first review the details65 of self-attention proposed in [ 1 ] . Based on the specific architecture , we elaborate the concerned66 attention dropout.67 3.1 Self-Attention68 Generally , a standard SAN block is mainly composed of an attention layer and several feed-forward69 layers ( actually there are residual connection , layer normalization , etc . as well ) . The input of it is a70 sentence or batch of sentences of length n , which is first embedded through an embedding layer . The71 embedded input E may go through three linear projections WQ , WK and WV referring to query , key72 and value layers respectively , and then obtain three matrices Q , K and V referring to the query , key73 and value components of self-attention . Subsequently , a dot-product of Q and K is taken and then74 normalized using Softmax function to obtain the attention matrix A . Then another dot-product of75 A and V follows . The mentioned calculation can be formalized as follow:76 Attention ( Q , K , V ) = Softmax ( Q ·KT√ dk ) · V ( 1 ) where √ dk is a scaling factor . Finally , the self-attention layer ends up with a linear projection WO77 to output.78 During the aforementioned process , we highlight a key phases , that is the attention matrix A , which79 is a dot-product of n×n from two separate linear projections WQ and WK . A is viewed as a feature80 map which stores the node-to-node significance in different scores . Various works show that there81 hides implicit but highly needed semantic clues.82 3.2 Dropout on Self-Attention83 Our dropout will apply to the attention matrix of the concerned attention layer . We first define two84 specific dropouts onto Eq . 1 , where both implementations are just as simple as in standard dropout85 via a mask matrix M .86 Weights Dropout . Weights dropout is applied to the attention matrix after Softmax function by87 default , which is formulated as:88 Attention ( Q , K , V ) = ( Softmax ( Q ·KT√ dk ) ⊙M ) · V ( 2 ) where M is a binary matrix with elements in { 0 , 1 } and ⊙ refers to element-wise multiplication.89 Scores Dropout . Different from weights dropout , scores dropout is applied before Softmax func-90 tion , which is formulated as:91 Attention ( Q , K , V ) = Softmax ( Q ·KT√ dk +M ) · V ( 3 ) Since the outer Softmax , we conduct an addition instead of multiplication , where elements in M92 are set to 0 for kept units and −inf for removed ones . Note that the Softmax takes a similar93 function as the scaling factor of 1/p in vanilla Dropout [ 13 ] , which balances the expectation of the94 network.95 Weights dropout is commonly used in self-attention based models , while scores dropout is less96 explored , which is our focus in this paper . For scores dropout , we need to pay attention to a special97 case , when all attentions are shut down , that is , all elements in M equal to −inf at the same time.98 Such case can be formulated as follow:99 Attention ( Q , K , V ) = Softmax ( M ) · V ( 4 ) Note that Softmax ( M ) obtains to a constant matrix , where each unit equals to 1/n . In this case,100 the attention matrix is fixed and consequently the WQ , WK and dot-product in between are skipped.101 4 Methodology102 In this paper , we propose Attention differentiable dropOut ( AttendOut ) , which contributes technique103 novelty in the following way : ( 1 ) dynamic and task-specific tuned ; ( 2 ) end-to-end trained ; ( 3 ) gra-104 dient optimized dropout method onto self-attention empowered PrLMs . We elaborate our approach105 with two parts , in which the first is composition , while the second is training algorithm.106 4.1 Elements of AttendOut107 Our training architecture is composed of three modules , A-Net ( Attacker ) , D-Net ( Defender ) and108 G-Net ( Generator ) . D-Net and A-Net are two identical models and trained simultaneously through109 standard gradient descent , while G-Net is a learnable dropout maker and trained through policy110 gradient . Now we elaborate each of them.111 Defender - Attacker As suggested , defender and attacker are two competitors playing a game112 with each other on specific criteria , e.g . training accuracy , training loss . Specifically , D-Net and A-113 Net are two identical self-attention empowered PrLMs , e.g . BERT , RoBERTa . However , they follow114 different dropout strategies . D-Net receives regular dropout as default in specific models , while A-115 Net receives additional dropout decision from G-Net onto its corresponding attention layers.116 Generator G-Net acts as a dropout maker through generating a mask matrix for each attention117 layer during training stage . As aforementioned , the common dropout strategies rely on randomness,118 which intends to shut down the co-adaption but not powerful enough . However , our dropout maker119 is an agent which is able to intelligently choose and learn dropout patterns for each sample . Specif-120 ically , after training for a fixed number of steps , we conduct evaluation for both A-Net and D-Net.121 When A-Net obtains a higher score than D-Net , which means attacker wins the game , G-Net will be122 rewarded positively . When defender wins , G-Net will be punished with a negative reward . In con-123 sequence , G-Net learns appropriate dropout patterns through the game between D-Net and A-Net,124 while assisting A-Net to win the game . On the other hand , A-Net needs to be stronger when training125 under such powerful dropout , which makes it much more robust from over-fitting . Compared to126 search-based dropout , G-Net is triggered by the difference between two model derivatives with and127 without dropout , instead of the final feedback on validation set , which makes it end-to-end-possible128 and sample-dependent.129 The design of G-Net is the most delicate part , which is also a self-attention based model with iden-130 tical number of layers with D-Net and A-Net . However , we make several improvements . 1 ) G-Net131 only exports the attention scores from attention layers with no extra output layers , from which we132 apply Gumbel [ 33 , 34 ] to sample the actions to obtain the dropout mask . 2 ) G-Net only makes133 one-head attention and share one group of parameters for all attention layers . 3 ) G-Net is excluded134 of feed-forward layers , which may obscure the impact of self-attention [ 11 , 10 ] .135 4.2 Training with AttendOut136 The core of training with AttendOut is to find a way to optimize G-Net , which receives signals from137 the difference between D-Net and A-Net . Supposing there is a list of dropout actions by G-Net:138 a1 : T = { a1 , a2 , a3 , · · · , aT } where T refers to the number of samples , for each action at , G-Net may achieve a reward rt . The139 optimization objective is to maximize the overall rewards of list a1 : T , denoted as R , that is:140 J ( θG ) = EP ( a1 : T ; θG ) [ R ] where R = ∑T t=1 rt . Since R is non-differentiable , we use policy gradient to update θG as in [ 17 ] :141 ∇θGJ ( θG ) = T∑ t=1 EP ( a1 : T ; θG ) [ ∇θG logP ( at|a ( t−1 ) :1 ; θG ) rt ] The above equation could be approximated as:142 1 m m∑ k=1 T∑ t=1 ∇θG logP ( at|a ( t−1 ) :1 ; θG ) rt For a model with n attention layers , each dropout decision is composed of n inner decisions of each143 layer . Additionally , each attention layer contains an attention matrix of l × l , namely l2 elements144 dropped or kept . Thus , we denote a dropout unit as dij , where i refers to the ith layer while j refers145 to the jth element of the attention matrix.146 However , nl2 dropout units bring a huge space , which makes it impossible to calculate the joint prob-147 ability . To this end , we introduce the independence assumption that each dropout unit is independent148 with each other . Under the relaxation , we can make the following probability likelihood:149 logP ( at|a ( t−1 ) :1 ; θG ) = 1 nl2 ∑ i , j logP ( dijt |d ij ( t−1 ) :1 ; θG ) where the summation ∑n i=1 ∑l2 j=1 is briefly denoted as ∑ i , j .150 Thus , the final gradient could be formalized as:151 ∇θGJ ( θG ) = 1 m 1 nl2 m∑ k=1 T∑ t=1 ∑ i , j ∇θG logP ( d ij t |d ij ( t−1 ) :1 ; θG ) ( rt − b ) ( 5 ) where b is a baseline function of moving average [ 35 ] . Note that we do not apply additional regular-152 izers like L0 and L1 penalty , which impose unnecessary bias.153 Algorithm 1 summarizes the overall procedure of training PrLMs with AttendOut . We first initialize154 all three networks . Note that D-Net and A-Net should be kept identical at the beginning of each155 training step . A straightforward strategy is to choose the better one to cover the other . To add156 randomness , we sample from D-Net and A-Net based on their evaluation performances , with higher157 probability for the better one . Then for each step , D-Net and A-Net are fed with the same mini-batch158 data and updated via standard gradient descent , meanwhile each batch will be cached . After training159 for T steps , which we denote as a dropout step , both D-Net and A-Net are evaluated on additional160 validation samples , which could be development set data , noisy training data or a small split of train-161 ing data . In this paper , we simply use development set . For efficiency , we make random sampling162 on it to retrieve T samples for evaluation . Based on the evaluation scores , G-Net is rewarded with163 { r1 , r2 , r3 , · · · , rT } and updated via Eq . 5 . At the end of each dropout step , the cached samples164 will be released and D-Net and A-Net will be re-initialized.165 Resource Usage We notice that training PrLMs with AttendOut may sacrifice time and memory167 cost . The detailed resource usage is shown in Appendix . Taking RoBERTa as an example , the168 algorithm requires two RoBERTa models as well as a smaller self-attention based generator , which169 is 1/3 of RoBERTa size . Considering cached samples , roughly speaking , AttendOut requires twice170 graphic memory as well as twice training time compared to a single model , which is a middle speed171 line between random-based dropout and neural architecture search ( Dropout [ 13 ] < AttendOut≪172 AutoDropout [ 19 ] ) . However , AttendOut contributes to remarkable performance gain compared to173 other attention dropout methods.174 Pre-training Our approach is both feasible for both fine-tuning and pre-training stage of PrLMs175 but expensive for the latter . However , we try to serve for the most delicate part of concerned issue,176 since pre-training is generally done on large-scale data with modest training epochs , which makes it177 less susceptible from over-fitting.178 We demonstrate the universal effectiveness of AttendOut on extensive natural language processing180 tasks . For all mentioned tasks , we apply our method on BERT [ 2 ] and its stronger variant RoBERTa181 [ 3 ] . Our implementations are based on PyTorch using transformers [ 36 ] . For further training details,182 please refer to Appendix.183 Our experiments include : ( 1 ) natural language understanding : General Language Understanding184 Evaluation ( GLUE ) benchmark [ 37 ] , a collection of nine natural language understanding tasks ( here185 we experiment on five of them , SST-2 , MRPC , QNLI , MNLI-mm and CoLA ; ( 2 ) document clas-186 sification : IMDB [ 38 ] , a sentiment analysis dataset where about 15 % of the documents are longer187 than 512 word-pieces ; ( 3 ) named entity recognition : CoNLL2003 [ 39 ] ; ( 4 ) part-of-speech tag-188 ging : English Penn Treebank ( PTB ) [ 40 ] ; ( 5 ) multiple choices question answering : SWAG [ 41 ] .189 We report both test and development results for GLUE sub-tasks since the large bias between them,190 while development results only for all the other tasks.191 Note that we only adjust the dropout steps and keep all other parameters the same for strict fair192 comparison . For example , the parameters we use in RoBERTa are identical with what we use in193 training with AttendOut including both D-Net and A-Net.194 6 Results195 6.1 Significance Analysis196 Pictorially in Table 1 , RoBERTa is strong enough as it outperforms BERT by a big margin , while197 AttendOut empowered RoBERTa still outperforms it on all five GLUE sub-tasks . For small-scale198 datasets , which are more likely to over-fit , AttendOut helps unfold remarkable performance gain199 ( 12.1 % / 3.5 % over BERT on CoLA , 1.7 % / 1.4 % over BERT on MRPC ) . However , for large-200 scale one like MNLI , which tends to be more stable , AttendOut still produces considerable boost,201 ( 1.4 % / 1.4 % over RoBERTa , 1.1 % / 0.7 % over BERT ) .202 Furthermore , AttendOut is shown universally effective as in Table 2 . For POS Tagging , BERT203 and RoBERTa have achieved very strong baselines , while AttendOut empowered ones are even204 stronger , ( 1.1 % over BERT on PTB ) . Similar results are seen on document classification and NER.205 For SWAG , however , AttendOut seems weakly effective ( 0.6 % over BERT , 0.4 % over RoBERTa ) .206 6.2 Visual Analysis207 Dropout Patterns Another concerned issue is the dropout proportions by AttendOut . Figure 3208 depicts the patterns on several datasets . We may find several interesting phenomenons . First , the209 overall patterns largely differ from datasets , which is fair since AttendOut is sample-dependent.210 However , we may observe something in common . Overall , the lower layers take higher dropout211 probabilities . For example on QNLI , the first layer almost remains steady with the probability of212 0.55 during the training process , while the fourth one continuously decays in a higher rate . Intu-213 itively , the first three layers undertake a similar trend in each dataset , while there might be an up and214 down for the fourth one as in SST-2 and CoLA . Especially for CoLA , we see unusual high dropout215 probabilities in the final period ( around 0.9 ) , which are close to complete dropout . We notice that216 CoLA is a small set with 8500 training samples , on which SAN model is more inclined to suffer217 from over-fitting . Therefore , PrLM on CoLA encounters more intensive dropout through AttendOut.218 Convergence Figure 4 depicts the accuracy trends of RoBERTa on SST-2 , QNLI , MNLI respec-219 tively . Due to a stronger dropout module , the one with AttendOut tends to fall behind ( SST-2,220 MNLI ) at the beginning of training . However , model becomes stronger since the second epoch221 ( SST-2 , QNLI ) . Especially on MNLI , RoBERTa obtains better results in the first two epochs and it222 drops in the last one , while with AttendOut , the performance is steadily rising for all three epochs.223 7 Ablation Study224 In this section , we conduct further experiments to demonstrate the effectiveness of AttendOut . Due225 to space limitation , we conduct corresponding experiments on development sets only.226 7.1 Attention Dropout227 Vanilla Dropout We conduct comparison with vanilla Dropout [ 13 ] , in which we dropout the228 attention matrix for all layers with Bernoulli distribution of p. Here , we choose the dropout proba-229 bilities in { 0.1 , 0.2 } .230 LayerDrop We also compare with LayerDrop [ 29 ] , which focuses on skipping the entire encoder231 blocks , Inspired of it , we design another strategy which randomly skips attention layers via Eq . 4.232 For fair enough comparison , we set the dropout probabilities to 0.2 for both methods , following the233 settings in [ 29 ] .234 Intuitively in Table 3 , vanilla Dropout with fixed probability does not produce noticeable gain ( 1.9 % 235 bellow RoBERTa on CoLA ) . However , AttendOut shows powerful advantage ( 4.1 % , 1.0 % and 1.0 % 236 over vanilla Dropout on CoLA , QNLI and MNLI ) , which stresses the necessity of dynamic dropout237 patterns rather than fixed static one . On the other hand , both layer-level regularizers are effective,238 while attention LayerDrop performs stronger and more stable on all the three . Especially on CoLA,239 it outperforms RoBERTa by 1.7 points , while LayerDrop meets a performance drop , which demon-240 strates that removing the attention layers act as a more effective regularizer than removing the entire241 SAN block as for self-attention based models.242 7.2 Pattern Approximation243 Guided by AttendOut , we design a dropout scheduler , in which we utilize piece-wise linearity to244 approximate the real curves as depicted in Figure 3 . Taking QNLI as an example , we initialize245 the dropout probabilities to 0.6 for all attention layers and set a a specific slope for each of them.246 Note that here the corresponding mask matrices are randomly-generated and subject to Bernoulli247 distribution . In AttendOut , however , the distribution are learned dynamically through self-attention248 of G-Net.249 As shown in Table 4 , RoBERTa with scheduled Bernoulli dropout works surprisingly well on both250 CoLA and QNLI , which outperforms RoBERTa by 0.8 and 0.6 points respectively , closer to At-251 tendOut , even if the strategy here is random-based and much looser . The guided scheduled dropout252 helps unfold the correctness of the dynamic dropout patterns learned by AttendOut as well as the253 self-attention based dropout maker.254 8 Conclusion255 This paper focuses on the co-adaption problem of deep self-attention networks , and presents a novel256 dropout method onto self-attention empowered pre-trained language models . Extensive experiments257 on multiple natural language processing tasks demonstrate that our proposed approach is universal258 and qualified to enable more robust task-specific tuning , which contributes to much stronger state-259 of-the-arts . We probe into the learned dropout patterns on different tasks , which empirically guide260 us to the very needed dynamic attention dropout design.261 References262 [ 1 ] Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , Lukasz263 Kaiser , and Illia Polosukhin . Attention is all you need . In Isabelle Guyon , Ulrike von Luxburg , Samy264 Bengio , Hanna M. Wallach , Rob Fergus , S. V. N. Vishwanathan , and Roman Garnett , editors , Advances265 in Neural Information Processing Systems 30 : Annual Conference on Neural Information Processing266 Systems 2017 , December 4-9 , 2017 , Long Beach , CA , USA , pages 5998–6008 , 2017.267 [ 2 ] Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . BERT : pre-training of deep bidi-268 rectional transformers for language understanding . In Jill Burstein , Christy Doran , and Thamar Solorio,269 editors , Proceedings of the 2019 Conference of the North American Chapter of the Association for Com-270 putational Linguistics : Human Language Technologies , NAACL-HLT 2019 , Minneapolis , MN , USA , June271 2-7 , 2019 , Volume 1 ( Long and Short Papers ) , pages 4171–4186 . Association for Computational Linguis-272 tics , 2019.273 [ 3 ] Yinhan Liu , Myle Ott , Naman Goyal , Jingfei Du , Mandar Joshi , Danqi Chen , Omer Levy , Mike Lewis,274 Luke Zettlemoyer , and Veselin Stoyanov . Roberta : A robustly optimized BERT pretraining approach.275 CoRR , abs/1907.11692 , 2019.276 [ 4 ] Zhenzhong Lan , Mingda Chen , Sebastian Goodman , Kevin Gimpel , Piyush Sharma , and Radu Soricut.277 ALBERT : A lite BERT for self-supervised learning of language representations . In 8th International278 Conference on Learning Representations , ICLR 2020 , Addis Ababa , Ethiopia , April 26-30 , 2020 . Open-279 Review.net , 2020.280 [ 5 ] Kevin Clark , Minh-Thang Luong , Quoc V. Le , and Christopher D. Manning . ELECTRA : pre-training281 text encoders as discriminators rather than generators . In 8th International Conference on Learning Rep-282 resentations , ICLR 2020 , Addis Ababa , Ethiopia , April 26-30 , 2020 . OpenReview.net , 2020.283 [ 6 ] Pengcheng He , Xiaodong Liu , Jianfeng Gao , and Weizhu Chen . { DEBERTA } : { DECODING } -284 { enhanced } { bert } { with } { disentangled } { attention } . In International Conference on Learning Repre-285 sentations , 2021.286 [ 7 ] Alec Radford , Karthik Narasimhan , Tim Salimans , and Ilya Sutskever . Improving language understanding287 by generative pre-training . 2018.288 [ 8 ] Weiqiu You , Simeng Sun , and Mohit Iyyer . Hard-coded gaussian attention for neural machine translation.289 In Dan Jurafsky , Joyce Chai , Natalie Schluter , and Joel R. Tetreault , editors , Proceedings of the 58th290 Annual Meeting of the Association for Computational Linguistics , ACL 2020 , Online , July 5-10 , 2020,291 pages 7689–7700 . Association for Computational Linguistics , 2020.292 [ 9 ] Yi Tay , Dara Bahri , Donald Metzler , Da-Cheng Juan , Zhe Zhao , and Che Zheng . Synthesizer : Rethinking293 self-attention in transformer models . CoRR , abs/2005.00743 , 2020.294 [ 10 ] Sinong Wang , Belinda Z. Li , Madian Khabsa , Han Fang , and Hao Ma . Linformer : Self-attention with295 linear complexity . CoRR , abs/2006.04768 , 2020.296 [ 11 ] Yihe Dong , Jean-Baptiste Cordonnier , and Andreas Loukas . Attention is not all you need : Pure attention297 loses rank doubly exponentially with depth . CoRR , abs/2103.03404 , 2021.298 [ 12 ] Saurabh Goyal , Anamitra Roy Choudhury , Saurabh Raje , Venkatesan T. Chakaravarthy , Yogish Sabhar-299 wal , and Ashish Verma . Power-bert : Accelerating BERT inference via progressive word-vector elimi-300 nation . In Proceedings of the 37th International Conference on Machine Learning , ICML 2020 , 13-18301 July 2020 , Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 3690–3699.302 PMLR , 2020.303 [ 13 ] Nitish Srivastava , Geoffrey E. Hinton , Alex Krizhevsky , Ilya Sutskever , and Ruslan Salakhutdinov.304 Dropout : a simple way to prevent neural networks from overfitting . J. Mach . Learn . Res. , 15 ( 1 ) :1929–305 1958 , 2014.306 [ 14 ] Li Wan , Matthew D. Zeiler , Sixin Zhang , Yann LeCun , and Rob Fergus . Regularization of neural networks307 using dropconnect . In Proceedings of the 30th International Conference on Machine Learning , ICML308 2013 , Atlanta , GA , USA , 16-21 June 2013 , volume 28 of JMLR Workshop and Conference Proceedings,309 pages 1058–1066 . JMLR.org , 2013.310 [ 15 ] Zhuosheng Zhang , Yuwei Wu , Junru Zhou , Sufeng Duan , Hai Zhao , and Rui Wang . Sg-net : Syntax-311 guided machine reading comprehension . In The Thirty-Fourth AAAI Conference on Artificial Intelligence,312 AAAI 2020 , The Thirty-Second Innovative Applications of Artificial Intelligence Conference , IAAI 2020,313 The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence , EAAI 2020 , New York , NY,314 USA , February 7-12 , 2020 , pages 9636–9643 . AAAI Press , 2020.315 [ 16 ] Hongqiu Wu , Hai Zhao , and Min Zhang . Code summarization with structure-induced transformer . arXiv316 preprint arXiv:2012.14710 , 2020.317 [ 17 ] Barret Zoph and Quoc V. Le . Neural architecture search with reinforcement learning . In 5th International318 Conference on Learning Representations , ICLR 2017 , Toulon , France , April 24-26 , 2017 , Conference319 Track Proceedings . OpenReview.net , 2017.320 [ 18 ] Hanxiao Liu , Karen Simonyan , and Yiming Yang . DARTS : differentiable architecture search . In 7th321 International Conference on Learning Representations , ICLR 2019 , New Orleans , LA , USA , May 6-9,322 2019 . OpenReview.net , 2019.323 [ 19 ] Hieu Pham and Quoc V. Le . Autodropout : Learning dropout patterns to regularize deep networks . CoRR,324 abs/2101.01761 , 2021.325 [ 20 ] Günter Klambauer , Thomas Unterthiner , Andreas Mayr , and Sepp Hochreiter . Self-normalizing neural326 networks . In Isabelle Guyon , Ulrike von Luxburg , Samy Bengio , Hanna M. Wallach , Rob Fergus , S. V. N.327 Vishwanathan , and Roman Garnett , editors , Advances in Neural Information Processing Systems 30 : An-328 nual Conference on Neural Information Processing Systems 2017 , December 4-9 , 2017 , Long Beach , CA,329 USA , pages 971–980 , 2017.330 [ 21 ] Avrim Blum , Nika Haghtalab , and Ariel D. Procaccia . Variational dropout and the local reparameteriza-331 tion trick . In Corinna Cortes , Neil D. Lawrence , Daniel D. Lee , Masashi Sugiyama , and Roman Garnett,332 editors , Advances in Neural Information Processing Systems 28 : Annual Conference on Neural Infor-333 mation Processing Systems 2015 , December 7-12 , 2015 , Montreal , Quebec , Canada , pages 2575–2583,334 2015.335 [ 22 ] Dmitry Molchanov , Arsenii Ashukha , and Dmitry P. Vetrov . Variational dropout sparsifies deep neural336 networks . In Doina Precup and Yee Whye Teh , editors , Proceedings of the 34th International Conference337 on Machine Learning , ICML 2017 , Sydney , NSW , Australia , 6-11 August 2017 , volume 70 of Proceedings338 of Machine Learning Research , pages 2498–2507 . PMLR , 2017.339 [ 23 ] Sungrae Park , Jun-Keon Park , Su-Jin Shin , and Il-Chul Moon . Adversarial dropout for supervised and340 semi-supervised learning . In Sheila A. McIlraith and Kilian Q. Weinberger , editors , Proceedings of the341 Thirty-Second AAAI Conference on Artificial Intelligence , ( AAAI-18 ) , the 30th innovative Applications342 of Artificial Intelligence ( IAAI-18 ) , and the 8th AAAI Symposium on Educational Advances in Artificial343 Intelligence ( EAAI-18 ) , New Orleans , Louisiana , USA , February 2-7 , 2018 , pages 3917–3924 . AAAI344 Press , 2018.345 [ 24 ] Hojjat Salehinejad and Shahrokh Valaee . Edropout : Energy-based dropout and pruning of deep neural346 networks . CoRR , abs/2006.04270 , 2020.347 [ 25 ] Yarin Gal , Jiri Hron , and Alex Kendall . Concrete dropout . In Isabelle Guyon , Ulrike von Luxburg , Samy348 Bengio , Hanna M. Wallach , Rob Fergus , S. V. N. Vishwanathan , and Roman Garnett , editors , Advances349 in Neural Information Processing Systems 30 : Annual Conference on Neural Information Processing350 Systems 2017 , December 4-9 , 2017 , Long Beach , CA , USA , pages 3581–3590 , 2017.351 [ 26 ] Shahin Boluki , Randy Ardywibowo , Siamak Zamani Dadaneh , Mingyuan Zhou , and Xiaoning Qian.352 Learnable bernoulli dropout for bayesian deep learning . In Silvia Chiappa and Roberto Calandra , edi-353 tors , The 23rd International Conference on Artificial Intelligence and Statistics , AISTATS 2020 , 26-28354 August 2020 , Online [ Palermo , Sicily , Italy ] , volume 108 of Proceedings of Machine Learning Research,355 pages 3905–3916 . PMLR , 2020.356 [ 27 ] Mingzhang Yin and Mingyuan Zhou . ARM : augment-reinforce-merge gradient for stochastic binary357 networks . In 7th International Conference on Learning Representations , ICLR 2019 , New Orleans , LA,358 USA , May 6-9 , 2019 . OpenReview.net , 2019.359 [ 28 ] Xinjie Fan , Shujian Zhang , Korawat Tanwisuth , Xiaoning Qian , and Mingyuan Zhou . Contextual dropout:360 An efficient sample-dependent dropout module . CoRR , abs/2103.04181 , 2021.361 [ 29 ] Angela Fan , Edouard Grave , and Armand Joulin . Reducing transformer depth on demand with struc-362 tured dropout . In 8th International Conference on Learning Representations , ICLR 2020 , Addis Ababa,363 Ethiopia , April 26-30 , 2020 . OpenReview.net , 2020.364 [ 30 ] Wangchunshu Zhou , Tao Ge , Furu Wei , Ming Zhou , and Ke Xu . Scheduled drophead : A regularization365 method for transformer models . In Trevor Cohn , Yulan He , and Yang Liu , editors , Proceedings of the 2020366 Conference on Empirical Methods in Natural Language Processing : Findings , EMNLP 2020 , Online367 Event , 16-20 November 2020 , pages 1971–1980 . Association for Computational Linguistics , 2020.368 [ 31 ] Zewei Sun , Shujian Huang , Xinyu Dai , and Jiajun Chen . Alleviating the inequality of attention heads for369 neural machine translation . CoRR , abs/2009.09672 , 2020.370 [ 32 ] Zhen Wu , Lijun Wu , Meng Qi , Yingce Xia , Shufang Xie , Tao Qin , Xinyu Dai , and Tie-Yan Liu . Unidrop:371 A simple yet effective technique to improve transformer without extra cost . In Proceedings of the The 2021372 Conference of the North American Chapter of the Association for Computational Linguistics - Human373 Language Technologies , Volume 1 ( Long Papers ) , 2021.374 [ 33 ] Chris J. Maddison , Daniel Tarlow , and Tom Minka . A * sampling . In Zoubin Ghahramani , Max Welling,375 Corinna Cortes , Neil D. Lawrence , and Kilian Q. Weinberger , editors , Advances in Neural Information376 Processing Systems 27 : Annual Conference on Neural Information Processing Systems 2014 , December377 8-13 2014 , Montreal , Quebec , Canada , pages 3086–3094 , 2014.378 [ 34 ] Xinwei Geng , Longyue Wang , Xing Wang , Bing Qin , Ting Liu , and Zhaopeng Tu . How does selective379 mechanism improve self-attention networks ? In Dan Jurafsky , Joyce Chai , Natalie Schluter , and Joel R.380 Tetreault , editors , Proceedings of the 58th Annual Meeting of the Association for Computational Linguis-381 tics , ACL 2020 , Online , July 5-10 , 2020 , pages 2986–2995 . Association for Computational Linguistics,382 2020.383 [ 35 ] Ronald J. Williams . Simple statistical gradient-following algorithms for connectionist reinforcement384 learning . Mach . Learn. , 8:229–256 , 1992.385 [ 36 ] Thomas Wolf , Lysandre Debut , Victor Sanh , Julien Chaumond , Clement Delangue , Anthony Moi , Pier-386 ric Cistac , Tim Rault , Rémi Louf , Morgan Funtowicz , Joe Davison , Sam Shleifer , Patrick von Platen,387 Clara Ma , Yacine Jernite , Julien Plu , Canwen Xu , Teven Le Scao , Sylvain Gugger , Mariama Drame,388 Quentin Lhoest , and Alexander M. Rush . Transformers : State-of-the-art natural language processing.389 In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing : System390 Demonstrations , pages 38–45 , Online , October 2020 . Association for Computational Linguistics.391 [ 37 ] Alex Wang , Amanpreet Singh , Julian Michael , Felix Hill , Omer Levy , and Samuel R. Bowman . GLUE:392 A multi-task benchmark and analysis platform for natural language understanding . In 7th International393 Conference on Learning Representations , ICLR 2019 , New Orleans , LA , USA , May 6-9 , 2019 . OpenRe-394 view.net , 2019.395 [ 38 ] Andrew L. Maas , Raymond E. Daly , Peter T. Pham , Dan Huang , Andrew Y. Ng , and Christopher Potts.396 Learning word vectors for sentiment analysis . In Dekang Lin , Yuji Matsumoto , and Rada Mihalcea,397 editors , The 49th Annual Meeting of the Association for Computational Linguistics : Human Language398 Technologies , Proceedings of the Conference , 19-24 June , 2011 , Portland , Oregon , USA , pages 142–150.399 The Association for Computer Linguistics , 2011.400 [ 39 ] Erik F. Tjong Kim Sang and Fien De Meulder . Introduction to the conll-2003 shared task : Language-401 independent named entity recognition . In Walter Daelemans and Miles Osborne , editors , Proceedings402 of the Seventh Conference on Natural Language Learning , CoNLL 2003 , Held in cooperation with HLT-403 NAACL 2003 , Edmonton , Canada , May 31 - June 1 , 2003 , pages 142–147 . ACL , 2003.404 [ 40 ] Mitchell P. Marcus , Beatrice Santorini , and Mary Ann Marcinkiewicz . Building a large annotated corpus405 of english : The penn treebank . Comput . Linguistics , 19 ( 2 ) :313–330 , 1993.406 [ 41 ] Rowan Zellers , Yonatan Bisk , Roy Schwartz , and Yejin Choi . SWAG : A large-scale adversarial dataset407 for grounded commonsense inference . In Ellen Riloff , David Chiang , Julia Hockenmaier , and Jun ’ ichi408 Tsujii , editors , Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process-409 ing , Brussels , Belgium , October 31 - November 4 , 2018 , pages 93–104 . Association for Computational410 Linguistics , 2018.411 Checklist412 1 . For all authors ... 413 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s contri-414 butions and scope ? [ Yes ] 415 ( b ) Did you describe the limitations of your work ? [ Yes ] See Section 4.2.416 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] 417 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to them ? 418 [ Yes ] 419 2 . If you are including theoretical results ... 420 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] See Section 4.2.421 ( b ) Did you include complete proofs of all theoretical results ? [ No ] 422 3 . If you ran experiments ... 423 ( a ) Did you include the code , data , and instructions needed to reproduce the main experimental424 results ( either in the supplemental material or as a URL ) ? [ Yes ] See supplemental material.425 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they were cho-426 sen ) ? [ Yes ] See Section 4.2 and appendix.427 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experiments mul-428 tiple times ) ? [ No ] 429 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type of GPUs,430 internal cluster , or cloud provider ) ? [ Yes ] See Section 4.2 and appendix.431 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 432 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] See Section 5.433 ( b ) Did you mention the license of the assets ? [ N/A ] 434 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ No ] 435 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re us-436 ing/curating ? [ N/A ] 437 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable infor-438 mation or offensive content ? [ No ] 439 5 . If you used crowdsourcing or conducted research with human subjects ... 440 ( a ) Did you include the full text of instructions given to participants and screenshots , if applicable ? 441 [ N/A ] 442 ( b ) Did you describe any potential participant risks , with links to Institutional Review Board ( IRB ) 443 approvals , if applicable ? [ N/A ] 444 ( c ) Did you include the estimated hourly wage paid to participants and the total amount spent on445 participant compensation ? [ N/A ] 446

Summary: This paper proposes a novel dropout strategy for self-attention, namely AttendOut. AttendOut trains 3 independent networks.  - Network G generates a $nxn$ dropout mask for the self-attention matrix for each layer conditioned on the input data.   - Network D is trained by applying the dropout mask generated by G - Network A is trained by applying the random dropout mask over self-attention.  Network D and A are trained with the task-specific objective. Importantly, the network G is trained with a reward defined by the advantage of D over A and optimized by the policy gradient.   Empirically, the proposed shows clear gains under language pretraining and fine-tuning.