FreeLB: Enhanced Adversarial Training for Natural Language Understanding
Authors: Chen Zhu
Yu Cheng
Zhe Gan
Siqi Sun
Tom Goldstein
Jingjing Liu
Venue:  (2020)

Abstract: Adversarial training, which minimizes the maximal risk for label-preserving input perturbations, has proved to be effective for improving the generalization of language models. In this work, we propose a novel adversarial training algorithm, FreeLB, that promotes higher invariance in the embedding space, by adding adversarial perturbations to word embeddings and minimizing the resultant adversarial risk inside different regions around input samples. To validate the effectiveness of the proposed approach, we apply it to Transformer-based models for natural language understanding and commonsense reasoning tasks. Experiments on the GLUE benchmark show that when applied only to the finetuning stage, it is able to improve the overall test scores of BERT-base model from 78.3 to 79.4, and RoBERTa-large model from 88.5 to 88.8. In addition, the proposed approach achieves state-of-the-art single-model test accuracies of 85.44% and 67.75% on ARC-Easy and ARC-Challenge. Experiments on CommonsenseQA benchmark further demonstrate that FreeLB can be generalized and boost the performance of RoBERTa-large model on other tasks as well.

Published as a conference paper at ICLR 2020
FREELB: ENHANCED ADVERSARIAL TRAINING FOR NATURAL LANGUAGE UNDERSTANDING
Chen Zhu', Yu Cheng", Zhe Gan', Siqi Sun', Tom Goldstein', Jiigjing Liu University of Maryland, College Park "Microsoft Dynamics 365 Al Research (chenzhu, tomg) 8as. adu, {yu. chang, zhe. sigi.sur, (t wwi.ee/moft....mmm
ABSTRACT
Adversarial training, which minimizes the maximal risk for label-preserving in- put perturbations, has proved be effective for improving the generalization of language models. In this work, we propose novel adversarial training algorithm, FreeLB, that promotes higher invariance in the embedding space, by adding adver- sarial perturbations to word embeddings and minimizing the resultant adversarial risk inside different regions around input samples. To validate the effectiveness of the proposed approach, we apply it to Transformer-baset models for natural language understanding and commonsense reasoning tasks. Experiments on the GLUE benchmark show that when applied only to the finetuning stage, it is able to improve the overall test scores of BERT-base model from 78.3 to 79.4, and RoBERTa-large model from 88.5 to 88.8. In addition, the proposed approach achieves state-of-the-an single-model test accuracies of 85.44% and 67.75% on ARC-Easy and ARC-Challenge further demonstrate that FreeLB can be generalized and boost the performance of RoBERTa-large model on other tasks as well.
Experiments on CommonsenseeAA benchmark
INTRODUCTION
Adversarial training is a method for creating robust neural networks. During adversarial training, mini-batches of training samples are contaminated with adversarial perturbations (alterations that are small and yet cause misclassification), and then used update network parameters until the resulting model learns resist such attacks. Adversarial training originally proposed as means to enhance the security of machine learning systems (Goodfellow et al., 2015), especially for safety-critical systems like self-driving cars (Xiao et al. 2018) and copyright detection (Saadatpanah et al. 2019).
In this paper, turn focus away from the security benefits of adversarial training, and instead study its effects on generalization. While adversarial training boosts the robustness, it is widely accepted by computer vision researchers that it is odds with generalization, with classification accuracy on non-corrupted images dropping as much as 10% on CIFAR-10, and 15% on Ima- genet (Madry al.] 2018 Xie et al., 2019). Surprisingly, people observe the opposite result for language models (Miyato et al. 2017 Cheng et al. 2019), showing that adversarial training can improve both generalization and robustness.
We will show that adversarial training significantly improves performance of state-of-the-ar models for many language understanding tasks. In particular, we propose novel adversarial training algo- rithm, called FreeLB (Free Large-Batch), which adds adversarial perturbations to word embeddings and minimizes the resultant adversarial loss around input samples. The method leverages recently proposed "free" training strategies (Shafahi et al. 2019)] Zhang et al.) 2019) to enrich the train- ing data with diversified adversarial samples under different norm constraints at no extra cost than PGD-based (Projected Gradient Descent) adversarial training (Madry et al., 2018), which enables us to perform such diversified adversarial training on large-scale of models. We observe improved invariance in the embedding space for models trained with FreeLB, which is positively correlated with generalization.
"oode available at httt://///gthhha...m/////aaaa///

---

Published as a conference paper at ICLR 2020
We perform comprehensive experimentss to evaluate the performance of variety of adversarial train- ing algorithms on state-of-th the-the-rrt art language understanding models and tasks. In the comparisons with standard PGD (Madry et al., 2018). FreeAT (Shafahi et al. 2019) and YOPO (Zhang et al. 2019). FreeL.B stands out to be the best for the datasets and models we evaluated. With Freel.,, achieve state-of-the-ar results several important language understanding benchmarks. On the GLUE benchmark, FreeLB pushes the performance of the BERT-base model from 78.3 to 79.4. The overall score of the RoBERTa-large models on the GLUE benchmark is also lifted from 88.5 to 88.8, achieving best results on most of its sub-tasks. Experiments also show that FreeLB can boost the performance of RoBERTa-large on question answering tasks, such as the ARC and Common- senseQA benchmarks. We also provide comprehensive ablation study and analysis to demonstrate effectiveness of our training process.
RELATED WORK
2.1 ADVERSARIAL TRAINING
To improve the robustness of neural networks against adversarial examples, many defense strategies and models have been proposed, in which PGD-based adversarial training (Madry et al., 2018) is widely considered to be the most effective, since it largely avoids the the obfuscated gradient problem (Athalye et al.) (2088). It formulates class of adversarial training algorithms (Kurakin et al. 2017) into solving minimax problem on the cross-entropy loss, which can be achieved reliably through multiple projected gradient ascent steps followed by a SGD (Stochastic Gradient Descent) step.
Despite being verified by /Athalye et al. (2018) to avoid obfuscated gradients, Qin et al. (2019) shows that PGD-based adversarial training still leads to highly convolved and non-linear loss surfaces when K is small, which could be readily broken under stronger adversaries. Thus, to be effective, the cost of PGD--bsed adversarial training is much higher than conventional training. To mitigate this cost, Shafahi et al. (2019) proposed a "free" adversarial training algorithm that simultaneously updates both model parameters and adversarial perturbations on single backward pass. Using similar formulation, Zhang et al. (2019) effectively reduce the total number of full forward and backward propagations for obtaining adversarial examples by restricting most of its adversarial updates in the first layer.
2.2 ADVERSARIAL EXAMPLES IN NATURAL LANGUAGES
Adversarial examples have been explored primarily in the image domain, and received many at- tention text domain recently. Previous works on text adversaries have focused on heuristics for creating adversarial examples in the black-box setting, pose to add distracting sentences to the input document in order to induce missclassification. Zhao ett al. (2018) generate text adversaries by projecting the input data a latent space using GANs, and searching for adversaries close to the original instance. [Belinkov & Bisk (2018) manipulate every word in sentence with synthetic or natural noise machine translation systems, [yyer et al. (2018) propose a neural paraphrase model based on back-translated data to produce paraphrases that have different sentence structures. Different from previous work, ours is not to produce actual adversarial examples, but only take the benefit of adversarial training for natural language understanding.
specific tasks. Jia & Liang] (2017) pro-
We are not the first to observe that robust language models may perform better on clean test data. Miyato et al. (2017) extend adversarial and virtual adversarial training (Miyato et al.) 2019) to the text domain to improve the performance on semi-supervised classification tasks. [Ebrahimi et al. (2018) propose a character/word replacement for crafting attacks, and show employing adversarial examples in training renders the models more robust. Ribeiro et al. (2018) show that adversarial attacks can be used valuable tool for debugging NLP models. Cheng et al. (2019) also find that crafting adversarial examples can help neural machine translation significantly. Notably, these studies have focused simple models text generation tasks. Our work explores how to effi- ciently use the gradients obtained in adversarial training to boost the performance of state-of ofte-off-the-n transformer-basee models.
2

---

Published as a conference paper at ICLR 2020
3 ADVERSARIAL TRAINING FOR LANGUAGE UNDERSTANDING
Pre-trained large-scale language models, such as BERT (Devlin et al, 2019). RoBERTa (Liu et al, 2019b), ALBERT (Lan al. 2020 and T5 (Raffel et al. 2019), have proven to be highly effective for downstream tasks. We aim to further improve the generalization of these pre-trained language models on the downstream language understanding tasks by enhancing their robustness in the em- bedding space during finetuning on these tasks. We achieve this goal by creating "virtual" adver- sarial examples in the embedding space, and then perform parameter updates on these adversarial embeddings. Creating actual adversarial examples for language is difficult; even with state-of-the art language models as guidance (e.g.. (Cheng et al. 2 remains unclear how to construct label-preserving adversarial examples via word/character replacement without human evaluations, because the meaning of each word/character depends on the context (Ribeiro et al., 2018). Since we are only interested in the effects of adversarial training, rather than producing actual adversarial examples, we add norm-bounded adversarial perturbations to the embeddings of the input sentences using a pradient-basedd method. Note that our embedding- based adversary is strictly stronger than more conventional text-based adversary, as our adversary can make manipulations on word embed- dings that are not possible in the text domain.
For models that incorporate various input representations, including word or subword embeddings, segment embeddings and position embeddings, our adversaries only modify the concatenated word or sub-word embeddings, leaving other components of the sentence representation unchanged, [] De- note the sequence of one-hot representations of the input subwords as = [21, Z2, ..., Zn.. the em- bedding matrix as V, and the language model (encoder) as a function y (X))), where X Vz the subword embeddings. y is the output of the model (e.g.. class probabilities for classification models), and 0 denotes all the learnable parameters including the embedding matrix V. We add ad- versarial perturbations d to the embeddings such that the prediction becomes y = So + 8). To preserve the semantics, we constrain the norm of 8 to he small, and assume the model's prediction should not change after the perturbation. This formulation is analogous to Miyato et al. (2017), with the difference that we do not require X to be normalized.
3.1 PGD FOR ADVERSARIAL TRAINING
Standard adversarial training seeks to find optimal parameters O" to minimize the maximum risk for any 8 within norm ball as:
min E. (Z
max T 6). |
(1)
where D is the data distribution, y is the label, and L is some loss function. We use the Frobenius norm to constrain 8. For neural networks, the outer "min" is non-convex, and the inner "max" is non-concave. Nonetheless, Madry et al. (2018) demonstrated that this saddle-point problem can be solved reliably with SGD for the outer minimization and PGD (a standard method for large-scale constrained optimization, see (Combettes & Pesquet 2011) and (Goldstein al., (0144), for the inner maximization. In particular, for the constraint [ [o| S 6, with an additional assumption that the loss function locally linear, PGD takes the following step (with step size a) in each iteration:
8,41 = 1118111 (8, + ag(())//la(6)))1l))
(2)
where ((6)) Vx 5c). v) is the gradient of the loss with respect to 8, and I1 performs a projection onto the e-ball. To achieve high-level robustness, multi-step adversarial ex- amples are needed during training, which is computationally expensive. The K-step PGD (K-PGD) requires K forward-backward passes through the network, while the standard SGD update requires only one. As a result, the adversary generation step in adversarial training increases run-time by an order of magnitude-4, catastrophii amount when training large state-of-the- language models.
3.2 BARGE-BATCEE ADVERSARIAL TRAINING FOR FREE
In the inner ascent steps of PGD, the gradients of the parameters can be obtained with almost no overhead when computing the gradients of the inputs. From this observation, FreeAT (Shafahi al.)
>SSubword embeddings" refers to the embeddings of sub-word encodings such as the popular Byte Pair Encoding (BPE) (Sennrich al. 2016).
3

---

Published as a conference paper at ICLR 2020
Algorithm "Free" Batge-Batch Adversarial Training (FreeLB-K)
Require: Training samples X (Z,y)), perturbation bound e, learning rate т, ascent steps K,
ascent step size (x
1: Initialize 0
2: for epoch = 1... Nop do
3: for minibatch C do
4: 6g x c-E,e)
5: go 2
6: for 1... K do
7:
Accumulate gradient of parameters 0
8:
9t + - + R ,,e Vo L(fo(X - 1).1
9:
Update the perturbation via gradient ascend
10:
gade + V8 L(fo(2 (X + 8c-1).
6e 11 1ll 0 a gadu F))
12: end for
<- T9K
14: end for
end for
2019) and YOPO (Zhang al. 2019) have been proposed to accelerate adversarial training. They achieve comparable robustness and generalization as standard PGD-trained models using only the same slightly larger number of forward- barkddb passes as natural training (i.e., SGD clean samples). FTeeAT takes one descent step on the parameters together with each of the K ascent steps on the perturbation. As result, FreeAT may suffer from the "stale gradient" problem (Dutta al.) 2018). where in every step t. 5, does not necessarily maximize the model with parameter O, since its update is based on VoL(fe... (X + 1). y). and vice versa. 0, does not necessarily minimize the adversarial risk with adversary d, since its update is based on VoI.foo (X + 8,-1), y). Such a problem may be more significant when the step size large.
Different from FreeAT, YOPO accumulates the gradient of the parameters from each of the ascent steps, and updates the parameters only once after the K inner ascent steps. YOPO also advocates that after each back-propagation, one should take the gradient of the first hidden layer as a constant and perform several additional updates on the adversary using the product of this constant and the Jacobian of the first layer of the network to obtain strong adversaries. However, when the first hidden layer is a linear layer as in their implementation, such an operation equivalent to taking larger step size the adversary. The analysis backing the extra update steps also assumes twice continuously differentiable loss, which does not hold for ReLU-based neural networks they experimented with, and thus the reasons for the success of such an algorithm remains obscure. We give empirical comparisons between YOPO and our approach in Sec. 4.3
To obtain better solutions for the inner max and avoid fundamental limitations the function class, we propose FreeLB, which performs multiple PGD iterations to craft adversarial examples, and simultaneously accumulates the "free" parameter gradients VeL in each iteration. After that, updates the model parameter 0 all at once with the accumulated gradients. The overall procedure shown in Algorithm I in which X + 5e is an approximation intersection of two balls I, = Bx+ (att) n Bx (c). By taking descent step along the averaged gradients at X ōo,.., X + 8K- we approximately optimize the following objective:
the local maximum within the
1 K
min E (Z,))~
Σ
max L(So(X +..).)) a.EI.
which is equivalent to replacing the original batch X with a K-times larger virtual batch, consisting of samples whose embeddings X do .... X 8K-1. Compared with PGD-based adversarial training (Eq. [T). which minimizes the maximum risk at a single estimated point in the vicinity each training sample, FreeL.B minimizes the maximum risk each ascent step at almost overhead.
Intuitively, FreeLB could be a learning method with lower generalization error than PGD. Sokolic et al. (2017) have proved that the generalization error of learning method invariant to a set of T' transformations may be up vT smaller than non-invariant learning method. According to
4

---

Published as a conference paper at ICLR 2020
their theory, FreeLB could have more significant improvement natural training, since FreeLB enforces the invariance to K adversaries from set of up to K different norm constraints while PGD only enforces invariance to a single norm constraint c.
Empirically, FreeLB does lead to higher robustness and invariance than PGD in the embedding space, in the sense that the maximum increase of loss in the vicinity of X for models trained with FreeLB is smaller than that with PGD. See Sec. 4.3 for details. In theory, such improved robustness can lead better generalization (Xu Mannor, 2012), which is consistent with our experiments. Qin et al. (2119) also demonstrated that PGD-based method leads to highly convolved and non-linear loss surfaces in the vicinity of input samples when K is small, indicating a lack of robustness.
3.3 WHEN ADVERSARIAL TRAINING MEETS DROPOUT
Usually, adversarial training is not used together with dropout (Srivastava et al. 2014). However, for some language models like RoBERTa (Liu al. 2019b), dropout is used during the finetuning stage. In practice, when dropout is turned on, each ascent step of Algorithm Tis optimizing ô for a different network. Specifically, denote the dropout mask as m with each entry my Bernuuli(p). Similar to our analysis for FreeAT, , the ascent step from 6,-1 to d, is based on V V () + 8,- 1--1); y), so 8 is sub-optimal for ((fe(m.)) (X + +)), y). Here 0 (m) is the effective parameters under dropout mask m.
The more plausible solution is to use the same m in each step. When applying dropout to any net- work, the objective for is to minimize the expectation of loss under different networks determined by the dropout masks, which is achieved by minimizing the Monte Carlo estimation of the expected loss. our case, the objective becomes:
min E D.)
Σ
max L(So(m) (X + 6c), d.EL.
(4)
K
where the I-sample Monte Carlo estimation should be & 2t= maxs, ET, L[Se(ma) (X + 6z), y) and can be minimized by using FreeL. with dropout mask mg in each ascent step. This is similar to applying Variational Dropout RNNs as used in Gal & Ghahramani (2016).
4 EXPERIMENTS
In this section, we provide comprehensive analysis FreeLB through extensive experiments on three Natural Language Understanding benchmarks: GLUE (Wang et al., 2019), ARC (Clark al.)
2018) and CommonsenseQA (Talmor et al. 2019). We also compare the robustness and general- ization of FreeLB with other adversarial training algorithms to demonstrate its strength. Additional experimental details are provided in the Appendix.
4.1 DATASETS
GLUE Benchmark. The GLUE benchmark is a collection of natural language understanding tasks, namely Corpus of Linguistic Acceptability (CoLA; Warstadt al. (018)). Stanford Senti-
ment Treebank (SST: Socher et al. (2013). ), Microsoft Research Paraphrase Corpus (MRPC; Dolan
& Brockett (205 Semantie Textual Similarity Benchmark (STS; (Agire et al. (007))) Quora
Question Pairs QQP::Iyer et al. (2017)) Multi NLI (MNLI; Williass et al. (2018). Question
NLI (QNLI: Rajpurkar al. (2016)), Recognizing Textual Entailment (RTE: Dagan al. (2006);
Bar Haim et al. (2006); Giampiccolo et al. (2007): [Bentivogli et af. (2009) and Winograd NLJ
(WNLI: Leveqque et al. (2011)) 8 of the tasks are formulated as classification problems and only STS-B is formulated as regression, but FreeLB applies to all of them. For BERT-base, we use the HuggingFace implementation"" and follow the single-task finetuning procedure as in Devlin et al. 2019). For RoBERTa, we use the fairseq implementatior? Same as Liu et al. (001b)), we also use
'The cardinality of the set is approximately min (K, - htwg:ghwwww..mmmmogiiiigccetthee
1).
5

---

Published as a conference paper at ICLR 2020
Method
MNLI
QNLI
QQP
RTE
SST-2
MRPC
CoLA
STS-B
(Acc)
(Acc)
(Acc)
(Acc)
(Acc)
(Acc)
(Mec)
(Pearson)
Reported
90.2
94.7
92.2
86.6
96.4
90.9
92.4
ReImp
85.61 (1.7)
96.56 (.3)
90.69 (.5) 67.57 (1.3)
92.20 (.2)
PGD
90.53 (.2) 94.87 (.2)
92.49 (.07)
87.41 (.9)
96.44 (.1)
90.93 (.2) 69.67 (1.2)
92.43 (7.)
FreeAT
90.02 (.2) 94,66 (.2)
92.48 (.08)
86.69 (15.)
96. 10 (.2)
90.69 (.4)
68,80 (1.3)
92,40 (.3)
FreeLB
90.61 (.1) 94.98 (.2)
92.60 (.03)
88.13 (1.2)
96.79 (.2)
91.42 (.7)
71.12 (.9)
92.67 (.08)
Table 1: Results (median and variance) on the dev sets of GLUE based on the RoBERTa- large model, from 5
runs with the same hyperparameter but different random seeds. Relmp is reimplementation of RoBERTa- large. The training process be very unstable
with the vanilla version. Here, both PGD STS-B and FreeAT on RTE demonstrates such instability, with one unconverged instance out five.
CoLA
SST-2
MRPC
STS-B 7k
QQP
MNL.-mmmm
QNLI
RTE
WNLI
AX
Model
Score
8.5k
67k
3.7k
364k
393k
108k
2.5k
634
BERT-basel
78.3
52.1
93.5
88
88.9/84.8
87.1/85.8
71.2/89.2
84.6/83.4
90.5
66.4
65.1
34.2
FreeLB-BBRRT
79.4
54.5
93.6
18.3..
87.7/86.7
72.7/89.6
85.7/84.6
91.8
70.1
65.1
36.9
MT-DNN-
87.6
68.4
96.5
92.7/90.3
91.1/9..7 73.7/89.9
73.
87.9/87.4
96.0
86.3
89.0
42.8
XLNet-Large
88.4
67.8
96.8
93.0//0.7
91.66/9111
74.2/90.3
90.2/89.8
98.6
86.3
90.4
47.5
RoBERTa'
88.5
67.8
96.7
92.3/89.8
92.22/919
74.3/90.2
90.8/90.2
98.9
88.2
89.0
48.7
Freel BB-RoB
88.8
68.0
96.8
93.1/90.8
92.4/92.2
74.8/90.3
91.1/90.7
98.8
88.7
89.0
50.1
Human
87.1
66.4
97.8
86.3/80.8
92.7/92.6 /6../98.
59.55/88888888888008 59.5/8
92.0/92.8
91.2
93.6
95.9
Table 2: Results on GLUE from the evaluation server, as of Sep 25, 2019. Metrics are the same the leaderboard. Number under each task's name is the size of the training results of BERT-base finetuned with FreeI_B, and FreeLBBRRoB is the ensemble RoBERTa-Larse models
FreeLB-BERT is the single-model
for each task. References: (Devlin et al. 2019); 2: (Liu et al. 2019a): 3; (Yang et al. 2019; 4; (Liu et al. 2019b).
single-task finetuning for all dev set results, and start with MNI.Lfinetuned models on RTE, MRPC and STS-B for the test submissions.
ARC Benchmark. The ARC dataset (Clark et al. 2018) is a collection of multi-choice science questions from grade-school level exams. It is further divided into ARC-Challenge set with 2,590 question answer (QA) pairs and ARC-Easy set with 5,197 QA pairs. Questions ARC-Challenge more difficult and cannot be handled by simply using retrieval and co-occurence based algo- rithm (Clark et al. 2018). typical question is:
Which property of mineral can be determined just by looking at it?
(A) luster [correct] (B) mass (C) weight (D) hardness.
CommonsenseeA Benchmark. The Commonsense@A dataset Talmor al., 2019) consists of 12,102 102 natural language questions that require human commonsense reasoning ability to answer. A typical question is
Where can stand on a river to see water falling without getting wet?
(A) waterfall, (B) bridge [correct], (C) valley, (D) stream, (E) bottom.
Each question has five candidate answers from ConceptNet (Speer et al. 2017). To make the ques- tion more difficult to solve, most answers have the same relation in ConceptNet to the key concept in the question. As shown in the above example, most answers can be connected to "river" by "At- Location" relation in ConceptNet. For fair comparison with the reported results in papers and leaderboard we use the official random split 1.11.
To
4.2 EXPERIMENTAL RESULTS
GLUE We summarize results on the dev sets of GLUE Table 1 comparing the proposed Freel.B against other adversatial training algorithms (PGD (Madry et al. 2018) and FreeAT (Shafahi et al.) )019 We use the same step size a and number of steps m for PGD, FreeAT and FreeLB. FreeLB is consistently better than the two baselines. Comparisons and detailed discussions about

---

Published as a conference paper at ICLR 2020
ARC
Easy Test
ARC-Challenge
ARC-Merge
CQA
Dev
Dev
Test
Dev
Test
Dev
Test
Test (E)
RoBERTa (Reported)
78.43
72.1
72.5
RoBERTa (Relmp)
84,39
84,13
64,54
64.44
77.83
77.62
77.56
RoBBRTaa
84,91
84,81
65,89
65.36
78.37
78.39
78.81
72.2
73.1
AristoRoBEKlav (MTL)
85.02
66.47
78.89
XI.Net + RoBERTa EMS))
67.06
FreeL RoBERTa (MTL)
84,91
85.44
70.23
67.75
79,86
79.60
Table 3: Results on ARC and CommonsenseQA (CQA), ARC-Merge is the combination of ARC-Easy and ARC-Challenge, "MTL" stands for multi-task learning and "Ens" stands for ensemble. Results of XLNct RoBERTa (MTL+Ens) and AristoRoBERTav (MTL) are from the ARC leaderboards. Test (E) denotes the test set results with ensembles. For CQA, we report the highest dev and test accuracies among all models. The models with 78.81/72.19 dev/test accuracy (as in the table) have 71.84/78.6 64 test/dev accuracies respectively.
YOPO (Zhang et al. 2019) are provided in Sec. 4.3 We have also submitted our results to the eval- uation server, results provided in Table 2 FreeLB B lifts the performance of the BERT-base model from 78.3 to 79.4, and RoBERTa-large model from 88.5 to 88.8 on overall scores.
ARC For ARC, a corpus of 14 million related science documents (from ARC Corpus, Wikipedia and other sources) is provided. For each QA pair, we first use retrieval model to select top 10 related documents. Then, given these retrieved documents" we use RoBERTa-large model encode (s) Retrieved Documents (/s) Question + Answer (/s), where (s) and (/s) are special tokens for
RoBERTa mode 8 We then apply a fully-connected layer to the representation of the [CI.S] token to compute the final logit, and use standard cross-entropy loss for model training.
Results are summarized in Table 3] Following [Sun et al. (2018), we first finetune the RoBERTa model on the RACE dataset (Lai et al., 2017). The finetuned RoBERTa model achieves 85, 50700% and 85.24% accuracy on the development and test set of RACE, respectively. Based on this, we further finetune the model on both ARC-Easy and ARC-Challema searching strategy (for 5 epochs), which achieves 84. 14.13% 9%%4.444 test accuracy on ARC-Easy/ARC-- Challenge. And by adding Freel.B finetuning, we reach 84.81 16.66,%%, significant boost on ARC benchmark, demonstrating the effectiveness of FreeI.B.
datasets with the same hyper-parameter
To further improve the results, we apply multi-task learning (MTL) strategy using additional datasets. We first finetune the model on RACE (Lai et al. 2017). and then finetune on joint dataset of ARC-Easy, ARC-Challenge, OpenbookQA (Mihaylov et al. 2018) and Regents Living Environmen
9 Based on this, we further finetune our model on ARC-Easy and ARC-Challenge with FreeLB. After finetuning, our single model achieves 67.75% test accuracy on ARC-Challenge and 85.44% on ARC-Easy, both outperforming the best submission on the official leaderboard
Commonsense(AA Similar to the training strategy in Liu et al. (2019b), we construct five inputs for each question by concatenating the question and each answer separately, then encode each input with the representation of the [CLS] token. A final score is calculated by applying the representation of [CLS] to fully-connected layer. Following the fairseq repository [T]] the input is formatted as: "(s) Q: Where can stand on a river to see water falling without getting wet? (/s) A: waterfall (/s)". where "Q:" and 'A:' are the prefix for question and answer, respectively.
Results are summarized in Table 3 We obtained a dev-set accuracy of 77.56% with the RoBERTa- large model. When using FreeLB finetuning, we achieved 78.81% %. 1.25% absolute gain. Com- pared with the results reported from fairseq repository, which obtains 78.43% accuracy on the dev- set, FreeLB still achieves better performance. Our submission to the CommonsenseQA leaderboard achieves 72.2% single-model test set accuracy, and the result of a 20-model ensemble is 73.1%, which achieves 1 among all the submissions without making use of ConceptNet.
We thank ooist team for providing retrieved documents and additional Regents Living Environ- ments dataset.
Equivalent to [CLS] and [SEP] token BERT.
1 hhtp:://Waaaaannaaaallleeeeeeeos and http:///eaderhoordaaleem
submissions/public
qa

---

Published as a conference paper at ICLR 2020
Methods
Vanilla
FreeLBB-3
BreeL.B-3
YOPO-3-2
YOPO-3-3
RTE
85.61 (1.67)
87.14
88.13 (1.21)
87.05 (I.36)
87.05 (0.20)
CoLA
67.57 (1.30)
69.31 (1.16)
71.12 (0.90)
70.40 (0.91)
69,91 (1.16)
MRPC
90.69 (0.54)
90.93 (0.66)
91.42 (0.72)
90.44 (0.62)
90.69 (0.37)
Table 4: The median and standard deviation of the scores on the dev sets of RTE, CoLA and MRPC from the GLUE benchmark, computed from 5 runs with the same hyper-paramettes except for the random seeds. We use FreeLB--i to denote FreeLB with m ascent steps, and FreeLB-3" denote the version without reusing the dropout mask.
Methods
RTE
CoLA M-Inc (R)
MRPC
M-Ine
M-Ine (R)
N-Loss
M-Inc (10-4)
N-Loss (107*)
M-Inc (10-2)
M-Inc (R)
N-Loss
(107*)
(10-²)
(107*)
(1079)
(1079)
(10 (10-3)
Vanilla
5.1
5.3
4.5
6.1
5.7
5.2
10.2
10.2
1.9
PGD
4.7
4.9
6.2
128.2
130.1
436.1
5.7
5.7
5.4
FreeLB
3.0
2.6
4.1
1.4
1.3
7.2
3.6
3.6
2.7
Table 5: Median of the maximum increase loss in the vicinity of the dev set samples RoBERTa-Laud model finetuned with different methods. Vanilla models are naturally trained RoBERTa's. M-Ine: Max Inc, M- Inc (R): Max Inc (R). Nat Loss (N-Loss) is the loss value on clean samples. Notice we require all clean samples here he correctly classified by all models, which results in 227, 850 and 355 samples for RTE, CoLA and MRPC, respectively. We also give the variance in the Appendix.
4.3 ABLATION STUDY AND ANALYSIS
In this sub-section, we first show the importance of reusing dropout mask, then conduct a thorough ablation study on FreeL.B over the GLUE benchmark to analyze the robustness and generalization strength of different approaches. We observe that it is unnecessary to perform shallow-layer updates on the adversary as YOPO for our case, and FreeLB results in improved robustness and generaliza- tion compared with PGD.
Importance of Reusing Mask Table (columns 2 to 4) compares the results of FreeLB with and without reusing the same dropout mask in each ascent step, as proposed in Sec. 3.3 With reusing, FreeLB can achieve a larger improvement over the naturally trained models. Thus, we enable mask reusing for all experiments involving RoBERTa.
Comparing the Robustness Table 5 provides the comparisons of the maximum increment of loss in the vicinity of each sample, defined as:
ALmax X max L(fo(X [
+8, (
(5)
which reflects the robustness and invariance of the model in the embedding space. In practice, we use PGD steps as in Eq, to find the value of A Lmaa (X,e). We found that when using a step size of 5-10-3 and 0.01 X F. the PGD iterations converge to almost the same value, starting from 100 different random initializations of for the RoBERTa models, trained with or without FreeLB. This indicates that PGD reliably finds ALmax for these models. Therefore, we compute A (X, c) for each X via a 2000-step PGD.
Samples with small margins exist even for models with perfect accuracy, which could give false sense of vulnerability of the model. To rule out the outlier effect and make A L Vxmax(,e) compa- rable across different samples, we only consider samples that all the evaluated models can correctly classify, and search for an for each sample such that the reference model can correctly classify all samples within the ball However, such choice of per-sample favors the reference model by design. To make fair comparisons, Table 5 provides the median of A L (X. e) with per-sample e from models trained by FreeLB (Max Inc) and PGD (Mac Inc (R)), respectively
Across all three datasets and different reference models, FreeLB has the smallest median increment even when starting from a larger natural loss than vanilla models. This demonstrates that FreeLB is more robust and invariant in most cases. Such results are also consistent with the models' dev set performance (the performances for Vanilla/PGD/FreeLBB models on RTE, CoLA and MRPC are 86.69/87. 41/89.21, 69.91/70.... 04770.44471.400 91.0,,9166779 91.67/91. 17/91. respectively).
For each sample. we start from value slightly larger than the norm constraint during training for and then decrease linearly until the model trained with the reference model can correctly classify after 2000-step PGD attack. The reference model is either trained with FreeI.B PGD.
8

---

Published as a conference paper at ICLR 2020
Comparing with YOPO The original implementation of YOPO (Zhang al. 2019) chooses the first convolutional layer of the ResNets as fo for updating the adversary in the "s-loop" As a result, each step of the "s-loop" should be using exactly the same value to update the adversary, and YOPO- m-n degenerates into Freel with a n-times large step size. To avoid that, we choose the layers up to the output of the first Transformer block as fo when implementing YOPO. To make the total amount of update on the adversary equal, we take the typer-parameters for FreeLB-m and only change the step size into a/n for YOPO-m-n. Table shows that FreeLB performs consistently better than YOPO on all three datasets. Accidentally, we also give the results comparing with YOPO-m-n without changing the step size a for YOPO Table 8 The gap between two approaches seem to shrink, which may be caused by using a larger total step size for the YOPO adversaries. We leave exhaustive hyperparameter search for both models as our future work.
CONCLUSION
In this work, we have developed an adversarial training approach, FreeLB, to improve natural lan- guage understanding. The proposed approach adds perturbations to continuous word embeddings using a gradient method, and minimizes the resultant adversarial risk in an efficient way. FreeLB is able to boost Transformer-baat -based model (BERT and RoBERTa) on several datasets and achieve new state of the art on GLUE and ARC benchmarks. Empirical study demonstrates that our method results in both higher robustness in the embedding space than natural training and better generaliza- tion ability. Such observation also consistent with recent findings in Computer Vision. However, adversarial training still takes significant overhead compared with vanilla SGD. How accelerate this process while improving generalization an interesting future direction.
Goldstein and Zhu supported in part by the DARPA GARD, DARPA QED for RML, and AFOSR MURI programs.
Acknowledgements:
REFERENCES
Eneko Agirre, Llu'is M M'arquez, and Richard Wicentowski (eds.). Proceedings of the Fourth In- ternational Workshop on Semantic Evaluations (SemEval-2007). Association for Computational Linguisties, 2007.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018.
Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernard Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. 2006.
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine trans- lation. In ICLR, 2018
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth PASCAL recognizing textual entailment challenge. 2009.
Yong Cheng. Lu Jiang, and Wolfgang Macherey. Robust neural machine translation with doubly adversarial inputs. In ACI., 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprini arXiv: 11880000555 2018.
Patrick L Combettes and Jean-Chrisse Christophe Pesquet. Proximal splitting methods in signal processing. In Fixed-point algorithms for inverse problems in science and engineering. 2011.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object clas- sification, and recognising tectual entailment, Springer, 2006.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI, 2019.

---

Published as a conference paper at ICLR 2020
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the International Workshop on Paraphrasing, 2005.
Sanghamitra Dutta, Gauri Joshi, Soumyadip Ghosh, Parijat Dube, and Priya Nagpurkar. Slow and stale gradients can win the race: Error-runtime trade-ofts in distributed sgd. In AISTATS, pp- 803-812, 2018.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. HotFlip: White-box adversarial examples for text classification. In ACL, 2018.
Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. NeurIPS, 2016.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, 2007.
Tom Goldstein, Christoph Studer, and Richard Baraniuk. A field guide to with a fasta implementation. arXiv preprint 1411.3406, 2014.
splitting
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Shankar Iyer, Nikhil Dandekar, and Kornl Csernai. First quora dataset release:
wtt.o/aa....////uuoo//oaa/oaataa Question pairs. 2017. -useesssooessttooPaairs
URL https: /topp:://www...// Tarrt-Qooor--atttt--Reeeeeeeee
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. In 2AACL,2018
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In EMNLP, 2017.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. ICLR, 2017.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704. 04683, 2017.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. ICIR, 2020.
Hector J Levesque, Ernest Davis, and Leora Morgenstemn. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, 2011.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In ACL, 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907. 1692, 2019b.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICLR, 2018.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv:/809.02788 2018.
Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Adversarial training methods for semi- supervised text classification. In ICLR, 2017.
Takeru Miyato, Shin-ichi Maeda. Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. TPAMI, 2019.
10

---

Published as a conference paper at ICLR 2020
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Alhussein Fawzi, Soham De, Robert Stanforth, Pushmeet Kohli, et al. Adversarial robustness through local linearization. In NeurlPS, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint 1910.10683, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging NL.P models. In ACL, 2018.
Parsa Saadatpanah, Ali Shafahi, and Tom Goldstein. Adversarial attacks on copyright detection systems. arXiv preprint 1906.07153, 2019.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2016.
A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. Davis, G. Taylor, and T. Gold- stein. Adversarial Training for Free! In NeurIPS, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In EMNLP, 2013.
Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel Rodrigues. Generalization emror of invari- ant classifiers. In AISTATS, 2017.
Robert Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. JMLR, 2014.
Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. Improving machine reading comprehension with general reading strategies. arXiv preprint 1rXiv:1810.13444 2018
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: question answering challenge targeting commonsense knowledge. In NAACL, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. arXiv preprint 1805. 12471, 2018
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. NAACL, 2018
Chaowei Xiao, Ruizhi Deng, Bo Li, Fisher Yu, Mingyan Liu, and Dawn Song. Characterizing adversarial examples based on spatial consistency information for semantic segmentation. In ECCV, 2018.
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L. Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In CVPR, 2019.
Huan Xu and Shie Mannor. Robustness and generalization. Machine learning, 2012.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NewrIPS, 2019.
11

---

Published as a conference paper at ICLR 2020
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Painless adversarial training using maximal principle. In NeurlPS, 2019.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. In ICLR, 2018.
12

---

Published as a conference paper at ICLR 2020
A ADDITIONAL EXPERIMENTAL DETAILS
A.1 PROBLEM FORMULATIONS
For tasks with ranking loss like ARC, CommonsenseQA, WNLI and QNLI, add the perturbation to the concatenation of the embeddings of all question/answer pairs.
Additional tricks are required to achieve high performance on WNLI and QNLI for the GLUE benchmark. We use the same tricks as Liu et al.] (2019b). For WNLI, we use the same WSC data provided by Liu et al. (2019b) for training. For testing, Liu et al. (219b) also provided the test set with span annotations, but the order is different form the GLUE dataset. We re-order their test set by matching. For the QNLI, we follow Liu et al. (2019b) and formulate the problem as pairwise ranking problem, which is the same for Commonsense We find the matching pairs for both training set and testing set by matching the queries in the dev set. We predict "entailment" if the candidate has the higher score, and entailmeen"" otherwise.
A.2 HYPER- PARAMEERRAMETERS
MNLI
QNLI
QQP
RTE
SST-2
MRPC
CoLA
STS-B
2E-1
I.5E-
4.56-1
1.5E-
6E-1
4E-1
2E-1
3E-1
TE-1
1E-1
1.5E-1
3B-2
1E-1
4E-2
2.5B-2
1E-1
2
2
2
3
2
3
3
3
Table 6: Additional hyper-parameters on GLUE tasks.
As other adversarial training methods, introduces three additional hyper-paramette step size a, maximum perturbation e, number of steps m. For all other hyper-parameters such as learning rate and number of iterations, we either search in the same interval as RoBERTa (on CommonsenseQA, ARC, and WNLI), or use exactly the same setting as RoBERTa (except for MRPC, where we find using a learning rate of 5 x 10 gives better results) We list the best combinations for a, c and vn for each of the GLUE tasks in Table For WSC/WNLL the best combination is e 1e 2, a 5e - 3, m 2. Notice even when ma < e, the maximum perturbation could still reach e due to the random initialization.
B VARIANCE OF MAXIMUM INCREMENT OF Loss
Methods
RTE
CoLA
MRPC
M-Inc (10 4)
M-Inc (R)
N-Loss
M-Ine (10 4)
M-Ine (R)
N-Loss
M-Inc
M-Inc (R)
N-Loss
(10 4)
(10 4)
(10 +)
(10 3)
3)
Vanilla
5.1(15087)
5.3(14346)
4.5(920)
6.1(13118)
5.1(41122)
5.2(477)
10.2(929)
10.2(955)
1,9(6)
PGD
4.7(10138)
4.9(1828)
6,2(752)
2(100)
(30.1(893)
436.1(1276)
5.7(321)
5.7(155)
5.4(54)
FreeLB
3.0(1614)
2.6(11114)
4.1(595)
1.4(1992)
1.3(6231)
7.2(615)
3.6(167)
3.6/601)
2.7(54)
Table 7: Median and Standard Deviation of the maximum increase in loss in the vicinity of the dev set samples for RoBERTa-Large model finetuned with different methods. Vanilla models are naturally trained RoBERTa's Nat Loss is the loss value on clean samples. Notice we require all clean samples here to be correctly classified by all models, which results in 227, 850 and 355 samples for RTE, CoLA and MRPC.
Table provides the complete results for the increment of loss in the interval, with median and standard deviation.
C ADDITIONAL RESULTS FOR ABLATION STUDIES
Here we provide some additional results for comparison with YOPO as complementary results to Table 4
We will release the complete results for comparing with YOPO and without variational dropout on each of the GLUE tasks in our next revision. From the current results, there is no need
13

---

Published as a conference paper at ICLR 2020
Methods
Vanilla
Freel.B-3
YOPO-3-2
YOPO-3-3
STS-B
92.20 (.2)
92.67 (.08)
(.170
92.60 (0.20)
SST-2
96.56 (.3)
96.79 (.2)
96.44 (.2)
96.33
QNLI
94.98 (.2)
94,96 (.1)
QQP
92.60 (.03)
92.55 (.05)
92.50 (.02)*
MNLI
90.61 (.1)
90.59 (.2)
90.45 (.2)
Table 8: The median and standard deviation of the scores on the dev sets of STS-B B, SST-2, QNLI, QQP and MNLI from the GLUE benchmark, cach computed from 5 runs with the same parameeasameters except for the random seeds (except for the results with YOPO QQP, which are from runs). Also note here use step step size of We FreeL.--m
parameters
size of a for the adversary of YOPO-m-n, YOPO effectively denote FreeLBB with ascent steps, and YOPO-3-m denote YOPO with n shallow-layer ascents.
in using extra shallow-layer updates that YOPO advocates, since this consistently deteriorates the performance while introducing extra computations.
14


Reviews:
[{"review_id": "rkeZXiHs9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present a new adversarial training algorithm and apply it to the fintuning stage large scale language models BERT and RoBERTa. They find that with FreeLB applied to finetuning, both BERT and RoBERTa see small boosts in performance on GLUE, ARC, and CommonsenseQA. The gains they see on GLUE are quite small (0.3 on the GLUE test score for RoBERTa) but the gains are more substantial on ARC and CommonsenseQA. The paper also presents some ablation studies on the use of the same dropout mask across each ascent step of FreeLB, empirically seeing gains by using the same mask. They also present some analysis on robustness in the embedding space, showing that FreeLB leads to greater robustness than other adversarial training methods\n\nThis paper is clearly presented and the algorithm shows gains over other methods. I would recommend that the authors try testing their method on SuperGLUE because it's possible they're hitting ceiling issues with GLUE, suppressing any gains the algorithm may yield.\n\nQuestions,\n-  In tables 4 and 5, why are only results on RTE, CoLA, and MRPC presented? If this is because there was not noticeable difference on the other GLUE datasets, please mention it in the text.\n- I realize that this method is meant to increase robustness in the embedding space, but did you do any error analysis on the models? Did they make different types of errors than models fine-tuned the vanilla way?\n\nCouple typos,\n- Section 2.2, line 1: many -> much\n- Section 4.2, GLUE paragraph: 88 -> 88.8 ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SyeaSzBZcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n-\tThis paper modifies and extends the recent “free” training strategies in adversarial training for representation learning for natural language.  The proposed “Free” Large-Batch Adversarial Training is well motived, in comparison with plain PGD-based adversarial training and the existing methods like FreeAT and YOPO, which virtually enlarges the batch size and minimize maximum risk at every ascent step. The contributions are solid. \n\n-\tThe proposed methods are empirically shown to be effective, in addition to being aligned with some recent theoretic analysis.  The models achieve SOTA on GLUE (by time the paper was submitted; it is not the best model now but that does not affect the contributions), ARC, and the commonsenseQA dataset.\n\n-\tThe paper conducted good analysis demonstrating the effectiveness of the proposed components, including detailed ablation analysis. \n\n-\tThe paper is well written. It is well structured and easy to follow.  A minor suggestion (just a personal view) is that the author(s) may consider using “natural  language” instead of just “language” in the title and may consider using more specific words like “representation” instead of “understanding”. But this is minor. \n\nI recommend an accept. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new algorithm for adversarial training of language models.  This is an important research area and the paper is well presented, has great empirical results and a novel idea. 
Decision: Accept (Spotlight)