How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?
Authors: Xinshuai Dong
Anh Tuan Luu
Min Lin
Shuicheng YAN
Hanwang Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Natural Language Processing
Pre-trained Language Models
Adversarial Robustness

Abstract: The fine-tuning of pre-trained language models has a great success in many NLP fields. Yet, it is strikingly vulnerable to adversarial examples, e.g., word substitution attacks using only synonyms can easily fool a BERT-based sentiment analysis model. In this paper, we demonstrate that adversarial training, the prevalent defense technique, does not directly fit a conventional fine-tuning scenario, because it suffers severely from catastrophic forgetting: failing to retain the generic and robust linguistic features that have already been captured by the pre-trained model. In this light, we propose Robust Informative Fine-Tuning (RIFT), a novel adversarial fine-tuning method from an information-theoretical perspective. In particular, RIFT encourages an objective model to retain the features learned from the pre-trained model throughout the entire fine-tuning process, whereas a conventional one only uses the pre-trained weights for initialization. Experimental results show that RIFT consistently outperforms the state-of-the-arts on two popular NLP tasks: sentiment analysis and natural language inference, under different attacks across various pre-trained language models.

How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?
Xinshuai Dong Nanyang Technological University & Sea Al Lab Setggiinsuai@outloooo com
Luu Anh Tuan
Nanyang Technological University anhtuan, luu@ntu. edu. sg
Min Lin Sea AI Lab
Shuicheng Yan
Hanwang Zhang
Sea AI Lab yansc&sea. com
Nanyang Technological University hanwangzhangEntu.. edu.
linmin@sea. com
Abstract
The fine-tuning of pre-trained language models has great success in many NLP fields. Yet, is strikingly vulnerable to adversarial examples, e,g., word substitution attacks using only synonyms can easily fool a BERT-based sentiment analysis model. In this paper, we demonstrate that adversarial training, the prevalent defense technique, does not directly fit a conventional fie-tuning scenario, because it suffers severely from catastrophic forgetting: failing to retain the generic and robust linguistic features that have already been captured by the pre-trained model. In this light, we propose Robust Informative Fine-T Tuning (RIFT), novel adversarial fine-tuning method from an inffrmationn-hhoretiici perspective. particular, RIFT encourages an objective model to retain the features learned from the pre-trained model throughout the entire fine-tuning process, whereas a conventional one only uses the pre-trained weights for initialization. Experimental results show that RIFT consistently outperforms the state-of-the-ars on two popular NLP tasks: sentiment analysis and natural language inference, under different attacks across various pre-trained language models.
1
Introduction
Deep models are well-known to vulnerable adversarial examples [64] 19 50 35). For instance,
fine-tuned models pre-trained very large corpora can be easily fooled by word substitution attacks using only synonyms [2] 58
32 [2] This has raised grand security challenges to modern NLP systems, such as spam filtering and malware detection, where pre-trained language models like BERT
[1 are widely deployed.
Attack algorithms 19
aim to maliciously generate adversarial examples to fool victim model, while adversarial defense aims at building robustness against them. Among the defense methods, adversarial training (64
76
40
is the most effective one 33. It updates model parameters using perturbed adversarial samples generated on-the-fly and yields consistently robust performance even against the challenging adaptive attacks 3 68].
19
48 45
However, despite its effectiveness in training from scratch, adversarial training may not directly fit the current NLP paradigm, the fine-tuning of pre-trained language models. First, fine-tuning per se suffers from catastrophic forgetting [46 to small fine-tuning data set, which may deviate too far from the pre-trained knowledge [25] 81]. Second, advessarialy fine-tuned models are more likely forget: adversarial samples are usually
i.e.. the resultant model tends over-fit
Our code will available at http::/gghhubb.coommmmggiiiiiua/
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

S: Feature by Objective Model
T: Feature by Pre-trained Model
Objective: Maximizing I(S: Y = I(SYY) Y)
HIS)
H(T)
H(Y)
I(S:T) Y)
Y: Class Label
I(S:)
Figure I: An illustration of the overall objective RIFT. Maximizing [(S; encourages features
of the objective model to be predictive of the class label, while maximizing ((S; 7| Y) encourages learning robust and generic linguistic information from the pre-trained model. (Random variable denotes extracted features of X by the objective model and T by the pre-trained language model)
out-of-distribution 38 63
thus they generally inconsistent with the pre-trained model. As result, adversarial mun-tuning fails to memorize all the robust and generic linguistic features already learned during pre- vee-training [65 57]. which are however very beneficial for robust objective model.
Addressing forgetting essential for achieving more robust objective model. Conventional wisdom such as pre-trained weight decay [8 10 and random mixout (37 mitigates forgetting by constraining 1p distance between the two models' parameters. Though effective to some extent, however, limited because change in the model parameter space only serves an imperfect proxy for that in the function space [5]. Besides, the extent to which an encoder fails to retain information also depends on the input distribution. Therefore, better way to encourage memorization is favorable.
In this paper, we follow an information-theorrtiee lens look into the forgetting problem: employ mutual information to measure how well an objective model memorizes the useful features captured before. This motivates our novel adversarial fine-tuning method, Robust Informative Fine- Tuning (RIFT). In addition to fitting a down-stream task as conventions, RIFT maximizes the mutual information between the output of objective model and that of the pre-trained model conditioned on the class label. It encourages an objective model to continuously retain useful information from the pre-trained one throughout the whole fine-tuning process, whereas the conventional one only makes use of the pre-trained weights for initialization. We illustrate the overall objective of RIFT in Figure and summarize the major contributions this paper as follows:
To the best of our knowledge, we are the first to investigate the intensified phenomenon of catastrophic forgetting in the adversarial fine-tuning of pre-trained language models.
To address the forgetting problem and achieve more robust models, propose novel approach named Robust Informative Fine-Tuning (RIFT) from an information-theoretica. perspective. RIFT enables an objective model retain robust and generic linguistic features throughout the whole fine-tuning process and thus enhance robustness against adversarial examples.
We empirically evaluate RIFT on two prevailing NLP tasks, sentiment analysis and natural language inference, where RIFT consistently outperforms the state-of-the-ard terms of robustness, under different attacks across various pre-trained language models.
2 Methodology
2.1 Notations and Problem Setting
In this paper, we focus on text classification task to introduce our method, while it can easily extended to other NLP tasks. We suppose random variables X, Y pD(x, (p,,,)), where X represents the textual input, represents the class label, pp is the data distribution, and at y are the observed values. Our goal is to build a classifier Fx()), where F.).,, referred to our objective model

---

Asserattl firatunng BERT
Standarrd Freduring 2emr
[Wea Weells (109slls
adurecatty fireuing koaserss
5/4
---
Sta card Tieuuing Recetts
5.7%
2100
5.0%
6%
4.5%
Adversrial Phstuing
39c
Standard frrsuugg BERT
Adversano Phitling NeERD
294
Standard fireturn ROTERTT
3.0%
174
2.5%
0%
fterations
(a) adversarial fine-tuning, the relative Lg dist-
(b) Trading clean accuracy off against robustness (by increasing / in Eq. ( induces increased relative L2 distance at the last epoch.
ance continuously grows as the fine-tuning proceeds.
Figure 2: Relative L2 distance between the pre-trained model and the objective model the parameter space, under different fine-tuning schemes IMDB, with small learning rate 2e-5.
in the rest of this paper, is feature extractor fine-tuned from the encoder of a pre-trained language model, e.g., Transformer [72), and q(+-)) is parameterized by using an MLP with softmaxed output. We favor classifier that is robust against ad versariaal attacks 644 i.e. maintaining high accuracy even given adversarial examples as inputs.
2.2 Adversarial Fine-Tuniie Fine-Tuning Suffers From Forgetting
Adversarial training 66
19 45
is currently the most effective defense technique
S Its training
objective can be generally formulated as:
min| E.YPP Beb(w)
max L(x, 2)]]],
where B(x)) defines the allowed perturbation space around x, and L is loss to encourage correct predictions both given vanilla samples and given perturbed examples. Such an objective function can also be applied to the fine-tuning of a pre-trained language model towards robustness, and refer to it as adversarial fine-tuning. The loss function in Eq. I can be specified following Miyato et al.
Zhang al. 79 in semi-supervised fashion as:
log (x) (KL((
where the Kullback-Leibh divergence encourages invariant predictions between and å.
Despite its effectiveness training from scratch, such an adversarial objective function may not directly fit the fine-tuning scenario. To begin with, fine-tuning itself suffers from catastrophic forgetting [46] 18 34). During fine-tuning, an objective model F.(.) tends to continuously deviate away from the traitrained one fit down-stream task [25] 81), and thus the useful information already captured before is less utilized as the tne-ttnnng proceeds. Further, adversarial fine-tuning suffers even more from the forgetting problem, the reasons of which are given what follows.
(i) Adversarial Fine- Tuning Tends to Forget: Adversarial fine-tuning targets at tackling adversarial examples, which are generally out of the manifold [38 63 of the pre-training corpora. To additionally handle them, an objective model would be fine-tuned towards a solution that is far away from the optimization starting point, i.e., the pre- pre-trained model. Figure (b) empirically shows this point: by increasing / in Eq. 2 we emphasize more on robustness instead vanilla accuracy, and consequently, at the last epoch the distance between models also increases. Besides, adversarial fine-tuning often entails more iterations to converge (several times normal fine-tuning), which further intensifies the forgetting problem, the objective model is continuously deviating away shown in Figure 2 (a).
(ii) Avversarial Fine-Tuning Needs to Memorize: Overfitting to training data dominant phe- nomenon throughout the adversarial training/fine-tung generalize better, adversarial fine-tuning should retain the generalizable features already captured
process [60] To mitigate overfitting and
by the pre-trained model [T]] 41. In addition, adversarial fine-tuning favors an objective model that

---

extracts features invariant to perturbations [70] 28 for robustness. As such, all those generalizable and robust linguistic information captured during pre-training 65| are particularly beneficial and should be memorized. We empirically validates that encouraging memorization does improve both generalization and robustness in Sec. 3.5
To address forgetting, previous methods such as pre-trained weight decay 8 10 and random mixout 37 have shown their effectiveness in stabilizing fine-tuning [81]. However, they focus only on the parameter space, in which they encourage an object model to be similar to the pre-trained one. Distance in the parameter space can only approximately characterize the change in the function space, and fails to take the data distribution into consideration. A more natural way to capture the extent to which a model memorizes or forgets, should be using the mutual information between outputs of the two models, and we provide our solution as follows.
2.3 Informative Fine- Time-Tuning
We first use an information tthoooeticaee perspective look into how a pre-trained model should be leveraged throughout the whole fine-tuning process.
We define random variable T (X) as the feature of X extracted by the pre-trained language model F.)., and as the observed value of T. Similarly, we define random variable S = (X) as the feature of X extracted by our objective model (.). and the observed value of S. We formulate an overall fine-tuning objective as follows. The motivation is to train F-(-) such that S is capable of predicting Y. as well as preserving the information from T..
max I(S; Y,T),
i.e. maximizing the mutual information between (i) the feature extracted by the objective model and (ii) the class label plus the feature extracted by the pre-trained model. Since the pre-trained model (-) is fixed deterministic function, the fine-tuning objective in Eq. in essense encourages F,(.) to output features that contain as much information as possible for predicting Y and T. This enables the objective model to learn from the pre- pre-trained language model via T throughout the whole fine-tuning process, and thus helps address the forgetting problem.
However, the objective defined in Eq. 3 is generally hard to optimize directly. Therefore, we decompose Eq. into two terms as follows:
I(S: Y,T) = (
(4)
where T(S; Y) measures how well the output of our objective model predict the label, and [S Y) measures when conditioned on the class label, how well the output features of the two models can predict each other. Visualization of each component can be seen in Figure for more intuitive understandings. We next introduce how each term in the right-hand side of Eq. 4 can be transformed into tractable lower bound for optimization.
(i) Maximizing I(S;Y): We treat (yyss). which is the classification layer that takes the features
extracted by the objective model input, variational distribution of p(yss), and derive variational lower bound on T(S; Y) following variational inference (33 as follows:
I(S;Y) H(Y) Enyy- log ((v|=)] (L(()))))((1s))
(5)
2 HYY) En [- [z)/)())))]
(6)
where H(Y) is a constant measuring the shannon entropy of Y, and Ex,y log {(vss)] is essentially the cross-entropy loss using {)y)s) for classification. Then, the objective of maximizing I(5; Y) can be achieved by minimizing Ex,([- log {(y[s)] instead.
(ii) Maximizing I(S;TI Y): The definition of the conditional mutual information /(S;T) Y) is as follows:
p(s, ((y
Y) Ey-po (w) [[S:T) Y = (x)) (nr [log
p(sly)p((h)) 1].
To achieve a tractable objective for maximizing Eq. 7 we employ noise contrastive estimation 20 49 and derive lower bound -Linto on the conditional mutual information. This can be summarized in the following Lemma (the proof of which can be found in Appendix A.2):

---

Maximize ((S.Y)
Maximize ((S:)
Muximize I(S:T) Y)
Figure 3: Visualization of the learned geometry using 500 random samples from IMDB by t-SNE 711. and denote samples from two different classes respectively. Different colors represent different data point IDs (approximately due to limited color space). In (c) and (d), (1)x) are projected to S with radius 1, while (x) with radius 0.9. For each sub-image: (a) The geometry of (x(x) before fine-tuning, represents the data manifold. (b) Maximizing I(S;Y) Y) encourages separating the whole data manifold into two class-specific data manifolds. (c) Maximizing T(S; T) by contrastive loss in essence encourages alignment between two circles and uniformity over the whole data manifold. (d) Maximizing / ( T Y) by contrastive loss in essence encourages alignment and uniformity inside each class-specific data manifold. Best view in color with zooming in.
Lemma 1. Given { N that is sampled i.idd from po(aly), 8; Fo(xi), and 4, = Fr(x;). elulssed I( (S:T Y) is lower bounded by -Linoo E Ey--pp() B(x,,)) IN log sy croenst log N].. and fy is a score function indexed by y-
By leveraging Lemmal I Linto can be computed using a batch of samples and then minimized for maximizing [S Y). The score function fy is defined as the inner product after non non-linear
where 9) and projections into space of hyper-sphere following 99 (a b) [a)( [a](0]al133011>> og parameterized by using MIPs.
2.4 Conditional Mutual Information Better Fits A Down-Stream Task
Above we have introduced the training objective of informative fine-tuning and decomposed it into I(S; Y) and T( Y) for optimization. However, one may wonder why not maximize I T( directly instead of [(: Y). From an information-theorrtieei perspective, we maximize I(S; YS: and TT, the intersection of the three circles in Figure [ i.e., the interaction information of them, repeatedly optimized and might induce confliction. To further elaborate this point, we look into contrastive loss and give explanation follows.
We consider contrastive learning by decomposing it into the encouragement of alignment and unifor- mity following [74). For example, maximizing ((S;T)= HS) (S is to decrease H (S)T) and increase H (S). Decreasing H (S(T)) corresponds to encouraging alignment in contrastive learn- ing, which aims to align 8; and & in the sense of inner product when they are projected into hyper-sphere. In the meanwhile, increasing (S) corresponds to uniformity in contrastive learning, which encourages 8; and s,, where i # j. to diffuse over the whole sub-space as uniformly possible. It can be seen in Figure (c) that, in the space of S alignment is achieved in that points from two different circles but with the same color are aligned, and uniformity is achieved in that all points diffuse over the whole S
However, when maximizing (S) the uniformity in the contrastive loss is encouraged over the whole data manifold. Diffusing uniformly over the whole data manifold can be against the objective of maximizing I(S: Y). which aims to separate the whole data manifold into class-specific parts as shown in Figure (b). In contrast, maximizing T(S:T) Y) only encourages uniformity inside each class-specific data manifold, which is complementary to maximizing T(S; Y), , as show in Figure 3 (d). In the meanwhile, the alignment is still enforced. More empirical support can be found in Sec.
5

---

Algorithm RIFT
Input: dataset D. syper-parametent of AdamW 43 Output: the model parameters and ø
1: Initialize 0 using the pre-trained model, and initialize & and randomly.
2: repeat
3: Sample y PD (t) and then {I,, v) pp(p)y)
4: for every Zi. y in the mini-batch {Ti, v) do
5: Find é; by solving Eq.
6: end for
7: Compute the loss function defined in Eq. 11 and update 8, p, and (p by gradients.
8: until the training converges.
2.5 Robust Informative Fine-Tuning
In this section, we are going to put the objective function of informative fine-tuning into the adversarial context and introduce Robust Informative Tuning (RIFT).
To robustly train a model, the adversarial examples for training should be formulated and generated first. As our end goal is to enhance robustness in down-stream tasks, the generation process of adversarial examples should focus on preventing a model from predicting the ground truth label. However, using label for generating adversarial examples in training often induces label leaking (36), ie., the generated adversarial examples contains label information which is used as a trivial short-cut generate & in a self-supervised fashion:
by classifier. As such, follow [8 79
arg max (L((()FF(())))((()FFCC) x'eB(n)
By solving Eq. 8 f is found induce the most different prediction from that of vanilla sample in terms of KL, inside the attack space B(x).
Next, we introduce how to robustly optimize each objective in the informative fine-tuning by using a.
(i) Robustly Maximizing I(S;Y): As shown in Sec. 2.3 maximizing I I(S; Y) can be achieved by minimizing cross- cross-entropy loss instead. To encourage adversarial robustness, this cross-entropy loss can be upgraded encourage both correct predictions on and invariant predictions between and We formulate such an objective function follows:
48
79
min Ly-task:
Cyessk
E
[- log q(vl (x))) + (KL(((FF((())))((((()))
(9)
2,yPPD
where denotes the parameters F.(-) and @ denotes the parameters of g(++). Minimizing Cr-msk in Eq. 1 corresponds to adversarilly maximizing I(SYY) for robust performance in down-stream task. By doing so, the adversarial example generated by Eq, S first and then both and are used to optimize the model parameters 0 and Φ.
(ii) Robustly Maximizing I(SITI Y): We aim to maximize the conditional mutual information I[ Y), but under an adversarial distribution of input data. We formulate such a term as I(Š:T|Y), Y), where random variable $ Fx(X), X Paav (ilu, 0, di BB, and sampling from Padva the adversarial distribution of input data, is to generate adversarial example & by Eq. S
To optimize 1($;T) Y), we propose -Crintto as lower bound it, and formulate the objective to minimize Lriinto as follows (similar to - -Lnnto by using Lemma
N
log (F(.t))))) Σ efo(3.)))) log NJ].
min Crinfo 0.5
Criinoo
E p-pD(y)
(10)
(a.
where 3 F.(2:), 44 F((x,, and y denotes the parameters of all the score functions fy- By Eq. 10 are able to encourage F-(-) to retain information from F:(-) in robust fasion.
Noted that, Lrinfo we do not use á; to extract features from the pre-trained model. This follows the spirit of Knowledge Distillation (2) (though the setting is different in that student is expected to perform identically to teacher in knowledge distillation, while in fine-tuning does not): the data used to extract features of teacher should be inside the domain for which the teacher is trained. A

---

Table Accuracy(%) of different fine-tuning methods under attacks on IMDB.
Method
Model
Genetic
PWWS
Method
Model
Genetic
PWWS
Standard
BERT
12.5
40.7=1.1
Standard
RoBERTa
42.1
45.6a3.1
Adv-Base
BERT
74.8
68.3
Adv-Base
RoBERTa
70.3 63.3
Adv-PTWD
BERT
73.9.0.4
29.1.a.7
Adv-PTWD
RoBERTa
69.3.1.4
64.4.0.3
Adv-Mixout
BERT
75.4
68, 68.8.0.6
Adv-Mixout
RoBERTa
70.6.1.
63.9
RIFT
BERT
77.2
=0.
RIFT
RoBERTa
73.5
66.3
(a) Accuracy (%) based on BERT-base-uneased.
Accuracy (%) based RoBERTa-base.
Table 2: Accuracy($b) different fine-tuning methods under attacks on SNLL
Method
Model
Genetic
PWWS
Method
Model
Genetic
PWWS
Standard
BERT
40.7
19.420.4
Standard
RoBERTa
43.4+1.2
20.4
Adv-Base
BERT
75.720.5
72.920.2
Adv-Base
RoBERTa
82.6.0.6
79.920.7
Adv-PTWD
BERT
75.2+1.0
72.620.5
Adv-PTWD
RoBERTa
81.2
78.9
Ady-Mixout
BERT
76.320.8
73.221.0
Adv-Mixout
RoBERTa
82.620.9
80.620.3
RIFT
BERT
77.5.0.9
74.3a1.1
RIFT
RoBERTa
83.5
81.1
(a) Accuracy (%) based on BFRRT-base-uncased.
(h) Accuracy (%) based RoBERTa-hase.
down-stream NLP task, e.g. sentiment analysis, has data domain that is generally sub-domain of the pre-training corpora, while à can be significantly off the pre-training data manifold.
Objective Function of Robust Informative Fine- Tuning (RIFT): The objective function of RIFT a combination of Cystsk and Leinno. defined as follows:
min tr + OtCrrinfo 0.0.0
(11)
where the hyper-parameter a controls what extent we encourage an objective model absorb information from the pre-trained one (ablation on a can be seen in Sec. 3.5). We summarize the whole training process Algorithm and include more implementation details in Appendix A.1.
Experiments
3.1 Experimental Setting
Tasks and Datasets: We evaluate the robust accuracy and compare our method with the state-of-the-
arts on: (i) Sentiment analysis using the IMDB dataset [44) (ii) Natural language inference using the SNLI dataset [6]. We mainly focus on robustness against adversarial word substitutions 58). as such attacks preserve the syntactic and semantics very well [3] 78 [2]. and are very hard to detect even by humans. Under this attack setting, any word in the input sequence can be substituted by semantically similar word of it (often its synonym). We evaluate robustness on 1000 random examples from the testset of IMDB and SNLI respectively following 12).
Model Architectures: We examine methods and compare with state-of thtt-oofthhearsss the following two prevailing pre-trained language models: (i) BERT-baas-uncased [1]. (ii) RoBERTa-base [41].
Attack Algorithms: Two powerful attacks are employed: (i) Genetic 2 based population
algorithm. Aligned with [2]. the population size and iterations are set as 60 and 40 respectively. (ii) PWWS [58) based word saliency. We only attack hypothesis SNLI aligned with [31 [2].
Substitution Set: We follow [3 to use the substitution set from
2 and the same language
model constraint is applied to Genetic attacks and not to PWWS attacks.
3.2 Comparative Methods
(i) Standard Fine-Ture Tine-Tuning: The standard fine-tuning process first initializes the objective model by the pre-trained weight, and then cross-entropy loss to fine-tune the whole model.
7

---

72
Accurecy under Genetic (16)
RFF
RFT Adv-Misou
Adv-Miouu Advotians
68
66
Accuracy
84
db
Varilla accuracy
Varilla accuracy 196)
(a) Accuracy (%) under Genetic attacks.
(b) Accuracy (%) under PWWS attacks.
Figure 4: Tradeoff curve between robustness and vanilla accuracy of BERT-based model on IMDB.
Table 3: Accuracy(%) of RIFT with maximizing Y(S;TT)) and 1(5 respectively.
Maximizing
Model
Genetic
PWWS
Maximizing
Model
Genetic
PWWS
I(S:T) Y)
BERT
77.2
70.1
I(S: Y)
BERT
77.5
74.3
I(S:T)
BERT
76.1
69.4
I(SIT)
BERT
76.6
72.1
I(S:T))
RoBERTa
73.5
66.3
I(S;T) Y)
RoBERTa
83.5
81.1
I(SIT)
RoBERTa
72.0
65.3
I(SIT)
RoBERTa
82.5
79.4
Accuracy (%) under attacks IMDB.
(b) Accuracy (%) under attacks SNLI
(ii) Adversarial Fine-Tuning Baseline (Adv-Base): We employ the sate-offtheeart defense against word substitutions, ASCC-Defense [2], as the adversarial fine-tuning baseline. This method is not initially proposed for pre-trained language models but can readily extend to perform adversarial fine-tuning. During fine-tuning, adversarial example 2, which is sequence of convex combinations, generated first and then both and & used for optimization using objective defined in Eq. 9
(iii) Ady Pre-Trained Weight Decay (Ad=-PTW)): fre-trained weight decay 8 10 penalizes X| aaj Wee 22 and mitigates catastrophic forgetting [75 37]. We combine it with the adversarial baseline for comparisons and À chosen as 0.01 on IMDB and 0.005 on SNLI for best robustness.
(iv) Ady + Mixout (Adv-Mixout): Motivated by Dropout 62 and DropConnect 73 Mixout [37 proposed to addresses catastrophic forgetting in fine-tuning. each iteration each parameter is replaced by its pre-trained counter-part with probability fine-tuning baseline for comparisons with our method and m chosen as 0.6 for best robustness.
We combine it with the adversarial
The proposed adversarial fine-tuning method. It can be deemed the adversarial fine-tuning baseline plus the Lrinfo term. We as 0.2 for all score functions fy. For best robust accuracy, is chosen as 0.1 and 0.7 on IMDB and SNLI respectively. Ablation study on a can be seen in Sec. 3.5
(v) Robust Informative Vine-Tuning (RIFT):
For fair comparisons, all compared adversarial fine-tuning methods use the same f on a same dataset, i.e., 3 10 IMDB and /3 5 SNLI, both of which are chosen for the best robust accuracy. Early stopping [60] is used for all compared methods according to best robust accuracy More implementation details and runtime analysis can be found in Appendix A. and A.3.
3.3
Main Result
In this section we compare our method with state-of-the-arts by robustness under attacks. As shown in Tables T and 2 RIFT consistently achieves the best robust performance among state-of-the-ards in all datasets across different pre- -reitra language models under all attacks. For instance, on IMDB our method outperforms the RoBERTa-baen based runner-up method by 2.9% under Genetic attacks and 1.9% under PWWS attacks. On SNLI based on BERT, we surpass the runner-up method by 1.2% under Genetic attacks and 1.1% under PWWS attacks. addition, RIFT consistently improves robustness upon the adversarial fine-tuning baseline, while the improvements by other methods are not stable. We observe that on SNLI, we surpass the runner-up by relatively small margin compared to IMDB. This may relate to the dataset property: input from SNLI has smaller attack space (on average 6.5'

---

Table 4: Ablation study on the hyper-parameter a.
Parameter a
Vanilla
Genetic
PWWS
Parameter a
Vanilla
Genetic
PWWS
0.00
78.1
74.8
68.3
0.00
79.4
75.7
72.9
0.05
78.4
76.5
69.2
0.50
80.0
76.9
73.8
0.10
78.3
77.2
70.1
0.70
80.4
77.5
74.3
0.30
78.3
76.2
69.5
1.00
80.6
77.3
73.2
(a) Acc (%) of RIFT based on BERT on IMDB.
(b) Ace (%) of RIFT based on BERT on SNLI.
combinations of substitutions on SNLI compared to 620H on IMDB), and thus smaller absolute space left for improvement. For results of vanilla accuracy please refer Appendix A.4.
3.4 Conditional Mutual Information Does Fit a Down-Stream Task Better
In this section empirically validate that maximizing [(S; (8) cooperates with a down-stream task better than maximizing I(S does. We plot two sets of results of our method with max- imizing [(S;T] Y) and T() respectively (using similar noise contrastive loss with all other hyper-pprameters the same). As shown in Table RIFT with maximizing [(S; T] Y) consistently outperforms that with maximizing I(S;T) in terms of robustness under all attacks both IMDB and SNLI, which serves as an empirical validation Sec. 2.4
3.5 Ablation Study and Tradeoff Curve Between Robustness and Vanilla Accuracy
In this section conduct ablation study on a, which is the weight of Cr-ino in Eq. It aims to control to what extent the objective model absorbs information from the pre-trained one. As shown in Table 4 good value of improves both vanilla accuracy and robust accuracy; e-g., IMDB increasing from 0 to 0.1 results in increased vanilla accuracy by 0.2% and increased robust accuracy under Genetic attacks by 2.4%. This demonstrates that RIFT does motivate the objective model to retain robust and generalizable features that are beneficial to both robustness and vanilla accuracy. When goes too large, the objective of fine-tuning would focus too much on preserving as much information from the pre-trained model as possible, and thus ignores the down-stream task.
One may wonder whether informative fine-tumm tie-ttnnngg itself can improve the clean accuracy upon normal fine-tuning. The answer is yes; e.g., on IMDB using RoBERTa, informative fine-tumin tine-tuning improves about 0.3% clean accuracy upon normal fine-tum fine-tuning baseline. The improvement is not very significant as normal fine-tuning targets at vanilla input only and thus suffers less from the forgetting problem. One common problem in contrastive loss is that the complexity grows quadratically with N, but larger N contributes to tighter bound on the mutual information
One potential improvement maintain dictionary updated on-the-fly like (21) and we will leave for future exploration.
(4 53
We finally show the trade-off curve between robustness and vanilla accuracy Figure As shown, RIFT outperforms the state-of- tiee--ffthhhrrttt the-arts in terms of both robust accuracy and vanilla accuracy. It again validates that RIFT indeed helps the objective model capture robust and generalizable information to improve both robustness and vanilla accuracy, rather than trivially trading off one against another.
4 Related Work
Pre-Trained Language Models:
Language modeling aims at estimating the probabilistic density of the textual data distribution. Classicial language modeling methods vary from CBOW, 3kip-Gram
[7]. to GloVe (5l) and ELMo [52]. l. Recently, Transformer [72] has demonstrated its power in
language modeling, e.g., GPT [55] and BERT [IT]. The representations learned by deep pre-trained
language models are shown to be universal and highly generalizable [77] 41, and fine-tuning
pre-trained language models becomes a paradigm in many NLP tasks 255
Adversarial Robustness: DNNs have achieved success in many fields, but they are susceptible to adversarial examples (44. In NLP, attacks algorithms include char-level modifications 24 13
15 54 sequence-level manipulations (29
59
30
82 and adversarial word substitutions
78 For defenses, adversarial training 64 19 generates adversarial examples
on- nn-thh--ly fly and is currently the most effective. Miyato et al. 48 first introduce adversarial training to

---

of Education (MOE) Academic Research Fund NLP using L2-ball to model perturbations. and later more scometry-alignee methods like first-order approximation 13,, axis-aligned bound [3T 27]. and convex hull [1]] become favorable.
The fine-tuning of pre-trained language models 25 can be very unstable [T], as during fine-tuning the objective model can deviate much from the
Fine-Tunin Tine-Tuning and Catastrophic Forgetting:
pre-trained one, and easily over-fits to small fine-tuning sets [25]. Such phenomenon is referred as
catastrophic forgetting [46. [8] in fine-tuning (25 37 81). Methods to address catastrophic forgetting
include pre-trained weight decay
g
10 81. learning rate decreasing [25]] and Mixout regularization
33.. These methods focus on parameter space to constrain the distance between two models while our method addresses forgetting following an
perspective. In continuous [42 [6 39 [ and regularization- based [34] [4] [26] also aims
learning, method like rehearsal-
at the forgetting problem, but under a different setting: they focus on balanced performance on tasks with previous data, while in our setting the pre-training corpora are not available and language modeling is not our concern.
has demon- strated its power in learning self-supervised representations ( (21]. The contrastive loss such as InfoNCE[4]], InfoMax [23, is proposed as maximizing the mutual information of features in dif- 69 66 [74]. This paper shares the same perspective
Contrastive Learning and Mutual Information:
Contrastive learning
49
23 61
ferent views, and further discussed in [53 67
of information theory, but looks into a different problem, addressing forgetting for fine-tuning. Addressing forgetting motivates our objective of maximizing the mutual information between the outputs of two models, which is different from conventional contrastive learning, and cooperation with fine-tuning motivates the use of conditional mutual information, which is quite different also. Fischer and Alemi [16] propose to restrict information for robustness in vision domain. Restricting information to some extent helps model ignore spurious features, but may be at the cost of ignoring robust features as well. contrast, the proposed method aims to retain the generic and robust features that are already captured by pre-trained language model, and thus better fits a fine-tuning scenario.
5 Discussion and Conclusion
In this paper, we proposed RIFT to fine-tune a pre- trained language model towards robust down-stream performance. RIFT addresses the forgetting problem from an information thhoorettceee perspective and consistently outperforms state-of-the-arts in experiments. We hope that this work contribute to the NL.P robustness in general and thus to more reliable NI.P systems in real-life applications.
Acknowledgements
This work is supported by the Singapore Ministry (AcRF) Tier 1 (S21/20) and Tier 2.
References
[1] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In NeurlPS, 2019.
[2] Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. Generating natural language adversarial examples. In EMNLP, 2018.
[3] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018.
[4] Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine translation. In ICLR, 2018
[5] Ari S Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space. In ICML, 2019.
[6] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. A large annotated corpus for learning natural language inference. In EMNLP, 2015.
10

---

[7] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks, In SP. 2017.
[8] Ciprian Chelba and Alex Acero. Adaptation of maximum entropy capitalizer: Little data can help lot. Computer Speech & Language, 20(4):382-399 2006.
[9] Ting Chen, Simon Korblithh Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[10] Hal Daumé III. Frustratingly easy domain adaptation. arXiv preprint arXiv:0907.1815,, 2009.
[11] Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[12] Xinshuai Dong, Anh Tuan Luu, Rongrong Ji, and Hong Liu. Towards robustness against natural language word substitutions. In ICLR, 2021.
[13] Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White-box adversarial examples for text classification. ACL, 2018
[14] Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty- guided continual learning with bayesian neural networks. In ICLR, 2020.
[15] Steffen Eger, Gözde Gül Şahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. Text processing like humans do: Visually attacking and shielding nlp systems. In NAACL, 2019.
[16] Ian Fischer and Alexander A Alemi. Ceb improves model robustness. Entropy, 22(10):1081, 2020.
[17] Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-hox generation of adversarial text sequences to evade deep learning classifiers. In SPW. IEEE, 2018.
[18] Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in mudient-basedd neural networks. arXiv preprint 1111v.1312.6211, 2013.
[19] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In ICLR, 2015.
[20] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, 2010.
[21] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[22] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.00 1,000,00551,,0000
[23] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
[24] Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. Deceiving google's perspective api built for detecting toxic comments. arXiv preprint
1388, 2017.
[25] Jeremy Howard and Sebastian Ruder. Universal language model tne-tuning for text classifica- tion. In ACL, 2018
[26] Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. Distilling causal effect of data in class-incremental learning. In CVPR, 2021.
[27] Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krishnamurthy Dvijotham, and Pushmeet Kohli. Achieving verified robustness to symbol substitutions via interval bound propagation. In EMNLP, 2019.
11

---

[28] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry, Adversarial examples are not bugs, they are features. arXiv preprint arXiv: 1905.0220 2019.
[29] Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example genera- tion with syntactically controlled paraphrase networks. NAACL, 2018.
[30] Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In EMNLP, 2017.
[31] Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. Certified robustness to adver- sarial word substitutions. In EMNLP, 2019.
[32] Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? natural language attack on text classification and entailment. AAAI, 2020.
[33] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
[34] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):351133526, 2017.
et al.
[35] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world, 2016.
[36] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning scale. In ICLR, 2017.
[37] Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In ICLR, 2020.
[38] Yingzhen Li, John Bradshaw, and Yash Sharma. Are generative classifiers more robust to adversarial attacks? In ICML, 2019.
[39] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12):2935-299 2947, 2017.
[40] Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. Deep text classification can be fooled. In I/CAI, 2018.
[41] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 2019.
[42] David Lopez-Pazz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In NeurlPS, 2017.
[43] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[44] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, 2011.
[45] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018
[46] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. Elsevier, 1989.
[47] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. NeurIPS, 2013.
[48] Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Adversarial training methods for semi- supervised text classification. In ICLR, 2017.
12

---

[49] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 187.00887, 2018.
[50] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In EuroS&P. IEEE,
2016.
[51] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543, 2014.
[52] Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-HLLT, 2018.
[53] Ben Poole. Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In ICML, 2019.
[54] Danish Pruthi, Bhuwan Dhingra, and Zachary C Lipton. Combating adversarial misspellings with robust word recognition. In ACL, 2019.
[55] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving lan- guage understanding by generative pre-training. https://s3-us--west2.amaaae www.sss-...ammzawsss.cooo wwwpsssse..aaaoooosoooooo/onnnne com//opnaii
2018.
[56] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, 2017.
[57] Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. NeurIPS, 2019.
[58] Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In ACL, 2019.
[59] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging nlp models. In ACL, 2018
[60] Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In ICML, 2020.
[61] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NeurIPS. 2016.
[62] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199) 2014.
[63] David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and generalization. In CVPR, 2019.
[64] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2013.
[65] Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert re-discovers the classical nlp pipeline. In ACL, 2019.
[66] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
[67] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. In NeurIPS, 2020.
[68] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks adversarial example defenses. In NeurlPS, 2020.
[69] Michael Tschannen, Josip Djolonga, Paul Rubenstein, Sylvain Gelly, and Mario Lucie. On mutual information maximization for representation learning. In ICLR, 2020.
13

---

[70] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry, Robustness may be odds with accuracy. arXiv preprint 1rXiv:180..21555 2018.
[71] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9111), 2008.
[72] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[73] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In ICML, 2013.
[74] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020.
[75] Georg Wiese, Dirk Weissenborn, and Mariana Neves. Neural domain adaptation for biomedical question answering. CoNLL, 2017.
[76] Eric Wong, Frank R Schmidt, and Zico Kolter. Wasserstein adversarial examples via projected sinkhorn iterations. In ICML, 2019.
[77] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS, 2019.
[78] Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. Word-level textual adversarial attacking as combinatorial optimization. In ACI, 2020.
[79] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. ICML, 2019.
[80] Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. Generating fluent adversarial examples for natural languages. In ACL, 2019.
[81] Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting faw-sample bert fine-tuning. In ICLR, 2021.
[82] Zhengli Zhao. Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. In ICLR, 2018
14


Reviews:
[{"review_id": "oskL_m1pJy", "reviewer": "Reviewer_jFCo", "summary": "This paper argues that applying adversarial training into the fine-tuning of the pre-trained language model suffers severely from catastrophic forgetting. It then proposes Robust Informative Fine-Tuning (RIFT) to encourage an objective model to retain the features learned from the pre-trained model. The results on sentiment analysis and natural language inference tasks show that the proposed method is able to outperform the strong baseline models in terms of Genetic and PWWS attacks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Strengths:\nThis paper tackles the adversarial fine-tuning problem from an information-theoretical perspective and tries to alleviate the catastrophic forgetting issue. The proposed methods (RIFT) are technically sound and the results on two classification-based tasks show that RIFT is more robust to the adversarial attacks.\n\n\n2. Weaknesses:\n- The improvements over the baseline models are relatively marginal. The paper argues that previous adversarial fine-tuning would cost severe catastrophic forgetting issue, and I expect the improvement to be higher if the propose method can well tackle this issue.\n- This paper needs analyses in terms of how the proposed method addresses the catastrophic forgetting issue compared to baseline models, since the main claim of the RIFT is to tackle this issue.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Since this paper aims to cope with the classification-based tasks, it would be better to conduct experiments on more tasks. For example, try on GLUE benchmark.\n2. How many times did you run the experiments for each task and calculate the averaged performance? Since the improvements for many tasks are marginal and the variances looks not low enough. It would be better if you conduct a statistical significant test.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "TMd0HSaLDFq", "reviewer": "Reviewer_NEaE", "summary": "This paper introduces a novel adversarial fine-tuning method, which they apply to language model fine-tuning. Their method, named RIFT, takes an information-theoretic perspective to define the new loss; this paper is very clearly written and includes mostly useful figures that help explain their novel approach. They show strong experimental results with a number of baselines, and run an ablation study for the hyperparameters introduced by their approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: As far as I am aware, their method is new. They derive their loss and clearly describe each of the components it is comprised of. \n\nQuality: I'm fairly confident their approach is technically sound, though they use a variational lower bound to bound one part of their loss and I always forget exactly the standard way variational lower bounds are setup. Here the bound is formed by just dropping the \"+KL(p,q)\" term; I'll leave it up to the other reviewers to determine if this is the correct way to form this lower bound.\nQuestion for the authors: did you try other choices for the score function f_y (line 144)? It would be nice to include some justification here.\n\nThe main claims are that the approach leads to improved robustness, and their do a good job of arguing why (from an information-theoretic perspective) their approach should lead to improve robustness, then actually showing it does using two different models. This is a strong claim, well-supported by the presented evidence.\n\nClarity: The submission is very clear. In fact, they might have even spent more space making it extra clear than is needed -- Figure 1 helps a reader understand at a glance what is being optimized, then the terms in figure one are optimized in Figure 3 and we can see the contribution from the different terms. Most of the paper also goes into detail deriving and explaining these terms, and the paper could benefit (only a little) from some additional experiments taking some of that space. I would also recommend including more information about the hyperparameters used for the experiments; presumably default hparams were used, but it would be good for you to be clear about exactly what they were.\n\nSignificance: If the experimental results hold on further experiments (e.g., with vision models) these results could be quite significant to the adversarial robustness community. Most of the paper is focused on the derivation of the new approach, and the clarity of that part is also valuable.\n\nDetailed comments:\nI would recommend including some vision experiments in the final version. The ablations are great -- it's really valuable to experimentally show the effect of alpha, for example. Figure 4 is also very strong, and you could consider moving that up in the paper, as the better robustness / accuracy tradeoff is exactly what most practitioners are looking for. I would recommend moving the related work section close to the end, but that's a style choice. \n\nIn general, robustness is not my main area of research, so if the other reviewers have strong opinions (e.g., about appropriate baseline comparisons) I would defer to them. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They do a sufficient job of making appropriate claims for the paper, and so implicitly describe limitations. They don't really describe societal impact, but for this kind of work that's okay.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "E90LUYEXSp", "reviewer": "Reviewer_kRk6", "summary": "This paper proposes using a mutual information objective for adversarially fine-tuned pretrained masked language models. The approach seems to reduce the problem of catastrophic forgetting of adversarial finetuning of pretrained masked language models and  achieve adversarial training approaches in sentiment analysis and natural language inference tasks. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Originality\n  * This paper proposed a new mutual information based objective for adversarially finetuning masked language models. There were few previous studies using information theory inspired adversarial training, but the authors extend it in the context of masked language model finetuning. \n\n- Quality: \n  * One of the claimed contributions is to show the catastrophic forgetting phenomenon of adversarial finetuning of pretrained language models. I found this claim a bit not solid. From Figures 1 and 2, it is based on the relative L2 distance of the pretrained and finefuned models in the parameter space. However it does not necessarily mean it is really forgetting all the generic linguistic features. It is more straightforward to justify this by showing downstream tasks performance or language model performance. In addition, it is common to use a smaller learning rate or early stopping in fine tuning. Hence, showing the enlarged discrepancy in parameters space in Figure 1 is not that informative. \n  * There is much more work in the image domain for adversarial training. For example, TRADES [1], ensembling, early stopping [2], etc. I wonder whether the adversarial baseline is too simple and doesn’t incorporate few newer findings in the community. \n\n    [1] https://arxiv.org/abs/1901.08573\n\n    [2] https://arxiv.org/abs/2103.01946\n\n- Clarity:\n  * When mentioning pretrained language models, it is not clear to me in the beginning whether the proposed approach is for masked language models (e.g., BERT) or for autoregressive language models (e.g., gpt-2).\n\n- Significance:\n  * The presented findings on finetuning on masked language models seems promising, but it is hard to justify as this is evaluated on new tasks. In addition, it will be more insightful to see quantitative examples or more analysis for the actual adversarial attack (success, fail ones) beyond reporting accuracy -- this is because adversarial attacks could potentially be imperfect in the NLP domain, or there might be gradient obfuscation problems in claiming robust accuracy. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Mentioned above in the quality, significance parts.\n", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a new training technique more robust adversarial training approach.  The method performs well under a variety of scenarios.  The paper is thorough and clear. In post-rebuttal discussion a few points were raised, which I would like the authors to include in their paper. 

Several reviewers wondered if this technique would work in the vision domain.  Reviewer kRk6 says "There is a line of work using information theoretical justification [3] in image domain, but these methods do not work well under stronger attacks in practice. " Could you please mention this, and hypothesize about why it might work well here, but not in the vision domain? 

Reviewer kRk6 also says "Especially in the adversarial training, one can tune the tradeoff between clean and adversarial performance. It is not clear whether the baseline and this work are tuned fairly." Please make sure the fairness of your baselines is thoroughly explained.

In the discussion period, the authors shared results on more runs and statistical significance tests.  Please incorporate those results in the final paper.
Decision: Accept (Poster)