Mask-tuning: Towards  Improving  Pre-trained Language Models' Generalization
Authors: Somayeh Ghanbarzadeh
Hamid Palangi
Yan Huang
Radames Cruz Moreno
Hamed Khanpour
Venue: Submitted to ICLR 2023 (2023)
Keywords: NLP
Pre-trained langugae model
out-of-distribution learning
robust generalization
fine-tuning

Abstract: Pre-trained language models have the known generalization problem. This issue emerges from the pre-trained language models' learning process that heavily relies on spurious correlations, which work for the majority of training examples but do not hold in general. As a consequence, the models' performance drops substantially on out-of-distribution datasets. Previous studies proposed various solutions, including data augmentation and learning process improvement. In this paper, we present Mask-tuning, an approach that alleviates the impact of spurious correlations on the fine-tuning learning process. To achieve this goal, Mask-tuning integrates masked language training into the fine-tuning learning process. In this case, Mask-tuning perturbs the linguistic relation of downstream tasks' training examples and computes masked language training loss. Then, the perturbed examples are fed into fine-tuning process to be classified based on their ground-truth label and compute the fine-tuning training loss. Afterward, Mask-tuning loss-- a weighted aggregation of masked language model training loss and fine-tuning loss-- updates the masked language model and fine-tuning through training iterations. Extensive experiments show that Mask-tuning consistently improves the pre-trained language models' generalization on out-of-distribution datasets and enhances their performance on in-distribution datasets. The source code and pre-trained models will be available on the author's GitHub page.

Under review conference paper at ICLR 2023
MASK-TUNING: TOWARDS IMPROVING PRE-TRAINED LAN- GUAGE MODELS' GENERALIZATION
Anonymous authors Paper under double-bling review
ABSTRACT
Pre-trained language models have the known generalization problem. This issue emerges from the pre-trained language models' learning process that heavily relies on spurious correlations, which work for the majority of training examples but do not hold in general. As consequence, the models' performance drops substantially on out-of-distribution datasets. Previous studies proposed various solutions, including data augmentation and learning process improvement. In this paper, we present Mask-tuning, an approach that alleviates the impact of spurious correlations on the fine-tuning learning process. To achieve this goal. Mask-tuning integrates masked language training into the fine-tuning learning process. In this case, Mask-tuning perturbs the linguistic relation of downstream tasks' training examples and computes masked language training loss. Then, the perturbed examples are fed into fine-tuning process to be classified based on their ground-truth label and compute the fine-tuning training loss. Afterward, Mask-tuning loss- a weighted aggregation of masked language model training loss and fine-tuning loss- updates the masked language model and fine-tuming through training iterations. Extensive experiments show that Mask- tuning consistently improves the pre- pre-traine language models' generalization on out-of- distribution datasets and enhances their performance in-distribution datasets. The source code and pre-trained models will be available on the author's GitHub page.
INTRODUCTION
One of the challenges in building a pre-trained language model with robust generalization is that training sets do not represent the linguistic diversity of real-world language. Thus its performance dramatically drops when encountering citoo--distribution datasets. This type of performance divergence on the in-distribution out-of-distribuioon datasets is named generalization gap. Previous studies (Zhang al., 2019: McCoy ctt al. 2019: Tu et al. 2020) have shown that pre-trained language models trained on the specific dataset are likely learn spurious correlations, which are prediction rules that work for the majority examples but do not hold in general. It means that the fine-tume tuning loss function cannot incentivize the language model to learn the linguistics patterns from the minority examples and generalize them to more challenging examples (e.g.. diitti--iitiiitionnn dataset). There are several solutions that have been referred the generalization gap as: annotation artifacts (Gururangan et al. 20018), dataset bias (He et al. 2019: Clark et al., 2019; MMhabadi et al. 2020), spurious correlation (Tu et al.) 2020 Kaushik et al. 2020), and group shift (Oren et al.) 2019).
However, most of these methods rely on the strong assumption of knowing the datasets' spurious correlations biased keywords in advance and also suffer from decreasing the model performance on the in-distribution
dataset.
One of the primary solutions for mitigating the generalization gap is directly increasing the number of minority examples in the training set and creating a more balanced training dataset, which has been performed in several studies named: data augmentation (Ng et al. 2020 Garg & Ramakrishnan 2020). domain adaptation

---

Under review as conference paper at ICLR 2023
(Gurraangan et al. 2020 Lee et al. 2020), multi-task learning (Pruksachatkun al. 2020 Phang al. 2020).
and adversarial data generation (Garg& & Ramakrishnan 2020 Li et al. (021). p. Although these solutions have achieved different levels of success, they suffer from the following challenges: a) They require coming up with large, related, and useful training examples. b) They assume that an appropriate intermediate task and dataset are readily available. e) They need to re-run the pre-training phase from scratch. Finally, d) some of these methods added complex procedure to the learning process. All these requirements need significantly high computational resources or human annotations (Zhou & Bansal 2020; Chen et al. (021).
In this paper. we propose Mask-tuming which is inspired by recent studies in computer vision (Hendrycks et al., 2020b; Mohseni et al. 2020) that showed frint-trainin improves the robustness of deep learning models. However, computer vision studies used mix of in-distribution and out-of out-of-distribution training examples for the training processes (labeled and unlabeled) besides changing the model's architecture by adding multiple auxiliary heads. In contrast, our approach (Mask-tuning) employs the original pre-trained language models' masked language modeling and fine-tuning and solely uses the downstream task's training dataset as labeled and unlabeled datasets. As a result, Mask-tuning applicable with any pre-traines language model that works with the original fine-tuning.
Mask-tuning reinforces the learning process of the pre-trained language models' fine-tuning by integrating the masked language model training into the fine-tuning process. The proposed approach hinders learning the spurious correlations by using masked language model loss incentivize the fine-tuning training loss when the correct output likely be according to the learned spurious correlations. We used masked language modeling to perturb the linguistics relation between input training examples by masking a certain percentage of input tokens, predicting those masked ones, and computing the masked language model loss. Then the perturbed example is fed to fine-tuning for classification according to the ground-truth label. Afterward. loss- weighted aggregation of masked language modeling loss and fine-tuning loss- updates the masked language model and fine-tuning through training iterations. Mask-tuning teak--tunn---seciallll the fine-tuning step after masked language modeling- learns that training examples with different linguistic relations can have the same semantics and belong to the same label. Our main contributions are as follows:
baask-tuning
We study the effect of integrating the masked language modeling into fine-tuning training process on the pre-trained language models' generalization and proposed Mask-tuning. Our proposed method is a plug-and-play tool applicable to any pre-trained language model without the need for making any changes to the pre-trained language models' architecture. Indeed, our approach solely uses the downstream task's s training dataset. To our knowledge, we are the first to integrate the masked language model into the fine-tuning learning process that includes training losses aggregation.
Conducting comprehensive experiments under a consistent evaluation process to verify the effective- ness of Mask-tuning. Using BERT, RoBERTa, and autoregressive BART language models, we show that Mask- tuning outperforms eight state-of-the-art approaches from literature on three downstream tasks and three in-distribution and five out-of- of-iistribution datasets.
Our ablation studies show the necessity of simultaneously running masked language model training with fine-tuning in mask-tuning. Extensive experiments show that when Mask- Mask-tuning trains only by fine-tuning loss applying each training step separately (only fine-tuning the perturbed training examples). it could not mitigate the generalization gap as well as Mask-tuning with joint-loss.
2 APPROACH
In this section, we formally introduce the proposed approach setup named Mask-tuning. We first define the details of the Mask-tuning setup that is also illustrated in Fig] Then we explain the perturbation strategy and insights behind the Mask-tuning.
2

---

Under review conference paper at ICLR 2023
D-Distribution
Film to affire lowe's power nelp people endure ormost unannoginodee homror Ground iruth label: Posilive
Masked Language Modeliig
MLM loss (seadd
I (Gucolll Its a5 (hrowll
Parurbeed ins Distribbutioon example:
Pre Pre-trained Language model
film to [wer] power helo people [rui]] unimaginable Groune-truth label: Postriv
Mank-tuning luns Wookk
Training
Fine-tuning
Fine-turning (Uona)
Figure 1: Illustration of the Mask-tuning's training process, combination of the pre-trained language model's Masked language modeling and Fine-tuning. The input the fine-tuning perturbed version of the di-iistribution training example. Both Masked language modeling and Fine-tuning are trained based on the Mask moss--tuning loss.
2.1 MASK-TUNING
As shown in Figurc Mask-tuning combines two training steps, masked language model and fine-tuning, enhance the training process of the fine-tuning to learn more linguistics patterns besides spurious correlations. For this aim, Mask-tuning perturbs the relation between the input tokens using the masked language model and trains the fine-tuning to classify the perturbed example based on the ground-truc -oound-truth label through the following steps:
First, bask-tuning initiates with masking a certain percentage the input tokens at random and then predicts those masked ones. In this case, the final hidden vectors corresponding the mask token(s) is fed into output softmax over the embedding vocabulary as in standard language model. In all of our experiments, we randomly mask 5% of all tokens each input sequence. the i-th token(s) chosen, Mask-tuning replaces the i-th token(s) with the [MASK] token(s). Then the final hidden vector for i-th token(s) will be used to predict the masked tsken(s) with the aggregation of the cross-entropy loss from all masked token(s) that we denote as Masked Language Model training loss (L (Fig.
Second, the training example with the predicted token(s), called perturbed example, fed into fine-tuning be classified based on the ground-truth label (y). Then (a y) is the fine-tuning function to predict the perturbed example's label (y') based on the perturbed example (f) and compute the fine-tuning training loss (L Fine,, where / is the pre-trained language model parameters for the fine-tuning. Afterward, weighted aggregation of these two processes (i.e., masked modeling and fine-tuning) computes Mask-tuning loss (C (Cakak-tuning) follows: Maskk-tning LMLM + (1 LFme
(1)
Where is weighting factor, employ it to adjust the contribution of the two training losses in computing
3

---

Under review conference paper at ICLR 2023
the Mask-tuning loss. For passing each training iteration, the training loss of both steps must be close to zero. Otherwise, the training continues to minimize Mask- Mask-tuning training loss. Thus, on the one hand, Mask-tuning loss forces masked language modeling to predict token semantically close to the original token, but with linguistic variations that still allows fine-tuning succeed. On the other hand, Mask-tuning avoids fine-tuning to be rewarded for being correct for the completed wrong predicted tokens (Section 2.3).
2.2 PERTURBATION STRATEGY
There are different noise injection strategies, such as additive Gaussian noise, dropout noise, or simple data augmentation; each matches a particular downstream task. In this study, we follow some data-augmentation studies (e.g.. Garg Ramakrishnan] 2020) that used masked-language modeling to directly increase the size of the training datasets and train the model more variety of examples. In contrast, language modeling as a robust noise injection method used in unsupervised pre- pre-training in the pre- pre-trained language models such BERT, We hypothesize that the successful masked language modeling that achieves state-of-art performance on in-distribution datasets generates realistic and diverse perturbed examples that considerably modify the relation between the input token without affecting the label. Thus, it is safe to induce consistency between two training phases.i.e.. masked language modeling and fine-tuning, Indeed, the extreme experiment results show that Mask-tuning's perturbation strategy matches different downstream tasks.
employ masked
2.3 THE EFFECT OF MASK-TUNING ON THE LEARNING PATTERNS
The key intuition behind Mask-tuning is breaking the linguistic relation between the words in an example and classifying the perturbed examples according to the ground-truth label. Then based on the perturbed examples, Mask-tuning learns that training examples with different linguistic relations can have the same semantics (label). Thus it makes the model robust when applied to out-of-distribttion (unseen) examples. For passing training iteration, both training steps' losses ( (i.e., 1) masked lan- guage and 2) fine-tuning) must be close to zero (output is correct), and Mask-tuning loss is close to zero. Otherwise, suppose the training loss of masked modeling is large and fine-tuning classifies the perturbed example correctly (fine-loss close to zero), likely based on spurious correlations statistical patterns. In this case, masked language modeling loss incentivizes the fine-tuning loss to be updated even with the correct output. We illustrate the effect of Mask-tuning loss by looking at the following scenarios:
D The masked language modeling training loss close to zero (CMLM & ); hence masked modeling is predicting the original token or a token that may be a close synonym. If the fine-tuning classifies the example correctly (Lrine = 0), the Mask-tuning loss (Lmask-tuning) is also close to zero, and there no need for further training. Otherwise, if the output of any of these steps is not correct, Mask-tuning loss is large enough to prompt masked modeling to generate further variations and fine- ane-tuning model train further to handle variations. Thus it makes it harder for fine-tuning to rely only on the majority examples of the given downstream in-distribution dataset.
> The masked language modeling training loss is significant (CMLM > 0 ); suppose, the predi- cated token(s) is not semantically related to the original token(s) and raise masked language modeling loss (MLM-losss). In this case, the Mask-tuning loss is large enough, regardless of the fine-tuning classification output, to force the Mask-tuning training's 's models to continue training to minimize the Mask-tuning loss.
4

---

Under review conference paper at ICLR 2023
3 EXPERIMENT
3.1 IN-DISTRIBUTION AND OUT-OP- DISTRIBUTION TASKS AND DATASETS
In typical fine-tuning setting, the language model is fine-tuned on the training examples from the downstream task and evaluated on a test/dev set drawn from the same distribution as the training set. However, for this study, we evaluated the fine-tuned model on the Dev set of the out-of-distribttion datasets as well as in- distribution Dev sets. Out-of-distribution datasets are drawn from a different distribution with more linguistic variation than the in-distribution training set. Pre-trained language models' performance on out-of-distribution datasets indicates their robustness and generalizability.
In this study, experiments have been conducted on three tasks from GLUE (Wang et al. 2019 paraphrase Identification (PI), Natural Language Inference (NI.), and Stanford Sentiment Treehank (SST-2). They have large-scale benchmark datasets, and their corresponding ofist--iistiibuttinn datasets are publicly available. Sentiment. Stanford Sentiment Treebank (SST-2) has been selected an in-distribution dataset for the senti- ment classification task. For out-of-distribution counterparts, the recent challenging sentiment classification datasets of IMDB Contrast set (IMDB- C-ntt Gardnnr et al. 2020) and IMDB Counter-factualle factually Augmented Dataset LLMDB-CAD) Kaushik et al. 2020) have been selected.
Natural Language Inference (NLI). Multi-Genre Natural Language Inference (MNLI) Williams et al. 2018) is a popular NLJ task's dataset containing about 400k premise/hypothesis pairs annotated with textual entail- ment information (neutral, entailment, and contradiction). MNLI includes two different dev sets, matched set (MNLI-m), derived from the same sources as those in the training set and mismatched set (MNLI-mm) that do not closely resemble any of those seen training time (Williams et al. 2018). pre-trained models were trained on MNLI as an in-distribution dataset and then evaluated on MNLI dev sets (m/mm) and two relevant outoof-distribution datasets, HANS and Adversarial NLI (AdvNLI
Paraphrase Identification (PI). Quora Question Pairs (QQP) et al., 2017) is widely used PI task's s dataset containing around 400k pairs of sentences annotated as either paraphrase or non-paraphrase. Mask- tunnng is trained on QQP PI task's in-distribution dataset and evaluated on QQP dev set and relevant out-of-distribution PAWS-QQF
ZZhag et al. 2019) with high lexical overlap but different semantic mean-
ings.
3.2 EXPERIMENT SETUP
Three pre-trained language models have been chosen for this study, BERT (Devlin et al. 2019), RoBERTa (Liu et al. 2019b), and BART (Lewis et al. 2020). Note that BART is an autoregressive language model with a different architecture than the other two. Recent studies (e.g.. (e.g., Hendrycks et al. (020a) illustrated that although language models of large sizes often perform well on both the in-distribution and out-of-iistribttion datasets, in comparison with base size language model, the larger model does not necessarily reduce the generalization gap. Thus we choose the base size of these models to show the efficacy of the lask-tuning method. Besides, the base size models are more efficient with a smaller size, making them deployable to a broader computational environment and applications.
We follow the original language model's parameters for two training steps of Mask Mask-tuning. Except for fine-tuning step, learning-rate and batch size are changed based on our experiment. After trying 8,16, and 32
$ http:://ghthhh.ccmmm/mommmo///// l/hans
$ thun:://hthab..mmmfccccckssssrat
5

---

Under review conference paper at ICLR 2023
Table I: Comparing the generalization performance on the out-of-distribution datasets using Mask-tuning (our study) and some previous approaches three different language models, BERT, RoBERTa, and BART. All models are trained on related in-distribution dataset and then evaluated on dev set of out-of-distribution exam-
ples. The column "Type" shows that each method is related which categories in the litrature (Section 6). The Fine-tuning (Original model) has been implemented using http:///hhbbb.ooo//gggggggg////t
Out-Of-Ditribttio
Sentiment
NLI
Paraphrase
Model
Method
Type
IMDB-Cont. IMDB-CAD
HANS
AdvNLI
PAWS
BERT scoe
Fine-tuning
original
79.08
87.00
56,90
24.12
32.80
Learned-Min Mixin+Il Clark el. 2019
Joint-train
66.15
DRiFt-Hand (He
Debiasing
66.20
PoE [Mahabooll An 20z4
Joint-train
66.31
Rogularzzco-com
2020
Debiasing
69.10
39.80
IPI-standard GGavas & vulie
2021
Data-aug.
56,70
Mask-tunin
Our study
82,75+0.3
88.32+0.1
69.52+0.22
26.32+0.3
46.74-0.5
RoBEKTauue
Fine-tuning
original
84.50
88 28.40
67 80
31.20
38.45
Span Cutodi Shen
Dara-aug.
HiddenCut Chiel ot al
2021
Data-aug.
87.80
90.40
71.20
32,80
41.50
IPT-Adapter (Ciavvs
Vaii
2021
Data-aug:
66.30
Mask- luning
Oar study
88.50-20.2
91.62:20.1
15.70+0.2
37.40-20.6
44.37-0.4
BART
Fine tuning
Onginal
82.48
86.03
56.30
30.51
32.27
Tuning
Our study
8300010.1
87,83-10.3
70.4810.5
35.3110.4
45.71-0.5
for batch size in several trials run, the batch size of 32 is used for double sentence datasets (i.e., MNLI and QQP) and 16 for single sentence datasets (i.e., SST-2). The best learning rate has been determined through grid search among {2 10 5 3 x 10 5, x 10 for a by grid search in 0
10 10-3). We empirically selected the optimal value I with 0.1 increments. For each downstream task, the best value of is o.6, 0.7, and 0.8 for SST, NLI, and PI, respectively. Also, through grid search, the masking percentage is set 5%, which achieved the best performance on in-distribution datasets. All experiments were performed with epochs and using an NVIDIA V100 GPU with five random seeds, and the average values for the five experiments with corresponding standard deviations reported. We used Wolf al. 2020 for the original Fine-tuning.
4 RESULTS
In each evaluation setup, we have reported the results of prior studies on in-distribution and out-of-distribution datasets for comparison purposes. According to the literature, these previously proposed methods belong to two categories: data augmentation and learning process improvement. According to the literature (Section 6.3), the previous studies on learning process improvement introduce a new learning method through joint training or debiasing algorithm. As can be seen from the Tables and 2 only Gilvva & Vulié (2021) reported their results on both BERT and RoBERTa models, and the others applied just one of them. Indeed report the accuracy results of the Mask-tuning when using the BART language model with different architecture than BERT on in-distribution and out-of-distribution datasets. To the best of our knowledge, no study has reported BART model performance on oui-of-distribution datasets. Thus we compare our proposed method's 's result with the original fine-tuning performance on out-of- distriiubbtt data.
4.1 PERFORMANCE ON OUT-OF-DISTRIBUTION DATASETS
The evaluation accuracy results on out-of-distributee datasets have been shown in Table I Mask-tuning gains the sate-of-theeart performance all three pre-trained language models, BERT, RoBERTa, and BART on all tasks and out-of out-of- disstribbtionn datasets. To the best of our knowledge, study has reported BERT- base
6

---

Under review conference paper at ICLR 2023
Table 2: The performance on the in-distribution datasets using Mask-tuning (our study) and some previous approaches on three different language models, BERT, RoBERTa, and BART. The results show Mask-tune tunnng did NOT hurt the models' performance on the in-distribution datasets. The Fine-tuning (original model) has been implemented using
In Distribution
Sentiment SST-2
NLI
Paraphrase
Model
Approach
MNLI (m/mm)
QQP
BERT base
Fine-tuning (Original model)
92.43
84.30/83.40
90.80
LeamnedMMxxin+ CClar et al.
2019
83.97/-
DRiFt-Hand (He et al.
2019
81.70/-
PoE (Mahabadh et al. 2020
84. 19/-
Regularized-com cont (Utama et al.
2020
84.3/84.8
91.50
IPT-standard (Glavas & Vulie 2021
84.40/-
Mask tuning (Our study)
93.11=001
04.7:10..2/551000111
91.54.0.11
RoBERTabcke
Fine-tuning (Original model)
94.49
87. 37.60/ 87.50
91.50
Span Cutoff (Shen et al. 2020
95.40
88 $8.40/
92.00
HiddenCut (Chen et al 2021
95.80
88 20/-
92,00
IPT IPT-Standard ((Glavaš & Vulié 2021
87.70/-
Mask- -tusing (Our study)
94.60-0.1
87.72.1.11
87.88 0.2
91.62:0.22
BART buse
Fine-tuning (Original model)
93.23
84,60/84.88
90.50
Mask-tuning (Our study)
93.80=0.2
86,08 /86.12 00.
91.03++.11
model performance ditto--iistribution datasets of the sentiment classification task, IMDB-Cont, and IMDB-CAD. Thus, we compare the original BERT fine-tuning and our approach.
In comparison with the BERT original fine-tuning, Mask-tuning improves the performance IMDB-Cont by +3.67, IMDB-CAD by +1.32, HANS by +12.62, AdvNLI by +2.2, and PAWS by +13.94. Moreover, Mask-tuning mitigates the generalization gap in RoBERTa model by improving original fine-tuning IMDB-Cont by +4, IMDB-CAD by +3.22, HANS by +7.9, AdvNLI by +6.2, and PAWS by +5.92. And on BART original fine-tuning, Mask-tuning enhances the performance on IMDB-Cont by +0.52, IMDB-CAD by +1.8, HANS by +14.18, AdvNLI by +4.8, and PAWS by +13.44.
4.2 PERFORMANCE ON IN- IN-DISTRIBUTION DATASET
Table 2 shows the performance of the different approaches on the in-distribution datasets three pre-trained language models BERT, RoBERTa, and BART. It is essential factor for evaluating approach not to drop the performance of a pre-trained language model on in-distribution datasets while improving its performance the citoo--distribution datasets. As it be seen from Table 2] in contrast with some of the previous approaches, Mask-tuning gains state-of-the- on the -ut-of-distribution datasets without hurting the performance the in-distribution datasets but even boosting it.
5 ABLATION STUDY
We perform ablation experiments to demonstrate the necessity of each component of the Mask-tuning for improving the pre-trained language models' generalization. The proposed method added two critical parts, masked language modeling and Mask-tuning loss, the fine-tuning learning process to alleviate the generalization problem. We perform two experiments to show the impact of the two components on the achieved performance. Two experiments & Mask-tuning with Fine-tuning Loss (Crine) In this experimental setup, we investigate the effect of masked language modeling loss (LMLM) on the performance of the proposed method. We Mask Mask-tuning
follows:
7

---

Under review conference paper at ICLR 2023
Table 3: Ablation results over the effectiveness of the joint-training loss in Mask-tuning. Mask tunning with (Linee) means the language model only trained with the fine-tuning loss. And Mask-tuning without Mask-tuning loss -tmninn means we first generate the perturbed examples and then fine-tune the language model based the perturbed examples.
Sentimen
Paraphrase
Dis SST-2
Out-ol-Di Di.
In-
Oui-oF-iis.
In-Dis
Out Dis..Ds.
Model
IMDE-Com.
IMDB-CAD
HANS
AdvNEI
QOP
PAWS
BERT tore
Finc-tunnn (Onginal modell
92.45
79.08
87.00
34, 30
56.90
24.12
90.80
32,80
Mask-tuming Crzer
84.99
83.57/84.22
38.15
Mack-tuming w/o L.Mfart-tuming
91.97
80.97
86.76
83.40084.00 00
55.60
24.20
91.00
34,82
Mask-tuning
93.11=0.1
82.75=0.3
88.32+0.1
64.75:40.285,.. 100000
69.52+002
26.32:+0.3
91.54:0011
46.74=0.5
RoBERTI Takexr
Tine cuning (Original modell)
94,49
64.50
$5,40
87.0087. 550
67.80
31.20
91.30
38.45
Mack-tuning C
93.00
87.31
65.16
30.00
90.09
41.00
Mask-tuning w/o Litur-tundnng
93.26
84,86
39 .0
87,50/87 40
71.69
30.70
91.20
39.91
Mask-tuning
94.60:20.1
08.50
91.62:10.1
87.72.10.
1/87. 0.2
75.70.10.2
37.40
91.62:1022
44.37=00.4
EART
Fine ane-cuning (Original model)
93.25
82.45
86.03
84.60844880
56.30
30.51
90.30
32.27
Mack-tuning C.
93.00
82.02
35.80
84.02/44. 10
55.12
32.40
89. 34
35.21
Mask-tuning Cstant-tuntns
25
89.95
34.62
Mask- Baskkttning
93.80.20.1
83.00
87.831..3
$6.08.10.2/86.122.2
70.48:10.5
35.31.10.4
91.03:10.1
45.71
on in-distribution downstream dataset and set the a to (Equation T.. Thus, the Mask-tuning training model is updated only based fine-tuning loss through this experiment. As masked language modeling loss from the mask-tuning loss hurts the Mask-tuning performance on both in-distribution and out-of-distribution datasets. Also the results show that the Mask-tuning with fine-loss could not improve the original fine-tuning performance but even decrease it. Hence, using masked modeling loss in the Mlsk-tuuiin learning process is essential. Furthermore, this experiment implies the impact of the mask-tuning loss to hinder the fine-tuning to predict only based on learned spurious correlation.
from Table 3 eliminating
Mask-tuning without Mask-tuning Loss (L (asask-turing))) This experiment aims show the significance of running three components of Mask-tuning (i.e., input perturbation, fine-tuning, and computing Mask- tuning loss) simultaneously improve the generalization of the fine-tunin fine-tuning's 's learning process. We initialize the experiment by applying masked language modeling and generating the perturbed examples. Then run the fine-tuning on perturbed examples. As we can see from Table [3 the performance of the Mask-tuning without Mask-tuning loss dropped dramatically. However, because of two rounds of separate training (first masked modeling and second fine-tuning) on the in nn-distribution dataset, this setting could slightly improve the original fine-tuning performance in some datasets.
the performance of the Mask-tunnin without Mask-tuning loss dropped dramatically. However, because of two round of separate training (first masked modeling and second fine-tuning) on the in-distribution dataset, this setting could slightly improve the original fine-tuning performance in some datasets.
6 PREVIOUS WORK
6.1 TRANSFER LEARNING
Recently, transfer learning has become popular solution to build more robust unsupervised pre-training (e.g., BERT (Devlin et al. 2019), RoBERTa (Liu et al. (019b)) for processing the challenging examples. Transfer learning for improving the pre-trained language models' generalization has been implemented through two different approaches, domain- and task- adaptation.. The domain comaainnddatatioon continues pre- pre-training on examples with the same domain as the downstream task dataset. Some studies proposed pre-train the language models in their domain of interest from scratch (e.g., (Huang et al.] [2019] Lee et al. (020)). In contrast, other studies suggested adding data from the domain of interest the existing pre-training data (e.g..
8

---

Under review conference paper at ICLR 2023
(Assentzer et al., 2019 Gururangan al. 2020)). Besides, task-adaptation focuses on sharing knowledge across different tasks (Liu et al. 2019a Pruksachatkun et al. 2020 Vu et al. 2020). In task-adaptation, the pre-trained model first fine-tunes on dataset most related to the target task and then fine-tuned again on the target task. The two studies have shown the benefits of transfer learning approaches, especially when target training data is limited. However, pre-training on a large-scale dataset computationally expensive. Furthermore, the transfer learning approach's 's effectiveness is highly limited to the size of the data, the similarity between the source and target tasks and domains, and task complexity (Vu et al. 2020)).
6.2 DATA AUGMENTATION
The most straightforward approach to improve the pre-trained language models' robustness is data aug- mentation. In this method, the downstream task dataset is directly enriched with examples from the target distribution using various techniques such as: increasing the size of the training dataset (Ng et al. 2020 Khashabi et al.] (2020), balancing the existing cues (Schuster et al., (201)), adversarial training (Jia & Liang
McCoy et al. 2019; Garg & Ramakrishnan 2020 Li et al. 2021), augmenting the standard training dataset with syntactic information (Min et al. 2020) Glavaš & Vulié (021). creating partial view to augment
the training data (Shen et al., 2020). or dropping span of hidden Information(Cheet et al. (022). Despite pre-trained language models benefit from data augmentation, collecting new datasets, especially at large scale, is costly. Also, the size of the required data for the best performance is fuzzy. Unlike traditional data augmentation, our Mask-tuning does not additional training examples. Instead, perturbs the original examples while keeping the size of the training dataset constant.
6.3 LEARNING PROCESS IMPROVEMENT
Another track of study to enhance the pre-trained models' generalization is introducing new learning tech- niques that are robust to cope with dataset bias (Belinkov et al. 2019; He et al.] | Clark et al. 2019 Oren
M0o0 et al.) 20199 Schuster et al.) 2019; Mahabadi et al., (00]2220) Zhou & Bansal| 2020; Moon et al.) [022]). These techniques focus on challenging examples keywords that do not allow the pre- pre-trained models to make shortcuts during training. Thus they need to design processes to recognize and handle the biased patterns and keywords in the training examples and re-training the model on extra text corpora. These changes often make the model more complicated and computationally expensive. However, the main weakness of these methods is their strong assumption on knowing the datasets biases or biased keywords in advance (Tu al. 2020
7 CONCLUSION
This study proposed Mask-tuning, a simple but effective approach for improving pre-trained language models' generalization. Mask-tuning employs masked language modeling for firstly perturbing the training example. As a result, this perturbation interrupts the linguistic relation between the input tokens. Secondly, the error of the perturbation process will play an inductive role in fine-tume fine-tuning training. Mask-tuning learns to classify the perturbed examples according to their ground-truth label. Finally, Mask-tuning trains the language model to be robust when encountering change the input training example. Also, Mask-tuning has overcome the limitation of the previous methods. The comprehensive experiments three in-distribution datasets and five out-of-distribution datasets from three different downstream tasks on three pre-traines language models (i.c., BERT, RoBERTa, and BART) illustrated that Mask-tuning outperformed the performance of the previous techniques on out-of-distribun datasets. The results also indicated that this generalization improvement achieved without hurting their performance on in-distribution datasets but even boosting it.
9

---

Under review conference paper at ICLR 2023
REFERENCES
Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. Proceedings of the 2nd Clinical Natural Language Processing Workshop, pp. 72-78, 2019.
Yonatan Belinkov, Adam Poliak, Stuart M Shieber, Benjamin Van Durme, and Alexander M Rush. Don't take the premise for granted: Mitigating artifacts in natural language inference. ACL (1), 2019.
Jiaao Chen, Dinghan Shen, Weizhu Chen, and Diyi Yang. Hiddencut: Simple data augmentation for natural language understanding with better generalization. arXiv preprint arXiv: /06. 06.000 2021.
Christopher Clark, Mark Yatskar, and Luke Zeutlemoyer Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL)), pp. 4060-4073, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. Evaluating models' local decision boundaries via contrast sets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 1307-1323 2020.
Siddhant Garg and Goutham Ramakrishnan. Bae: Bert-based adversarial examples for text classification. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp- 6174-6181, 2020.
Goran Glavaš and Ivan Vulić. Is supervised syntactic parsing beneficial for language understanding tasks? empirical investigation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pp. 3090-3104, 2021.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah Smith. Annotation artifacts in natural language inference data. In NAACL-ILLT (2), 2018.
Suchin Gururangan, Ana Marasovié, Swabha Swayamdipta. Kyle Lo. Iz Beltagy. Doug Downey, and Noah A Smith. Don't 't stop pretraining: Adapt language models domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342-83828 8360, 2020.
He He, Sheng Zha, and Haohan Wang. Unlearn dataset bias in natural language inference by fitting the residual. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pp. 132-142, 2019.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2744-2751 2020a.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning improve model robustness and uncertainty. 2020b.
Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv: 1044::190 2019.
10

---

Under review conference paper at ICLR 2023
Shankar Iyer, Nikhil Dandekar, and Koml Csernai. First quora dataset release: Question pairs, 2017. URL https://data. quora. com/First- con/First-Quora Dataset-Releann Question-Pairs, 2017.
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2021- 2031, 2017.
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. Learning the difference that makes a difference with counterfettalll--ugmeeeed data. In International Conference on Learning Representations, 2020.
Daniel Khashabi, Tushar Khot, and Ashish Sabharwal. More bang for your buck: Natural perturbation for robust question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 163-170, 2020.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-122) 2020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, 2020.
Dianqi Li, Yizhe Zhang, Hao Peng. Liqun Chen, Chris Brockett, Ming-Ting Sun, and William B Dolan. Contextualized perturbation for textual adversarial attack. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5053-5069, 2021.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 4487-4496, 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 2019b.
Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. End-to-end bias mitigation by modelling biases in corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8706-8716, 2020.
R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In 57th Annual Meeting of the Association for Computational Linguistics, ACL 2019, pp. 3428-3448.
Association for Computational Linguistics (ACL), 2019.
Junghyun Min, R Thomas McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. Syntactic data augmentation increases robustness to inference heuristics. In Proceedings of the 58th Annual Meeting of the Association 2020.
for Computational Linguistics, pp.
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable dit-o---itttibutionn detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5216-5223, 2020.
Seung Jun Moon, Sangwoo Mo, Kimin Lee, Jaeho Lee, and Jinwoo Shin. Masker: Masked keyword regular- ization for reliable text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 13578 13586, 2021.
11

---

Under review conference paper at ICLR 2023
Nathan Ng. Kyunghyun Cho, and Marzych Ghassemi. Ssmba: Self-supervised manifold based data aug- mentation for improving out-of-domain robustness. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 1268- 1283, 2020.
Yonatan Oren, Shiori Sagawa, Tatsunori Hashimoto, and Percy Liang. Distributionally robust language modeling. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pp. 4218-4228, 2019.
Jason Phang, lacer Calixto, Phu Mon Htut, Yada Pruksachatkum, Haokun Liu, Clara Vania, Katharina Kann, and Samuel Bowman. English intermediate-tane training improves zero-shot cross-lingual transfer too. In Proceedings of the 1st Conference of the Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 557-575, 2020.
Chapter of the Association for Computational
Yada Pruksachatkun, Jason Phang. Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel Bowman. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5231-5234444 2020.
Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3410-3416, 2019.
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. A simple but tough-to- beat data augmentation approach for natural language understanding and generation. arXiv preprint MiXiv22009113816 2020.
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 8:621 2020.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Mind the trade-off: Debiasing nlu models without degrading the in-distribution performance. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8717-8729, 2020.
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke Subhransu Maji, and Mohit lyyer. Exploring and predicting transferability across nlp tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1882-7926, 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018, pp. 1112-1122. Association for Computational Linguistics (ACL), 2018
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemnie,, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest,
12

---

Under review conference paper at ICLR 2023
and Alexander M. Rush. Transformets: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL https://ww/ acl welb.e..oo org/ httg//anthllooyy220000
emn comnp--demos 6
Yuan Zhang, Jason Baldridge, and Luheng He. Paws: Paraphrase adversaries from word scrambling. In NAACL-HLT' (I), 2019.
Xiang Zhou and Mohit Bansal. Towards robustifying nli models against lexical dataset biases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8759-8771, 2020.
13


Reviews:
[{"review_id": "ek6Ba_cprQv", "reviewer": "Reviewer_U4CM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new mask-tuning method to improve the generalization of pre-trained language models, that incorporates the masked language model training with the fine-tuning. Extensive experiments on several in-distribution and out-distribution datasets with three pre-trained language models BERT, RoBERTa, and BART illustrate that mask-tuning outperforms previous methods.", "summary_of_the_review": "An easy-to-follow paper solving an interesting problem, but with unclear motivation and unconvincing experiments.", "strength_and_weaknesses": "Strength:\n\n-- The paper is easy to follow and well-writing overall.\n\n-- The research topic on improving the generalization of PLMs is quite interesting and valuable. The proposed mask-tuning method is simple and intuitive.\n\nWeaknesses:\n\n-- Unclear motivation: The motivation for mask-tuning on improving the generalization is quite unclear to me. It claims that \"the mask-tuning learns that training examples with different linguistic relations can have the same semantics and belong to the same label\". The masked language model training loss can be large during the training in the beginning. In this case, the generated perturbed example can be high noise that doesn't semantic correlate with the original data sample. This seems like introduce the wrong perturbed examples to train the fine-tuning loss. Did the authors try to solve this or am I having any misunderstanding?  More illustration and analysis on the training process of masked language loss and fine-tuning loss in experiments are expected to help understand the mechanism of mask-tuning.\n\n-- Unconvincing experiments: Why not use the full dev sets of the the GLUE benchmark as in the previous work HiddenCut (Chen et al., 2021)? More results on the benchmark dataset are expected to illustrate the advantage of the proposed method. Moreover, it is expected to have visualization or case studies such as the generated perturb examples to help understand the mechanism of mask tuning.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eZsRZhqMdG", "reviewer": "Reviewer_Kw5T", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a fine-tuning technique which improves both in-distribution and out-of-distribution generalization of large language models on various downstream tasks. Specifically, the proposed method involves a pre-training style training during finetuning, along with the original finetuning objectives. Their proposed method appears to work well compared with the baseline on BERT, RoBERTa and BART model variants, specifically improving the out-of-distribution results.", "summary_of_the_review": "The paper is clearly rushed and would require major rewrite to be considered for publication in this conference. While the proposed method is empirically better than the baselines, the choice of the said baselines is poor and does not reflect the central claim on out-of-domain generalization. Finally, the proposed approach is not exactly novel. While this isn't a hard negative, the lack of thorough comparison of the proposed method with similar approaches is what makes the paper a poor contribution.\n", "strength_and_weaknesses": "## The good\n\nThe fine-tuning technique makes sense on why it should improve the performance. The authors also include thorough experiments on an array of in-distribution and out-of-distribution datasets, which is commendable. The experimental results are also sound and contains the average and standard deviation over multiple runs, hence exhibiting robustness. Some models do get a substantial amount of improvement using their method, compared to the baseline.\n\n## The bad\n\n- Methodologically, the approach is similar to Domain/Task Adaptive Pre-training, where better downstream performance is achieved by continuing pre-training on the fine-tuning corpus (cite:gururangan2020). Hence, it is not suprising that this approach yields better results. The authors cite this work in a different context, but fail to draw similarities/dissimilarities with it.\n- Different from TAPT/DAPT, authors here introduce \"perturbation\" on the input. It is not clear what is the motivation behind it. The authors directly jump into introducing a \"perturbed\" example in Section 2.1, without motivating the reader why it is needed.\n- I'm not clear with what exactly the perturbation strategy is which the authors employ, even though they have a dedicated section on it (Section 2.2). They talk about injecting a \"noise\", without giving a definition of the said noise, nor examples. My best guess it is probably some sort of synonym-based perturbation, based on the phrase \"may be close to a synonym\" in Section 2.3.\n- The authors claim (in Section 4.2) that their method gains SOTA on OOD datasets compared to previous approaches (Table 2). However, previous approaches are only compared with respect to in-distribution tasks (Table 1). Hence, their claim is not backed by empirical results.\n- Finally, it is nowhere mentioned in the paper how many steps of training the authors perform on their mask tuning approach. Since the authors perform perturbation, their model is actually similar to a data augmentation strategy (in the related work they argue it isn't) and the model trumps the baseline due to more amount of training updates. The paper should include a discussion on this, and compare with baselines having the same amount of training steps.\n\n## Questions\n\n- P2, L6: \"some of these methods added a complex procedure to the learning process\": what is this exactly? How are those methods less complex than the one proposed by the authors?\n- In Section 6.2, the authors claim their method is different from data augmentation as they do not use additional examples. However, they perturb the same training examples - isn't that also a kind of data augmentation?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LetWLILqtg", "reviewer": "Reviewer_ahpL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an approach called \"Mast Tuning\" that combines the objectives of masked-language modeling (MLM) and standard fine-tuning (FT) to train a more robust model that generalizes to out-of-domain (OOD) datasets well. They take a pretrained language model, perturb the training data of the finetuning stage and jointly train on the MLM and FT objectives. The finetuning is performed on the perturbed examples with the hypothesis that after the masked-LM prediction stage, they are sufficiently diverse. The method outperforms 8 existing approaches on 3 downstream tasks (sentiment analysis, paraphrase detection, NLI) on a total of 5 out-of-domain datasets.", "summary_of_the_review": "The paper needs significant clarifications and rewriting to make it more clear and easy to read. There are many crucial issues and the claims made in this paper are not well justified by the experimental choices. Many unfound claims are made in this work. However, the idea seems very simple and should be thoroughly investigated because it can have a strong impact on training general models. I urge the authors to take my review in a constructive manner to improve this work.", "strength_and_weaknesses": "Strengths:\n* The idea proposed in this work is fairly simple, consequently simple to implement and test widely.\n* Decent ablations are present for the proposed idea and the method achieves good performance on the OOD datasets.\n* The experiment details are adequately present with different hyperparameters.\n\nWeaknesses:\nTLDR: \n* This paper is not well written. It was hard to follow, some sections are not clear and hard to understand, and there is significant abuse of standard terminology throughout the paper. There are also significant grammatical and language issues (I have put them separately and won't consider them as weakness if they can be fully addressed). This paper needs significant polishing.\n* The claims made in this paper are not well justified (see details below). The results are weak and don't justify the main claims. \n* The approach does not seem to be very novel: Use MLM to perturb examples, Jointly train MLM and Finetuning (but I will give the benefit of doubt to the authors as I'm not aware of the related work extensively).\n* The authors claim that a low MLM loss signifies that the model predicts a token semantically close to the original token. That is not true. The MLM loss does not take semantic similarity of words into account. (see 2.3 2nd para line 1, 2.1 last para 3rd last line).\n* They hypothesize that \"MLM creates realistic and diverse perturbed examples that considerable modify the relation between the input token without affecting the label\" (Section 2.2)-> but have not shown any examples of it. Nor is any modification of any relation happening.\n* The choice of datasets raises questions. It is claimed that \"OOD datasets are drawn from a different distribution with more linguistic variation than in distribution sets\" (line 4 section 3.1). Q) How do you measure this linguistic variation? Is the OOD datasets chosen standard?\n* Baseline models have not been explained at all. Is the baseline that \"does pretraining using MLM on the target dataset followed by standard finetuning\" included? Are the baselines models, or just Data augmentation techinques that can be plugged on top of other models?\n  * Hard to compare your results with the baselines as most of the numbers are not there for OOD dataset. It is claimed that the numbers are missing because the baselines have not reported them, but to fully justify the efficacy of this method, i urge the authors to implement / run the baselines on the given datasets for a decent comparison. For instance, if any baseline is just doing data augmentation using some technique, I feel it can very well be run on top of BART to get the results of that \"data augmentation technique\" for comparison. All in all, I'm not convinced by the quality of comparison to baselines. Why is there no \"transfer learning\" baseline (or the one I suggested - MLM to improve representation learning and then FT)?\n\n\nDetails and questions:\n* What does \"linguistic relations\" even mean? It is claimed multiple times that \"Mask tuning perturbs the linguistic relation of the examples in the downstream task\". I was not sure what this even meant until probably after the experiments section when I realized that just some tokens are changed. I don't think changing some tokens is related to \"linguistic relations\" or just perturbing tokens can be claimed as \"perturb the linguistics relation between input examples\".\n* The word semantics is used very lightly. To be precise, the authors claim that \"training examples have same semantics if they belong to the same label\". That is absolutely not true. Semantics of text does not mean that they predicting the same label.  (mentioned at: last line of intro before contributions, Section 2.3 line 4)\n* What does \"biased keyword\" in last line of 1st para of intro mean? This is not explained at all. I understood this at the very end after reading the related works!\n* Intro second para : \"c)  They (baselines) need to re-run the pretraining phase from scratch\". Why? I think this is just the \"finetuning stage\". This is not adequately justified.\n* 3rd para of intro line 2 - \"joint-training improves\" - Q) joint training of what?\n* Intro para 4: Line 3 - \"MLM loss to incentivize the FT training loss when\" - Q) incentivize to do what?\n* Section 2.1 line 3 - \"perturbs the relation between the input tokens using the\" -Q) what does this even mean? Also, you're never perturbing the relations between tokens. You're just perturbing the tokens.\n* Section 2.1 para 3 - p(y'=y|x) is not the \"fine-tuning function\".\n* What does \"for passing each training iteration\" even mean? (eg: usage in first line of page 4, section 2.3).\n* Section 2.2 -hypothesis that MLM modify the relation between input token by generating realistic and diverse perturbations is not verified at all. There are no qualitative examples of pertubations. No stats on how many tokens are *actually* perturbed by the LMs. etc.\n* In 2.3, 2nd para : \"Mask tuning loss is large enough to prompt masked modeling to generate further variations and fine-tuning model to train further to handle variations\" - It was never discussed that those examples without variations are dropped. Or any kind of \"further training\" on some specific examples is performed. Q) Are you filtering examples at all?\n* In 2.3 3rd para line 1 - again wrong assumption. High MLM score does not mean predicted tokens are not semantically related.\n* Q) why did you evaluate on dev set and not test sets? Do the baselines also report results on the dev set?\n* Explain why the chosen datasets are OOD. Eg: Are they created from different domains? (news vs social media)?\n* The OOD Phrase identification dataset \"has a high lexical overlap but different semantic meanings\" Q) How was this judged? Does this really make the dataset OOD? (I doubt).\n* In 3.2 line 5 : \"larger model does not necessarily reduce the generalization gap. Thus we choose base\" - How is this a justification to use the base model?\n* What is the Original finetuning model? It is just specified in 3.2 \" We used Wolf et al. (2020) for the original finetuning\" Please describe this approach in a sentence.\n* In Related work: Please talk a bit about how your work is different from domain adaptation.\n* The claim in related work Data augmentation 6.2 last line : \"MLM does not use additional training examples\" -> I'm pretty sure that there are data augmentation works that also dont add new examples, but just perturb existing ones (but I might be wrong). I urge the authors to double check this.\n\nGrammar and language:\n*Abstract:\n  * Abstract line 1: \"Pretrained language models have the known generalization problem\" : Q) to what? Domain, languages, tasks, etc? It is later clarified (OOD datasets) but good to add it in this sentence.\n* Intro:\n  * Line 5 of intro - \"pretrained language models trained on the specific dataset\" - Q) pretrained or finetuned?\n  * Lines 6-8 in intro are confusing.\n  * Line 9 in intro - \"solutions that have ~been~ referred the generalization gap as\" - This sentence does not make sense.\n  * last 4 lines of para 3 are hard to read. Not sure what is meant in the last line (\"As a result, mask tuning is applicable with any pretrained lm that works with the original fine-tuning\").\n  * Para 4 line 4 - \"correct output likely be according to\" -> \"correct output is likely due to\"\n* 2.1\n  * Para 3 line 1 - \"called perturbed example, fed into fine-tuning to be classified based on true label y\" - this sounds weird. \n  * last line in page 3 - \"<equation>\\nWhere \\alpha is a weighting factor, we employ it\" -> \"<equation>,\\nwhere \\alpha is a weighting factor. We employ it ...\"\n  * on the one hand -> on one hand.\n  * last line - \"completed wrong predicted tokens\" -> \"wrongly predicted tokens\".\n* 2.2\n  * line 6 - \"successful masked language modeling that\" -> insert \"objective\" between modeling and that.\n* 2.3\n  * Be consistent with your usage of your own terminology. Sometimes \"fine tuning loss\", sometimes \" fine loss\"  is used (section 2.3).\n* 3.2\n  * Line 2 of 2nd para - \"fine tuning step, learning rate\" -> \"fine tuning step where learning rate\".\n  * 5th last line character 1 - \"o.6\" -> \"0.6\"\n* 4.2\n  * last line : \"but\" ->\", sometimes, \"\n\n\nOther suggestions:\n* Make references to figure, table as links. \n* The \"mask tuning\" nomenclature is confusing in Section 5 (2nd last para) as it refers to the main proposed approach as well as one component of the loss of that approach.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DYSaf5jr8l", "reviewer": "Reviewer_Sc8b", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a multi-task learning method to fine-tune pre-trained LMs (e.g., BERT, BART) and evaluate on in-distribution and out-of-distribution data. In addition to the MLM loss, the proposed training process also applies a mask LM to replace randomly sampled words in an input sentence and uses the sentence's original label for standard fine-tuning with a cross-entropy loss.  ", "summary_of_the_review": "The proposed methodology in this paper does not seem correct to me, as there is no guarantee that the predicted words would not change the label. Fine-tuning with the proposed method may reinforce the errors made by MLM. Even though the paper shows better results of their method on the selected datasets, it's questionable whether it could be reliably applied to other datasets.", "strength_and_weaknesses": "Strength:\n- Out-of-distribution generalization is an important topic.\n\nWeakness:\n- The writing in this paper should be improved. Some explanations/descriptions are unclear.\n- The major concern is reusing the label even though there is no guarantee that the predicted words would not change the label. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject