Difference-Masking: Choosing What to Mask in Continued Pretraining
Authors: Alex Wilf
Syeda Nahida Akter
Leena Mathur
Paul Pu Liang
Sheryl Mathew
Mengrou Shou
Eric Nyberg
Louis-Philippe Morency
Venue: EMNLP 2023 Findings (2023)
Keywords: Machine Learning
Self-Supervised Learning
Multimodal
NLP
TL;DR: A novel strategy for deciding what to mask during continued pretraining to do well on various downstream tasks.

Abstract: The self-supervised objective of masked prediction has led to promising performance gains on a variety of downstream tasks. However, while most approaches randomly mask tokens, there is strong intuition that deciding what to mask can substantially improve learning outcomes. We investigate this in continued pretraining setting in which pretrained models continue to pretrain on domain-specific data before performing some downstream task. We introduce Difference-Masking, a masking strategy that automatically chooses what to mask during continued pretraining by considering what makes a task domain different from the pretraining domain. Empirically, we find that Difference-Masking outperforms baselines on continued pretraining settings across four diverse language-only and multimodal video tasks.

DIFFRENCE-VASKING::
Choosing What to Mask in Continued Pretraining
Alex Wilf", Syeda Nahida Akter", Leena Mathur, Paul Pu Liang, Sheryl Mathew, Mengrou Shou, Eric Nyberg, Louis-Philippe Morency Carnegie Mellon University
(awilf, sak Bakkerreee J@cs. cmu. cdu
Abstract
Random Masking
Difference-Masking
The self-supervised objective of masking-and- predicting has led to promising performance gains on a variety of downstream tasks. How- ever, while most approaches randomly mask tokens, there is strong intuition that deciding what to mask can substantially improve learn- ing outcomes. We investigate this in continued pretraining setting in which pretrained models continue to pretrain domain-spennt specific data be- fore performing some downstream task. We introduce DIFFERENCE- -AAEEENNEE-ASAKNNG a mask- ing strategy that automatically chooses what to mask during continued pretraining by con- sidering what makes a task domain different from the pretraining domain. Empirically. we find that PEFFERENCE-MASKIN forms baselines on continued pretraining set- tings across four diverse language-only and multimodal video tasks.
bensfty Bescrtioes the ielattoonshi between the and substance
Density describes the relasionship between the
what is nasked during continued pretraining
automatically lects what to mask based on what makes the task domain different from the pretraining domain, enhancing model learning the end task.
Figure 1: DIFFERENCE-
most masking approaches randomly choose which parts of the data to mask. Although it is under- studied in SSL, deciding what mask is critical component in human education (Pajares and Miller, 1997; Bjork and Linn, 2006). Educators designing "fill-in- tit-iinthhe-he blank" assessments for students must de- cide what content to mask in order to effectively assess student understanding of a domain (Bae and Lee, 201 8). For example, in real-world "fill-in- the-blank" chemistry test, a teacher might choose to mask domain- con") to assess student learning, instead of masking Commain-irrleeve words ("example", "process"). In this paper, we propose DIFFERENCE- MASKING, a novel approach for automatically se- lecting what to mask during continued pretraining. Our strategy first identifies anchors that describe what makes target domain different from the pre- training domain and then determines what to mask during continued pretraining based on similarity those anchors.
outper-
1 Introduction
Inspired by the distributional hypothesis in the lan- guage domain (Harris, 1954), masking is a self- supervised learning (SSL) objective in which a model attempts to reconstruct hidden portions of data from the surrounding context. Masking has enabled breakthrough performance on tasks from variety of domains, such language, vision, and speech (Devlin et al., 2019; Li et al., 2021; Hsu et al., 2021: Ericsson al., 2022), motivating inter- est in researching how masking strategies influence representation learning SSL.
words ("density", "sili-
Masked prediction has recently been applied to adapt pretrained models to specific downstream tasks by continuing to pretrain models on in- domain unlabelled data (Dery et al., 2023). Mask- ing in this continued pretraining setting been shown be particularly effective when the target do- main differs substantially from the pretraining do- main (Gururangan al., 2020).
In experiments spanning four diverse language- only and multimodal video datasets (ACL-ARC, ChemProt, TVQA, and Social-IQ), we find that DUTERENCE-MASKINS outperforms strong base- lines, supporting our hypothesis that masking based what is different about task provides strong representation for continued pretraining. We pro- vide intuitions explain the strong performance of DIFFERENCE- MASKING, along with extensive
While prior work has studied how the amount masked influences model learning (He et al., 2022),

---

Difference-Wasking for Continued Pretraining
Language
Choose whether to mask w based on similarity with anchors
Language
Density ity describas the relationship between the nass volune
Density describes the relatfonship be between the mass (velune substancee
Xy, date
shich words should be nosked?
fn-domala tack
Multimodal Video
Multimodal Video
mochers Sp
DOON
Fyyer
Ryar
shoule word
object bounding boxes should be nasked?
Figure 2: DISTRRECCI-MASKING: MASKING: an approach to choosing what to mask during continued pretraining that prioritizes masking concepts that make the target domain different from the pretraining domain. DIFFERENCE- MASKING does this by first selecting anchor topics relating to the downstream task, and then by masking words or bounding boxes based on their similarity to those anchor topics.
analyses and ablations to better understand the per- formance of our method. Our code is publicly available.
ing how much each token contributes to improved downstream task performance. is important to note that Selective Masking uses supervised downstream task labels, whereas DIFFERENCE- MASKING is entirely self-supervised.
2
Related Work
Masking relies on the distributional hypothesis, which posits that the meaning of word can be inferred from its context (Harris, 1954). Masking in NLP has functioned as an effective SSL strategy when training models such BERT (Devlin et al., 2019) and XL-Net (Yang et al., 2019). Although random masking has been more closely studied in NLP than non-random masking, there are three closely related works ours from NLP.
Prior work from the vision community has also contributed an understanding of masking strate- gies, primarily by using the attention of the model during SSL training to determine what to mask. MST (Li et al., 2021) uses attention maps to de- termine "non-cssential regions" to mask, while At- tnMask (Kakogeorgiou et al., 2022) does the op- posite by masking the most attended-to regions. Unlike DuTERENCE-MASKEN do not take into account domain-specific informa- tion when determining their masking strategy. This can be an impediment to performance when the model's attentions do not already contain infor- mation about what is important in given input sequence.
these approaches
EntityBERT (Lin et al., 2021) masks tokens based on whether they are part "entities" rec- ognized by a domain-specific pretrained named- entity-recognizer. Salient Span Masking (SSM) (Guu et al., 2020) is a similar method that uses wew.dd-enttt--rconntteee
model to mask out sin- gle entity for the downstream task of open-domain QA. However, these approaches require domain- specific pretrained eagge,, and the masking strategy they determine is the same for any do- main to which that tagger is applied. In contrast, COFFRENCEEMASKKING determines what to mask without pretrained tagtes, and its masking strategy can change depending the unlabelled data in the task domain.
3 DIFERENCE-MASKING
This section describes the motivation and imple- mentation of DIFFERENCE- MASKIEEG our self- supervised method to determine masking strategy for continued pretraining. The overall process is depicted visually in Figure 2.
3.1 Problem Setting
Selective Masking (Gu al., 2020) uses data from the downstream task to decide which tokens mask during continued pretraining by estimat-
We are given model which has been pretrained on multi-domaii data drawn from domain distribution X PT (e.g., model such RoBERTa pretrained
2

---

on a large multi-domain corpus). We are interested in how adapt this pretrained model to a specific target domain Xr without observing task labels Y. Continuing pretraining on XT has emerged as a popular solution approach to this problem (Guru- rangan et al., 2020; Dery et al., 2023).
general domains Xpr. We term these words difference anchors that summarize the con- cepts unique to Xr.
Masking based on differences: Using these difference anchors, we determine the likeli- hood that cach word should be masked based on its similarity to these difference anchors. We sample from this probability distribution to decide what to mask during MLM contin- ued pretraining.
3.2 Motivation and Notation
If the masking objective is used to train models to learn word representations (Harris, 1954; Devlin et al., 2019), natural question emerges: which words is most important that our models learn represent? We believe that this question may be important to effectively continue pretraining on specialized domains. We expect that continued pretraining can benefit from a masking strategy that considers what makes a task-domain different.
These steps are explained in detail in the following subsections.
3,4 Finding Difference Anchors: TF-ICF
Our goal is to determine set of corpus-level dif- ference anchors that are representative of the dif- ferences between the pretraining domain Xpr and the task domain Xy. Since our goal to design a simple yet effective method for finding these differ- ences, we use of a modified version of the widely used TF-IDF scoring function from the field of sta- tistical NLP (Jones, 1972). TF-IDF determines the ratio of how frequently word appears in a docu- ment compared how frequently the word appears in other documents in a corpus. Because we are attempting find words that make target cor- pus Xy different from general pretraining corpora X XPт, the score of a word is highest when it appears frequently in our corpus (Xx) and infrequently in the multi-domain pretraining corpus (X PT). We denote our approach as TF-ICF for term- cerm-frequency. inverse-corpus catpss-cooruuu-eqqeeeecy expressed by the follow- ing scoring function:
This leads to the intuition behind DIFFERENCE- MASKING: to train what makes a target domain different from the pretraining domain. For example, in corpus about chemistry we would expect that the task of masking and predicting words strongly related to chemistry such as "molecule" will lead to better learning outcomes than words such as "analysis", which could be related to chemistry in addition to many other domains.
Formally, term XTrPr as the concepts likely to appear in both Xy and XPT (e.g.. "analysis"), and we term XTyPT as the concepts that make the domain Xr different from XPT (c.g.. "molecule"). With this notation, we can now express our intu- ition in terms of mutual information with the down- stream task Y: we intuit that concepts common in Xr but uncommon in XPT (i.e., in XTyrr) share higher mutual information with the task label than concepts found in both domains (XTrPT) do:
TF-ICF(u:; freq (w, XT)
(2)
freq w., PP)
I(T//PT: Y)= > I(XTOPr;Y)
(1)
To effectively capture word frequencies in the general distribution of the English Language used for pretraining (XPT), we use unigram counts de- rived from the Google Web Trillion Word Corpus (Brants and Franz, 2006; Norvig, 2009).
The goal of DIFFERENCE- MASKKING then is to learn representations during masking that capture the information unique to the domain (XT/PT) which is more relevant for the downstream task.
We score all words in XT with this metric and choose the top K as anchors A to represent the do- main, where K' is hyperparameter of our method. We analyze the impact of this hyperparameter in Section 5.3.
3.3 Our Approach: DIFFERENCE-MASHIN
To learn masked representations that capture the information unique to the domain (XT/PT), our proposed DUFFRENCE-MASSE ceeds in two steps:
approach pro-
3.5 Masking Based on Differences
1. Finding difference anchors: We first deter- mine which words are most commonly found in domain Xy and not commonly found in
DIFFERENCE-MASKING then masks words based on similarity to anchors A. Formally, we define
3

---

similarity between a word and an anchor word A as the cosine similarity of the words' BERT (Devlin et al., 2019) embeddings.
Experiments are performed to allow each model to learn long as needed during continued pre- training, only stopping when validation error in- creases (early-stopping). Each result is averaged across five random seeds. Hyperparameter settings and data preprocessing details can be found in Ap- pendix A.
sim(w, A&) = cos (BERT(()))))) w), BERT(AA)) (3)
In order to choose words to mask, we generate probability distribution a over the words in the sentence x to represent the probability that each word should be masked. We determine the weight of each word Wy by calculating its similarity score with the most similar anchor word in A (we explore other strategies in our experiments). This value is normalized over the length of the sequence ensure the probability distribution sums to 1.
4.1 Datasets
Language-only Datasets As in Cururangan et al. (2020); Dery et al. (2023), conduct experiments with the ChemProt dataset (Kringelum al., 2016), relation classification task that uses chem- istry documents. ChemProt is low-resource clas- sification task with large amount of in-domain un- labeled data, making it realistic setting in which SSL is helpful in continued pretraining.
maXkck sim A&)
a(w;)
(4)
Eز=1 maxIck sim(uj, A&)
We also conduct experiments with the ACL- ARC task (Jurgens et al., 2018), citation intent task based on the ACL Anthology Reference Cor- pus (Bird al., 2008) used in continued pretraining experiments in (Gururangan et al., 2020). We use train, validation, and test splits for both datasets from (Dery et al., 2023; Gururangan et al., 2020).
DIFFERENCE-MASEE MASKKING then masks terms by sampling from distribution a without replacement, and the model attempts to reconstruct the masked terms from the surrounding context.
Multimodal Implemention of DIFFERENCE- MASKING To apply our method to the visual domain, we draw on work from the vision commu- nity in which visual representations are grouped at the object level (Baradel et al., 2018; Sajjadi et al.. 2022) and use object labels (e.g. person, car...ttc) from a state-of-the-art object detector (Wang et al., 2021: Zhang et al., 2016) to calculate similarity with the anchor words. A detailed description of our implementation of SETPEREECE-MMASKIN in the multimodal setting can be found Appendix
Multimodal Datasets We also experiment on continued pretraining for two challenging multi- modal video understanding tasks. TVQA (Lei et al., 2018) 8) dataset containing 21,792 videos from 6 American television shows and questions and answers related to the videos. Each question is paired with answer choices (one correct answer and 4 incorrect answers), and corresponding video, audio, and subtitles.
Social-IQ (Zadeh et al., 2019) contains 1,250 videos of social situations and questions and an- swers pertaining to the videos. Each question has corresponding video, audio, and subtitles. We use the train, validation, and test splits from the pub- licly available datasets.
4 Experimental Settings
Our experiments evaluate whether DIFFERENCE- MASKING'S 's masking strategy leads to performance improvements on challenging language-only and multimodal video understanding tasks. We fol- low the experimental setting from (Gururangan et al., 2020), in which unlabelled data from the downstream task domain is used for continued pre- training before eventually performing downstream task finetuning. This is a popular SSL setting be- cause it represents a camputaaionnally-fassbbb way to test the effectiveness of self-supervised represen- tation learning methods (c.g. without recreating a pretrained model), and it realistic to modern approaches which rely heavily on pretrained mod- cls (Dery et al., 2023).
We use performance metrics consistent with prior work (Gururangan et al., 2020; Dery et al., 2023): Fl score for ACL-ARC and classification accuracy for ChemProt, TVQA, and Social-IQ.
4.2 Baseline Methods
Random Masking Most masking approaches choose tokens or words to mask with uniform ran- dom probability (Devlin et al., 2019; Yang et al.. 2019). We consider both the token-level and word- level approaches in our experiments. Formally, the probability that word or token in sequence
4

---

Language-Only
Multimodal
Masking Strategy
ACL-ARC
ChemProt
Social-IQ
TVQA
Random Masking (Word)
62.05
81.90p.51
Random Masking (Token)
63.741.97
82.82p.23
69.05o.32
73.750.31
MST (Li et al., 2021)
6..6
83.17
68.370.49
81.14
AttnMask (Kakogeorgiou et al., 2022)
6..301.67
83.530.56
70.180.71
81.570.12
DGA (Ke et al., 2023)
27.20
70.67
Selective Masking (Gu et al., 2020)
69.061.80
82.94..47
EntityBERT (Lin et al., 2021)
71.090.25
82.04o.40
Salient Span (Cole et al., 2023)
71.940.58
82.41..21
CIPFERENCE-MASKIIN
74.042.01
83.94
71.37
81.73 1..731.13
Table 1: We find that DIFFERENCE MASKIIGG outperforms strong baselines in both the language and multimodal experimental settings. We note that our entirely self-xupervised method also outperforms Selective Masking, which uses labelled data to inform its masking strategy. Values average results over five trials, subscripts are standard deviations.
of length N will be masked in andom-masking is
DGA (Ke et al., 2023) is another relevant work that proposes masking strategy for NLP model adaptation. However, unlike the methods described above, DGA chooses which attention heads to mask instead of choosing which tokens to mask, assigning importance to attention heads based on the gradient of the loss between the model's rep- resentations of two differentty-masked versions of the same input. Additionally, DGA encourages the model to learn integrated representations of the target domain and general knowledge using a contrastive loss.
1
(5)
N
AttnMask (Kakogeorgiou et al., 2022) is domain-agnostic token-based masking approach in which the likelihood of masking given token is proportional to how attended-to that token is by the [CLS] token, averaged across the different heads of the transformer. Formally, this approach can be seen as defining a function fatt which takes in model fo. sequence of tokens æ, and index i and outputs how attended-to token is.
EntT (Lin et al., 2021) masks tokens based on whether they are part of "entities", as defined by domain-specifer named-entity recognition (NER) model. The original paper uses the PubMedBERT model, trained originally on the clinical domain. We also implement Salient Span Masking (Guu al., 2020), which in this case is the same as the Entity approach applied only to mask a single word in the sentence. To apply the approach to the ChemProt and ACL-ARC domains requires NER models effective in those domains. For ChemProt used the BioBERT model (Lec et al., 2019) fine-tuned in NER task with BC5CDR- chemicals (Li al., 2016) and the BCACHEMD (Krallinger et al., 2015) corpus and for ACL-ARC we used the popular SciBERT model (Beltagy et al., 2019).
ag 9ost (Jot x,
(6)
MST a02, is an approach very similar AttnMask, except that it masks "non-essential regions", effectively corresponding to an inverse weighting based on the attention of the model to the token Zz.
0i X gatt Gate(fo. (fo, x, i)-
(7)
Selective Masking (Gu et al., 2020) chooses to- kens to mask based on whether adding each token will improve downstream task accuracy as mea- sured by the difference between the downstream task performance when using the full sequence R versus using only the sequence up to and in- cluding the token I;, Notably, this approach uses downstream task labels guide the choice of mask in continued pretraining, whereas DIFFERENCE- MASKING is self-supervised.
4.3 Experimental Methodology
Language-only We reproduce the experimental setting from AANG (Dery et al., 2023), which em- ploys pretrained 110M RoBERTa model with
as x P(y x)) P(y m74)
(8)
5

---

one for continued pretraining and one for the downstream lask. Our hyperparameters and other detailed configuration notes are described in Appendix A.
heads:
than all the baselines, including Random Masking (82.82%), AutMask (83.53%) and Ent (82.04%). Similarly to Selective Masking, the En- tityBERT and DGA masking strategies were orig- inally tested on much larger datasets, which may suggest limitation of these methods in the low- resource continued pretraining setting.
Multimodal We conduct our multimodal experi- ments using strong pretrained model: MERLOT- Reserve (Zellers et al., 2022), a large multimodal transformer pretrained with contrastive multi- modal prediction objective on a dataset of 20 mil- lion Youtube videos.
DIFFERENCE-MASKING also demonstrates bust performance in multimodal settings. On the Social-IQ dataset, CMPFERENCE-MASKING achieved an accuracy of 71 7,.7%, outperforming the Random Masking (69.05%), AunMask (70.18%). and MST (68.37%) methods. We were unable to compare our approach with Selective Masking and EntityBERT on these datasets due to the language- only design of their entity taggers. In contrast, our method is not limited the language domain, and, in fact, performs well in the multimodal set- ting. And on the TVQA dataset, DIFFERENCE- MASKING achieved an accuracy of 81.73%, out- performing the Random Masking approach sub- stantially (73.75%) and the AttnMask approach marginally (81.57%).
To experiment with masking strategies in the multimodal setting, we continually pretrain a 200M MERLOT-ReseTvee.e model by masking- and-predicting visual patches. We evaluate the learned representation quality by freezing the model and finetuning only the linear classifier layer on the downstream task following (Wilf et al., 2023)'s methodology.
detailed description of our implementation of DIFFERENCE- MASKING in the multimodal setting can be found in Appendix B, and our hyperparame- ters can be found in Appendix A.
5 Results and Discussion
These results highlight the effectiveness and ver- satility of the MILFRENCE-MMSKKIN approach across various language and multimodal datasets.
5.1 Comparison with Baseline Approaches
Our experiments compare our proposed DIFFERENCE-MASKING
5.2 What is masked?
with established baselines including Random Masking (at the word and token level), AttnMask (Kakogeorgiou al., 2022), MST (Li al., 2021), Selective Masking (Gu et al., 2020), DGA (Ke et al., 2023), Ent (Lin et al., 2021), and Salient Span Masking (Cole et al., 2023). The results are summarized in Table 1. We find that DIFFERENCE- MASKING shows strong results compared to baselines across language-only and multimodal video understanding tasks.
In this section, investigate what is masked by DIFFRENCE-MASKING and its link to down- stream task performance.
On the ACL-ARC task, we find that the most frequently masked words in the ACL-ARC task had an interesting grounding in human intuition. The ACL-ARC task is a citation intent task on a corpus comprising ACL papers. As the subject of ACL papers can vary widely, comprising multiple sub-domains and research fields, we were curious how DIFFERENCE- MASKING'' masking strategy would handle this domain.
Notably, our approach demonstrated superior performance on the ACL-ARC dataset with an ac- curacy of 74.00 74.04%, marked improvement over the random token baseline (63.74%) and a substan- tial improvement over the best baseline (Salient Span Masking, 71.94%). Our approach also sur- passed Selective Masking (69.06%). This is surpris- ing because Selective Masking uses downstream task labels to inform its masking strategy whereas DIFFERENCE-MASKING is self-supervised.
We found that the most frequently masked words closely-aligned with the ACL paper submission tracks describing the high-level topic categories for papers. For example, some of the most fre- quently masked words were "learning", "infor- mation". "translation", "semantic", and "lexical". These words closely correspond to submission tracks "Machine Learning for NLP", "Information Extraction", "Machine Translation", and "Seman- tics: Lexical". Since submission tracks for ACL can be seen as a set of topics that span the space
Results on the ChemProt dataset are also en- couraging, showing that DIFFRENCE-MASKING achieves an accuracy of 83.94%, marginally better
6

---

ACL-ARC Dataset
ChemProt Dataset
Frequently Masked Words
ACL Tracks
Frequently Masked Words
Task Labels
systen
Dialogue and Interactive Systems
activity
activator
model
inhibited
inhibitor
Language Grounding
language
inhibitor
agonist
information
Information Extraction
celts
substrate
translation
Machine Translation
increased
downreuulator
Machine Learning
learning
for NLP
human
antagonist
Figure 3: The most frequently masked words chosen by the DIFFERENCE- DIFFERENCE-MASKING MASKING algorithm across the ChemProt and ACL-ARC tasks. We find that for the ChemProt dataset, the masks find automatically through unlabelled data partially recover the end task labels.
of ACL papers, this supports our hypothesis that masked words chosen by DIFFERENCE- MEEEENCE-MAKIIGGG MASKING align with what makes this domain different.
Method
TVQA
Social-IQ
Random 17%
15%
AttnMask
38%
19%
On the ChemProt task we also found an inter- esting pattern in what was masked. The objective of the ChemProt task is to determine type of relation corresponding to a type of biochemical interaction between entities in the text, where la- bels include words such as "activation", "inhibitor". and "antagonist". Interestingly, we find that some of the words DIFFERENCE-MASEE mask most often are the same words as the labels for the downstream task. This result is also vi- sualized in Figure 3. Some of the most-masked words by CEOFERENCE-MASKIINE are "activity"", followed by "inhibitedd", "inhibitor", and "antago- nist". This is fascinating result because it suggests that in masking what makes the ChemProt domain unique, DIFFERENCE- MASKING is determining a self-supervised objective that is highly similar to the downstream task without accessing the down- stream task labels.
DIFFERENCE-MASHNE
40%
90%
Table 2: For each method, we analyze what percent of tokens are chosen masked from within bounding boxes over people opposed objects.
types including those that target visual reasoning about non-human entities and non-visual reasoning from specifically text or audio modalities.
chooses to
As such, would expect that our continued pre- training strategy would choose to prioritize mask- ing tokens representing human body language more often in Social-IQ than in TVQA. We found that this was in fact the case. Interestingly, we found that AttnMask baseline also picked up on a similar trend in its attempt to mask based on where atten- tion already focuses, although the trend is much more pronounced in our approach.
The findings in Table 2 demonstrate that DIFFERENCE-MASKNN chooses to mask substan- tially fewer visual tokens corresponding to peo- ple than to objects in TVQA, (40%) in compari- son to Social-IQ (90%). On the Social-IQ dataset, where the performance difference is more pro- nounced over the closest baseline (t 1.76% over AttnMask), the difference between the proportion of tokens masked from people by these approaches is also most pronounced (90% in DIFFERENCE- MASKING vs 19% in AttnMask).
In the multimodal setting we also find an inter- esting grounding of how CEOFERENCF-MAKKINE chooses masks in human intuition. Reasoning about social interactions is believed by many psy- chologists to rely heavily on understanding visual body language cues (De Stefani and De Marco, 2019; Keck et al., 2022). Social-IQ is designed to test these kind of social intelligence capabilities with subtle questions such as "How do the men in the room feel about each other?" and "Do the peo- ple in this video feel comfortable about the clown being t"ere?". . In contrast, TVQA tests more gen- eral video understanding with question and answer
7

---

5.3 Sensitivity Analysis
sive enough to determine strong masking strategy, and too many anchors may begin overfit to niche concepts that are not representative of the domain. We find that there is indeed a "sweet spot", and interestingly that is the same for both datasets: 20. These results are visualized in Figure 4.
Similarity Function As described in Sec- tion 3, DIFFERENCE-MASKIN determines mask- ing probabilities by comparing the anchor repre- sentations to the token representation. Because the token representation is a single vector and the anchors are group of vectors, similarity can be de- fined multiple ways. Table I shows results from the neighbor approach to determining simi- larity described in Section 3.5, motivated by the intuition that a domain can have many sub-domains and if a token is close to any one of these concepts it should be prioritized for masking. For example, the ACL-ARC corpus has many sub-domains, in- cluding the over twenty different submission tracks described in Section 5.2. If a paper about lin- guistics, may be important to mask words similar to "language", whereas if a paper is heavy on ML theory, another anchor might be more appropriate to mask in order to best understand the work.
ChemProt
0.030
Pham
0.825
0.70
0.6
0.815
K(D Anchorst
Figure 4: Performance on both tasks best at the hyper- parameter K 20 anchors. We hypothesize that each task may have an optimal setting of this hyperparameter.
An alternative approach that could be to deter- mine scores by relation to the centroid of the anchor embeddings: in essence, determining whether the token in question is similar to the anchors ag- gregale. We would expect that this approach would perform similarly to ours on narrowly-defin dataset such as ChemProt, but substantially differ- ently on a multi-domain dataset such as ACL-ARC. We evaluate this alternative in Table 3.
6 Conclusion
In this paper we introduce DIFFERENCE- MASKING, a method for identifying what makes a target domain unique and using this information to guide strategy that chooses what to mask during SSL continued pretraining. We find that our method outperforms strong baselines actoss diverse language and multimodal video understand- ing tasks. We provide a detailed discussion of what is masked in SEIFERENCE-MASSINE and why our method performs well on various tasks. The cross- task applicability of DiFFERENCE-MARKIN supports the effectiveness of our framework for SSL pretraining in language, vision, and other domains.
ACL-ARC
ChemProt
Centroid 69.02
83.66
Nearest-Neighhor
74.04
83.94
Table 3: Ablating
SAADEENNCE-MAKIKNGG anchor- scoring function based on nearest-neighhor and re- placing it with one based similarity with the an- chor embeddings' centroids leads to performance degra- dation. This provides evidence our hypothesis that the neirest- scoring function helps make DIFFERENCE- MASKING robust to anchor selections.
7 Limitations
As described Section 3. DIFFERENCE- MASKING is based on the intuition that it is more beneficial to mask based on what is unique (XT/PT) about a downstream task's 's domain. How- ever, it is challenging to find what makes domain unique; therefore, our method is approximation of XT We believe future work may find it fruitful investigate additional methods for ap- proximating this, including modifications on the TF-ICF method we proposed. Scction 5, we pro- vided intuition, empirical results, and analysis to
We find that the nearest-neighhor strategy does, in fact, outperform the centroid strategy, especially on the ACL-ARC task. This supports our intuition that the nearest-ncigee neighbr strategy is particularly helpful when there is complex or peaky domain.
Number of Anchors In considering the relation- ship between the anchors and the downstream task, we also investigate how the choice the number of anchors (K) impacts the downstream performance. We expect that too few anchors will not be expres-
8

---

understand why our method outperformed attention masking basclines by larger margin on Social-IQ than on TVQA. A broader investigation of why CUFFERECCEMMSKINNN
Mark Joseph. Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir Radev, and Yee Fan Tan. 2008. The acl anthology reference corpus: A reference dataset for bibliographic research in computational linguistics. In Proceedings of the sixth international conference on language resources and evaluation (I.REC(08).
during pretraining is ben- eficial by a larger margin to some downstream tasks than to others would be helpful to the community.
Robert A Bjork and Marcia C Linn. 2006. The science of learning and the learning of science. Aps Observer, 19(3).
8 Ethics Statement
We believe that self superviee learning is a promis- ing direction for the machine learning community. This does not discount the salient arguments made about the social and enviromental risks of large models (Bender et al., 2021; Strubell et al., 2019). We believe that works such as ours, which study SSL in a resource-constrainee context, both in- crease access to those with limited compute re- sources and conform to a more environmentally- sustainable way of doing research.
Thorsten Brants and Alex Franz. 2006. All n-gram are belong to you. https://ai. googleblog..oooooooooooooooooooo com/ wat///alloounnn---eaeee are-loor--- you html. Accessed: 2023-05-22.
Jeremy R. Cole, Aditi Chaudhary, Bhuwan Dhingra, and Partha Talukdar. 2023. Salient span masking for temporal understanding. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 3052- 3060, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
Acknowledgements
Elisa De Stefani and Doriana De Marco. 2019. Lan- guage, gesture, and emotional communication: An embodied view of social interaction. Frontiers in Psychology, 10:2063.
This material is based upon work partially sup- ported by National Science Foundation awards 1722822 and 1750439, and National Institutes of Health awards R01MH 25740,, ROIMH132225, M009999951 and R21 MH133077. Any opin- ions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors, and no official endorsement should be inferred.
Lucio M. Dery, Paul Michel, Mikhail Khodak. Gra- ham Neubig, and Ameet Talwalkar. 2023. AANG : Automating auxiliary learning. In The Eleventh Inter- national Conference on Learning Reprexentations.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171 -161-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
References
Minryoung Bae and Byungmin Lee. 2018. Effects of text length and question type on test-takers' perfor- mance on fillin-the-blank items in korean esat. En- glish Teaching, 1344)1149-174
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations.
Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori. 2018. Object level visual reasoning in videos. Proceedings of the European Conference on Computer Vision (ECCV), pages 105-121.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scib- ert: Pretrained language model for scientific text. In EMNLP.
I Linus Eriesson, Henry Gouk, Chen Change Loy, and Timothy M Hospedales. 2022. Self-supervised repre- sentation learning: Introduction, advances, and chal- lenges. JEEE Signal Processing Magazine, 39(3):2- 62.
Emily M Bender, Timnit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Confer- ence Fairness, Accountability, and Transparency. pages 623..
Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang. Zhiyuan Liu, and Maosong Sun. 2020. Train no evil: Selective masking for task-guided pre-training. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages
Steven Bird, Robert Dale, Bonnie Dorr, Bryan Gibson,
9

---

6966-6974, Online. Association for Computational Linguistics.
Ole Lund, Tudor I Oprea, and Olivier Taboureau. 2016. Chemprot-3.00 a global chemical biology dis- eases mapping. Database, 2016.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't t stop pretraining: Adapt language models to domains and tasks. In Proceedings of ACI..
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim. Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 1234)::234-240.
Kelvin Guu, Kenton Lee, Zora Tung. Panupong Pasu- pat. and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International Con- ference Machine Learning, pages 3929-3938. PMLR.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg- 2018. TVQA: Localized, compositional video ques- tion answering. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 1369-1379, Brussels, Belgium. Association for Computational Linguisties.
Zellig S Harris. 1954. Distributional structure. Word, 10(2-3)116-162. 16-112..
Jiao Li, Yueping Sun, Robin J. Johnson, Danicla Sci- aky. Chih- Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu, 2016. BioCreative V CDR task corpus: resource for chemical disease relation extraction. Database, 2016. Baw068
Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Pi- otr Dollár, and Ross Girshick. 2022. Masked autoen- coders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 56000-16009
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdel- rahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEFACM Trunsactions on Audio, Speech, and Language Processing, 29:3451-3460.
Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. 2021. Mst: Masked self- supervised transformer for visual representation. Ad- vances in Neural Information Processing Systems, 1011316--31766
Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation.
Chen Lin, Timothy Miller, Dmitriy Dligach, Steven Bethard, and Guergana Savova. 2021. EntityBERT: Entity-centric masking strategy for model pretrain- ing for the clinical domain. In Proceedings of the 20th Workshop on Biomedical Language Processing. pages 191-201, Online. Association for Computa- tional I. Linguistics.
David Jurgens, Srijan Kumar, Raine Hoover, Dan Mc-
Farland, and Dan Jurafsky. 2018. Measuring the evolution of scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391-406,
Peter Norvig. 2009, Natural language corpus data. Beautiful data, pages 219-242.
Ioannis Kakogeorgiou, Spyros Gidaris, Bill Pso- mas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. 2022. What to hide from your students: Attention-guided masked image modeling. In Computer Vision ECCV 2022, pages 300-318. Springer Nature Switzerland.
Frank Pajares and M David Miller. 1997. Mathematics self-efficacy and mathematical problem solving: Im- plications of using different forms of assessment. The journal of experimental education, 65(3):213-228.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8763. PMLR.
Zixuan Ke, Yijia Shao, Haowei Lin, Hu Xu, Lei Shu, and Bing Liu. 2023. Adapting language model while preserving its general knowledge. arXiv preprint arXiv:2301.089::
Johannes Keck, Adam Zabicki, Julia Bachmann, Jörn
Munzert, and Britta Krüger. 2022. Decoding spa- tiotemporal features of emotional body language in social interactions. Scientific Reports,
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Ma- hendran, Sjoerd van Steenkiste, Filip Pavetic. Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. 2022. Object scene representation transformer. Advances in Neural Information Processing Systems, 35:9512-952
Martin Krallinger, Obdulia Rabal, Florian Leitner. Miguel Vazquez, David Salgado. Zhiyong Lu, Robert Leaman, Yanan Lu, Donghong Ji, Daniel M Lowe. et al. 2015. The chemdner corpus of chemicals and drugs and its annotation principles. Journal of chem- informatics, 71).1117.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, Florence, Itll, July 28- August
Jens Kringelum, Sonny Kim Kjaerulff, Søren Brunak,
10

---

2019, Volume 1: Long Papers. pages 3645-3650.
Association for Computational Linguistics.
Zhongdao Wang, Hengshuang Zhao, Ya-Li Li, Shengjin Wang, Philip Torr, and Luca Bertinetto. 2021. Do different tracking tasks require different appearance models? Advances in Neural Information Processing Systems, 34:726-738.
Alex Wilf, Martin Q. Ma. Paul Pu Liang, Amir Zadeh, and Louis-Philiipe Morency. 2023. Face-to-fad contrastive learning for social intelligence question- answering. In 2023 IEEE 17th International Con- ference Automatic Face and Gesture Recognition (FG), pages 1-7.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for guage understanding. Advances in neural informa- tion processing systems, 32.
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. 2019. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Puttern Recognition, pages 8807-8817.
Rowan Zellers. Jiasen Lu, Ximing Lu, Youngjae Yu. Yanpeng Zhao. Mohammadreza Salehi, Aditya Kusu- pati, Jack Hessel, Ali Farhadi, and Yejin Choi. 2022. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li. and Yu Qiao. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters, 23(10):1499-1555 1503.

---

A Detailed Experimental Settings
instead of the token-level in our language exper- iments. We found that K performed much better than other values, where the selected anchor word was "person". We considered two possible bounding boxes associated with people: bounding boxes over faces and bodies. We evaluated both options and found that considering entire bounding boxes over people's bodies (including their faces) performed the best. These results are shown in Table 5.
In this section, we provide an overview of the exper- imental conditions utilized in our study. To ensure fair comparisons with our basclines, we maintain consistent set of hyperparameters for both con- tinuous pretraining and fine-tuning, For language tasks, we largely adhere to the hyperparameters ployed in (Gururangan et al., 2020). Throughout our experiments, we maintain masking ratio of 25% in both language and multimodal settings. We adopt a static masking strategy, replacing masked tokens with random values.
Masking Strategy
TVQA
Social-IQ
Random Masking
73.75
69.05
DIFFERENCE CEPERRECE-MASKING -AAKING (Face)
81.51
69.13
Hy perparameters
Language
Multimodia
Langwage
Multimolal
DITFERENCI-MASKING (Body)
81.73
71.37
learning rase
0.0001
0000005
1000.06
5.001-06
noon epocks
eval_every.oo spocks
30
Table 5: Results of DIFFERENCE-MASKING on mul-
puttence
timodal video understanding benchmarks TVQA and Social TQ. DIFFERENCE- leads improve- ment of 8% and 2% accuracy over random accuracy.
Table 4: List of hyperparameters used both continu- ous pretraining (CPT) and finetuning (FT).
We reproduce MERLOT-Reeseve original train- ing on TVQA: we decompose samples in Social- IQ and TVQA from the form (Question, All An- swers, Video Information) into a list of 3-tuples: (Question, Candidate Answer, Video Information). MERLOT scores each candidate answer indepen- dently, given the question and video, and trained with loss that encourages the model to minimize estimated likelihood of incorrect answers and max- imize likelihood of correct answers.
We extracted body detection coordinates using UniTrack (Wang et al., 2021) and face detection coordinates using MTCNN (Zhang et al., 2016). Apart from the bounding box strategy, we also experimented with masking patches chosen by dif- ferences between CLIP embeddings (Radford et al., 2021) of the anchor and the vision patch directly (without bounding box labels). Our experiments validate that the CLIP-based masking strategy per- forms poorly compared to our bounding box strat- egy. One possible reason can be that CLIP not robust enough for video datasets which led to mask- ing patches that are not relevant to the anchor word "person".
From video frames, into 16x16 patches as determined by MERLOT- Reserve's 's backbone image transformer ViT (Doso- vitskiy et al., 2021). The language experiments took nine hours of runtime each on a single 12GB GPU, and the multimodal vision experiments re- quired six hours on a single TPU v2-8.
mask image patches
TVQA
Social-IQ
CLIP (Radford et al., 2021)
73.58
68.75
DIFFERENCE-MASKING 81.73
71.37
B Masking Video Tokens
Table 6:
We validate our hypothesis that masking patches using DIFFERENCE-MASKEN is more effec- tive than masking using CLIP similarity.
Following the intuition from language, we hypoth- esize that masking and predicting small patches of an image may be testing local capabilities (e.g. determining what an eye looks like from the rest of the face) rather than global capabilities (e.g. de- termining what person's face looks like from the rest of the scene, including other people's faces). Accordingly, instead of masking low-level image patches, mask groups of patches corresponding to higher level semantic entity: bounding boxes over objects in the image. We see this approach as a visual analogue for masking at the word-level
C Masking Language Tokens
In Section 4.3 we describe the motivation for us- ing a word-level strategy in our implementation of DiFFERENCE- mentation could be to assign each token in word the same masking likelihood, and mask tokens only by this probability. This could result in some tokens
An alternative imple-
12

---

from the same word being masked where others not. Our intuition is that for specialized domains such as chemistry, subword tokens may be trivial to predict from their neighbors, but whole words may not be trivial to predict given the context. For example, word such as "phosphates" would be to- kenized into "phos" and "-phates". We expect that it may be trivial to predict "phos" given "-phates" vice versa, but it may be hard (and may promote a better understanding of the task) predict the word "phosphates" given the context.
Empirically, we find that this decision improved performance substantially, as shown the results in Table 7 below.
ACL-ARC
ChemProt
Token
0.6501
0.8224
Word
0.7404
0.8394
Table 7: We validate our hypothesis that masking tokens using SEFFERENCE-MASKONN at the word-level is more effective than masking the token-level.
13


Reviews:
[{"review_id": "VRWYbz27YT", "reviewer": "Reviewer_LPF7", "summary": "", "questions": "Please see Reasons To Reject.\n\n\n=====Post-rebuttal:\n\nI appreciate the authors' diligence in their rebuttal, which partially addressed my concern. The concept of Difference-Masking continues to be an elegant and effective strategy, as convincingly demonstrated by the authors.\n\nHowever, there are a few points that I believe still warrant further clarification or consideration in the final version:\n1. About Distribution: While I acknowledge the authors' efforts in addressing the drawback to store the pretraining data, I still recommend a deeper exploration. The rebuttal discussed the use of distributions from a large dataset as a proxy for the pretraining distribution. To enhance the paper's robustness, I suggest a clearer distinction between using approximations and exact pretraining dataset statistics.\n \n2. Masking Fairness in Video/Image Patching: the authors explained that the masking selection for patches is based on object region, while random masking is based on patch level -- this is somehow unfair. The authors may: (a) authors may use their strategy to conduct patch selection for masking, or (2)  compare their method to randomly mask the patches for random objects.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our manuscript. We appreciate your positive feedback on the clarity of the paper's writing and the straightforwardness of our method. It is heartening to see that you recognize Difference-Masking's strong performance across both language-only and multimodal tasks, and found the analysis detailed.\n\nTo address your concerns:\n\n–-- \"The solution's suboptimal aspect arises from the need to store the entire pretraining corpus when selecting anchors for continued pretraining, making it impractical. Other masking strategies do not require such storage.\" –--\n\nWe should perhaps clarify a misconception. We don't actually store the entire pretraining corpus for anchor selection. Instead, we determine the anchor \"term frequency\" based on the continued pretraining corpus. This corpus is notably smaller than the primary pretraining corpus, and we can't get around storing the continued pretraining corpus because it is required for continued pretraining.\n\nFor the \"inverse corpus frequency,\" we see where the potential confusion arises. While determining this value directly would require the entire pretraining corpus, we've leveraged ingenuity from prior works and approximated approximated large pretraining corpora using word distributions from vast online data sources. A notable example is the study in [1] that employs the Google Books dataset to approximate a large pretraining distribution. In line with this approach, we've turned to the Google Web Trillion Word Corpus to get our approximation. \n\nThe beauty of our method lies in its simplicity: we just need word counts from our smaller, continued pretraining corpus and an aggregate measure that's already been meticulously worked out by others who've delved deep into word distributions on the web. We'd like to direct you to Section 3.4 where we've discussed this in detail. And of course, if there's any point you'd like us to elaborate on, we'd be happy to!\n\n--- \"The extension of Difference-Masking to the multimodal setting is limited. The authors rely on labels/bboxes for anchor and mask selection, which might not be available for other multimodal tasks. The application of Difference-Masking to methods that patchify image/video input instead of extracting object regions is unclear.\" –--\n\nThis is an interesting point, and one that warrants a deeper dive here and in our paper. The key contribution of Difference-Masking is to mask what makes the domain unique. We find that masking at a higher level of semantic abstraction is useful – in Appendix C we find that Difference-Masking applied at the word level is better than Difference-Masking applied at the token-level, and we find the same for Video Tokens at the bounding box level vs the patch level in Appendix B. There are many ways to consider \"higher level semantic entities\": for language we move from tokens to words, for vision we move from patches to bounding boxes, for audio we could move from raw audio to audio separated by word-boundaries...etc. \n\nWe'd like to respectfully challenge one particular point: \"The application of Difference-Masking to methods that patchify image/video input instead of extracting object regions is unclear.\" Our multimodal experiments use a vision backbone that actually does patchify image/video input. Our method calculates the probability of masking a patch based on the bounding box from which that patch is drawn. This principle is adaptable and not specific to vision. For example, we could imagine an extension to another modality where a semantically meaningful \"object\" is defined, for example audio by figuring out the corresponding word for a given raw audio token. We refer you to Appendix B and the MERLOT-Reserve paper [2] for details on the patching & masking process.\n\nAdditionally, we could imagine applying this method based on multimodal embedding similarity, for example taking similarity between vision patches and text tokens, as determined by a multimodal model such as CLIP. In early experiments, we found that this did not perform nearly as well as our approach based on bounding boxes, perhaps because of CLIP’s difficulty with multimodal compositionality [3].\n\n–--\"Some hyperparameters appear to be sensitive. The analysis in Figure 4 shows that even within the small range [15, 30], different numbers of anchors can lead to considerable performance differences, e.g., for ACL-ARC when K=15 and K=20.\"–--\n\nOur method does appear to be sensitive to the setting of K; we've investigated further and would like to present a nice intuition we’ve found for why we see the upside-down-U shaped graph in Figure 4. When the number of anchors K is equal to the number of unique words in the dataset, our method should converge to the random masking strategy. This is because when that happens, every word will have perfect similarity to a nearby anchor (itself), so each word will be equally weighted, defaulting to a random uniform masking probability. When the number of anchors K is very small (e.g., 1), we're adding a strong inductive bias into the masking strategy based on an aggregate measure of the dataset determining which word is most unique to dataset. \n\nSo searching for the optimal the number of anchors K is meaningfully a question of coverage: how close are anchors in embedding space to different words that we'd like to mask? This can be roughly approximated by the variance of the anchors, and we've run an additional analysis and found that performance on the different datasets is actually tightly correlated with the variance of the anchor embeddings: using Pearson's Correlation Coefficient, ChemProt is .8463 (meaning that as the variance of the anchors increases the performance increases) and for ACL-ARC the correlation is -.9576 (as the variance of the anchors increases, the performance decreases). These are interesting results, and we actually see this as a benefit of our approach because it allows us to nicely calibrate the masking strategy to the domain, and to learn something about the domain in the process. For example, it seems that ACL-ARC benefits from a masking strategy that is more tightly coupled in the embedding space, whereas ChemProt benefits from a higher variance masking strategy. As this is the only hyperparameter of our method, and our experiments only take a few hours to run, it seems worthwhile for future work to search through to understand. We are happy to include these results in our paper as well.\n\n–\n\nThank you for these insightful contributions to our work, and for engaging with it so carefully! We appreciated your comments and are happy to make these changes.\n\n[1] Liu, Fangyu, et al. \"Do ever larger octopi still amplify reporting biases? Evidence from judgments of typical colour.\" arXiv preprint arXiv:2209.12786 (2022).\n[2] Zellers, Rowan, et al. \"Merlot reserve: Neural script knowledge through vision and language and sound.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[3] Kamath, Amita, Jack Hessel, and Kai-Wei Chang. \"Text encoders are performance bottlenecks in contrastive vision-language models.\" arXiv preprint arXiv:2305.14897 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper's writing is clear, and the proposed method is straightforward and easy to understand\n- The performance of Difference-Masking on downstream language-only (ACL-ARC, ChemProt) and multimodal (Social-IQ, TVQA) tasks surpasses strong baselines, and the analysis provided is detailed.", "reasons_to_reject": "- The solution's suboptimal aspect arises from the need to store the entire pretraining corpus when selecting anchors for continued pretraining, making it impractical. Other masking strategies do not require such storage.\n- The extension of Difference-Masking to the multimodal setting is limited. The authors rely on labels/bboxes for anchor and mask selection, which might not be available for other multimodal tasks. The application of Difference-Masking to methods that patchify image/video input instead of extracting object regions is unclear.\n- Some hyperparameters appear to be sensitive. The analysis in Figure 4 shows that even within the small range [15, 30], different numbers of anchors can lead to considerable performance differences, e.g., for ACL-ARC when K=15 and K=20.", "questions_to_authors": "Please see Reasons To Reject.\n\n\n=====Post-rebuttal:\n\nI appreciate the authors' diligence in their rebuttal, which partially addressed my concern. The concept of Difference-Masking continues to be an elegant and effective strategy, as convincingly demonstrated by the authors.\n\nHowever, there are a few points that I believe still warrant further clarification or consideration in the final version:\n1. About Distribution: While I acknowledge the authors' efforts in addressing the drawback to store the pretraining data, I still recommend a deeper exploration. The rebuttal discussed the use of distributions from a large dataset as a proxy for the pretraining distribution. To enhance the paper's robustness, I suggest a clearer distinction between using approximations and exact pretraining dataset statistics.\n \n2. Masking Fairness in Video/Image Patching: the authors explained that the masking selection for patches is based on object region, while random masking is based on patch level -- this is somehow unfair. The authors may: (a) authors may use their strategy to conduct patch selection for masking, or (2)  compare their method to randomly mask the patches for random objects.", "paper_topic_and_main_contributions": "This paper introduces a method called Difference-Masking for continued pretraining. The approach prioritizes masking concepts that differentiate the target domain from the pretraining domain. It achieves this by selecting anchor words/regions relevant to the continued pretraining tasks and masking similar concepts during the process. The experimental results on language-only benchmarks and multimodal benchmarks demonstrate the effectiveness of this masking strategy for continued pretraining tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dS1kHcc3WJ", "reviewer": "Reviewer_Yfzq", "summary": "", "questions": "See above", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback and for recognizing the significance of our domain-adaptive pre-training approach. We've always believed that there's room to improve upon traditional masked language modeling, and that this line of research could be helpful in domains that pretrained models struggle with. We're grateful to see that you appreciate our effort in refining the process! Looking forward to addressing your comments below.\n\n— “There are still other methods that are trying to improve the domain-adaptive pre-training (e.g., [6]), which the proposed method should also consider compared with. The author should compare the pros and cons of changing the data (like masking) with changing the neurons (like [6])” ---\n\nThis is an interesting baseline to run! We considered baselines in our paper that modified the masking strategy, but did not consider baselines that changed the neurons. We’ve implemented this paper, and found that our approach outperforms it by a wide margin: On ACL-ARC (where the metric used is Macro F1 [6] ), DGA scored 67.20, whereas Difference-Masking outperformed it by over 6% absolute to 74.04. On ChemProt (where the metric is accuracy [6]), Difference-Masking outperformed DGA as well: DGA scored 70.67 and Difference-Masking performed at 83.94 – an over 13% absolute increase. We ran these results over 5 trials, and are happy to incorporate the results into the paper. In their original paper, DGA performed continued pretraining on a much larger corpus; it is possible that their poor performance is due to their method’s reliance on a larger amount of information, whereas Difference-Masking performs well in the low-resource setting. We plan to include this in the next version of our paper.\n\n--- “Since the proposed method only masks the difference, will the model forget about the general knowledge in the LM? How the proposed method address this? Intuitively, a good pre-training objective should be able to consider both the shared and the different. ---\n\nThis is another interesting point. It may be the case that the model will sacrifice some of the general knowledge in order to perform well on the downstream task. We agree that in some cases a good pretraining objective would balance the shared and the different, but we would argue that there are tasks for which this line of research would be very useful – where we don't need to preserve generality, because our primary concern is that the model perform well on the downstream task. For example, in the clinical domain it is critical that models perform strongly, even if this comes at a cost to its capabilities in another field – [2] implements this. In fact, some approaches such as [4] (also in the clinical domain) go as far as to pretrain from scratch in the target domain so achieve performance gains – we do not pretrain from scratch because the datasets are relatively small and to scrape large enough amounts of data from these domains to perform an effective pretraining (see [5]) would be computationally infeasible. Our method is dramatically less computationally and data expensive than a full domain-specific pretraining, leveraging the models' pretrainings to transfer to the target domain with relatively little compute.\n\n--\n\n[1] Gururangan, Suchin, et al. \"Don't stop pretraining: Adapt language models to domains and tasks.\" arXiv preprint arXiv:2004.10964 (2020).\n[2] Lee, Jinhyuk, et al. \"BioBERT: a pre-trained biomedical language representation model for biomedical text mining.\" Bioinformatics 36.4 (2020): 1234-1240.\n[3] Tuhin Chakrabarty, Christopher Hidey, and Kathy McKeown. 2019. IMHO fine-tuning improves claim detection. In NAACL.\n[4] Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. ClinicalBERT: Modeling clinical notes and predicting hospital readmission. arXiv:1904.05342.\n[5] Kaplan, Jared, et al. \"Scaling laws for neural language models.\" arXiv preprint arXiv:2001.08361 (2020).\n[6] Adapting a Language Model While Preserving its General Knowledge, Ke et al., EMNLP 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Domain-adaptive pre-training is important.   \n2. The conventional masked language modeling based on random masks is intuitively too naive. Improving it with masking only the important things make intuitively sense.", "reasons_to_reject": "1. There are still other methods that are trying to improve the domain-adaptive pre-training (e.g., [1]), which the proposed method should also consider compared with. The author should compare the pros and cons of changing the data (like masking) with changing the neurons (like [1])  \n2. Since the proposed method only masks the difference, will the model forget about the general knowledge in the LM? How the proposed method address this? Intuitively, a good pre-training objective should be able to consider both the shared and the different.\n\n\n[1]: Adapting a Language Model While Preserving its General Knowledge, Ke et al., EMNLP 2022\n\n", "questions_to_authors": "See above", "paper_topic_and_main_contributions": "This paper improves continued pre-training, a.k.a. pre-finetuning, post-training or domain-adaptive pre-training by masking only the important/different words.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "See above"}, "extra_text": {}}, {"review_id": "lh6bkMWUmP", "reviewer": "Reviewer_o2x7", "summary": "", "questions": "Even though this may be out of scope for this paper, I am wondering is continual pretraining needed? Especially for VL tasks including VideoQA I think the common setting is to finetune on the dataset using pretrained VL models.", "limitations": "", "rating": 1, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your review and the insights provided about our work on continued pretraining. It's heartening to know that you recognize the importance of studying effective masking strategies, especially given its prevalence in VL and language model pretraining. We are glad that the significance of the continual pretraining setting and our exploration across both language-only and multimodal tasks resonated with you. We'll now proceed to address your concerns.\n\n--- \"Novelty of the proposed masking strategy is limited.\" ---\n\nWe respectfully disagree, but find this difficult to engage with because of a lack of explanation. Although tf-idf has been widely used in NLP, its application to determining what to mask has not been explored before (especially using the idea of tf-idf across corpuses, as we do in our paper). Additionally, our experiments and analyses show our method's usefulness in important application areas that may not be covered by standard pretraining. Underlying our use of tf-idf is the core idea to mask what makes a domain unique without labels, which is significantly different from the baselines we considered, which either mask randomly ([1]), by focusing on models' attentions ([2],[3]), by using a domain-specific entity tagger which requires pretraining, and does not \"rank\" masking probabilities by uniqueness ([4]), or by using downstream labels (which our method assumes we do not have access to) to influence the masking strategy ([5]).\n\nPerhaps this assessment of our novelty is due to our method's simplicity. Although Difference-Masking is relatively simple to implement, we do not believe this to be a weakness of the method – in fact quite the opposite. Because our method is simple to implement and operates in a setting with relatively small amounts of data, we think this may be an important baseline for future work that seeks to continually pretrain models on target domains – especially because it performs much more than complex baseline approaches on the low-resource setting.\n\nWe hope this addresses R3's concern.\n\n--- \"Only a 110M RoBERTabase model and a MERLOT-Reserve model are tested. It is hard to tell if the proposed method can be generalized to a more common scenario for example for large language models of billions of parameters.\" ---\n\nIn this paper, we study how to determine which tokens to mask during continued pretraining. Encoder-based transformer models such as RoBERTa and MERLOT-Reserve randomly mask-and-predict a portion of the input tokens. We replace this random masking strategy with our proposed approach (as well as other non-random baselines) and find strong improvements. Decoder-based transformer models such as today’s multi-billion-parameter LLMs do not selectively mask-and-predict; they always predict the next token. Therefore, it is unclear how to apply “masking strategies” to a training paradigm that does not selectively mask (through random selection or otherwise).\n\nHowever, in response to this question, we attempted to do so to the best of our abilities by creating an instruction-tuning version of masked-language modeling with one open source LLM, Vicuna-7B. We found that Vicuna did actually perform slightly better with our method of selecting masks. Our experimental details and results are below:\n\nWe conduct two stages of finetuning on pretrained Vicuna-7B using LoRA. To adapt autoregressive models in our setup, we design an instruction-tuning task for the first stage (CPT) where we replace some words (randomly vs Difference-Masking: TF-ICF) in the text (15%) with “%%%%” and ask the model to predict the replaced words. We only use the training set of the downstream tasks (ACL-ARC, Chemprot)  for CPT. In the second stage (FT), we finetune the CPT-ed model on the downstream tasks. \n\nHere is our exact hyperparameter setting: Batch size: 32, LR: 3e-4, Epochs: 3 (early stopping), LoRA: r=8, lora_alpha=16, lora_dropout=0.05, Schedule: Linear warmup with cosine decay, Warmup steps: 128, Precision: bfloat16.\n\nAnd the prompts we use for instruction tuning:\n\nCPT Prompts:\n\nprompts = f\"Given a text from a scientific paper where some words are replaced by %%%%% , the assistant outputs those words in the text. \\n\\n Text: {text} \\n\\n Output:\"\n\nFT Prompts:\n\nprompts = f\"Given a text from a scientific paper which contains two phrases inside << >> and [[ ]], the assistant outputs the relations of those two phrases in the text \\n\\n Text: {text} \\n\\n Relation:\"\n\nWe run these experiments across five trials and find that Vicuna-7B performs slightly better on ACL-ARC (.5476) using Difference-Masking CPT as opposed to random (.5396).  On ChemProt too, our method performs slightly better: .8020 compared to .7982. The gap in these experiments may be narrower due to the fact that this kind of “masked language modeling” within an instruction tuning setup is not common for this class of models. This is our best attempt to simulate masking within these models.\n\nA deeper interpretation of this point, however, may be that it is not worthwhile to conduct research on masking strategies in encoder-based models such as RoBERTa and MERLOT in the era of LLMs such as the GPT models. To this point, it is worth mentioning that masking is particularly useful in other modalities right now, such as vision [9] and audio [10]. Understanding masking strategies may be useful for those modalities, if not for NLP at the moment. However, continued pretraining appears to be a relevant and meaningful research direction at the moment [1,11,13,14].\n\n--- \"In the multimodal setting, only the VideoQA is evaluated and for language-only setting, only ACL-ARC task is evaluated.\" ---\n\nWe would like to clarify some misconceptions about our experimental setting: it is incorrect that \"for language-only setting, only ACL-ARC task is evaluated\". We reference the ChemProt task, the other language-only dataset often in our paper, including in Table 1, Section 1, Section 4, Section 5.1, Section 5.2, Figure 3, Section 5.3, and Table 3, where we describe the dataset, our experimental setting, our findings, and detailed analysis involving the ChemProt task.\n\nIt is true that both of our multimodal datasets (TVQA and Social-IQ) are video-based and use the question-answering paradigm for evaluation, but this is the case of many multimodal datasets and tasks ([6]). We test on four distinct datasets, two language-only and two multimodal video. We hope this clarifies any misconceptions about the nature of our experimental setting.\n\n--- Questions For The Authors: \"Even though this may be out of scope for this paper, I am wondering is continual pretraining needed? Especially for VL tasks including VideoQA I think the common setting is to finetune on the dataset using pretrained VL models.\" —\n\nContinued pretraining has been shown to be particularly effective in settings where there is limited labelled data from the downstream task, and a larger amount of unlabelled in-domain data [1,13,14]. As labelled data can be costly to annotate (which is the case for VL tasks including VideoQA [12]), but unlabelled data is much easier to come by, it becomes increasingly important to have domain-adaptation strategies that can learn from larger amounts of in-domain unlabelled data. The limited label-setting is what makes the continued pretraining necessary. Our method is an approach to performing this continued pretraining better by determining what to mask based on the downstream domain. Continued pretraining can also be important in related settings applied to language and vision models, such as mitigating catastrophic forgetting [11].\n\nRegarding soundness and reproducibility, we carefully document our experimental setting and our method and describe the analyses and the baseline masking approaches in great detail. In the appendices, we provide our exact hyperparameter grid search applied to the original settings from Don't Stop Pretraining [1] and MERLOT-Reserve [7], as well as describe in detail how we implement masking at the bounding box and word level for our experiments. In addition, we have released our anonymized code and use publicly available dataset [8] for anyone to reproduce. We hope R3 will reconsider our soundness and reproducibility given these clarifications.\n\n--\n\n[1] Gururangan, Suchin, et al. \"Don't stop pretraining: Adapt language models to domains and tasks.\" arXiv preprint arXiv:2004.10964 (2020).\n\n[2] Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. 2021. Mst: Masked self- supervised transformer for visual representation. Advances in Neural Information Processing Systems, 34:13165–13176.\n\n[3] Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. 2022. What to hide from your students: Attention-guided masked image modeling. In Computer Vision – ECCV 2022, pages 300–318. Springer Nature Switzerland.\n\n[4] Chen Lin, Timothy Miller, Dmitriy Dligach, Steven Bethard, and Guergana Savova. 2021. EntityBERT: Entity-centric masking strategy for model pretrain- ing for the clinical domain. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 191–201, Online. Association for Computa- tional Linguistics.\n\n[5] Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan 686 Liu, and Maosong Sun. 2020. Train no evil: Selective 687 masking for task-guided pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6966–6974, Online. Association for Computational Linguistics.\n\n[6] Baltrušaitis, Tadas, Chaitanya Ahuja, and Louis-Philippe Morency. \"Multimodal machine learning: A survey and taxonomy.\" IEEE transactions on pattern analysis and machine intelligence 41.2 (2018): 423-443.\n\n[7] Zellers, Rowan, et al. \"Merlot reserve: Neural script knowledge through vision and language and sound.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[8] https://anonymous.4open.science/r/Difference-Masking-842D/README.md\n\n[9] Peng, Zhiliang, et al. \"A unified view of masked image modeling.\" arXiv preprint arXiv:2210.10615 (2022).\n\n[10] Niizumi, Daisuke, et al. \"Masked spectrogram modeling using masked autoencoders for learning general-purpose audio representation.\" HEAR: Holistic Evaluation of Audio Representations. PMLR, 2022.\n\n[11] Cossu, Andrea, et al. \"Continual pre-training mitigates forgetting in language and vision.\" arXiv preprint arXiv:2205.09357 (2022).\n\n[12] Vondrick, Carl, Donald Patterson, and Deva Ramanan. \"Efficiently scaling up crowdsourced video annotation: A set of best practices for high quality, economical video labeling.\" International journal of computer vision 101 (2013): 184-204.\n\n[13] Dery, Lucio M., et al. \"AANG: Automating Auxiliary Learning.\" arXiv preprint arXiv:2205.14082 (2022).\n\n[14] Gupta, Kshitij, et al. \"Continual Pre-Training of Large Language Models: How to (re) warm your model?.\" arXiv preprint arXiv:2308.04014 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The masking strategy is commonly used in VL and language model pretraining. Studying how to more effectively perform the masking is well-motivated and it is an important problem.\n\nThe continual pretraining setting is interesting.\n\nBoth language-only and multimodal tasks are evaluated.", "reasons_to_reject": "Novelty of the proposed masking strategy is limited. \n\nOnly a 110M RoBERTabase model and a MERLOT-Reserve model are tested. It is hard to tell if the proposed method can be generalized to a more common scenario for example for large language models of billions of parameters.\n\nIn the multimodal setting, only the VideoQA is evaluated and for language-only setting, only ACL- ARC task is evaluated.", "questions_to_authors": "Even though this may be out of scope for this paper, I am wondering is continual pretraining needed? Especially for VL tasks including VideoQA I think the common setting is to finetune on the dataset using pretrained VL models.", "paper_topic_and_main_contributions": "This paper studies the continued pretraining setting where pretrained models continue to pretrain on domain-specific data before performing some downstream task. The paper introduces a masking strategy that automatically chooses what to mask during continued pretraining by considering what makes a task domain different from pretraining domain. On continued pretraining settings the proposed method performs better across four diverse language-only and multimodal video tasks.", "excitement": 1, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a simple and elegant modification to masking for domain adaptation in continued pre-training by focusing on masking tokens that are different in the new domain compared to the old domain. The authors also provide good experimental results, outperforming more complicated baselines on two NLP datasets (ACL-ARC, ChemProt) and two multimodal datasets (Social-IQ, TVQA).

The authors have addressed the reviewer concerns in the comments (clarifications for Reviewer LPF7), results on an additional baseline (Reviewer Yfzq) and additional experiments + clarifications for Reviewer o2x7.
Decision: Accept-Findings