The Distributional Hypothesis Does Not Fully Explain the Benefits of Masked Language Model Pretraining
Authors: Ting-Rui Chiang
Dani Yogatama
Venue: EMNLP 2023 Main (2023)
Keywords: Distributional Hypothesis
MLM
Pretraining
TL;DR: We show that the distributional hypothesis can not provide a complete explanation for the efficacy of MLM pretraining.

Abstract: We analyze the masked language modeling pretraining objective function from the perspective of the Distributional Hypothesis.
We investigate whether the better sample efficiency and
the better generalization capability of models
pretrained with masked language modeling can
be attributed to the semantic similarity encoded in the pretraining data's distributional property.
Via a synthetic dataset, our analysis suggests that distributional property indeed leads to the better sample efficiency of pretrained masked language models, but 
does not fully explain the generalization capability.
We also conduct an analysis over two real-world datasets 
and demonstrate that the distributional property does not explain the generalization ability
of pretrained natural language models either.
Our results illustrate our limited understanding of model pretraining and provide future research directions.

The Distributional Hypothesis Does Not Fully Explain the Benefits of Masked Language Model Pretraining
Ting-Rui Chiang and Dani Yogatama
University of Southern California (tingruic, yogatama) (tingruuic.yogataaaaeeuuss. edu
Abstract
Previous analyses of pretrained model focus on probing the pretrained representations (Tenney et al., 201 2019b,a; Liu et al., 2019a; 9a; Hewitt and Man- ning, 2019; Wu et al., 2020; Rogers et al., 2020; Zhang et al., 2021; Immer et al., 2022). However, these models are rarely used a static feature extractor. Practitioners often fine-tune them with downstream tasks.
We analyze the masked language modeling pre- training objective function from the perspective of the distributional hypothesis. We investi- gate whether better sample efficiency and the better generalization capability of models pre- trained with masked language modeling can be attributed to the semantic similarity encoded in the pretraining data's distributional property. Via a synthetic dataset, our analysis suggests that distributional property indeed leads the better sample efficiency of pretrained masked language models, but does not fully explain the generalization capability. We also conduct anal- yses over two real-world datasets and demon- strate that the distributional property does not explain the generalization ability of pretrained natural language models either. Our results illustrate our limited understanding model pretraining and provide future research direc- tions."
We analyze whether mask language model pre- training allows models to leverage semantic knowl- edge derived from the distribution of words in the pretraining data for downstream tasks. This re- search question follows the distributional hypoth- esis (Harris, 1954; Firth, 1957). postulates that semantically related words have a similar con- text distribution, which we elaborate more rig- orously distributional property of pretrain- ing data in $2. Because it draws connections be- tween word semantics and data distribution, has been widely accepted as an explanation for the efficacy non-contextullized (type-based) word embeddings (Levy and Goldberg, 2014) such as Word2 Vec (Mikolov et al., 2013) and GloVe (Pen- nington et ul., 2014). Recently, Sinha et al. (2021) shows that word order does not greatly affect the efficacy of masked language model pretraining (De- vlin et al., 2019; Liu et al., 2019b), thus conjectur- ing that the distributional hypothesis could be an explanation. In this work, we continue studying this conjecture by investigating whether pretrained models can utilize the semantic knowledge derived from the data distribution as suggested by the dis- tribution hypothesis.
1 Introduction
Despite the rise of the prompting paradigm with the scaling breakthrough of very large language models (Brown et al., 2020), understanding the mechanism model fine-tuning remains to be an important endeavor. Fine-tuning relatively small models such BERT (Peters et al., 2018; 8; Devlin et al., 2019; Liu et al., 2019b) has significant practical implica- tions since these models are computationally more efficient than large language models such as GPT-3 (Brown al., 2020). Distilling large language model to a small model by fine-tuning is also a promising direction (Lang et al., 2022; Hinton et al., 2015). The development of large language models also involves fine-tuning, e.g. doing reinforcement learning from human feedback (Bai et al., 2022). In this work, seek to understand how pretraining benefits downstream fine-tuning.
Among different types of semantic knowledge the distribution of the pretraining data may encode, focus on semantic equivalence, i.c., synonym, which Harris (1954) suggests be encoded in the data distribution. In $3, we theoretically show that prior knowledge about semantic equivalence alone is enough to lead to two desirable propertics that pretrained language models have: better sample
'Scripts for the experiments in this paper are available at https:// //githubb com/ uuc- was--ttagtttchii Comagotchi/DH-WLL /DH-MLM.

---

efficiency and generalization capability (Tu al.,
single token). Having these semantic equivalence sets has the following implications:
2020; Hendrycks et al., 2020: Tänzer al., 2022). In $4, as our first step to understanding how masked language modeling training objective ex- ploits the distributional property of the pretraining data, set up toy experiment. We construct synthetic pretraining corpus and a downstream task. In $5, we show that the distributional property ex- plains the sample efficiency of pretrained masked language models, but does not always lead to bet- ter generalization on its own; still depends on the type of distribution shifts. Our results also suggest that better parameter initialization statistics (Wu et al., 2022) do not explain the benefit of masked language model pretraining either.
In the pretraining data: The distributional hy- pothesis suggests that the pretraining data has a distributional property: for any two features a, b in the same synset and any 1, the distribution of their neighboring words satisfies
p(æ1, 22, " .nnla) 22 p(#1, 22, .nn(6). (1)
For example, the phrase "is delicious" has the same meaning as the phrase "taste good". Therefore, if the training data has this distributional property, then we expect its distribution to satisfy
In $6, we conduct a similar experiment in the real-world setting. We analyze a pretrained BERT model and two real-world tasks SST- 2 (Socher et al., 2013; Wang et al., 2018) and MNLI (Williams et al., (). We find that the semantic (in)equivalence the model learns from the pretraining task is independent of how fine-tuned model treats words as synonyms. This indicates that pretrained models do not generalize better by modeling the distributional property of the pretrain- ing data either.
p(ar| "is delicious")) (tastts good").
In downstream task: Given an input æ for a downstream task, substituting a feature a with an- other feature b in the same synset does not change the meaning of because and b are synonyms. Therefore, the substitution does not change the la- bel of æ either:
f (xx f (Replace(z; a, b)),
(2)
where f is the task's labeling function that maps an input æ its label y-
To summarize, our main findings include: i) We show that semantic equivalence encoded in the dis- tributional property of the pretraining data makes pretrained models more sample-efficient. ii) There is a type of generalization capability independent the distributional property of the pretraining data. iii) The distributional property of the pretraining data does not explain pretrained models' gener- alization capability in the real world. Therefore, conclude that the distributional hypothesis by Harris (1954) alone is not enough for complete explanation. Future work may study the interplay between other complex semantic relationships, data distribution, and model pretraining to better under- stand pretrained models' generalization behavior.
The Benefit of Modeling Semantic Equivalence
In this section, we show why having prior knowl- edge about the semantic equivalence sets (synsets) helps downstream performance.
3.1 Sample Efficiency
Understanding which symbols are semantically re- lated makes a classification problem easier. From the perspective of the model, the inputs are quences of symbols. Having prior knowledge about the relationship between these symbols decreases the number of training examples the model requires to learn the target labeling function.
2 Properties of Pretraining Data and Downstream Tasks
For example, consider a task with four training examples: ("It is delicious". True), ("It is awful". False), ("It is tasty", True), ("It is bad", False). If the model does not know the semantic relation- ship between the predictive features "awful", "bad", "tasty", and "delicious", then it is impossible for the model learn the underlying target labeling function from these four samples, because each of these four words appears only once the dataset.
Since we focus on semantic equivalence in lan- guage, the concept of "synsets" from Word- Net (Miller, 1998), i.e., the sets of synonyms. Here we extend the concept of synsets to allow each synset to contain multi-token elements, such as phrases or sentences. Elements in a synset are fea- tures and each of them is a sequence of tokens (or

---

Pre-define two Markov chains as pg pg
synset contains two features:
8; (ai, bz).
Choose or Pg
3. Sample features a1 Q2 03 a1 a2 by b2 bj by bz not satisfying DH
This provides us with two semantically isomorphic sets of features La = {ar)== and L = {b:]/=1 Each feature analogizes to a word or a phrase natural languages, depending on whether each fea- ture corresponds to one or multiple tokens. We discuss the multi-token setting in A.
2. Sample synsets
P (1) sample
01 02 03 01 02
a1 b2 a3 b1 02
PS sample
ag 02 b1 03 b2
03 02 01 03 02
satisfying DH
4. Map features to tokens
To understand how masked language model pre- training would help models utilize the semantic isomorphism between Pa and P. we start with a setup where the semantic isomorphism is encoded with a simple distribution that language models can learn easily. Specifically, we first randomly gener- ate two Markov chains P(!) p!?) and use them to generate sentences (Figure 1):
@1 - Q1 01 + 0103
-1 0101
b1 + B1
b 32
b 02
single-token
mulit-token
mulit-token sharing
vocab. sharing
Figure 1: The generation process of the pretraining data (84.1). The words in the orange color denote indepen- dent experimental variables.
1.
We randomly choose l from {1,2), which decides whether use the first Markov chain p!l or the second one p(o)
However, the knowledge that some
symbols satisfy Eqq. makes the task f("It is delicious") f (Replace is delicious", "delicious", "tasty"). In other words, it reduces the feature space. Based on statistical learning theory (Valiant, 1984; Blum and Mitchell, 1998), this smaller feature space reduces the sample complexity of the model.
easier, e.g.
2. We then draw a sequence of synsets $1, $2. 11 based on the distribution defined by the chosen Markov chain pS (s1, 82, st).
Then for 1, 2, ... ,I, we draw a feature T3 E 8,. At this step, we can control the dis- tributional property as an independent exper- imental variable. We can generate a dataset with the distributional property in Eq. by drawing features from s; {aa, bi) uniformly random. Or can generate a dataset with- out the distributional property by always draw- ing as or when or = respectively.
3.2 Generalization Capability
If a pretrained model is able to preserve its knowl- edge about semantic equivalence among words af- ter being fine-tuned, then will help the model to generalize better. For example, consider a simpli- fied case with two training examples ("It is deli- cious", True) and ("It is awful", False), as well as test example "It is tasty". Generalizing to this test example is possible only when the model un-
4. Finally, map each feature single token. (In A.1, we describe multi-token setups where we map each feature multiple tokens.)
derstands the semantic relationships between "deli-
cious", "awful" and "tasty". That is, if model has an inductive bias in Eq. 2, as long as e containing a is in the training set, the model will be able to gen- eralize to testing sample Replace(2; a, b). where b is an unseen feature semantically equivalent to
4.2 Downstream Task
We construct synthetic downstream task aligned with the semantic equivalence specified in Eq. 2. We define pattern-matching task where the label- ing function that maps sequence of features label based the underfying synsets sequence that generates the feature sequence. If the sequence of synsets matches one of a predefined set of pat- terns, then we label the sequence as positive and otherwise negative.
4 Synthetic Data Construction
4.1 Pretraining Datasets
For pretraining, we construct a lseudo-language with the properties described in $2. This language has n synsets Σ = {0,, "2 ,n, and each

---

We define these patterns with regular expressions in this form:
the model has the knowledge about the semantic isomorphism between &a and 4,. then the model will be able to use the examples from both domains to learn a single labeling function. However, if the model has no such prior knowledge, then it will need to learn two labeling functions for A-D1 and B-D2 respectively with only half of the total amount of data. Thus, if the model can learn to uti- lize the semantic equivalence encoded in the distri- bution property (Eq. 1) of the pretraining data, then models pretrained with dataset having the property in Eq. will perform better.
M' M' S2 E Sz "
where Σ (or, 02 02. Jnt. St, S2, S3 Σ are 3 randomly selected sets of synsets and * is a Kleene star. In other words, sequence of synsets matches the pattern if synsets in St, S22, S3 appear in the sequence in order.
We generate four downstream datasets as fol- lows. First, we can choose to generate the sequence of synsets from Ps or p!? We will use DI and D2 to denote the samples generated from these two distributions respectively. Second, for each synset Sj. we can choose to always use features in 4, or Øg. We will use A and B to denote the samples constituted with these two feature sets respectively. Therefore, there are 4 combinations of choices: A-
Generalization capability. We assess general- ization capability by evaluating the model on a test set whose distribution is different from the model's training set. Specifically, we fine-tune the model with training set in A-DI and test it with test sets in all of the 4 distributions. We also explore whether having small amount of data from tar- get domain can improve the generalization. Thus we also experiment with the setting where 10% of the training samples are from B-D2.
DI, B-DI, A-D2, and B-D2. We will use these combinations test the generalization capability and the sample efficiency of the pretrained models.
5 Synthetic Experiment Results
There are distribution shifts in two directions in this setting. One direction is the distribution shift from feature o, Qg (e.g. from A-D1 to B- DI). We refer to this direction as a vocabulary shift, because the vocabulary is changed. The other direction is the shift from [(]) to P! (e.g. from A-D1 to A-D2). We refer to it semantic shift, because the correlation of the synsets is changed.
5.1 Experiment Design
We first generate two pretraining datasets of the same size, one of which has the distributional prop- erty in Eq. 1 while the other one does not (accord- ing to $4.1). We then use these two datasets to pretrain two Transformer models (Vaswani et al., 2017) with the masked language modeling objec- tive (the w/ DH and w/o DH model). Finally, fine-tune these two models with a downstream task dataset generated in $4.2 (more details B). We report the performance of the models after fine- tuning them with different numbers of downstream examples. By comparing the performance of the downstream task, we can infer to what extent the distributional property contributes to the efficacy of MIM pretraining. Despite the complexity of natural language, we posit that a portion of seman- tie equivalence in natural language encoded in a distribution as simple as the one of this pseudo language. Therefore, positive results obscrved in this toy experiment would also apply to real-world setups. In the following, we describe how we use the four downstream datasets (i.e., A-D1, B-D1, A-D2, and B-D2) for evaluation:
These two distribution shifts exist in real-world NLP scenarios. The vocabulary shift is analogous to cross-lingual generalization. It is similar to fine- tuning multi-lingual pretrained model with lan- guage and doing zero-shot transfer to test set in another language (Artetxe et al., 2020). As for the semantic shift, it is related to the spurious correla- tions in training data (Poliak et al., 2018; Gururan- gan et al., 2018; McCoy et al., 2019; Chiang et al., 2020; Gardner et al., 2021; Eisenstein, 2022).
5.2 Other Baselines
In addition to the model pretrained with a dataset that does not have the distributional property (Eq. 1), we also have other baselines:
From-scratch: We fine-tune the model from scratch with randomly initialized parameters.
Sample efficiency. We fine-tune the models with a dataset in which 50% of examples are in A-D1 and the other 50% are in B-D2. In this setting,
CBOW: We pretrain CBOW embeddings with the data that has the distributional property

---

Ratio 0.4
training
The performance of the CBOW initialization is also better than training from scratch. Thus, the dis- tributional property in Eq. I indeed a valid expla- nation for non-contextualized embeddings. How- ever, its improvement is not as stable as the w/ DH pretrained MLM. It suggests that in addition to the property of distribution, some other factors also attribute model's better sample efficiency.
0.2
0.6
0.8
0.9
Accuracy
0.7
0.5
A-21
A-D1 DH
B-01 w/
B-D1 we
-0h A-D2 w/ DH
A-02 w/o OH
8-02 w/ DH
+ 6-02 w/o DH
Ratio of training data
0.6
5.3.2 Generalization Capability
0.9
Generalization in the presence of vocabulary shifts. The results in Figure 3a and 3b indicate that the distributional property Eq. can help the generalization when there are vocabulary shifts to some extent. We observe that the w/o DH model does not generalize to B-D1 and B-D2 at all, while the w/ DH model can generalize from domain A- DI to the test set in B-D1 and B-D2. However, the generalization diminishes the amount of fine- tuning data increases. The model may suffer from catastrophic forgetting after we fine-tune more. Having some fine-tuning data in B-D2 mitigates this problem. As shown in Figure 3b, when the 10% of the fine-tuning data is in B-D2, the w/ DH model outperforms the w/o DH model much for the test sets in B-DI and B-D2, and the generalization does not diminish.
Accuragy
0.7
-0 D1 CBOW +h D1 scratch - 8:01 CBOW + 8:D1 scratch
A-02 CBOW
02 scretch 8-02 CBOW 9-02 scretch
Figure 2: Performance on the synthetic downstream task when the models are fine-tuned with 50% A-DI and 50% B-D2 ($4.2).
and use it to initialize the embedding layer of randomly initialized Transformer.
Shugfte-weight: We randomly shuffle the weights in each module of the MLM pre- trained model before fine-tuning it, which keeps the statistics of the parameters. This is motivated by a previous work showing the efficacy of parameter initialization statistics (Wu et al., 2022). We inspect whether the statistics explain the efficacy of the model.
The CBOW model also generalizes to B-D1 and B-D2. Its generalization to B-D1 does not suffer from catastrophic forgetting. Thus, the distribu- tion property in Eq. I indeed largely explains the efficacy of non-contextualized word embedding.
Generalization to the semantic shifts. We find that the distributional property in Eq. 1 can not ex- plain it. From both Figure 3a and Figure 3b, we can see that both the w/o DH model and the w/ DH model generalize A-D2 better than the model trained from scratch. Moreover, the w/ DH model does not perform better. Meanwhile, the CBOW model performs only slightly better than training from scratch. Thus, generalization to the semantic shifts may be related to the nature of MLM pre- training instead of data properties (Chiang, 2021).
5.3
Experiment Results" 2
5.3.1 Sample Efficiency
We find that the distributional property (Eq. 1) can largely explain MLMs' better sample efficiency. In Figure 2, we can see that the w/ DH model's performance on A-D1 and B-D2 grows faster than the performance of the w/o DH model. This implies that pretraining with data satisfying Eq. indeed improves the sample efficiency. The w/ DH model also has better performance on B-D1 and A-D2. It is aligned with our intuition that Eq. 1 can help the model utilize the isomorphism between 4. and &, and learn a single general function instead of two functions for A-DI and B-D2.
5.4 Summary
Our results suggest that pretraining with data whose distribution encodes the semantic equivalence be- tween words (E'q. 1) improves pretrained MLM's sample efficiency, but the generalization capability in the presence of vocabulary shifts diminishes as the number of training examples grows.
Modells initialized with shuffled weights have similar per- formance as models trained from scratch so we omit them in the figures for clarity.

---

Ratio of training data
Ratio of training data
0.2
0.4
0.6
0.8
0.4
0.6
0.8
0.9
Accuracy
Accuracy
0.8
0.5
0.5
Fine-tuning with 100% A-D1
Ratio of training data
Ratio of training data
0.2
0.4
0.6
0.8
0.4
0.6
0.8
Accuraty
0.8
Accuracy
0.8
0.7
0.5
A0-A-D1 W/
w/o w/
w/o
CBOW
B-D1
-0-A-02 w/ DH
A-D2 w/o DH
B-02 w/ DH
B-02 w/o DH
-0-
A-D2 CBOW + A-D2 scratch -0- B-D2 CBCW + B-D2 scratch
(b) Fine-tuning with 90% A-DI and 10% B-D2
Figure 3: Performance the synthetic downstream task described in $4.2.
Our experiments also shed light on the effect of non-contextualized word embeddings and the statis- tics of parameter initialization. We find that the distributional property (Eq. 1) indeed contributes to non-cone non-connextuulized embeddings' better sample efficiency and better generalization capability. As for the statistics of the parameter initialization, find that models with shuffled weights marginally outperform models trained from scratch for all the settings. suggests the statistics of parameters may not explain the efficacy of pretraining.
where the data also has this property. However, the negative results regarding the generalization capability may be due to the oversimplification of our toy setting. Therefore, in this section, we study generalization in real-world scenarios.
Unfortunately, it is difficult to conduct fully controlled experiment as do with the toy set- tings. It is impossible find out all the synsets in a natural language. It is also difficult to divide features into isomorphic subsets as the feature sets t and o, we have in the toy settings. Therefore, we will measure the relationship between the dis- tributional property in Eq. 1 and generalization in indirect manner.
Additionally, also observe phenomena that the distributional property (Eq. 1) cannot explain. We find that the distributional property unrelated to the semantic distribution shifts. We also observe that using CBOW embeddings not as stable as using an MLM pretrained model. Therefore, the distributional property alone does not fully explain the efficacy of model pretraining.
6.1 Experiment Design
We want to measure to what degree we can at- tribute models' generalization capability to the dis- tributional property (Eq. 1) of the pretraining data. Thus, we design experiment using the follow- ing premise: if fine-tuned model f generalizes well because the pretrained model can learn seman- tic equivalence between features by modeling the distribution property in Eq. I, then whether f can generalize should depend on whether fo models Eq. successfully.
As for the multi-token setting, sometimes the distribution property does not improve sample ef- ficiency but still helps generalization. We include the results in A.2 for brevity.
6 Analyzing Models for Real-world Datasets
Based on this premise, design experiment in the following way:
In show that the distributional property in Eq. can improve sample efficiency, least when the feature at the single-token level. We posit that this conclusion applies to the real-world scenario
Given a validation or testing sample æ in a downstream task, we pick a feature a from æ

---

and generate a noisy paraphrase b ($6.2).
on the task. For example, in the NLI task, each example involves two sentences.
We measure how well the fine-tuned model f generalizes by how much its prediction changes after we replace a in a with b:
We use WordNet (Miller, 1998) and back- translation (Sennrich et al., 2016; Yu et al., 2018) to generate semantically similar features. For word- level feature a, randomly select word b from one of its synonyms in WordNet that has the same POS tag. For a feature at higher level, we use back-translation to paraphrase it because we find that back-translation generates more diverse para- phrases than publicly available paraphrase models do. We use Facebook FAIR's s WMT19 translation models (Ng et al., 2019) and use German as the intermediate language. When generating para- phrase b for feature a in I, we use the context before a in x as the prefix of the generation pro- cess, so b has the same syntax role as a. We include some examples in Table
Dy(0, a, b) = (V][()) (Replacc(,,, a, b)],
where TV is the total variation distance.
We also measure whether the pretraining pro- cess encodes the semantic equivalence be- tween a and in the model, which we will claborate on later in $6.3.
Finally, we measure the Pearson correlation coefficient between fu (a, b) and D a, b). We should observe that D is high only when J is high if the fine-tuned model generalizes totally based on the semantic equivalence en- coded by the distributional property in Eq. 1.
Note that and b do not need to be perfectly sim- ilar. Although an imperfect (a, pair may cause the fine-tuned model to change its prediction (hav- ing high (5(æ, a, b), a pretrained model should also identify a and b as dissimilar (having high Du (a, b)). Therefore, noises in our paraphrasing generation process do not affect our analyses.
We analyze bert-base-ue uncased model for single-sentence and multi-sentence task, the SST- 2 task, and the MNLI task. The MNLI dataset comes with its constituency parsing along with the POS tags. As for the SST-2 dataset, it does not include the POS tags, so we parse with the Stanford parser (Manning et al., 2014). addition to analyzing the final fine-tuned model, we report the results for intermediate checkpoints. We also analyze models trained with different amounts of data to inspect the effect of training data size.
6.3 Measuring Semantic Distance from Pretrained Model
If pretrained models learn the semantic relationship between features by modeling the distributional property in Eq. during MLM pretraining, then the distribution predicts will reflect its knowl- edge about the semantic relationship. Specifically, if MLM pretraining encodes the semantic equiv- alence between two features a, b in a pretrained model fo. then the model should predict similar context distributions for and b. Therefore, we can check whether the pretraining process encodes the semantic equivalence between feature and by
6.2 Noisy Feature Paraphrases
We use validation and test examples in this anal- ysis and extract features at different levels: word level, phrase level, sentence level, and example level. For word-level features, we first measure the word saliency (Li et al., 2016; Ren et al., 2019) and extract the word in the example with the highest saliency. For the phrase-level features, we consider both short phrases and long phrases in an example. For short phrases, we measure the saliency of phrases no longer than 4 words and choose the one with the greatest saliency. For long phrases, we choose the longest phrase in the exam- ple. A sentence-level feature is a whole sentence in a testing sample, while an example-level feature may involve more than one sentence depending
D (a,b) b) = [Contcoo)) [ (context/b)),
where TV is the total variation distance.
We use templates to construct queries for the context distribution conditioned on feature. A desirable template should convert feature into grammatical sentence containing mask token such that the distribution the mask token can reflect the semantics of the feature. Therefore, we design different templates to take the features' syn- tax role into consideration:
'i... we measure the impact of a feature by measuring the change of the fine-tuned model's prediction after we mask the feature.

---

Templates for word-level and phrase-level fea-
12
101
150
20p
25k
376
406
tures We expect that the verb after a noun can reflect its meaning because different things act
Correleton
in different ways. For example, both cats and
dogs walk, but only dogs bark. Therefore, the
feature is noun or noun phrase, then we use "<feature> [mask]" as its template. Based on similar intuition, if the feature is adjective or adjective phrase, then is <feature>". If the feature verb or verb phrase, we use the template "[mask] <feature>". Sentences with this syntax structure are pervasive in the pretraining data, so these templates construct natural sentences as the queries for the semantic relationship in the pretrained model.
Step
0.14
template "[mask]
Carrelation
0.40
0.00
9.06
50.04
(a) MNLI (up) and SST-2 (down) different steps. We mea- sure the correlation per 100 steps the beginning.
0.5
0.3
0.1
Templates for sentence-level features We can sentence very high-level feature. For this
Conrelation
Correlation
0.08
0.06
0.04
0.04
kind of feature, use templates. The first
0.02
0.02
one more natural: "<feature> with [mask]." where we attach prepositional phrase after the feature using "with". Because the word after "with" largely depends on the semantics of the context be- fore it, the distribution at the "[mask]" reflects the semantics of the feature. We also experiment with template proposed by Jiang al. (2022) because they show that "">feature>" means [MASK]" can extract high-quality sentence representations.
(b) MNLI (left) SST-2 (right) with different data sizes.
Figure Correlation between ((x,a,))) and D
share the vocabulary (in Appendix A.2). However, in general, the correlation is low, indicating that the model's knowledge learned from the distributional property (Eq. 1) of the pretraning data can hardly explain models' generalization.
Templates for example-level features highest level, we can treat an example of a task as feature. That is, for tasks that involve multiple sentences, such as a natural language inference task, feature. For this use task-specific templates from Gao et al. (2021). These templates elicit the model's prior knowledge about the task. We will inspect to what extend such prior knowledge is preserved after we fine-tuned the model.
the
7 Related Work
we treat pair of sentences high-level feature,
This work aims to inspect how MLM pretraining infuses inductive bias for downstream tasks. is different from previous studies that treat pretrained models as static models. For example, Wei et al. (2021) and Chiang (2021) study the efficacy of MLM theoretically by analyzing the distribution predicted by the model. Tenney et al. (2019b,a); Liu et al. (2019a); Hewitt and Manning (2019); Wu et al. (2020); Zhang et al. (2021) study the rich linguistic structure encoded in the representation. Some works focus on the fine-tuning process but the pretraining loss. For instance, Hao et al. (2019) shows MLMs have better loss landscape. Agha- janyan et al. (2021) shows that pretrained models have lower intrinsic dimensionality. Malladi et al. (2022) shows neural tangent kernels (Jacot et al., 2018) can explain MLM's efficacy some extent. Some works study the connection between pre- training and downstream tasks. Li et al. (2021) show that lon-lang pretraining tasks help NLP
6.4 Results and Discussion
Figure shows that the correlation between D ((raat) and D is below 0.15 for both MNLI and SST-2. For MNLI, higher-level features have a higher correlation. As for SST-2, task-level fea- tures still have the highest correlation, but short- phrase-level features have the second-highest corre- lation. It may be because the SST-2 model utilizes lower-lever features more. We also observe that the correlation does not diminish to 0 regardless of the training steps or data sizes. suggests that the real-world scenario may be more similar to the high-level toy setting where isomorphic feature sets

---

downstream tasks, while Lu al. show that lan- guage models help non-language downstream tasks, Lovering et al. (2021) uses some linguistic tasks to show that the linguistic structures in MLMs can serve as an inductive bias. Zhang and Hashimoto (2021) investigate whether MLMs' efficacy can be explained by the task-relevant cloze-like masks alone. Our work follows this line and explains MLMS' general machine-learning characteristics. Beyond the scope of NLP, the concept of con- necting training examples is very common ma- chine learning. For example, co-training (Blum and Mitchell, 1998) uses multiple views of fea- tures to build connections between training sam- ples. Recently, Saunshi et al. (2019); HaoChen et al. (2021) show that contrastive self-supervise, learning reduces sample efficiency by pulling the representation of similar samples together. Sim- ilarly, Shen al. (2022) shows that contrastive learning connects samples in different domains and thus helps generalization. However, those studies mainly focus on contrastive learning for continuous data instead of discrete languages.
relatively simple case, where tasks are pattern- matching tasks. We do not consider, for example, tasks that require reasoning. $4, we study the ef- fect of the distributional property using Markov dis- tributions. It only suggests that model can achieve better sample efficiency by utilizing the seman- tic relationship encoded in this simple first-order distribution. How the models are able to utilize semantics encoded in more complicated distribu- tions, such as the distribution of natural languages, remains unknown. In $6, we only consider the dis- tribution of a single context token; however, the semantics of a feature may be encoded in the dis- tribution in a subtler way. Finally, we show the inefficiency of the distributional hypothesis as an explanation by providing counterexamples with small language models. We leave the study of large language models for future work.
A Acknowledgements
We appreciate Joshua Robinson for providing valu- able feedback on this paper. We also thank the members of my lab, tamagotchi lab at USC, for helpful discussions. Finally, we thank the USC NLP cluster maintenance team.
8 Diseussion
We showed in a synthetic experiment that pretrain- ing with masked language model objective allows the model to make use of semantic equivalence that simple distributions (c.g. Markov models) can en- code to improve models' sample efficiency. How- ever, we found that it can not explain generalization in the presence of distribution shifts.
References
Armen Aghajanyan, Sonal Gupta, and Luke Zettle- moyer. 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the I 11th Inter- national Joint Conference on Natural Language Pro- cessing (Volume Long Papers), ). pages 7319-7328, Online. Association for Computational Linguistics.
Our work leads to two future research directions. First, we showed the limitations of our understand- ing of pretraining. What semantic relationships are important, how data distributions encode those rela- tionships, and how modeling data distributions help downstream performance remain open questions. Second, our results in $6 show the unpredictability of models' generalization behavior. Investigating this behavior may help us develop more robust NLP
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623-4637, Online, Association for Computational Linguistics.
models.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint X2v:2220.005880
Limitations
Our main goal is to provide new perspective to analyze the efficacy of pretraining and to show that the distributional hypothesis is not a sufficient ex- planation. We only study semantic equivalence between features. Other relationships, such as hypernyms or causal relationships, are not in the scope of this paper. Additionally, we only consider
Avrim Blum and Tom Mitchell. 1998. Combining la- beled and unlabeled data with co-training. In Pro- ceedings of the Fleventh Annual Conference on Com- putational Learning Theory, COLT 98, page 92-100, New York, NY, USA. Association for Computing Ma- chinery.

---

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
2018. Annotation artifacts in natural language infer- ence data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 107-112 12, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019. Visu- alizing and understanding the effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP) pages 4143- 4152, Hong Kong, China. Association for Computa- tional Linguisties.
Ting-Rui Chiang. 2021. On a benefit of mask language modeling: Robustness to simplicity bias. arXiv preprint arXiv:2110.05300
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. 2021. Provable guarantees for self- supervised deep learning with spectral contrastive loss. In Advances in Neural Information Process- ing Systems, volume 34, pages 5000-5011, 1. Curran Associates, Inc.
Ting-Rui Rui Chiang, Hao-Tong Yc, and Yun-Nung Chen. 2020. An empirical study of content understanding in conversational question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7578-755
Zellig S Harris. 1954. Distributional structure. Word. 1062-3):146-162.
Jacob Devlin, Ming-Wei Chang. Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 2744-2751, Online. Association for Computa- tional Linguistics.
Jacob Eisenstein. 2022. Informativeness and invariance:
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4129 4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Two perspectives on spurious correlations in natural language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4326-4331, Seattle, United States. Association for Computational Linguisties.
John Firth. 1957. A synopsis of linguistic theory, 1930- 1955. Studies in linguistic analysis, pages 10-32.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in neural network. arXiv preprint (rXXi::50030255:
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1ith International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers). pages 38 3816-3830, Online. Association for Computa- tional Linguistics.
Alexander Immer, Lucas Torroba Hennigen, Vincent Fortuin, and Ryan Cotterell. 2022. Probing as quanti- fying inductive bias. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1839- 1851, Dublin, Ireland. Association for Computational Linguistics.
Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and Noah A. Smith. 2021. Competency problems: On finding and removing artifacts in language data. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 1801-1813, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Arthur Jacot, Franck Gabriel, and Clement Hongler. 2018. Neural tangent kernel: Convergence and gen- eralization in neural networks. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc
Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang. 2022. Prompt- BERT: Improving BERT sentence embeddings with
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith.

---

prompts. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8837, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
more, Maryland. Association for Computational Lin- guistics.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuris- ties in natural language inference. In Proceedingg of the 57th Anmual Meeting of the Association for Com- putational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguisties.
Hunter Lang. Monica N Agrawal, Yoon Kim, and David Sontag. 2022. Co-training improves prompt-based learning for large language models. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 11985-12003, PMLR.
Tomas Mikolov, llya Sutskever, Kai Chen, Greg Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. In Advances in Neural Information Processing Sys- tems, volume 26, Curran Associates, Inc.
Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. In Ad- vances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.
George A Miller, 1998. WordNet: An electronic lexical database. MIT press.
Jiwei Li. Will Monroe, and Dan Jurafsky, 2016. Un- derstanding neural networks through representation erasure. arXiv preprint arXiv: 1612.08200.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ou, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shaved Task Pupers, Day 1), pages 314-319, Florence, Italy, Association for Computational Linguistics.
Liunian Harold Li, Haoxuan You, Zhecan Wang, Alireza Zarcian, Shih-Fu Chang, and Kai-Wei Chang. 2021. Unsupervised vision-and-lame and-language pre-training with- out parallel images and captions. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 3339-5350, Online. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532 1543, Doha, Qatar. Association for Computational Linguistics.
Nelson F. Liu, Maut Gardner, Yonatan Belinkow, Matthew E. Peters, and Noah A. Smith. 2019a. Lin- guistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volune 1 (Long and Short Pa- pers), pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguisties.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisticx: Humun Lunguage Tech- Rologies, Volume I (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint 190..
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language infer- ence. In Proceedings of the Seventh Joint Confer- ence on Lexical and Computational Semantics, pages 180-191, New Orleans, Louisiana. Association for Computational Linguistics.
Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick. 2021. Predicting inductive biases of pre- trained models. In International Conference on Learning Representations.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mor- datch. Pretrained transformers as universal computa- tion engines.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial exam- ples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1085- 1097, Florence, Italy. Association for Computational Linguistics.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora, 2022. kernel-based view of language model fine-tuning, arXiv preprint 1r.0v:2210.06643.
Christopher Manning. Mihai Surdeanu, John Bauer, Jenny Finkel. Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language pro- cessing toolkit. In Proceedings of 52nd Annual Meet- ing of the Association for Computational Linguis- lics: System Demonstrations, pages 55-60, Balti-
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.

---

Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. 2019. A theoretical analysis of contrastive unsu- pervised representation learning. In Proceedings of the 36th International Conference on Machine Learn- ing, volume 97 of Proceedings of Machine Learning Research, pages 5628-5637. PMIRR PML..
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. 2020. An empirical study on robustness to spuri- ous correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 8:621
L. G. Valiant, 1984. A theory of the learnable. Commun. ACM, 27(1 1) 1142.
Rico Sennrich. Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 86-96, Berlin, Germany, Association for Computational Lin- guistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018, GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Kendrick Shen, Robbie M Jones, Ananya Kurnr, Sang Michael Xie, Jeff Z. Haochen, Tengyu Ma, and Percy Liang. 2022. Connect, not collapse: Ex- plaining contrastive learning for unsupervised do- main adaptation. In Proceedings of the 39th Inter- national Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 19847-19878, PMLR.
Colin Wei, Sang Michael Xie, and Tengyu Ma. 2021. Why do pretrained language models help in down- stream tasks? an analysis of head and prompt tuning. In Advances in Neural Information Processing Sys- tems, volume 34, pages 16158-161777 Curran Asso- ciates, Inc.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. Masked language modeling and the distributional hy- pothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 2888-2913, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume (Long Pupers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642 Seattle, Washington, USA. Association for Computational Linguistics.
Yuhuai Wu, Felix Li, and Percy Liang. 2022. Insights into pre-training via simpler synthetic tasks. In Ad- vances in Neural Information Processing Systems.
Michael Tänzer, Sebastian Ruder, and Marek Rei. 2022. Memorisation versus generalisation in pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 7564-7578, Dublin, Ireland. Association for Computational Lin- guistics.
Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. 2020. Perturbed masking: Parameter-free probing for ana- lyzing and interpreting BERT. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4166-4176, Online, Asso- ciation for Computational Linguistics.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a.
BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 4593 4601, Florence, Italy, Association for Computational Linguistics.
Adams Wei Yu, David Dohan, Quoe Le, Thang Luong, Rui Zhao, and Kai Chen. 2018 Fast and accurate reading comprehension by combining self-attention and convolution. In International Conference on Learning Representations.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak. R Thomas McCoy, Najoung Kim, Ben- jamin Van Durme, Sam Bowman, Dipanjan Das, and Fillie Pavlick. 2019b. What do you learn from con- text? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations.
Tianyi Zhang and Tatsunori B. Hashimoto. 2021. On the inductive bias of masked language modeling: From statistical to syntactic dependencies. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5131-5146. Online. Association for Computational Linguistics.

---

Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021. When do you need bil- lions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 11 11 125, Online. Association for Computational Linguistics.
Ratio of training data
0.5
0.8
0.75
Accuracy
0.7
cles
6.0
0.55
(a) separated vocabulary
A Toy Experiments in the Multi-token Setting
Ratio of training data
0.5
0.75
A.1 Independent Experimental Variables
Accuracy
0.7
coes
Feature Levels. We can also control whether the semantic relationship in this pscudo language is at a single-token level or at a multi-token level, namely whether each feature corresponds to one or more tokens. This multi-token setting is interesting because the distribution of the token sequences will have higher-order dependence than Markov chain. Additionally, multi-token features also make this language more similar to natural languages. We decide the mapping between features and token sequences before we start generating pas- sages. We use two vocabulary sets Va and V3 for the feature sets 4a and 4,, respectively. In the single-token setting, each feature in 4, and 4b, is bijectively mapped to a token in Va and V3 respec- tively. In the multi-token setting, each feature in to 3 randomly selected
6.6
0.58
6.5
D-01 w/o OH CROW scratch
D1
W/
CBOW B-01 scretch
1--2 w/ Dil
A-02 w/o DiI A-02 CBOW
A-02 scratch
B-D2 w/ DM
8-02 weo DH von 1-02 CROW B--22 scratch
(b) shared vocabulary
Figure 5: The downstream performance of fine-tuning with 50% A-D1 and 50% B-D2 in the multi-token fea- ture setting.
indicate that MLM pretraining is not always able to infuse the inductive bias for the muti-token se- mantic relationship. As for the model initialized with CBOW embeddings, it barely outperforms the model trained from scratch. It shows that non- contextualized embeddings are not able to capture the semantic relationship at higher level.
L. and i corresponds tokens from Va and Vg respectively.
Vocabulary sharing between la and dy. When the features are multi-token, Va and Vig can be the same while keeping the mapping between features and token sequences bijective. Therefore, we can choose whether to share the vocabulary between Pa and 4,, The no-sharing setting analogous to the multilingual setting in the real world, while the vocabulary-shartee setting is more similar to the monolingual setting.
A.2.2 Generalization Capability
When P. and iG have separated vocabularies, we have similar observations as in the single-token- level setting. When all the fine-tuned data is in A-D1, the w/ DH model generalizes to B-D1 and B-D2, and the generalization also diminishes when there is more data. As for the performance on A- D2, the w/ DH and w/o DH models outperform training from scratch too, though the advantage is not as apparent as in the single-token setting.
A.2
Results on the Multi Token Setting
When 10% of the fine-tuning data is in B-D2, its generalization to B-D1 persists. However, w/ DH does not improve the generalization to B-D2 as much as in the single-token-level setting. Interest- ingly, the w/ DH model outperforms the w/o DH model for A-D2. It implies that the distributional property in Eq. helps the generalization to seman- tic shift, which is different from the observations in the other settings. It is possibly because the w/ DH model has better sample efficiency. As for CBOW, does not help generalization either, indicating that CBOW not able to capture the multi-token
A.2.1
Sample Efficiency.
For the multi-token-leeel setting, Eq. helps the sample efficiency only when the feature sets 4. and 4 have separated vocabulary. In Figure 5b, we can see the w/ DH model consistently outper- forms the w/o DH model when there is no vocabu- lary sharing. However, the improvement not as apparent in the single-token-level setting. When the feature sets T. and b have shared vocabu- lary, as shown in Figure 5a, the w/o DH model performs better than the w/ DH model. The results

---

Ratio of training data
0.2
0.4
0.6
0.8
1
0.9
Accuracy
0.8
0.7
0.6
(a) Fine-tuiing with 50% A-DI and 50% B-D2
Ratio of training data 0.4
0.2
0.6
0.8
1
0.9
Accuracy
0.8
0.7
0.6
0.5
tiee-tuning with 100% A-D1
Ratio of training data 0.4
0.2
0.6
0.8
1
0.9
Accuracy
0.8
0.7
0.6
0.5
A-D1 w/ DH
A-D1 w/o DH - A-D1 CBOW
- A-D1 scratch
B-D1 w/ DH
B-D1 w/o DH B-D1 CBOW
B-D1 scratch
A-D2 w/ DH
- A-D2 W/O DH
- A-D2 CBOW
- A-D2 scratch
B-D2 w/ DH
B-D2 w/o DH - B-D2 CBOW
- B-D2 scratch
Fine-tuning with 90%
B-D2
Figure 6: Performance on the synthetic downstream task described in $4.2 (enlarged plots).

---

Ratio of training data
Ratio training data
0.0
0.8
Acouracy
0.7
0.5
(a) Fine-tuned with 100% A-DI, separated vocahulary
(b) Fine-tuned with 100% A-DI, shared vocabulary
Ratio of training data
Ratio of training data
0,2
0.4
0.5
0.8
0.9
0.9
Accurecy
0.7
0.6
0.5
1-01
A-D1
A-Di
-MAA02
ino A-D2 scratch
(c) Fine-tuned with 90% A-DI and 10% B-D2, separated vo-(d) Fine-tuned with 90% A-DI and 10% B-D2, shared vocabu- cabulary
lary
Figure 7: Performance of the downstream task the multi-token feature setting.
semantic relationship.
not improve for 5 epochs consecutively. As for the labeling function, we use 5 patterns in total, and [S = S = Sy] = 12.
When 4, and 4,, share the vocabulary, we ob- serve mixed results. We find that when all the fine-tuning data is in A-D1, the w/ DH model out- performs the w/o DH model on B-D1 persistently. It implies that, in this setting, the distributional propery in Eq. helps the model generalize to vo- cabulary shift better than in the single-token setting. However, when 10% of the fine-tuning data is from B-D2, the advantage over the w/o DH model is less apparent. Additionally, regardless of whether use data in B-D2 fine-tune the model, Eq. 1 does not help the generalization to B-D2. MLM pretrained does not outperform the model trained from scratch on A-D2 either. implies that MLM pretraining does not help the generalization to se- mantic shift in this setting.
C Experiment Details for $6
We use the Trainer API in the Python package trans- formers v4.21.2 with the default hyper-parametent When doing back-translation, we use beam size equal to 5. Among the generated candidates whose score is not less than the score of the best candi- dates by more than 1, choose the one with the highest edit-distance to the original input.
B Experiment Details For $5
We use 6-layer transformer (Vaswani et al., 2017) with 12 heads. p()) and p(2) are fully-connected randomly generated Markov chains. Each node in the Markov chains has two outward edges point- ing to two random nodes (as in Figure 1). Each node in a chain has uniform probability to be the starting node. For the pretrainng corpora, ple 100k sequences, each of which contains 256 synsets. The dataset for the downstream task con- tains another 100k sequences, each of which con- tains 100 synsets. When fine-tuning the model, we early-stop when the validation performance does

---

Feature
Examples
Word
the movie understands like few others how the depth and breadth of emotional closeness (intimacy) give the physical act all of its meaning and most of its pleasure.
in an effort, suspect, not to spite (offend) by appearing either too serious or too lighthearted it spites by just being wishy.washy.
if you are an actor who can relate to the search for inner peace by dramatically depicting the lives of others onstage then esther 's story is a compelling pursuit (quest) for truth.
dazzles with its fully-written characters its determined stylishness ( (which always relates to characters and story ) and johnny dankworth 's best soundtrack in days (years).
the title not only draws (describes) its main characters but the lazy people behind the camera well.
Short phrase
if you are an actor who can relate to the search for inner peace by dramatically depicting the lives of other people (the lives of others) onstage then esther 's story is a compelling quest for truth.
the film is powerful, approachable (accessible) and funny.
the best film about baseball to hit theaters since "Field of Dreams" (field of dreams).
literate presentation that wonderfully weaves a homicidal event (a murderous event) 1873 with murderous rage 2002.
Long phrase
although laced with humor and few fanciful touches, the film is refreshingly earnest look young women (is refreshingly serious look at young women).
The three stakeholders of vera mollå, gil and bardem (vera 's three actors mollà gil and bardem -) excel in insightful, empathetic performances.
coughs and stutters at his own post-modern vanity. (coughs and sputters its own postmodern conceit.)
the old-world meets-new mesh is embodied in the soundtrack of the film, joyous effusion of disco-Bollywood that lifted my spirit out of the theatre at the end of the monsoon wedding (is incarnated in the movie 's soundtrack, joyful effusion of disco bollywood that, by the end of monsoon wedding, sent my spirit soaring out of the theater).
Table Examples of the noisy paraphrase pairs used in Section 6. The text bold face parts the noisy paraphrase (b) based on the original feature in the parentheses (a).

---

0.75
0.75
0.75
0.5
0.5
0.75
(a) word
(b) short phrase
(c) long phrase
0.75
0.25
0.25
0.25
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
(d) sentence, template meun
(e) sentence, template with
(i) task
Figure 8: Scatter plots of (D (()a)) for the MNLI experiment in $6, where the a-axis is Do and the y-axis is Dy.
0.25
0.25
0.25
0.75
0.75
0.5
0.75
(a) word
(h) short phrase
(c) long phrase
0.75
0.75
0.75
0.5
0.5
0.5
(d) sentence, template mean
(c) sentence, template with
(f) task
Figure 9: Scatter plots of (D D) for the SST-2 experiment in $6, where the z-axis is Dj, and the y-axis is Dy.


Reviews:
[{"review_id": "qzTuLyUCDK", "reviewer": "Reviewer_GgVT", "summary": "", "questions": "1. When replacing the words, how do you handle the case for ambiguous words? For example, the word “bass” can take the “musical instruments” or “a type of fish” meanings.\n2. For semantic shift, do you also consider the semantic change of a word over time?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your recognition of our contribution. Regarding your questions, please see below our answers:\n\n**Q1: Handle the case for ambiguous words**\n\nFor phrase-level and sentence-level features, because we use back-translation to generate their noisy paraphrases, we expect that the translation model is able to infer the correct word sense according to the context. As for word-level features, we do not take the multiplicity of word senses into consideration. However, through manually inspecting the samples, we do not find many ambiguous words in our noisy paraphrase/synonym pairs. We will include more examples in our later revision.\n\n**Q2: Consider the semantic change of a word over time for the semantic shifts**\n\nIn our toy experiment, the relationship between the synsets ($\\Sigma$) and features ($\\Phi_a$ and $\\Phi_b$) is static. If we see the synset of a feature as its semantic, we do not consider the semantic change of the words. However, we acknowledge that the semantic change of a word may cause the change of their occurrence distribution and thus cause a semantic distribution shift. This could be an interesting direction for future research!\n\nThank you again for your insightful questions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and easy to follow. It highlights the boundaries of our understanding regarding models' pretraining prompting further examination of the intricate interplay between semantic relationships, data distribution and model pretraining to better understand the generalization behaviour of pretrained models. ", "reasons_to_reject": "I do not have a specific reason to reject the paper. ", "questions_to_authors": "1. When replacing the words, how do you handle the case for ambiguous words? For example, the word “bass” can take the “musical instruments” or “a type of fish” meanings.\n2. For semantic shift, do you also consider the semantic change of a word over time?\n", "paper_topic_and_main_contributions": "The authors in this paper investigate whether mask language model pretraining allows models to use semantic knowledge extracted from the distribution of words in the pretraining data for downstream tasks. They show that prior knowledge about semantic equivalence alone is enough to result in better sample efficiency. Additionally, demonstrate the existence of a form of generalization capacity that is independent of the distributional property of the pretraining data. Furthermore, the distributional property of the pretraining data does not explain pretrained models'  generalization capability in the real world. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3quumzdMg5", "reviewer": "Reviewer_wjUa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer wjUa for their thoughtful comments. We agree with the reviewer that our work does not consider all semantic relationships. However, we would like to point out that\n\n1. Even though the Distributional Hypothesis has been widely accepted as an explanation for the efficacy of non-contextualized word embeddings (Levy and Goldberg, 2014) and as a plausible hypothesis for the efficacy of MLM pretraining (Sinha et al., 2021), the meaning of “the Distributional Hypothesis” in this context (i.e. what semantic relationships can be encoded in the data distribution in what way) and how it is helpful for pretraining (i.e. among the encoded semantic relationships what relationships are helpful and in what way) has never been clearly stated. One contribution of our work is that we make a first step to formalize the statement. It is the well-defined scope of our formulation that allows us to scientifically inspect the hypothesis. Our efforts may inspire future exploration of other semantic relationships.\n2. We theoretically and empirically analyze the benefit of modeling semantic equivalence. In particular, in Section 5, we show that semantic equivalence alone is enough to partially explain MLM pretrained models’ sample efficiency. \n\nTherefore, we believe that our work is beneficial for future researchers to better understand why MLM pretraining is beneficial.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper are as follows:\n\nNovel Perspective: The paper adopts a novel perspective by analyzing the masked language modeling (MLM) pretraining objective function through the lens of the Distributional Hypothesis. \n\nSynthetic Dataset Validation: By conducting experiments on a synthetic dataset, the paper offers a controlled environment to study the influence of distributional properties. This adds validity to the findings and helps establish causal relationships between distributional properties and sample efficiency.\n\n\n\n\n", "reasons_to_reject": "The weaknesses of this paper are as follows:\n\nLimited Scope: The paper explicitly states that it only considers a relatively simple case of pattern matching tasks, and semantic equivalence between features is the primary focus. This limited scope may restrict the generalizability of the findings to more complex NLP tasks or tasks involving different types of relationships between features.\n\nOmitted Relationships: The paper explicitly states that it does not consider other relationships, such as hypernyms or causal relationships, between features. However, these relationships could be relevant to understanding the efficiency of pretraining and generalization in certain NLP tasks.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper analyzes the masked language modeling (MLM) pretraining objective function from the perspective of the Distributional Hypothesis. It investigates whether the improved sample efficiency and generalization capability of pretrained MLM models can be attributed to the semantic similarity encoded in the distributional property of the pretraining data.\n\nThe main contributions of the paper are as follows:\n\nThe paper conducts an analysis using a synthetic dataset to explore the influence of the distributional property on the sample efficiency of pretrained MLM models. The results suggest that the distributional property does contribute to better sample efficiency.\n\nThe paper also extends the analysis to two real-world datasets to assess the impact of distributional property on the generalization ability of pretrained natural language models. The findings indicate that the distributional property alone does not fully explain the models' generalization capability.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oGBfnyFvl9", "reviewer": "Reviewer_ggfY", "summary": "", "questions": "Question A: This question is relevant to my 2nd reason to reject. As you mentioned in the limitations, the single-token context you considered in section 6.3 is quite limited. I wonder why not directly compare the contextualized embeddings of words/phrases/sentences from the pre-trained LM?\n\nQuestion B: In A.2, do you think the conclusion will change if you do span-based MLM as opposed to the standard token-based MLM?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank reviewer ggfY for their careful inspection of our work and the thoughtful comments. Below is our response:\n\n**Figure 2 is hard to read**\n\nWe apologize that we sacrifice the accessibility of figures a little bit to get more space. We will try to make the figures easier to read with one additional page if our paper is accepted, e.g. enlarge the figures, include the caption in every subfigure. If feasible, we will also include figures in the appendix and each of them will only include the curves required for a specific comparison purpose.\n\n**Compare the contextualized embeddings**\n\nThank you for this insightful question. We agree that comparing the contextualized embeddings is an intuitive way to infer the semantic equivalence learned by the model. However, compared with using the probability predicted by the model, the relationship between the contextualized embeddings and the data distribution is less clear. Because in this work we focus on understanding how modeling the data distribution is helpful, we prioritize using the probability predicted by the model in our experiment.\n\n**Using a span-based MLM in A.2**\n\nIt is a very interesting question. We think it is possible that the conclusion may change if we use a span-based MLM. If time permits, we will run a few experiments and include the result in our later version.\n\n**Typos Grammar Style And Presentation Improvements**\n\n- 1: We use Pearson correlation. We will state it in our later revision.\n- 2-4: We will fix it!\n- 5: You are correct. We will proofread this section again.\n\nThanks a lot for pointing out these issues!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The synthetic experiment is very well-designed and performed. It clearly shows how MLM pretraining affects the sample efficiency and generalization of fine-tuning and will provide inspiration for future work in this direction. I especially like the way they caption vocabulary shift and semantic shift which is analogous to cross-lingual transfer and spurious correlations in real settings. \n\n2. Although the results of the real-data experiment are less clear, the proposed method in this investigation is reasonable and can also inspire future work.\n\n3. Analyzing how pretraining helps downstream fine-tuning is an interesting and important topic to study.\n\n4. The paper is well-written and easy to follow.", "reasons_to_reject": "1. Figure 2 is hard to read. Lines are cluttered, and I have to refer to the caption constantly to find a line that I want to see. \n\n2. Compared to the synthetic experiment, the real-data experiment result is less clear and less informative. It is understandable that real-data experiment lacks control and thus are difficult to obtain clear conclusion from. Nonetheless, I think the experiments could have been done more thoroughly, e.g., try different ways of measuring the semantic distance from the pre-trained model.", "questions_to_authors": "Question A: This question is relevant to my 2nd reason to reject. As you mentioned in the limitations, the single-token context you considered in section 6.3 is quite limited. I wonder why not directly compare the contextualized embeddings of words/phrases/sentences from the pre-trained LM?\n\nQuestion B: In A.2, do you think the conclusion will change if you do span-based MLM as opposed to the standard token-based MLM?", "paper_topic_and_main_contributions": "This paper aims to analyze how the masked language model (MLM) pertaining helps downstream fine-tuning from the distributional hypothesis perspective. In particular, they analyze whether learning semantic equivalence leads to better sample efficiency and generalization capability.\n\nTo this end, they first design a well-controlled synthetic experiment. They show that the learned semantic equivalence during pretraining indeed improves sample efficiency during fine-tuning, but it does not fully explain the model's better generalization capability. Especially, it does not help when there is a semantic distribution shift. \n\nSecond, they extend the experiment to a real-data setting. They get the change of the fine-tuned and pre-trained models, respectively, when replacing a word/phrase/sentence/example with its paraphrase in the input.  Then they measure the correlation between these two changes. If they are highly correlated, then it means downstream finetuning directly makes use of the distributional knowledge learned in pretraining. However, they only observe low correlations. \n\nOverall, they show that the distributional hypothesis, in particular, semantic equivalence, can partially explain the benefit of MLM pretraining but not fully. \n\nThe contributions of this paper are made toward both computationally-aided linguistic analysis and NLP engineering experiment directions.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Please mention which correlation measure is used in section 6.4.\n\n2. In line 507, the footnote number should be at the end of the previous line.\n\n3. in line 491, MLNI -> MNLI\n\n4. in line 028, significalt -> significant\n\n5. in line 1119, do you mean \"when there is NO vocabulary sharing\"?"}}, {"review_id": "p2EzeZCLvu", "reviewer": "Reviewer_GJHm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments.We will try our best to improve clarity. Could you please let us know which specific parts are unclear and how we can possibly improve our work? We will appreciate it if you can help us make our work better.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work may be referable for readers in related line of research. ", "reasons_to_reject": "- The details of the synthetic data and the real tasks a not clear, which makes the analyses and figures difficult to grasp the meaning.\n- The overall conclusion of the work is somehow negative. So that the work still has spaces to be improved.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper provides investigations on the distributional property of pre-trained language models. Evidence on synthetic data and real tasks is provided.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main contribution of this paper is formal definition of the "distributional hypothesis" which allows the authors to set up a synthetic dataset on which to measure the impact of this phenomena on language models. They conclude that this definition of semantic equivalence explains partially the models' sample efficiency (but does not fully explain their generalization capability).
As one could expect, a set-up which aims to measure this on real data is less conclusive which might be explained due to a myriad of cofounding variables.

This type of contribution is compelling due to its originality, and this was mentioned by the reviewers. The main drawbacks is the limited conclusions that can be drawn from such a synthetic set-up, something that is expected from a first work in this direction.
Decision: Accept-Main