Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models
Authors: Zhengxuan Wu
Nelson F. Liu
Christopher Potts
Venue: ICLR 2022 Submitted (2022)
Keywords: transfer learning
pretrained language model

Abstract: There is growing evidence that pretrained language models improve task-specific fine-tuning even where the task examples are radically different from those seen in training. What is the nature of this surprising cross-domain transfer? We offer a partial answer via a systematic exploration of how much transfer occurs when models are denied any information about word identity via random scrambling. In four classification tasks and two sequence labeling tasks, we evaluate LSTMs using GloVe embeddings, BERT, and baseline models. Among these models, we find that only BERT shows high rates of transfer into our scrambled domains, and for classification but not sequence labeling tasks. Our analyses seek to explain why transfer succeeds for some tasks but not others, to isolate the separate contributions of pretraining versus fine-tuning, to show that the fine-tuning process is not merely learning to unscramble the scrambled inputs, and to quantify the role of word frequency. These findings help explain where and why cross-domain transfer occurs, which can guide future studies and practical fine-tuning efforts.

Under review as a conference paper at ICLR 2022
IDENTIFYING THE LIMITS OF CROSS-DOMAIN KNOWLEDGE TRANSFER FOR PRETRAINED MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
There is growing evidence that pretrained language models improve task-specific fine-tuning even where the task examples are radically different from those seen in training. What is the nature of this surprising cross-domain transfer? We offer a partial answer via systematic exploration of how much transfer occurs when models are denied any information about word identity via random scrambling. In four classification tasks and two sequence labeling tasks, we evaluate LSTMs using GloVe embeddings, BERT, and baseline models. Among these models, we find that only BERT shows high rates of transfer into our scrambled domains, and for classification but not sequence labeling tasks. Our analyses seek to explain why transfer succeeds for some tasks but not others, to isolate the separate con- tributions of pretraining versus fine-tuning, to show that the fine-tuning process is not merely learning to unscramble the scrambled inputs, and quantify the role of word frequency. These findings help explain where and why cross-domain transfer occurs, which guide future studies and practical fine-tuning efforts.
INTRODUCTION
Fine-tuning pretrained language models has proven to be highly effective across a wide range of NLP tasks; the leaderboards for standard benchmarks are currently dominated by models that adopt this general strategy (Rajpurkar et al.) [2016] 2018: Wang et al. 2018: Yang et al.) 2018 Wang et al.) 2019]. Recent work has extended these findings in even more surprising ways: Artetxe et al. (2020), Karthikeyan et al. (2019), and Tran (2020) find evidence of transfer between natural languages, and Papadimitriou Jurafsky (2020) show that pretraining language models on non-linguistic data such music and computer code can improve test performance natural language.
Why does pretraining help even across what appear to be fundamentally different domains, and what are the limits of such cross-domain transfer? In this work, we seek to inform these questions via a systematic exploration of how much cross-domain transfer we see when the model is denied any information about word identity. In this setting, we vary the pretraining and fine-tuning examples dramatically while holding other aspects of the task constant. This allows us to quantify the extent of transfer, and it can yield insights into the wide-ranging transfer results cited above.
Figure [ gives an overview of our core experimental paradigm: starting with two identical copies of a single pretrained model for English, we fine-tune one on English examples and the other on scrambled English sentences, using a scrambling function F' (section 3). and then we evaluate the resulting models. We apply this paradigm to four classification tasks and two sequence modeling tasks, and we evaluate bag-of of--ods-ords baselines, LSTMs with GloVe initialization and rich attention mechanisms, and BERT. Our central finding is that only BERT is able achieve robust cross-domain transfer, and for classification tasks but not sequence labeling ones.
To try to understand why such transfer successful for some tasks but not others, we pursue a number of hypotheses. First, we consider whether using scrambling function F that matches word frequencies is required for transfer, and we find that such matching plays a small role, but not enough to account for the observed performance (section 7.I) Second, we assess whether frequency matching might actually be inserting semantic consistency into the scrambling process by. for example, systematically creating substitution pairs like goodlgreat and professor/teacher (section (.22). However, we find no evidence of such semantic consistency. Third, we try to isolate the contribution of pretraining versus fine-tuning by fine-tuning randomly initialized models of different

---

Under review as a conference paper at ICLR 2022
Model
Copy
Model
Model
Fine- Fine-turiin tan-ttnnnng
Dataset
Scrambling fumetion
Scrambled
Tin--umine
Dataset
(truin)
(train)
Fine-tuned
Fine-tuned
Model
Modelg
Evaluation
Evaluation
Serambling
Scrambled (test)
Dataset (test)
Figure 1: An overview of our experiment paradigm. Starting with model (e.g., pretrained BERT, GloVe-inttillized LSTM, etc.), we copy it and fine-tune it on the regular and scrambled train set using a scrambling function F. The model is then evaluated regular and scrambled test sets. Our paper explores different options for F and a number of variants of our models to try to quantity the amount of transfer and identify its sources.
sizes (section 7.3 and by freezing the BERT parameters, such that only task-sp parameters updated (section 7.4). These variations lead to a substantial drop in transfer, suggesting that fine- tuning is vital, although our LSTM results show that the BERT pretrained starting point is also an essential component. Fourth, we ask whether the fine-tuning process is primarily learning to reassociate scrambled words with their sources, and find that is not (section 7.5). While these findings do not fully account for the transfer we observe, they offer a partial explanation which should help guide future studies of this issue and which can help with practical fine-tuning work.
RELATED WORK
2.1 EVIDENCE FOR TRANSFER
Transferability across domains is often used to benchmark large pretrained models such as BERT (Devlin et al. (2019)). RoBERTa (Liu et al. 2019b), ELECTRA (Clark et al. 2019), and XL- Net (Yang et al. (019). To assess transfcrability, pretrained models are fine-tuned for diverse down- stream tasks Wang et al. 2018 2019). Recently, pretrained Transformer-based models (Vaswani et al. 2017) have even surpassed estimates human performance on GLUE (Wang et al.) (018) and SuperGLUE (Wang et al. 2019). While the benefits of pretraining reduced when there is a large train set (Hernandez et al. 202T), there is little doubt that this pretraining process helps in many scenarios.
2.2 STUDIES OF WHY TRANSFER HAPPENS
There are diverse efforts underway to more deeply understand why transfer occurs. Probing tests of- ten involve fitting supervised models on internal representations in an effort to determine what they encode. Such work suggests that BERT representations encode non-trivial information about mor- phosyntax and semantics (Tenney et al. 2019 Liu et al. 2019a] Hewitt & Manning, 2019 Manning ct al. 2020) and perhaps weakly encode world knowledge such as relations between entities (Da & Kasai 2019 Petroni al. 2019), but that they contain relatively little about pragmatics or role- based event kndwledge (Ettinger 2020). Newer feature attribution methods (Zeiler & Fergus 2014 Springenberg al. 2015 (0risShrikumaa et al. 2017 Binder al.] 2016 Sundararajan et al. (2017) and intervention methods (McCoy et al. 2019 Vig et al.] 2020; Geiger et al.) (020)) are corroborating these findings while also yielding a picture of the internal causal dynamics of these models.
2

---

Under review as a conference paper at ICLR 2022
Scrambling Method
Sentence
Original English (No Scrambling) Similar Frequency Random
"the worst titles in recent cinematic history" engaging semi is everyone dull dark" "kitsch theatrically tranquil andys loaf shorty lauper"
Table 1: An example from the SST-3 dataset and its two scrambled variants.
Another set of strategies for understanding transfer involves modifying network inputs or internal representations and studying the effects of such changes task performance. For instance, Tamkin
et al. (2000) show that BERT's performance on downstream GLUE tasks suffers only marginally some layers
reinitialized before fine-tuning, and Gauthier & Levy (2019), [Zanzotto cet al. 22200), Pham et al. (2020), and Sinha et al. (2021) show that BERT-like models are largely insensitive to word order changes.
2.3 EXTREME CROSS- PROSS-DOMAIN TRANSFER
Cross-domain transfer is not limited to monolingual cases (Karthikeyan et al. 2019). With modifica- tions to tokenizer,
BERT improves performance on downstream multilingual NLU tasks (Artetxe et al. 2020; Tran 2020). Papadimitriou & Jurafsky (2020) show that pretrain- ing language models on structured non- non-lingum longlinguistic data (e.g., MIDI music or Java code) improves test performance natural language. Our work complements and advances these efforts along two dimensions. First, we challenge models with extremely ambitious cross-domain settings and find that BERT shows high degree of transfer, and conduct a large set of follow-up experiments to help identify the sources and limitations of such transfer.
EXPERIMENTAL PARADIGM
We now describe the evaluation paradigm summarized in figure [section] 3.1 the scrambling functions F' that we consider (sections 3.2
with special attention
3.1 EVALUATION PIPELINE
Figure Dshwss shows our main evaluation paradigm for testing the transferability of a model without word identity information. On the left side, we show the classic fine-tuning pipeline (i.e., we fine-tune on the original English training and evaluate the original English test set). On the right side, show our new evaluation pipeline: starting from a single model, we (1) fine-tune it with a corrupted training split where regular English word identities are removed and then (2) evaluate the model on a version of the evaluation set that is corrupted in the same manner. The paradigm applies equally to models without any pretraining and with varying degrees of pretraining for their model parameters.
3.2 SCRAMBLING WITH SIMILAR FREQUENCY
To remove word identities, we scrambled each sentence in each dataset by substituting each word with a new word w/ in the vocabulary of the dataset. For Scrambling with Similar Frequency, we use the following rules:
1. w and w' must have the same sub-token length according to the BERT tokenizer; and 2. w and w' must have similar frequency.
The first rule is motivated by the concern that sub-token length may correlate with word frequency, given that rarer and longer words may be tokenized into longer sub-tokens. The second rule is the core of the procedure. The guiding idea is that word frequency is often reflected learned embeddings (Gong al., 2018). so this scrambling procedure might preserve useful information and thus help to identify the source of transfer. Table an example, and Appendix Ceanndii(pppovids. details about the matching algorithm and additional examples of scrambled sentences.
3

---

Under review as a conference paper at ICLR 2022
Standard Madels (Train and Test on English)
Scrambled Models (Train and Test on Serambledd English)
Dataset
BERT-Scrambled Similar Frequency
LSTN-Scrambled Similar Frequency
BERT
LSTM
Bow
Dunny
Random
Random
SST-3
(.02)
(.01)
(.00)
.33 (.02)
.65 (.01)
.64 (,02)
.57 (.02)
.56 (.02)
SNLI
91 (.02)
.78 (.02)
66 (.02)
.33 (.01)
.84 (.0))
82 (,02)
.72 (.00)
.71 (.01)
QNLI
91 (.02)
.68 (.02)
.2 (.0)
(01)
.82 (.01)
.79 (.02)
.62 (.01)
.61 (.01)
MRPC
(.01)
.72 (.02)
(.02)
.50 (.02)
.82 (.02)
.78 (.02)
.69 (.00)
.68 (.00)
EN-EWT
97 (.0))
88 (.02)
66 (.01)
.09 (.01)
.66 (.01)
81 (02)
(.01)
(.1))
CoNLL-20033
95 (.01)
.75 (.01)
.28 (.02)
.02 (.01)
.74 (.1))
(.02))
.61 (.02)
.56 (.01)
Table 2: Model performance results for models trained on original English and on scrambled En- glish. Standard deviations are reported for all entries.
3.3 RANDOM SCRAMBLING
To better understand the role of frequency domain transfer, we also consider a word scrambling method that does not seek to match word frequencies. For this, simply shuffle the vocabulary and match each word with another random word in the vocabulary without replacement. We include the distributions of the difference in frequency for every matched word pair in Appendix to to make sure a word is paired with a new word with drastically different frequency in the dataset. We also tried pair words by the reverse order of frequencies, which yielded similar results, we report only random scrambling results here.
4 MODELS
In this section, describe the models evaluated within paradigm. Appendix additional details about how the models were designed.
B provides
BERT For our BERT model (Devlin al., 2019a), we import weights from the pretrained BERT- base model through the HuggingFace transformert library (Wolf et al.) (020). For sequence clas- sification tasks, we append a classification head after the [CLS] token embedding in the last layer of the BERT model. If an imput example contains pair of sentences, we concatenate them using a [SEP] token in between. For sequence labeling tasks, we append a shared classification head to each token embedding in the last layer of the BERT model.
LSTM We contextualize our results against a strong L.ST--based model (Hochreiter & Schmidhu- ber 1997). We lower-case each input sentence and tokenize by separating on spaces and punc- luation. We then use 300-dimensional GloVe embeddings (Pennington al. 2014 as inputs to single-layer recurrent neural network with LSTM cells, with a hidden size 64. We use dot-product attention (Luong al. 2015) to formulate context vector for each sentence. Finally, context vector through a multilayer perceptron (MLP) layer to get the final prediction. For an input example with pair of sentences, we concatenate two sentences together before feeding them into our LSTM encoder. For sequence labeling tasks, we directly feed the hidden state at each position to the MI.P layer get the final prediction.
pass the
Bag-of- Words (BoW) Model We compare against BoW classifier, which serves as proxy of model performance when only given word co-occurrence information. For each sentence in a dataset, first formulate BoW vector that uses unigram representations of an input sentence. Then, we feed the BoW vector through a softmax classifier. For examples with a pair of sentences, we create two BoW vectors for each sentence, and concatenate them together before feeding them into the linear layer for predicting labels. For sequence labeling tasks, we use Conditional Random Fields models (CRFs: Lafferty et 200I) with character-level unigram BoW features.
We include random classifier that generates predictions randomly proportional to the class distribution of the training set. We use this model to further contextualize our results.
Dummy Model
We use the Common Crawl cased version: http:///lb..ttaffrrrdde//aaaaass 8488
4

---

Under review as a conference paper at ICLR 2022
Dataset
Type
#Train
#Dev
#Test
#Class
SST-3
Sequence Classification
159k
1.1k
2.2k
SNLI
Sequence Classification
550k
10k
3
QNLI
Sequence Classification
108k
5.7k
5.7k
2
MRPC
Sequence Classification
3.7k
408
1.7k
2
EN-EWT UPOS
Sequence Labeling
14k
2k
3.5k
18
CoNLL-2003 NER
Sequence Labeling
12.5k
2k
2.1k
9
Table 3: Summary information for each task.
5 TASKS
We consider six sequence classification and sequence labeling tasks (Table
Sequence Classification We select four NLU datasets for sequence classification. We consider sentiment analysis (SST-3; Socher et al. 2013), where SST-3 is variant of the Stanford Sentiment Treebank with positive/negative/neuraa labels; we train on the phrase- and sentence-level sequences in the dataset and evaluate only its sentence-level labels. Additionally,
include natural lan- guage inference (QNLI; Demszky et al.] 2018 and SNLI; Bowman et al., (0155) and paraphrase (MRPC: Dolan & Brockett 2005). QNLI is derived from version of Stanford Question Answering Dataset. For sequence classification tasks, we use Macro-F1 scores for SST-3, and accuracy scores for other NLU tasks.
Sequence Labeling In contrast sequence classification, where the classifier only considers the [CLS] token of the last layer and predicts single label for sentence, sequence labeling requires the model to classify all tokens using their contextualized representations. We select two datasets covering distinct tasks: part-of-speech detection (POS) and named entity recognition (NER). We used Universal Dependencies English Web Treebank (EN-EWT) (Silveira et al. 2014) for POS, and CoNLL-2003 (Tjong Kim Sang & De Meulder (003) for NER. For sequence labeling tasks, used Micro-FI (i.e., accuracy with full labels) for POS and Fl scores for NER.
RESULTS
In this section, we analyze the fine-tuning performance of BERT on scrambled datasets. Table 2 shows performance results. We focus for now the results for Scrambling with Similar Frequency. Additionally, we also include baseline models trained with original sentences for comparison pur- poses. When training models each task, select models based on performance during fine-tuning. We average performance results with multiple random seeds to get stabilized results. See Appendix for additional details on our training and evaluation procedures.
the dev split
6.1 SEQUENCE CLASSIFICATION
Comparing the second column (BERT model that is trained and tested on English) with the sixth col- umn (BERT model that trained and tested on Scrambled English with Similar Frequency Scram- bling) in Table 2 we see that BERT maintains strong performance for all sequence classification tasks even when the datasets are scrambled. More importantly, we find that BERT fine-tuned with scrambled dataset performs significantly better than the LSTM model (with GloVe embeddings) trained and evaluated on standard English data
For example, on the MRPC task. BERT evaluated with scrambled data experiences less than 5% performance drop, and shows significantly better performance (a 13.9% improvement) than the best LSTM model. BERT evaluated with scrambled QNLI experiences the biggest drop (a 9.89% de- crease). However, this still surpasses the best LSTM performance by a large margin (a 20.6% improvement).
also presents performance results for other baseline models, which can be used to assess the intrinsic difficulty of each task. Our results suggest that BERT models fine-tuned with scrambled
5

---

Under review as a conference paper at ICLR 2022
LSTN-Scrambled Similar Frequency
0.0
Dataset
LSTN-Baseline
GloVe
GloVe
-10.0%
SST-3
.62 (.1)
.57 (02) .58 (.01)
Score
0%
SNLI
.78 (.02)
(.00)
(.00)
0.5
116.1%
-19.4%
QNLI
.68 (.02
(01)
.61 (.1)
0.4
-50 0%
-30 00.07
MRPC
(.0))
(.00)
(.00)
0.3-
-46.7%
455.0%
EN-EWT
85 (,02)
(01)
.99 (01))
-5.7%
75 (.01)
(.02)
(.01)
Original
Similar Freq.
Random
Dummy Model
+ SST-3
SNLI
QNLI
MRPC
Table 4: Performance results for LSTM models trained on regular English and on English with Scrambling with Similar Fre- quency, with GloVe embeddings and with randomly initialized embeddings.
Zero-shot evaluation with the Bag-of- Word (BoW) model on scrambled datasets and the dummy model. Numbers are the differences between the cur- rent points and the first points percentages.
Figure 2:
tasks remain very strong across the board, and they remain stronger than best LSTM baseline models (those trained and tested on regular English) in all the classification tasks.
The overall performance of the LSTM models is worth further attention. The LSTMs are far less successful at our tasks than the BERT models. However, it seems noteworthy that scrambling does not lead to catastrophic failure for these models. Rather, they maintain approximately the same performance in the scrambled and unscrambled conditions. This might seem first like evidence of some degree of transfer. However, as we discuss in section 7.3 the more likely explanation that the I.STM is simply being retrained more or less from scratch in the conditions.
6.2 SEQUENCE LABELING
For more complex setting, we fine-tuned BERT with sequence labeling tasks, and evaluated its transferability without word identities (i.e., using datasets that are scrambled in the same way in our sequence classification tasks). The second set (bottom set) of Table shows performance results for sequence labeling tasks where the goal of the BERT model is to classify every token correctly. As shown in Table BERT experiences a significant drop when evaluated with a scrambled dataset for sequence labeling task. For LSTMs trained with scrambled sequence labeling tasks, also observe bigger drops compared with sequence classification tasks. For CONLL-2003,, LSTM with GloVe embeddings drops (a 18.7% decrease) from its baseline counterpart. Our results suggest that transfer learning without word identities is much harder for sequence labeling tasks. One intuition is that sequence labeling tasks are more likely rely on word identities given the fact that classification (i.e., labeling) is at the token-level.
ANALYSIS
7.1 FREQUENCY EFFECTS
Preserving word frequencies during scrambling may lead to higher performance when training and evaluating on scrambled datasets. To assess how much of the observed transfer relates to this fac- tor, can compare Scrambling with Similar Frequency (SSF) with Random Scrambling (RS), as described in section 3 As shown in Table performance drops slightly if we use RS, For sequence classification tasks, RS experiences 1-5% drops in performance compared with SSF. For sequence labeling tasks, the difference is slightly larger: about 2-6%. This suggests that word frequency is indeed one of the factors that affects transferability, though the differences are relatively small, dicating that this is not the only contributing factor. This is consistent with similar findings due to Karthikeyan et al. 2019 for multilingual BERT.
7.2 DOES SCRAMBLING PRESERVE MEANING?
Another explanation is that our scrambling methods tend to swap words that are predictive of the same labels. For example, when substituting words with similar frequencies in SST-3,

---

Under review as a conference paper at ICLR 2022
SST-3
SNLI (Sequence Classification)
QNLI
(Sequence Classification)
(Sequence Classification)
0.75
0.30
0.-
0.65
13,1% 11.1%
-0.3%
9. 0.
8.4%
0.7
08-
77.0% 12.3% 3%.12.35
0.55
1.77%
-10
0.75
0.
0.0%
Score
MRPC (Sequence Classification)
EN-EWT (POS) (Sequence Labeling)
CoNLL-2003 (NER)
(Sequence Labeling)
1.00
1.0
0.35
0.0
0.75
0.80
0.0%
1.3%
-2.0%
22.5
4.3%
4.9%
0.50
13
17.4%%
14 14.1%
0.8
0.7
30 on 31.0% 28.8% 20--000000 20.0% 20.
0.70
0.6
.3.9%
45..4%
03
10
10
10
Number of Transformer Layers
Train and test on English
Train and test Scrambled twits peetraining)
Train and test on Scrambled (without pretraining)
Figure 3: Performance results when fine-tum tne-tuning end-to-end for different number of Transformer lay- ers. Annotated numbers are the differences between the red lines and the green lines in percentages. Scoring for each task is defined in Section 5
"good" may be swapped with "great" since they may have similar frequencies in a sentiment anal- ysis dataset. To rule this out, we conducted zero-shot evaluation experiments with our BoW model on sequence classification tasks. The rationale here is that, to the extent that our swapping preserved the underlying connection between features and class labels, this should show up directly in the per- formance of the BoW model. For example, just swapping of "good" for "great" would hardly affect the final scores for each class. If there are great many such invariances, then it would explain the apparent transfer.
Figure 2 shows the zero-shot evaluation results of our BoW model on all sequence classification datasets. Our results suggest that both scrambling methods result in significant performance drops, which suggests that word identities are indeed destroyed by our procedure, which again shines the spotlight on BERT as the only model in our experiments to find and take advantage of transferable information.
7.3 TRANSFER OR SIMPLE RETRAINING?
Our results on classification tasks show that English-pretraie BERT can achieve high performance when fine-tuned and evaluated on scrambled data. this high performance uniquely enabled by transfer from BERT's pretrained representations, or is BERT simply re-learning the token identities from its scrambled fine-tuning data?
To distinguish between these two hypotheses, we first examine whether models can also achieve high performance when fine-tuned and evaluated on scrambled data. We study models of varying capacity by modulating the number of BERT Transformer blocks. See Appendix B for details about the training procedure for these andoollyinitialiinn models.
BERT
We compare these varying-depth English. To modulate the capacity of these pretrained models, we progressively discard the later Transformer layers (i.e.. we make predictions from intermediate layers). Comparing these models is a step toward disentangling the performance gains of pretraining from the performance gains relating to model capacity.
aaddoml--iitializee models against BERT models pretrained on
Figure 3 summarizes these experiments. The red line represents our fine-tuning results, across dif- ferent model sizes. The shaded area represents the performance gain from pretraining when training and testing on scrambled data. Pretraining yields consistent gains across models of differing depths, with deeper models seeing greater gains.
For sequence labeling tasks, the patterns are drastically different: the areas between the two lines are small. Since the random-ing aandom-nitialized and pretrained models achieve similar performance when fine-

---

Under review as a conference paper at ICLR 2022
SST-3
QNLI
MRPC
(Sequence Classification)
(Sequence Classification)
(Sequence Classification)
0.85
0.
0.80
Score
0.8
0.7
08-
0.34
0.68
0.72
0.7
0.60
0.75
0.00
0.2
0.33
0.34
0.53
01.
0.69
10
12
10
Number of Transformer Layers
+ Train and test Scrambled (fine tune end-to end)
Train and test Scrambled (fme-tune classsier only)
Figure 4: Performance results when fine-tuning only the classifier head by freezing all proceeding layers in BERT (red line) vs. fine-tuning end-to-end, which includes the classifier head and all pro- ceeding layers in BERT (green line). Numbers are scores for the red lines. Scoring for each task is defined in Section 5
tuned and tested scrambled data, pretraining is not beneficial. This suggests that BERT hardly transfers knowledge when fine-tuned for sequence labeling with scrambled data.
Table 4 shows results when training LSTMs without any pretrained embeddings. Unlike with BERT. Vl initialization (a pretraining step) hardly impacts model performance across all tasks. Our leading hypothesis here is that the LSTMs may actually relearn all weights without taking advantage of pretraining. All of our LSTM models have parameter sizes around 1M, whereas the smallest BERT model (i.e., with a single Transformer layer) around 3.2M parameters. Larger models may be able to rely more on pretraining.
Overall, these results show that we do see transfer of knowledge, least for classification tasks, but that there is variation between tasks on how much transfer actually happens.
7.4 ASSESSING TRANSFER WITH FROZEN BERT PARAMETERS
We can further distinguish the contributions of pretraining versus fine-tuning by freezing the BERT parameters and seeing what effect this has on cross-domain transfer. |Ethayarajh (019) provides evidence that early layers are better than later ones for classifier fine-tuning, so we explore the effects of this freezing for all the layers in our BERT model.
As shown in Figure 4 performance scores drop significantly if we only fine-tune the classifier head and freeze the rest of the layers in BERT across three of our tasks. However, we find that per- formance scores change significantly depending on which layer we append the classifier head to. Consistent with Ethayarajh|s findings, contextualized embeddings predictive. For example, if we freeze BERT weights and use the contextualized embeddings from the 2nd layer for SST-3, the model reaches peak performance compared with contextualized embed- dings from other layers. More importantly, the trend of the green line follows the red line in Figure especially for SST-3 and QNLI. The only exception is MRPC, where the red line plateaus but the green line keeps increasing. This could be artifact of the size of the dataset, since MRPC only contains around 3.7K training examples. Our results suggest that pretrained weights in successive self-attention layers provide a good initial point for the fine-tuning process.
lower layers tend to be more
7.5 PROBING FOR WORD IDENTITY REASSOCIATIONS
We further investigate the learning dynamics of our fine-tuned models. Specifically, we study whether our fine-tuned models reassociate word identities with tokens for our sequence classifi- cation tasks. To do this, measure the cosine similarities between words and their scrambled counterparts before and after the fine-tuning process P To the extent that these similarities are in- creased after fine-tuning, their scrambled counterparts. However, we find essentially no evidence for such changes. As shown in figure 5 the correlation distributions before fine-tuning and after are extremely similar. This suggests that our fine-tuned models rarely reassociate word identities in the embedding layer.
have evidence that fine-tuning has learned to ressociate words with
We only consider shared words in the model vocabulary and our scrambling maps, which includes 30% of words in the model vocabulary.
8

---

Under review as a conference paper at ICLR 2022
S (Sequence Classification)
SNLI (Sequence Classification)
QNLI (Sequence Classification)
MRPC
(Sequence Classification)
Frequency
Pine tuned
Pine-tuned
60
2.
64
64
-82
02
-0.2
6.0
-12
Correlation
Figure 5: Correlations between cosine similarities of word embeddings before fine-tuning v.s. fine-
tuning with scrambled datasets. Measurements of correlations are defined in Section 7.5
SST-3 (Sequence Classification)
SNLI
QNLI (Seuuence Classification)
MRPC (Sequence Classificationn
SSeuence Classification)
Selectivity
1.0%
1.0%
10
12
Number of Transformer Layers
Figure 6: Accuracy of word identity probes when applied to hidden states of each layer comparing to the control task introduced by Hewitt & Liang (2019). Measurements of accuracies are defined in Section 7.5
To push this analysis a step further, we probe whether word identities are recovered through trans- former layers by adapting the probing method with control task from Hewitt & Liang (2019). For- mally, we use an MILP classifier to predict the word identity for w using the contextualized hidden representations of its scrambled counterpart w'. For our control task, ask the probe to predict random word identities. The difference in performance between these two conditions is know as selectivity, and it estimates the degree to which the word identities are recoverable, taking the power of the probe model into account. As shown in figure our results suggest that relatively little infor- mation about the scrambling map latent in these representations, across tasks and model layers.
8 CONCLUSION
In this paper, we propose an evaluation pipeline for pretrained models by testing their transferability without word identity information. Specifically, we take an English pretrained BERT off-the-shelf and fine-tune with a scrambled English dataset. We conduct analyses across six tasks covering both classification and sequence labeling. By evaluating performance against multiple baselines, we aim to assess where BERT can transfer knowledge even without word identities. We find considerable transfer for BERT compared to even powerful baselines, by only for classification tasks.
What the source successful cross-domain transfer with BERT? We find that word frequency contributes, but only to limited extent: serambling with matched word frequencies consistently outperforms scrambling with unmatched word frequencies, but transfer still occurs robustly even with random scrambling. We are also able determine that both pretraining and fine-tuning are im- portant and interacting factors in this transfer; freezing BERT weights during task-specific training leads to much less transfer, but too much task-specific training erodes the benefits of pretraining and in turn reduces the amount of transfer observed.
These analyses begin to piece together a full account of these surprising transfer results for BERT, but they do not fully explain our experimental results. Recent literature suggests at least two new promising avenues to explore. First, Sinha et al. (202T) seek help characterize the rich distribu- tional prior that models like BERT may be learning, which suggests that higher-order notions of frequency play significant role transfer. Second, the findings of [Ethaaraahh (2019) may be in- structive: through successful layers, BERT seems to perform specific kinds of dimensionality reduc- tion that help with low- dimensiiiional classification tasks. Our results concerning layer-wise variation consistent with this. And there may be other paths forward. The more can learn about the extent of cross-domain transfer, the more effectively we can train and fine-tune these models on challenging tasks.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Mikel Artetxe, Sehastian Ruder, and Dani Yogatama. On the cross-lingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pp. 4623-4637, 2020.
Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Glaus-Robert Müller, and Wojciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks, pp. 63-71. Springer, 2016.
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pp. 632-642, 2015.
Kevin Clark, Minh- Minh-Thang Luong, Quoe V Le, and Christopher D Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2019.
Jeff Da and Jungo Kasai. Cracking the contextual commonsense code: Understanding commonsense
reasoning aptitude of deep contextual representations. EMNLP 2019, pp. 1, 2019.
Dorottya Demszky, Kelvin Guu, and Percy Liang. Transforming question answering datasets into natural language inference datasets, 2018.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies. Volume (Long and Short Papers), pp. 4171 4171-4111 1-116,,,, Minneapolis, Minnesota, June 2019a. Association for Computational Linguistics. doi: 11118653/1/N11..1 1423. URL https: 7/www. www.llwwbborr/attoooog/////////e
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, 2019b.
William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop Paraphrasing (/WP2005), 2005.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geom- ctry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),, pp. 55-65, 2019.
Allyson Ettinger. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguisties, 4:3448. 2020.
Jon Gauthier and Roger Levy. Linking artificial and human neural representations of language. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pp- 529- 539, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 10./5////D11.... 19D 1050. URL
Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference mod- els partially embed theories of lexical entailment and negation. Proceedings of the Third BlackboxNLP Workshop Analyzing and Interpreting Neural Networks for NLP, pp. 163-173, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2000 blackboxnlp- 1.16. Utthhttss://ww..a..eeb org http://////ooog/////oaaaaooooao. www.kbbbbbbxolp11166666
Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Yan Liu. Frage: Frequency- agnostic word representation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URL @555ebbecce44667F9bbbbe 3 11117766665500-P-P00
www.eeddiiinnnniiisssn.omomm www. http:///ceeeiissseerrrrcc//////p
10

---

Under review as a conference paper at ICLR 2022
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021.
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2733-2743, 2019.
John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pa- pers), pp. 4129-4138, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, 1997.
K Karthikeyan, Zihan Wang, Stephen Mayhew, and Dan Roth. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, 2014.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning. ICML '01, pp. 282-289, San Francisco, CA. USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558607781.
Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 1073 1094, 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692, 2019b.
Minh-Thiee Thang Luong, Hieu Pham, and Christopher D Mamning. Effective approaches to attention- based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421, 2015.
Christopher Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117448:330466-3005 2020.
R. Thomas McCoy, Tal Linzen, Ewan Dunbar, and Paul Smolensky. RNNs implicitly implement tensor product representations. In In Proceedings of the 7th International Conference Learning Representations, New Orleans, USA, May 2019.
Isabel Papadimitriou and Dan Jurafsky. Learning music helps you read: Using transfer to study
linguistic structure in language models. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6829-6839, 2020.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 1532-1543, 2014.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJNNLP), pp. 2463-2475 2019.
Thang M Pham, Trung Bui, Long Mai, and Anh Nguyen. Out of order: How important is the sequential order of words in a sentence in natural language understanding tasks? arXiv preprint arXiv:2012. 15180, 2020.
11

---

Under review as a conference paper at ICLR 2022
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383 Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 10533////DD11611 1244. httpst//::///
aclweb. org/
anthol.ogy/D16-1264
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics (Volume 2: Short Papers). pp. 784-789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 11116///1//18///1 1/P18-2124, anthology/P18-2124
URL https: /wwww/ acl org/
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3145-3153, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL httt:://proceeee///// http:///oocceciinnn.mmrrrrrsrrrs ptwss//70/shrikumeeee 17..
Natalia Silveira, Timothy Dozat, Mathe-Catherine de Marneffe, Samuel Bowman, Miriam Con- nor, John Bauer, and Christopher D. Manning. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), 2014.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. Sit.:20040066444, 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, 2013.
J Springenberg, Alexey Dosovitskiy, Thomas Brox, and M Riedmiller. Striving for simplicity: The all convolutional net. In ICLR (workshop track), 2015.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In
Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3319-3328, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http:// proceedings. mlr S
html
Alex Tamkin, Trisha Singh, Davide Giovanardi, and Noah Goodman. Investigating transferability in pretrained language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1393-1401, Online, November 2020. Association for Computational Linguis- tics. doi: 10. 10.1880 001118661112200 htt333//11222..mnniii hwt..//....iiinnnemmmp.....mm 125.. URL httph:://////t htt.://wwwwwwl..l..coc. acwweb. org/ www./oogg/o.aiaeoe anthology/ 2020. findings-emnipp
125)
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 459934601) 2019.
Erik Tjong Kim Sang and Fien De Meulder. Introduction to the CONLL-20003 shared task: Lunguage-indeepedeen named entity recognition. In Proceedings of the Seventh Conference Natural Language Learning at HLT-NAACL 2003, pp. 142-147, 2003. URL hnttps://www. wwwpee..rg//anoooog/////-------
Ke Tran. From english to foreign languages: Transferring pre-trained language models. arXiv preprint arXiv:2002.07306, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural In- formation Processing Systems, pp. 5998-6008, 2017. URL /http://papers. www..cccssp.iioee ct/ ccper/ 7181 71Bi-attentiior-i--al---yo
12

---

Under review as a conference paper at ICLR 2022
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias, 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill. Omer Levy, and Samuel Bowman, Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP, pp. 353- 305,555 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose lan- guage understanding systems. In H. Wallach, H. Larochelle. A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http:: http::/::////proeeeggsess www.eeeedinnn.neeiiiiiie httprrrrppppccppppp////////ieee
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick VOn Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Syslem Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653//1/2020.coo www.l-demosls.......mmmmmm aclweb. wwtgg/anthllogy/20000 eew.://eoommmmddmms..m...
URL httpss:/www.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question an- swering. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Pro- cessing, pp. 2369-2380, Brussels, Belgium, October-November 2018. Association for Computa- tional Linguistics. doi: 10.18653/v1/D18- 1259.. URL https: //ww: hwtp://wwwwwwwww.bbo////b www.eb.www...................... wrg.:/www..rrr///eeeggeeee org harggg/tttog///// D18-1259
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding, In Advances in neural information processing systems, pp. 5753-5763, 2019.
Fabio Massimo Zanzotto, Andrea Santilli, Leonardo Ranaldi, Dario Onorati, Pierfrancesco Tom- masino, and Francesca Fallucchi. Kermit: Complementing transformer architectures with en- coders of explicit syntactic interpretations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.
2020.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelarrs (eds.), Computer Vision ECCV 2014, pp. 818-833, Cham, 2014. Springer International Publishing.
APPENDIX FOR 'IDENTIFYING THE LIMITS OF TRANSFER FOR PRETRAINED MODELS'
KNOWLEDGE
A DATASETS
Table in our main text shows statisties for the six datasets included in our experiments. We use the Dataset interface provided by the Hugging Face library Wolf et al. (2020) to foster reproducibility. For each scrambling test, we use the same splits as the original datasets.
B MODEL AND TRAINING SETUP
BERT Model Our BERT model has 12 heads and 12 layers, with hidden layer size 768. The model uses the WordPiece tokenizer, with maximum sequence length of 128. We fine-tune our model
13

---

Under review as a conference paper at ICLR 2022
with a dropout probability of 0.1 for both attention weights and hidden states. We employ early stopping with patience of 5. This ensures a fair comparison between different settings.
We original BERT Adaam optimizer [Kingma & Ba 2014) with the default cross-entropy loss as our loss function. Through our experiments, we discover the mitiil learning rate plays an important role for performance across all datasets. Thus, we optimize over wide range of initial learning rates including {2e-b, 4e-b, 6e- 8e le 4e Se S) For each initial learning rate, we repeat our experiments for different random seeds. Table 2 shows the best averaged performance. To foster reproducibility, our training pipeline is adapted from the Hugging Face library Wolf et al. (2020). We use x GeForce RTX 2080 Ti GPU each with 11GB memory. The training process takes about hour to finish for the largest dataset and 15 minutes for the smallest dataset.
LSTM Model Similar to our BERT model, we use a maximum sequence length of 128. We employ a training batch of 1024, and early stopping with patience of 5. This ensures a fair comparison between different settings. It is worth to noting that we find that BERT converges with scrambled datasets as quickly as (i.e., with same amount of steps) fine-tuning with original datasets.
We use the Adam optimizer with the cross-entropy loss as our loss function. We experiment with 1e=o) and choose the best one to report averaged performance results over 3 runs with different random seeds. We use x GeForce RTX 2080 Ti GPU each with 11GB memory. The training process takes less than hour to finish for all datasets.
learning rates of (le-3, 11e- 1e_1, le-5,
BoW Model Similar with the BERT model, we use dev sets to select the best model during training. We employ early stopping with patience of 5. This ensures fair comparison between different settings.
We use the Adam optimizer with the cross-entropy loss our loss function. We experiment with learning rates of {le-5, (1e-5,1e_,, (e- and choose the best one to report averaged performance results over 3 runs with different random seeds. For Conditional Random Fields models (CRFs), we use sklearn- sklearn-crfsuit crfsuite te library with default settings All models are trained using CPUs. The training process takes less than 15 minutes to finish for all datasets.
Dummy Model We use the dummy classifier in the sklearn library with stratified strategy as our random model.
Non-pretrained BERT Model For training from scratch, we try two stop conditions. First, we employ early stopping with a patience of 5. Next, we also try another condition where we left the model run for 500 epochs for every dataset except MRPC. For MRPC, we train it for 5000 epochs due to its small data size. We select the best performance out of these two options. This ensures the model to explore in the parameter space exhaustively and fair comparison between fine-tuned models and main-from-seratch models.
We the BERT Adam optimizer with the cross-entropy loss our loss function. We fix the initial learning rate at le-4, and choose the best one to report averaged performance results over 3 runs with different random seeds. We use 8 x GeForce RTX 2080 Ti GPU each with 1IGB memory. The training process takes about 4 hours to finish for the largest dataset and 50 minutes for the smallest dataset. For the fixed epoch approach, the training process takes about 16 hours to finish for the largest dataset, and 5 hours for the smallest dataset.
C FREQUENCY MATCHING
To study the effect of word frequencies on the transferability of BERT, we control word frequencies when scrambling sentences. Figure 77 shows the differences in frequencies of matched pairs. Our results show that the difference in frequency for a frequency-matched pair is significantly smaller than randomly matched pair.
dummy wuw..yyyaummllssssfffer...tt
14

---

Under review as a conference paper at ICLR 2022
To match word frequency during scrambling, first preprocess sentences by lower-casing and separating by spaces and punctuation. We then use the original BERT WordPiece tokenizer to deter- mine the sub-token length for each word, where the sub-token length is the number of word pieces a word contains. To randomly match words with similar frequencies, we first bucket words by their sub-token length. Then, iterate through words within each bucket in the order of word frequen- cies. For each word, we use the round-robin method to find the closest neighbor with the closest frequency.
A perfect match is not always possible as not every word can be paired with another word with an identical word frequency. We include the distributions of the difference frequency for every matched word pair in Appendix/Cto illustrate word frequencies are preserved.
D SCRAMBED SENTENCE
In Table 5 and Table 6 we provide one example sentence from each dataset destructed by our scrambling methods. We also include the original English sentence (OR) at the top for each dataset.
15

---

Under review as a conference paper at ICLR 2022
Scrambling Method
the worst titles in recent cinematic history a engaging semi is everyone dull dark kitsch theatrically tranquil andys loaf shorty lauper
Original Sentence
Similar Frequency
Random
(a) Scrambled Examples from the SST-3 dataset with different types of scrambling methods.
Scrambling Method
premise: a lady wearing a batman shirt is walking along the boardwalk.
Original Sentence
hypothesis: a woman swimming in a lake.
premise: .ar, peach playing the outside hands lay a
Similar Frequency
hypothesis: with the baseball man helmet a
premise: moist cleaver surf moist blades smurf hover bugger unto locals pinnies cotton
Random
hypothesis: moist songs hover starves blacktop moist beam
(b) Scrambled Examples from SNLI dataset with different types of scrambling methods.
Scrambling Method
question: what objects do musicians have to have in order to play woodwind instruments ?
Original Sentence
sentence: despite their collective name, not all woodwind instruments are made entirely of wood.
question: pubs people bomb first and first, areas and october confessor wit- nesses of
Similar Frequency
senfence: video its rebels states in his world confessor witnesses) under guam ? hall the
question: warranties mundine encountered froschwiller nir entering nir litatio pachomius entering mille says me diaspora
Random
sentence: mosfet bigua satisfactory merv gooding daewoo kennedy says mc iditarod scrofula depositing unprotected ubaidian oran
(c) Scrambled Examples from the QNLI dataset with different types of scrambling methods.
Table 5; Comparisons between the original English sentence and scrambled sentences.
16

---

Under review as a conference paper at ICLR 2022
Scrambling Method
sentencel: the court then stayed that injunction, pending an appeal by the cana- dian company.
Original Sentence
sentence2: the injunction was immediately stayed pending an appeal to the fed- eral circuit court of appeals in Washington.
sentencel: cents executive airways for simon to needs economy from custody no the
Similar Frequency
sentence2: simon loss airways needs economy share sending cents stores of dollar the
sentencel: najaf render analyzed threatening earners bethany hurlbert melville 517 riyadh birdie najaf hail weighs warden
Random
sentence2: najaf bethany roared jackson threatening melville 517 riyadh eves najaf credentials manfred render mission noting deceptive things warden
(a) Scrambled Examples from the MRPC dataset with different types of scrambling methods.
Scrambling Method
relations with Russia which is our main partner have great importance Kuchma said.
Original Sentence
overseas O NEW are 4 city children Draw after Wasim Mia. on turning 's providing 585 soliciting Pushpakumara Grabowski dissidents Kuwait flick-on Sorghum Pushpakumara Goldstein Batty secure Pushpakumara OHNKEL,RUO
Similar Frequency Random
Gama 603 LUX
(b) Scrambled Examples from the EN-EWT dataset with different types of scrambling methods.
Scrambling Method
We walked in to pick our little man at 10 minutes to closing and heard laughter from kids and the staff, any murder themselves good Iraq second my family Your hell a.? phenomenal n't death every the
Original Sentence
Similar Frequency
northward Darfur Bert stink Minimum descriptive A gunning Turns discomfort TERRIBLE stink Washington passcode Ham's 's blurred human 15 passcode agree faction Goldman
Random
(c) Scrambled Examples from the CONLI--203 dataset with different types of scrambling methods.
Table 6: Comparisons between the original English sentence and scrambled sentences.
17

---

Under review as a conference paper at ICLR 2022
Frequency (LOG)
Frequency (LOG)
104
104
102
102
0e+00
2e+03
4e+03
6e+03
8e+03
0e+00
1e+04
2e+04
3e+04
4e+04
Difference Frequencies
Difference Frequencies
(a) Scrambling with similar frequency SST-3.
(b) Random scrambling SST-3.
Frequency (LOG)
Frequency (LOG)
104
104
102
102
0e+000
Se+04
1e+05
2e+05
0e+00
2e+05
4e+05
6e+05 8e+05 1e+06
Difference in Frequencies
Difference in Frequencies
(c) Scrambling with similar frequency for SNL.I
(d) Random scrambling for on SNLI.
Frequency (LOG)
Frequency (LOG)
104
104
102
102
0e+000
1e+04
2e+04
3e+04
4e+04
0e++00 24 5e+04 8e+04 1e+05 05+ 2e+05
Difference in Frequencies
Difference in Frequencies
(e) Scrambling with similar frequency for QNLI.
(() Random scrambling for QNLI.
Frequency (LOG)
104
Frequency (LOG)
10a
102
101
0e+00
5e+02
1e+03
2e+03
2e+03
0e+00
2e+033
4e+03
6e+03
8e+03
Difference in Frequencies
Difference in Frequencies
(g) Scrambling with similar frequency for MRPC.
(h) Random scrambling for on MRPC.
Frequency (LOG)
Frequency (LOG)
104
102
102
0e+000
5e+02
1e+03
2e+03 2e+03
20+03
0e+00
2e+033 4e+03 6e+03 8e+03 1e+04
Difference in Frequencies
Difference in Frequencies
(i) Scrambling with similar frequency for EN-EWT.
(j) Random scrambling for on EN-EWT.
Frequency (LOG)
Frequency (LOG)
10a
104
102
102
0e+00
1e+03 2e+03 3e+03 4e+03 5e+03 Difference in Frequencies
0e+00
2e+03 4e+03 6e+03 8e+03 1e404 Difference in Frequencles
(k) Scrambling with similar frequency for CoNLL- 2003.
(1) Random scrambling for on CoNLL-2003.
Figure 7: Distributions of difference in word frequency for each dataset.
18


Reviews:
[{"review_id": "v0MV94nFjfz", "reviewer": "Reviewer_6a8W", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. the paper is mostly well-written and easy to follow.\n2. the experiments are well-designed to support the claims\n\nWeakness:\n2. I'm not convinced by the results about BERT in S7.3. The confounder factor -- pre-training, is not considered. For instance, for the pre-trained BERT, the word frequencies information is already captured in the parameters (along with many other features that also hold for scrambled text). Such that when fine-tuned on scrambled data (word frequency preserved), it's not surprising that pre-trained BERT can clearly perform better. And the reason why LSTM is more robust is that the model size is very small such that it can relearn all weights. We can observe the same trend for 1-layer-BERT on classification tasks, the gaps are generally smaller than other model variants. Some claims in the paper can be further substantiated if the author can experiment with a BERT that is pre-trained on scrambled English.\n\n3. I also have some doubts about S6.2, in which the author claim that the bad performance is because sequence labeling tasks are more likely to rely on word identities. However, as explored in Hewitt & Liang, those labeling supervisions (although in POS tagging) can be easily learned by fine-tuning. Can authors find a better explanation for the poor performance on labeling tasks?\n\n1. The method proposed in this paper is widely used in many other studies, e.g., in adversarial attacking. But it is still somewhat novel since it is applied to new domains. Also, the observations made are marginally novel or significant, there are not many new results compared to prior observations. The title is about pre-trained models, but only BERT is discussed throughout the paper.", "summary_of_the_paper": "This paper proposes an evaluation pipeline for pre-trained models by testing their transferability\nwithout word identity information. Specifically, they take an English pre-trained BERT off-the-shelf\nand fine-tune it with a corrupted English dataset. Those corrupted texts are designed to remove word identity information while preserving the word frequency information. They conduct experiments on 6 tasks.", "summary_of_the_review": "This paper presents some interesting observations, but the methods or findings are only marginally significant or novel.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tABD54QKiT", "reviewer": "Reviewer_vdgZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe overall presentation is regular. I think authors fail to properly state all the possible explanations to support the existence of transfer learning with scrambled or random input, \nIf transfer learning were the explanation of why scrambled input keeps showing transfer learning capabilities an experiment training with the original data while tested on scrambled or vice versa is needed. Why didn't the authors try that configuration?\nThe analysis section can be largely improved. Three out of four experiments show inconclusive results. If results cannot be interpreted, authors should find other experiments to support (or refute) their claims.\nDespite the interesting results and large number of experiments, the inconclusive results make unclear the scientific contribution of this research piece.\n\nSome details:\n\n_Paper's title is somehow misleading \"Cross-Domain Knowledge Transfer for Pretrained Models\". Authors use just BERT as pretrained model and a LSTM with GloVe embeddings as part of the experimentation.\n\n\n_ The introduction and the Conclusions are inconsistent:\n\"we evaluate LSTMs using GloVe embeddings, BERT, and baseline models\" vs \"we take an English pretrained BERT off-the-shelf and fine-tune it with a scrambled English dataset\"\n\n\n_table 4 is shown on page 6, but referenced on page 8. Please bring the table closer to its reference.\n\n_ \"Our leading hypothesis here is that the LSTMs may actually relearn all weights without taking advantage of pretraining\". \n", "summary_of_the_paper": "This work aims to identify the source of transfer learning on neural models.  To this end they set up a series of experiments where models are trained and tested with english data, and, with scrambled (or randomly replaced) data, keeping token-wise sentence length.\nScrambled sentences also use replacement tokens keeping token frequency with the original token.\nModels are tested on standard benchmarks on text classification and sequence modelling, showing a drop in performance when the input is scrambled, and a similar drop with randomized input.\nFurther analysis are meant to test of the scrambling maintains sentence semantics (it doesn't'), if BERT is just retraining (there is some transfer but inconclusive), keeping BERT frozen (it depends on the layer) and finally identifying word identity reassociation (it seems there is not reassociation).\n", "summary_of_the_review": "Finding are unclear\nThe paper is difficult to follow\nWriting can be improved\nExperiments are inconclusive.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "i0K0dUeqqZr", "reviewer": "Reviewer_4jKu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and does a reasonable job evaluating the robustness of architectures under an artificial form of vocabulary shift. However, the authors fail to convince me why studying this distributional shift would bring new and relevant insights to the table. I think for two reasons: \n\n(a) The shift is not interesting in the sense that it is related to cross-domain or cross-lingual or cross-time shifts, or any other shifts that we observe in the wild. \n(b) The shift does not seem to add value over other synthetic shifts that have already been studied, e.g., word-level shuffling, character-level shuffling, etc. \n\nYou could have studied many other scrambling functions, so why this one? That said, I liked §7.3, which nicely baselines the scrambled model performance. §7.5 is also nice, except that it’s hard to see why any network would learn to reassociate words in the context of learning a particular task. (Learning to reassociate words is not necessary to solve most tasks. Maybe in machine translation?)\n\nWriting: The paper is overall well-written, but I found the use of the word ‘scrambling’ for a left-to-right in-order replacement of words, misleading. In related work, scrambling refers to scrambling of word order. I would also have liked to see a stronger motivation for focusing on frequency. \n\nOther weaknesses: \n\n(i) BERT is old now. We have already seen a lot of studies of BERT (in the so-called ‘Bertology’-literature), but isn’t it time to move on? I’m a little worried about community-wide overfitting of our intuitions about language models. There’s the popular alternatives (RoBERTa, GPT-2, t5, etc.), but also a lot of faster, fairer, more interpretable alternatives. \n\n(ii) The two sequential labelling tasks are very similar. POS is a give-away of NER. How about finegrained sentiment, coreference resolution, semantic tagging, grammatical error detection, etc.?\n\n(iii) As shown in related work, the GLUE tasks are relatively easy (e.g., solvable without word order information). Maybe also consider harder sequence classification problems?\n\n(iv) Using only a small subset of GLUE tasks looks like cherry-picking. \n\n(v) It is unclear how sensitive the results for each architecture are to hyper-parameter changes. ", "summary_of_the_paper": "The paper studies the generalization of several architectures (BOW, GloVe, LSTM, BERT) under scrambling of vocabularies - with and without frequency constraints. ", "summary_of_the_review": "The paper is well-written, but poorly motivated. I simply do not see what we learn from the paper (in the intersection of 'previously unknown' and 'relevant'). In addition, while the experiments are well-designed, the datasets seem cherry-picked, and the models out-dated. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "XlpgD1isw2U", "reviewer": "Reviewer_dnE4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think the paper is well-written and debates interesting points in transferability.\nHowever, I want to see some measures of intrinsic evaluation related to the scrambled datasets\nas well (e.g., perplexity). My concern is that these scrambling techniques might push your\ndataset outside of real word adversarial attacks. I would suggest comparing them all together\nto support validity of the scrambling methods. Furthermore, could you every tried replacing\nword using ontology or a collection of semantically related words? Please add similar\nexperiment to show that scrambling with synonyms and antonym have different effects (or\npositive vs negative replacements).", "summary_of_the_paper": "This paper elaborates on transfer learning and domain adaptation in language models. The\nauthors argue that there are limits on how much information could be transferred when a\nmodel is scrambled or somehow randomized. They used different strategies to randomize\ntraining data including frequency matching and completely random replacement. They suggest\nthat only Bert shows high rates of transfer into their scrambled domains in classification tasks.\nTheir experimental results exhibit importance of pretraining on sequence labeling tasks where\nrandomness occurs. For example, they showed that identity of words are important and when\nthey are swapped with completely different frequencies (random) the performance drops\nsignificantly. They also supported the view of Sinha et al. (2021) and Ethayarajh (2019) that BERT\nmodel may preserves frequency better and this is the reason behind its superiority.", "summary_of_the_review": "I believe regardless  of how good some of transfer learning models perform, we need to elaborate on situations those models collapse. Therefore I think having similar papers truly exhibit weakness and strengths of these models is necessary. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work performs an analysis of the generalization ability of pre-trained models under the condition of vocabulary scrambling. The paper is well written and easy to understand. However, a full story and investigation into the cause of the observed transfer under word scrambling is lacking. For example, do more powerful models transfer less because pre-training is more effective?

While the effect described in the paper is interesting, it lacks a solid connection to important areas such as adversarial attacks, cross-lingual domain shifts and doesn't seem to have any effect on the application of fine-tuning to pre-trained models. The experimental section could also be improved with comparisons to other more recent models such as RoBERTa and GPT-2. Even though these more recent models are still Transformer-based models it can help answer the question if more powerful models transfer less under word scrambling as raised by reviewer 6a8W. The results on LSTMs seem to imply this case. We thank the authors for including additional results on DeBERTa but this was insufficient to change the authors' opinion of the value of the paper.
Decision: Reject