Where to start? Analyzing the potential value of intermediate models
Authors: Leshem Choshen
Elad Venezian
Shachar Don-Yehiya
Noam Slonim
Yoav Katz
Venue: EMNLP 2023 Main (2023)
Keywords: Intertraining
fine tuned
intermediate
finetune
TL;DR: Rank good models, not similar models

Abstract: Previous studies observed that finetuned models may be better base models than the vanilla pretrained model. Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset.
Here, we perform a systematic analysis of this \emph{intertraining} scheme, over a wide range of English classification tasks. Surprisingly, our analysis suggests that the potential intertraining gain can be analyzed \emph{independently} for the target dataset under consideration, and for a base model being considered as a starting point. Hence, a performant model is generally strong, even if its training data was not aligned with the target dataset. Furthermore, we leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings. Last, we release an updating ranking of best models in the HuggingFace hub per architecture.

Where to start? Analyzing the potential value of intermediate models
Leshem Choshen", Elad Venezian; Shachar Don- Yehiyaj Noam Slonim, Yoav Katz
IBM Research, MIT, "The Hebrew University (leshem.choshen @, cladv @ whl..ssaacaa...d--yhhyaae noams@il., katz@ Jibm.com
Abstract
whether it is worthwhile spending resources on lecting a base model. Assuming the potential gain is high, the following natural question is which base models are most promising. out of countless options available through hubs Hugginglaace (e.g, Wolf et al., 2020). We propose pragmatic methods to answer both questions, supported by extensive experiments.
Previous studies observed that finetuned mod- els may be better base models than the vanilla pretrained model. Such model, finetuned on some source dataset, may provide a better start- ing point for a new finetuning process on de- sired target dataset. Here, we perform system- atic analysis of this intertraining scheme, over wide range of Einglish classification tasks. Surprisingly, our analysis suggests that the po- tential intertraining gain can be analyzed inde- pendently for the target dataset under consider- ation, and for base model being considered starting point. Henee, a performant model is generally strong, even if its training data was not aligned with the target dataset. Further- more, we leverage our analysis to propose practical and efficient approach to determine if and how to select base model in real-world settings. Last, release updating rank- ing of best models in the HuggingFace hub architecture!!
We begin with two observations: (i) some target datasets are intertraining -eprrraiiinngssnssiiiveee, i.e., have the potential to gain significantly from intertraining, while others are not, and are typically indifferent to the base model selection. Furthermore, revealing this property of the target dataset can be done effi- ciently, by examining the gains obtained when us- ing a single representative base model as a starting point; (ii) some base models are of high quality, i.e. finetuning on them provides consistent improve- ments on target datasets, but most base models are inferior and degrade performance. Furthermore. ranking base models by quality can be done one target task and efficiently, via linear probing, namely training only the base model classification head, over a single representative dataset.
1 Introduction
Finetuning pretrained models (Devlin al., 2019). is currently the standard and best approach for adjusting such models to perform a downstream task (Chen et al., 2022). The resulting finetuned models are typically used for inferring the labels of new examples that are reminiscent of the data used for finetuning. However, was shown (Phang al., 2018a) that finetuned models, trained on some source dataset, may represent better base models, namely better starting point for a new finetuning process on a desired target dataset. This scheme, often referred to as intertraining, is the focus of the present work.
Thus, argue that preferable base model can be selected independently of the target dataset. This in contrast to the common perception (c.f. $7) that the alignment of the target dataset and the source dataset - used to generate the base model is major factor in determining intertraining suc- cess. We substantiate our observation of indepen- dence by conducting experiments on comprehen- sive set of target datasets and base models, compris- ing models obtained under controlled conditions well models from HuggingFace. In addition these findings, we analyze attributes of the source and target datasets that affect gains ($6).
Given target dataset, one may wonder what could be the intertraining gain, to determinc
As some models are just better, not due the choice of current dataset, it makes sense to rank the models once and pick the best ones. But even
"These authors contributed equally to this work. 'https: //tbm gi thub. io/model -o/mmdel-reeccclingg

---

ranking thousand models is costly. In $8, we rely on our analysis propose a practical approach to efficiently select models in a real-world setting. Moreover, instead of expecting others to rank the models, we share an updating site site featuring the best models currently found. So far, we tested over 2.5K models.
We focus on text classification for ease of evalua- tion, but assume the tasks are diverse enough for our conclusions to extend to other settings.
General. Containing GLUE (Wang et al., 2018) and SuperGLUUE classification datasets (Wang et al., 2019a), excluding test only and regression datasets. The datasets cover a wide range of tasks, from sentiment analysis through linguistic accept- ability to natural language inference. It is the most commonly used benchmark in related work ($7).
2 Preliminaries
In this paper, we use the following terminology. A dataset is a set of examples and labels. Our goal is to maximize accuracy on the test of the target dataset, "target" for short. We discuss the differ- ence between domain, task, and dataset in App. A.
NLI. Containing Natural Language Inference and Entailment tasks. Datasets of this group all share the same task. There is some overlap between NLI and General: in Fig. and mean comparison we subtract the overlapping datasets from General.
A pretrained (PT) model is a self-supervised model, c.g., RoBERTa (Liu et al., 2019). A fine- tuned model is a PT model that was further trained over some source dataset, denoted henceforth as "source". We assume access to many such mod- els, c.g., through HuggingFace. A base model finetuned model.
Twitter. Containing 6 Twitter datasets collected by TweetEval (Barbieri et al., 2020). The tasks range from irony detection to emoji prediction. Datasets of this group all share the same domain.
can be either a PT model When finetuning over the target train data, one can start from any base model. Intertraining refers to starting from finetuned model a base model, and in this case, we refer to this base model as an intermediate model. We denote by S the accuracy score obtained over the target test set, 1, after finetuning some base model m over the target train set. The intertraining gain of model w.r.t. using the PT model, is thus defined via goin (m,t) t) stn <'pT- Note that the gain may be negative. Given a set of intermediate models, .M m. ... mm, the intertraining max-gen is defined as maxmeM (sfn - s'pr). Thus, theoret- ically, max-gain is achieved by finetuning all the available intermediate models and picking the one best performing on the target test set. To avoid overfitting and reduce costs, our goal is to find an intermediate model with gain that is close as possible the max-gain, without explicitly fine- tuning all the intermediate models.
3.2 Models
Unless stated otherwise, our PT model of choice is RoBERTA (Liu et al., 2019). We acquire interme- diate models in two ways:
In-house.
Obtained by finetuning the PT model over General, NLI, and Twitter dataset groups as the source datasets, with 5 seeds. Since we con- trol the datasets and have knowledge about their features, this enables us to find relations between the dataset properties and the intermediate models generated from these datasets.
66 RoBERTa models downloaded from HuggingFace (see App. $E for more details), Since these models do not carry information ex- cluding their names, this set allows to validate our claims on "real-world" model distribution.
Off-the-shelf.
3 Experimental Setup
3.3 Models/Targedt experiments
We test many intermediate models on various tar- get datasets. We finetune each intermediate model and the PT the target train, and report the inter- training gain over the target test. In the In-house models/targets experiment, all 22 datasets from the General, NLI, and Twitter groups act as both source and target and gains are average seeds. the
Our experimental setup is described next. The pa- rameters for reproducibility are detailed in App. B.
3.1 Dataset Groups
Our experiments are based on 3 groups of English text classification datasets described next (App. C).

---

Target Dataset
Telturr Swetteenn
Twitts Emecion
Hade
Twchor tonry
Twitter Emook
FANRR
MARIC
CPAL
VAJI
MALI
General
Twither
MUTRC
6:13
1:34
1:72
3:33
8.16
0100
0.53
19:26
837
334
3:9
1.48
0.4
2:00
Q0P
141
5:50
050
231
4:20
3.64
548
000
3.11
4.7
16.99
0.25
4.67
-332
0.18
2.00
5.12
0.30
-.18
2.00
ISC
1:5
244
8.74
648
4.93
0/2
107
0.99
4.2
9.82
2.49
2.2
00
1.54
648
-145
MRPC
4.54
1.75
940
13
0.32
285
000
0.1
0.00
3.77
0.90
12
-1.0
RkniiQ
0.29
0.20
230
020
333
1A4
4.00
0.99
0.28
0.00
2.83
15.00
0.7
087
-353
2.2
2.42
3.57
020
0.7
0.1
0.75
COPA
693
1.16
027
0:29
-143
001
4.25
0.3
4.04
004
0.80
3.44
0.2
938
0.1
SS122
4:42
0.9
0.51
1.4
0.44
4.00
-307
0.00
0.11
4.17
0.1
C8
Source Daraset
0:00
3.25
0.16
2.42
5.93
0.34
6:13
3.22
1.90
0.00
3.00
0.5
0.8
8.43
3.41
351
3.20
0.8
033
0.3
0.16
RTC
5:00
4.82
0.96
450
648
2.43
4.70
600
4.19
7.35
422
4:50
0.25
-443
-323
330
020
147
69
000
43:
0.00
ESMLI
<2.38
4.75
2.06
4.2
12.43
4.05
12.57
4.00
4.48
-308
4.14
2881
4.40
4.2
1/7
0.66
Twrter Hate
3.50
-120
0.52
-170
0.30
2:50
6:12
4.67
4.48
-100
Twittar Offansive
132
0.49
3.06
1.75
0.1
4:31
4.19
080
0.06
1.07
004
4.50
4.2
020
2.40
-327
1.92
0.54
0.7
0.33
0.2
1.15
Twitter Irony
0:24
8:14
0UY
2064
0.2
0160
2.11
011
4.79
8:40
-311
4.29
00"
4.58
0.5
-10
Sondiment
Ensji
0.27
9.55
4:44
021
-35
342
333
max-gain
2:1
0.14
0:3
0.99
028
2352
251
18.10
0201
022
000
10."
087
028
173
0.0
1.4
028
048
Figure Results of in-house models/targets experiment. Columns correspond to target datasets and Rows correspond to intermediate models generated based on same datasets source. The 22 datasets come from the General, NLI and Twitter groups. Each value indicates intertraining gain w.r.t. the PT model, averaged over 5 seeds. Sorted by group and source average gain (bottom row). Positive significant cells (>2 STD) are italicized.
Off-the-sheff models/targets experiment, we down- load the 66 source models from Huggingface and test on the 14 General datasets as targets.
The rows and columns in Fig. are similarly ordered - first the General datasets, then the NLI datasets, and last the Twitter datasets. Loosely speaking, we do not recognize an approximate green block structure across the diagonal; namely, do not observe clear intertraining gain for simi- lar tasks (NLI); nor for similar domains (Twitter). However, some columns and some rows depict higher intertraining gains, while for others, the im- pact is minor. Taken together, these observations suggest little dependence between the source used to generate the intermediate model and the perfor-
4 Results
Most models worse than PT and about I in 6 are better, providing positive intertraining gain. The in-house models/targets results are depicted in Fig. and STDs and reference results App. $D. App. $E reports results with off-the-shelf RoBERTa and T5 intermediate models.

---

ate models (see $8). In the following, we propose simple yet efficient method to obtain such static ranking which is made once, without account- ing for the target. A more comprehensive ranking alternative is described in App. $F.
uce (Aeg Irlatianing Gani!
Given an intermediate model m, we train a lin- ear probe (LP) i.e.,, train only the classification head over MNLI, and consider the gain, de- noted L MNL.).. Evidently, this gain is a good proxy for the quality of m. Specifically, let ang be the average gain of m over a set of tar- get datasets. As depicted in Fig. 2, we observe that LP(m, MNLI) and gm are highly corre- lated in the in-house models/targets experiment (Spearman: 0.46, Pearson: 0.78, p 0.01) and the off off-the-shelf models/tagget experiment (Spearman: 0.51, Pearson: 0.66, p < 0.01). In other words, if available intermediate models are ranked by LP on one dataset L LP,,, L.), highly ranked mod- els represent the most promising starting points on average. The high correlation means this connec- tion holds not only for the top-ranked models, but throughout. Moreover, the replication on off-the- shelf models shows this is robust not only across targets but across sources.
MN21)
Figure 2: Linear probing MNLI (x) is enough to predict finetuning gains (y) averaged over 14 General datasets. Each point corresponds to one off-the-shelf base model.
mance over the target. This is in contrast to the common assumption ($7) that the source and target need to be similar for intertraining to work. Next, we delve deeper into these observations.
4.1 Target Sensitivity to Intertraining
Considering Fig. columns, we notice that for some target datasets (e.g., ESNLI) intertraining makes little difference, while for others (e.g.. COPA) the impact is quite significant. We argue that this target property can be predicted via an effi- cient and straightforward method. Specifically, the gains of one strong intermediate model should re- semble the max-gaims of a group of models. Indeed, MNLI highly correlates both with the max-gain of in-house models tested on the 22 targets in Fig. (Spearman: 0.89, Pearson 0.99) and off-the-shelf models tested on the 14 General targets (Spearman: 0.90, Pearson: 0.94, p < 0.01 for all). The repli- cation off- the-shlf models shows that this is general result and not reflection of MNLI being top of the in-house group. Overall, we find sensitiv- ity a characteristic of the target dataset separated from the source factor.
To further support this claim, ing to find the most promising intermediate models. For each target t, we consider the gain obtained by the top-ranked model and the max-gain obtained by one of the three top-ranked models, denoted Ii and K(y)- respectively. In comparison, we consider the max-gain obtained while considering all avail- able models, denoted (moz)) We further denote loss 9(max) - I/1 and loss? = gines gaa In other words, loss represents the potential gain loss when using the top statically ranked model versus using the best available model for the target under consideration. Similarly, loss represents the lost gain when using the best model out of the top 3 ranked models versus using the best available model. Note, that even in this latter case, finding the best model involves finetuning only 3 models over the target train data, which is far less demand- ing compared to finetuning all available models.
this rank-
4.2 Ranking Intermediate Models
Considering Fig. rows, we notice that some inter- mediate models - e.g., MNLI provide relatively high gains for many targets. We observe that this is a general phenomenon - stronger models are typically stronger for many datasets.
In Table 1, we report statistics for lossf and Toss over the in-house and off-the-shelf mod- els/targets experiments. Although the ranking is turget-independen,, the top 3 ranked models typ-
Identifying such models in advance could be practically valuable, since for a new target, one would consider only the highly ranked intermedi-
MMILI is not unique, many datasets showed promising results in initial trials.

---

ically provide most of the potential intertraining gain. For example, instead of checking all 66 avail- able models, using this simple strategy of checking the top 3 ranked models, each of the 14 targets lost at most 1.62 points.
General
NLI
Twitter
General -0.37
-2.68
-0.54
NLI 1.26
0.63
-0.03
Twitter -0.4
-2.39
0.53
Table 2: Intermediate models trained sources from the same domain (Twitter) or task (NLI) as the target, yield greater gain. Numbers represent the average gain
@ @Top
Avg.
Max
# targets s.t. Joss >
Models
of intermediate models of source group (rows) on a given target group (columns).
Loss loss
0.37
2.11
3/22
In-house
0.2
1.15
1/22
lossj
2.33
12.0
8/14
Off-the-shelf
prove General datasets even more than NLI ones. Possibly, NLIs are just good intermediate models. Twitter models, however, do seem to improve Twit- ter targets more (ANOVA < 0.01), by 1 point. Hence, it seems the effects are mixed.
loss
0.34
1.62
2/14
Table I: Lost Gain (loss,)) small when choosing the top-ranked models. Columns present the aggregation across target datasets of the lost gain: average, max and the number of targets that lose least one point. Rows present either in-house (22 models and 22 targets) or off-the-shelf (66 models and 14 targets) experiments.
In summary, as a group, a similar source tends to yield greater gains than an unrelated source. How- ever, in the rest of this section, we find this effect is of secondary importance to predicting model gains. A similar source may be more beneficial than dom one, but well chosen source produces larger benefits regardless of similarity.
5 Source and Target Interaction Analysis
Next, we analyse the interaction between the source dataset and the target dataset. Obviously, such interaction may impact the gain. On one extreme, since finetuning on the same data twice does not improve performance, intertraining is valuable when the source and target data are identical. On another extreme, consider partitions of the same data. Obviously, training over half the data as the source would be beneficial for the other half, the target. Thus, we do not question that interaction may exist, but rather investigate how common and strong it is.
Symmetry a similarity bound. We consider dataset similarity from another perspective. Sim- ilarity is a symmetric relation. Hence, if source- target similarity was a main reason for gains, we would expect that when the source helps the tar- get B, the source B would help as well. We assess the symmetry of the in-house models/targets experiment. We find that gains are far from sym- metric (details in App. SG). Thus, (symmetric) similarity seems like weak explanation of which source data would help which target.
Interaction between dataset groups. Our Gen- eral dataset group consists of diverse datasets, while the other groups share domain (Twitter) task (NLI). We analyze whether datasets that share a domain task with the target represent better source datasets than others.
Regression. As additional support for the rela- tively small impact of the source-target interaction, show that the interaction is hardly needed to pre- dict the scores. Specifically, simple regressor can approximate the gain without considering such in- teractions. The regression fits 3 sets of coefficients. For each target two coefficients f,, 1 which one may think of as capturing average gain and sensi- tivity to inter-training per target; and for each base model bj- which one may think of as capturing av- erage inter-trainin gain, per base model. We then define gain (base,, targety) (b, + tt) t where i and are the base model and target indices, re- spectively. Note that by construction, this regressor
Table 2 depicts the average gain of each source group each target group. Comparing targets (ta- ble columns), find the models trained on similar tasks, as group, have distinct behavior (ANOVA p < 0.01). On average, using NLI intermediate models on NLI targets, yields a gain 0.63, com- pared to a strong negative gain when using interme- diate models from other groups. Similarly, there is a same-grome game-group gain of 0.5 on Twitter.
Comparing sources (table rows), while NLI is best improved by NLI models, NLI models im-
'Even if similarity was a strong indicator we would not expect full symmetry, as other factors such as sizes do matter.

---

has 2Ñ + n parameters, while trying model Ná interactions; thus, it does not have enough degrees freedom to explicitly model all source/target in- teractions. Still, it obtains satisfactory performance, as shown next. As a baseline, we fit the same re- gressor after randomly shuffling all the gains in the experiment. This shuffling ensures no information comes from the specific source and target, while maintaining the gain value distribution. We mini- mize Mean Squared Error (MSE) and fit with SGD until convergence.
ANLI
MILI
Before we present the results, it would be benefi- cial to give some intuition MSE. As MSE is the square of the error, an average MSE of 4, means that the average prediction was accurate up (V4) points on average or more accurate than that (as outliers have more effect on MSE than on the average error rate).
5
130
203
-00
200
9900
3035
Figure For "good" sources the average gain increase the source training size increases, while for 'bad' sources it decreases.
We obtain an MSE of 4.2 on the in-house models/targets experiment (3.3), versus 9.6, 0.9 when the gains are shuffled. Thus, know- ing the source and the target provides significant information about the expected gain, with no need of keeping designated parame- ter for each source-target combination. We fur- ther compare this regressor with two other gressors, one that considers only base model in- formation Cquin(bases targetz) by) and one that considers only target related information (gain(base; targetz) 12). The MSE fit is 10.4 and 8.2, respectively, compared to 10.8, a 0.4 on shuffled gains. This suggests both the source and the target impact the intertraining gain, with po- tentially somewhat stronger influence by the target.
76
QUP
In conclusion, our observations suggest that when considering intertraining potential, loosely speaking it is enough to consider two separate is- sues (i) choosing a base model; and (ii) determin- ing the expected effect on the target.
829 Aag Gan)
25
.00
6 Factors Contributing to Gains
GBM
Target 122
So far, we showed the effects of the intermediate models are largely separate from those of the target. We proceed to examine how specific factors cach contribute to intertraining gains.
Figure 4: The average gain across targets decreases as the target training size increases.
6.1 Source and target sizes
Following the above observations, we ask what makes a base model good, or a target sensitive?

---

We examine the effects of architecture ($6.3) and source score ($6.2), but start by examining the data sizes effect on the gain: First, we identify effects of the datasets sizes in controlled experiments. Next, we assess the extent to which the effect of dataset size is evident in previous experiments. For more related analysis we refer to Phang et al. (2018a); Pruksachatkun et al. (2020).
the average is taken across sources. Still, targets with small training data seem be more sensitive to intertraining. Specifically, the 5 targets with the smallest training data (CB, CoPA, WSC, WNLI, RTE) are also those for which we observe the max- imal gain difference across all sources.
6.2 Similar Source Score, Different Gain
One can expect that two models with similar perfor- mance source dataset would also have similar intertraining gains. Our results suggest otherwise. We finetune 20 models over MNLI source and use them as intermediate models on the General target group. We compare the scores on the source test data to the average score obtained on the target datasets. Source task scores vary between 86.5 and 87.5 while General target average varies between 74.5 and 79, without correlation (c.f. App. I).
Effect of dataset size. We first control the source dataset train size. We create intermediate mod- els on 4 source datasets - the top 2 and lowest 2 in-house models, according to the static ranking ($4.2). For each, we limit the training to 50 - 3200 samples and use the General group datasets tar- gets. Evidently, for good sources (ANLI, MNLI), more training data yields greater gains (Fig. 3). However, additional data decreases performance for bad sources (MultiRC, QQP). We conclude that the amount of source data amplifies the effect de- termined by the type of data.
McCoy et al. (2019) found that finetuned models that show similar performance on their test data, still have different performance on out-of-domain challenge sets, suggesting the models learnt differ- ent generalizations schemes. Juneja al. (2022a) suggested that those models converged different basins in the loss space. They tagged models from one basin that tend to generalize better as good, and the rest as bad. We check whether good models are also better intermediate models for other tasks. We took their BERT models finetuned on MNLI intermediate models 12 good and 12 bad models and used the General datasets targets, finding that good models are indeed better for intertraining (3.6 65 avg. gain for good vs. 2.16 for bad).
We experiment on the effect of target size, using General sources and General targets with train data of more than 1600 examples. We limit the target train sizes to between 50 - namely, few-shot set- ting and 1600. As depicted in Fig. 4, the gain decreases the target training size increases, im- plying larger potential for intertraining when the target training data limited. Interestingly, for 3 targets we see positive gains on small datasets, which drops to negative gain as training data in- creases, and then seem to rise again towards zero. This 'U-shape' effect hints at two competing ef- fects that should be better examined in future work (c.f. App. H).
The differences discussed above are due to using different seeds. In App. I.1 show that hyper- parameter selection can also impact the quality of an intermediate model, regardless of the score ob- served on the source test data.
Training size effects in practice. We examine whether the effect above is strong in comparison to other unknown factors. Considering the in-house models/targets experiment, the Pearson Correlatio between source training size and source average gain is 0.75. Out of the 5 largest sources (ESNLI, MNLI, QQP, ANLI, and QNLI), 3 are also the source tasks with the highest average gain (MNLI, ANLI and ESNLI) and QQP the source dataset with the recond-lowes gain (negative). This is in line with our previous observation that additional source training data magnifies the positive or the negative intertraining effect of the source data.
In summary, similar training and/or similar per- formance the source test, do not translate similar intertraining gains on new target tasks.
6.3 Effect of different architectures
We validate our main conclusions across differ- ent architectures. To that end, we repeat the in- house models/targets experiment with BERT (De- vlin et al., 2019) and T5 (Raffel et al., 2020) archi- tectures. (sce full tables in App. J).
We observe no significant correlation between target training size and target average gain, where
We start by showing that the loose source-target

---

coupling holds across architectures. We then show that different souree datasets rank differently across architecture, but that target sensitivity is similar.
et al., 2021; Bassignana et al., 2022a,b; Vu et al., 2020). or when to prefer multitask (Weller et al., 2022). We study a much larger number of sources and targets, aiming to describe their natural distri- bution (c.f. $9) and also find that while specific choice may help, given enough models, it is safe to identify the models that just excel generally.
To show the source-target independence, peat the regression fit ($5). As before, the fit ex- plains each urchitecture's gains much better than when data is shuffled (BERT MSE 10.5, random 30.1, o = 4.17; T5 MSE 8.11, random 13.51, a = 1.5). Neither the average gain of intermediate mod- els - trained over various sources, nor the average gain for target tasks, correlate across different archi- tectures. However, the target sensitivity, measured by max-gain, is correlated across all architectures (pairwise Pearson 0.6 0.94, p < 0.05). Thus, al- though the source-targe decoupling and the target sensitivity are shared across architectures, a source dataset that produces high gains in one architecture might not do so in another.
re-
Recent work focuses on fusing multiple basc models rather than picking just one (Choshen et al., 2022; Matena and Raffel, 2021; Wortsman et al., 2022; Yadav et al., 2023). We expect our under- standing to aid in choosing a subset of models to fuse well.
Multi-task learning is another related field. It studies finetuning on different tasks at once (Aribandi et al., 2021; Aghajanyan et al., 2021) and recently also way to recycle models to do so was proposed (Don-Yehiya et al., 20226). In contrast our analysis, similarity between tasks aids multi-task learning (Abnar et al., 2021).
notable exception is the MNLI source dataset which achieves the highest gain in all three archi- tectures. Possibly, some data sources provide a strong intermediate model regardless of PT, with MNLI as a prominent example.
Deciding which information should accompany publications is an active research field, covering datasets (Holland et al., 2018; Gebru et al., 2021). human annotation sheets (Shimorina and Belz, 2022), and models AMcMillan-Major et al., 2021; Mitchell et al., 2019). Our work proposes to report upon sharing a model the aspects shown to affect its quality, such LP (m, MNLI). For datasets, propose to report intertraining sensitivity.
7 Related Work
Various works use intertraining, often following the assumption of task alignment necessity (Ein-Dor et al., 2022; Don-Yehiya et al., 2022a; Awasthy et al., 2020), namely, that the source acts as weak labeled data (Shnarch et al., 2018; Yu et al., 2021). While we consider intertraining through finetun- ing, adaptation to the target (Shnarch et al., 2022) the domain (Gururangan et al., 2020) was also suggested. Such adaptation may be applied to any base model, and complementary to the choice among base models. The need for alignment was also previously debated in the context of pretrain- ing tasks (Krishna et al., 2021; Rothe et al., 2021; Zhang et al., 2020; Ram et al., 2021).
Practical recommendations
Based on the observations ($4) and analysis ($5), we propose methodology for efficiently utilizing intertraining in real-world settings. We suggest to collaboratively rank all available base models for intertraining and to utilize this list whenever intertraining is applied to new target.
The properties of intertraining were studied in other contexts. Phang et al. (2018a) suggested the intertraining scheme. Pruksachatkun et al. (2020) probed the linguistic knowledge changes after in- tertraining, noting correlations to some target tasks, and hypothesized why some source tasks are good for intertraining, Mosbach al. (2020); Chang and Lu (2021) replicated the existence of good sources, but rejected the hypothesis. Others tried to find which tasks have an affinity to each other (Poth
New model. When new model becomes avail- able, we encourage practitioners to assess and share its quality. This can be done efficiently by lin- ear probing on MNLI ($4.2) or comprehensively (App. $F) by finetuning on various datasets.
We created statically ranked lists for RoBERTa- base and T5-small App. $E. Moreover, we apply our methods to many architectures and 36 test sets in an updating site reporting the best models.

---

New target. When considering intertraining on a new task, we recommend checking the target dataset sensitivity, and then choosing the base model. Since the model's rank hardly depends on the target dataset, we suggest using the static ranking. Specifically,, we propose to finetune the top-ranked model, and compare the results to those obtained when finetuning the respective PT model. Assuming the gain justifies it, one should consider a few additional intermediate models, in descend- ing order, according to the allocated resources.
in many targets while degrading performance in others.
Moreover, while testing 22 source datasets, many of which previously untested, we did not find new top source for intertraining. The best we found for RoBERTa was already known be good (MNLI; Phang et al., 2018b; Pruksachatkun et al., 2020). With that, by checking dozens off- the-shelf models, we did uncover new intermediate models that seem to outperform MNLI (c.g.. STS- B for RoBERTa and QQP for T5 c.f. App. $E). More work is needed to assess the potential inter- training gain of the available datasets and models.
9 Discussion
We ran thousands of finetuning experiments, spanning a vast number of tasks and base mod- els. Thus, while we believe it is unlikely that producing experiments will result in different outcomes, the large scale of our experiments places heavy burden on trying to replicate our results. Moreover, the ffflff models used in our ex- periments might not be hosted publicly in the fu- ture.
In $8, we highlighted practical recommenda- tions for intertraining. Those, together with a sys- tematic analysis of what affects intertraining, cover our main contributions. We hope this analysis would also aid future work on interactions between datasets; intertraining practices; and reusing fine- tuned models.
Our experiments mainly characterize what is probable rather than what is possible. We do not create specific models or aim to improve a specific task. Instead, we investigate what is likely to be found in typical practical settings. Accordingly, our findings are probabilistic in nature: Most models are not beneficial intermediate models, but there are enough that are. Mostly, beneficial models are beneficial for many targets.
Another limitation is that we could not upload all of the models to a shared location. This project was very computationally demanding, but more than that, it was demanding in terms of disk space, hence we had to delete many models along the way.
Finally, for practical reasons, our results are lim- ited classification tasks in English. We hope that future work will aim to test our conclusions beyond this scope. Overall, in the space of classification, we see our results as robust, testing on 22 datasets (about double the amount of previous works (Pruk- sachatkun et al., 2020)). We hope the diversity of targets brings large enough differences between datasets that the results would hold in other scopes.
As side effect, we do identify specific strong source models. MNLI was already considered beneficial source dataset (Phang al., 2018a), 8a). finding which we corroborate in a comprehensive manner. Furthermore, when considering off-the- shelf models we find models that outperform (e.g.. STS-B based for RoBERTa, and Quora for T5). To facilitate finding additional and better base models we will continuously report in the website website the best models per architecture.
10 Limitations
One obvious limitation is that our work is empir- ical in nature. As such, we report observations, sampled or common, with theoretical guaran- tees, and one should recognize the existence of exceptions. Specifically, even though have not observed there might exist target tasks that benefit greatly from a certain type of intermediate models; or intermediate models that help greatly

---

References
Transferability estimation for encoder selection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4218-4227, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. 2021. Exploring the limits of large scale pre-training. ArXiv, abs/2110.0200
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre- pre-tineunnigg ArXiv, abs/2101.11038
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The sixth pascal recognizing textual entailment challenge. In TAC.
Oana-Marie Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natu- ral language inference with natural language expla- nations. In NeurIPS.
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao. Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Hon- glei 7huang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al. 2021. Ext5: Towards extreme multi- task scaling for transfer learning. arXiv preprint arXiv:2111.10952.
Ting- Yun Chang and Chi-Jen Lu. 2021. Rethinking why intermediate-task fine-tuning works. In EMNLP.
Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. 2022. Revisiting parameter- efficient tuning: Are we really there yet? ArXiv, abs/2202.07962.
Parul Awasthy, Bishwaranjan Bhattacharjee, John Kender, and Radu Florian. 2020. Predictive model selection for transfer learning in sequence labeling tasks. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pages 113-118, Online. Association for Computa- tional Linguisties.
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. arXiv preprint 2rXiv:2004.166:
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro. Danilo Giampiczolo, and Bernardo Magnini. 2006. The second pascal recognising textual entailment challenge.
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. 2022. Fusing finetuned models for better pretraining. ArXiv, abs/2204.0300
Francesco Barbieri, Jose Camacho Camacho-Collados, Luis Es- pinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644-1650, Online. Association for Computational Linguistics.
Christopher Clark, Kenton Lee, Ming-Wo Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/uo questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2924-2936, Minneapolis, Min- nesota. Association for Computational Linguistics.
Francesco Barbieri, Jose Camacho-Collados Francesco Ronzano, Luis Espinosa-Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. 2018. SemFval-20188 Task 2: Multilingual Emojj Prediction. In Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, United States. Association for Computational Linguistics.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In MLCW.
Marie-Catherine de Marneffe, Mandy Simons, and Ju- dith Tonbauser. 2019. The CommitmentBank: Inves- tigating projection naturally occurring discourse. To appear in Proceedings of Sinn und Bedeutung 23. Data can be found at https: //gi thub. wtt.b.comoo....coooooooo.....to. com/mcdm/ Commi tmentBank/.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 54-63, Min- neapolis, Minnesota, USA. Association for Compu- tational Linguistics.
Jacob Devlin, Ming-Wer Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies (NAACL-HLT) Association for Computational Linguistics. Elisa Bassignana, Max Müller-FBerstein Mike Zhang, and Barbara Plank. 2022a. Evidence intuition: Transferability estimation for encoder selection. ArXiv, abs/2210.11255.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Elisa Bassignana, Max EberseeE, Mike Zhang, and Barbara Plank. 2022b. Evidence > intuition:

---

Shachar Don-Yehiya, Leshem Choshen, and Omri Abend. 2022a. PreQuEL: Quality estimation of ma- chine translation outputs in advance. pages 11 170- 11183.
Kundan Krishna, Jeffrey Bigham, and Zachary C Lipton. 2021. Does pretraining for summariza- tion require knowledge transfer? arXiv preprint arXiv:219.04995
Shachar Don- Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. 2022b. Cold fusion: Collaborative descent for distributed multitask finetuning, arXiv preprint arXiv:2212.01378.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. 2022. Fine-tuning can distort pretrained features and underperform out- of-distribution. ArXiv, abs/2202.10054.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd schema challenge. In Thir- teenth International Conference on the Principles of Knowledge Representation and Reasoning.
Liat Ein- Sor, Ilya Shnayderman. Artem Spector, Lena Dankin, Ranit Aharonov, and Noam Slonim. 2022. Fortunately, discourse markers can enhance language models for sentiment analysis.
Hector Levesque, Ernest Davis, and L.. Morgenstem. 2011. The winograd schema challenge. In KR.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daumé, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64:86 92.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv, abs/1907. 16692.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B. Dolan. 2007. The third pascal recogmizing textual entailment challenge. In ACL- PASCAL@ACL.
Ilya Loshchilov and Frank Hutter. 2019, Decoupled weight decay regularization. In ICLR.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. ArXiv, abs/2004. 109644
Michael Matena and Colin Raffel. 2021. Merging mod- cls with weighte averaging. arXiv preprint 1oXiv:2111009833
Thomas McCoy, Junghyun Min, and Tal Linzen. 2019. Berts of feather do not generalize together: Large variability in generalization across models with similar test performance. arXiv preprint 1rXiv:1911.02900
Stephen Hanson and Lorien Pratt. 1988. Comparing biases for minimal network construction with back- propagation. Advances in neural information pro- cessing systems, 1.
Angelina McMillan Major, Salomey Osei, Juan Diego Rodriguez, Pawan Sasanka Ammanamanchi, Sebas- tian Gehrmann, and Yacine Jernite. 2021. Reusable templates and guides for documenting datasets and models for natural language processing and gener- ation: case study the HuggingFace and GEM data and model cards. In Proceedings of the 1st Work- shop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 121-135, Online. Association for Computational Linguistics.
Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmiclinski. 2018, The dataset nutrition label: A framework to drive higher data quality standards. ArXiv, htss/880500667.
Jeevesh Juneja. Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. 2022a. Linear connectiv- ity reveals generalization strategies. arXiv preprint 1rxiv:2205.12411.
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. 2022b. Linear con- nectivity reveals generalization strategies. ArXiv, abs/2205.12411
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. Pro- ceedings of the Conference on Faimess, Accountabil- ity, and Transparency.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking be- yond the surface: A challenge set for reading compre- hension multiple sentences. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Teclnologies (NAACL-HLT)) Association for Computational Linguistics.
Saif M. Mohammad and Felipe Bravo-Marquezz 2017. Emotion intensities tweets. In Proceedings of the sixth joint conference on lexical and computational semantics (*Sem), Vancouver, Canada.
Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sob- hani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval- 2016 task 6: Detecting stance in tweets. In Pro- ceedings of the International Workshop on Semantic Evaluation, SemEval 16, San Diego California.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint
1412.6980.

---

Marius Mosbach, Anna Khokhlova, Michael A. Hed- derich, and Dietrich Klakow. 2020. On the interplay between fine-tuning and sentence-level probing for linguistic knowledge in pre-trained transformers. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 68-82, Online. Association for Computational Linguistics.
Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Glober- son, and Omer Levy. 2021. Few-shot question an- swering by pretraining span selection. arXiv preprint arXiv:2101.0045
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S. Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning. In 2 AAAI Spring Symposium Series.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4885-4901, Online. Association for Computa- tional Linguisties.
Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval- 2017), pages 502- 518, Vancouver, Canada. Association for Computa- tional Linguisties.
Jason Phang. Thibault Févry, and Samuel R Bowman. 2018a. Sentence encoders on STILTs: Supple- mentary training on intermediate labeled-data tasks. arXiv preprini 181 1.00080.
Sascha Rothe, Joshua Maynez, and Shashi Narayan. 2021. A thorough evaluation of task-specific pre- training for summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 140-145, Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018b. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks, ArXiv,
abs/1811.01085 1.00888.
Anastasia Shimorina and Anya Belz. 2022. The human evaluation datasheet: template for recording details of human evaluation experiments in NLP. In Pro- ceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval), pages 54-75, Dublin, Ireland. Association for Computational Linguistics.
Mohammad Taher Pilehvar and Jose Camacho-Colladoo 2019. WiC: The word-in- word-in-context dataset for evalu- ating context-sensitive meaning representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies (NAACL- HLT). Association for Computational Linguistics.
Eyal Shnarch, Carlos Alzate, Lena Dankin, Mar- tin Gleize, Yufang Hou, Leshem Choshen, Ranit Aharonov, and Noam Slonim. 2018. Will it blend? blending weak and strong labeled data neural network for argumentation mining. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pages 599-605, Mclbourne, Australia. Association for Computational Linguistics.
Clifton Poth, Jonas Pfeiffer, Andreas Rücklé and Iryna Gurevych. 2021. What to pre-traie treit on? Efficient intermediate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10585-10605, Online and Punta Cana, Dominican Republic. Association for Computational I Linguistics.
Eyal Shnarch, Ariel Gera, Alon Halfon, Lena Dankin, Leshem Choshen, Ranit Aharonov, and Noam Slonim. 2022. Cluster & tune: Boost cold start per- formance in text classification. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 7639-7653 Dublin, Ireland. Association for Compu- tational Linguistics.
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang. Clara Vania, Katharina Kann, and Samuel R. Bow- man. 2020. Intermediate-rask transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231-5247, Online. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631 16422, Seattle, Washington, USA. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research, 21:1- 67.
Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 task 3: Irony detection in En- glish tweets. In Proceedings of The 12th Interna- tional Workshop on Semantic Evaluation, pages 39- 50, New Orleans, Louisiana. Association for Compu- tational Linguistics.

---

Tu Vu, Tong Wang. Tsendsuren Munkhdalai, Alessan- dro Sordoni, Adam Trischler, Andrew Mattarella- Micke, Subhransu Maji, and Mohit lyyer. 2020. Ex- ploring and predicting transferability across NLP tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7882-7926, Online. Association for Computational Linguistics.
Mitchell Wortsman. Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Loper Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Komblith, and Ludwig Schmidt. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increas- ing inference time.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raf- fel, and Mohit Bansal. 2023. Resolving interference when merging models. ArXiv, abs/2306.01708.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- prect Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: stick- ier henchmark for general-purpose language under- standing systems. In NeurlPS.
Yue Yu, Simiao Zuo. Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. 2021. Fine-tuning pre- trained language model with weak supervision: A contrassive-reggulrized self-training approach. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 1063-1077, Online. Association for Computa- tional I Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels. Belgium. Association for Com- putational Linguistics.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the Type and Target of Offensive Posts in Social Media. In Proceedings of NAACL.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: multi-task benchmark and analysis plat- form for natural language understanding. In Interna- tional Conference on Learning Representations.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020. Pegasus: Pre- theetrriiing with extracted gap-sentences for abstractive summarization. In In- ternational Conference on Machine Learning, pages 11328-11339, PMLR.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Orion Weller, Kevin Seppi, and Matt Gardner. 2022. When to use multi-task learning vs intermediate fine- tuning for pre- rraitri encoder transfer learning. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 2: Short Papers), pages 272-282 Dublin, Ireland. As- sociation for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 11 1112-1122 122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of thee--------arrr natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.

---

A Task, Domain and Dataset
2011). MULTIRC (Khashabi et al., 2018), WIC (Pilehvar and Camacho-Colladoo 2019), WSC (Levesque et al., 2012)
A task is defined by the input and the output. The input in context is a text instance. The out- put could be, e-g.. posttive/negatiiv/neerrall for sentiment analysis task, entailed/not-entailee for textual entailment task, etc.
MNLI (Williams et al., 2018), QNLI Ra- jpurkar al. 2016, RTE (Dagan et al., 2005; Bar-
NLI:
Haim et al., 2006; Giampiccolo et al., 2007; Ben- tivogli et al., 2009), WNLI (Levesque et al., 201 I),
A domain is the type of text found in the amples, regardless of the labels. For example, domain may be financial or comments for twitter.
ESNLI (Camburu et al., 2018), adversarial NLI (Nic et al., 2020).
A dataset for our purpose is set of examples and their labels, divided into train, dev, and test folds. Being such, each dataset has domain (char- acterizing its examples) and a task (for which its labels are annotated). Hence, we consider a subset dataset as an another dataset. Note that in the literature those terms are often not well defined and may even be interchangeable (Wang et al., 2019b).
Twitter: Emolnt (Mohammad and Bravo- Marquez, 2017), Emoji (Barbieri et al., 2018), Irony (Van Hee et al., 2018), OffenseEval (Zampieri et al., 2019), HatEval (Basile et al., 2019). Sentiment Analysis (Rosenthal et al., 2017)
Whenever the test is held out (such as is GLUE and SuperGLUE), we extracted 1K or 10% of the training examples as test set, the smaller. We did not experiment with the small Stance (Moham- mad et al., 2016) Twitter dataset originally found in TweetEval (Barbiieri et al., 2020) to reduce noise. In the T5 experiment ($E) we used stance datasets well, to have large group. For MNLI use the mismatched validation a test and the matched as a validation set.
B Hyperparameters
For RoBERTa, we train for 10 epochs with linear learning rate 5e-5 with warm-up of 0.6% of train- ing, batch size of 256, early stop epsilon 0.001 accuracy points, patience of 20 x 50 x 256 ex- amples, validate every 50 x 256 examples, op- timizer ADAMW(Loshchiloo and Hutter, 2019). with weight decay 0.01 or 0. For BERT-baas- uncased we use 2c-5 learning rate and never use decay. For T5 use le-4 learning rate and train until early stopping occurs. We used A100 and V100 GPUs. Finetuning times vary, but all end within a couple of hours, most in less than an hour, some up to 8 hours.
D In-house models/targets additional information
We report in Table 3 the score of finetuning RoBERTa without intertraining.
We also report the standard deviation for each cell the experiment, i.e., taking into account dif- ferences due to finetuning the intermediate model and target dataset in figure 5. For each seed, we finetune the PT over the source dataset to produce the base model, and use it to finetune the target task. It means that each seed utilises different base model. Note that $6.2 suggest different seeds may create base models with different quality. Note that to assess the variability of the averages reported in the main text ($4) the Standard Error of the Mean is required, this the STD divided by the square of the number of seeds SE = STD.V5.
C Datasets used
All datasets could be downloaded from hugging- face datasets. As used groups of datasets report here the full list of datasets they contain.
General GLUE: CoLA (Warstadt et al., 2019), SST2 (Socher et al., 2013), MRPC (Dolan and Brockett, 2005), QQP (data. comFFrst MNLI (Williams et al., 2018), QNLI Rajpurkar et al. 2016, RTE (Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), WNLI (Levesque et 2011)
E Models in the wild
E.1 Models used
SuperGLUE: BoolQ (Clark et al., al. 2019), CB (de Marneffe et al., 2019), CoPA (Roemmele et al.,
We collected manually 66 RoBERTa-base models. From their names most were finetuned from vanila

---

Target Dataset
Twibe Ortunaaee
Twiteer Excotion
Twiter Enaol
MARROO
000
VNSI
DOPA
ONLY
RTE
ESHAL
arg
General
NLF
Twitter
0.10
103
1:1
671
11:20
2:1
0.80
3:2
0.20
0.4
011
1.3
2.00
033
138
044
2.0
0.0
0.2
127
W8C
235
028
108
123
3/5
2:12
14:44
0.0
0:12
2.05
033
2.40
2,22
134
1.77
0.8
OCLA
2.80
028
280
121
3.2
300
620
002
0.2
245
0.7
1.00
General
MMC
BadlD
1:88
6:22
6.03
0.2
170
6:00
0.08
0.5
1.8
1.02
0:3
9.70
0.16
1.2
642
2.00
016
0.2
032
1.43
3312
1/07
6164
0:34
0.80
0.24
69
827
1.00
7/8
0:81
011
010
114
1182
1175
084
2.21
100
Source Delaeset
GNU
3:8
657
90.00
3.24
0G0
066
4.00
042
5:15
3.66
514
043
1:30
067
0.2
079
2.35
1.2
0.2
239
191
1.77
1:15
123
4:2
070
120
222
275
0.6
3.77
323
1.1
037
123
077
031
0112
1.42
0.00
084
241
151
ANLI
264
040
577
140
025
075
1112
9:01
0.0
253
2:1
1.20
0.4
0.12
021
1.48
0.4
307
000
046
233
130
Twitter Hate
0..0
1079
161
172
0.4
1.41
1.70
023
1194
696
1.81
0.11
171
Twitter Cifensive
366
Twiter irony
3.34
0.06
144
144
15c
600
111
321
3.00
11.11
012
0.5
1.30
044
233
1:4
Twitter Senrimmnt
3:36
697
0.00
0.6
023
475
4.64
1.5
642
1.33
Twetee trme
871
141
575
5:6
1.0
022
207
235
Twitter Eirtton
3.04
261
0.52
337
6.5
0.7
0.0
027
1.71
1.5
Figure 5: Standard deviation of in-house models/targets experiment. Rows correspond to intermediate models. generated based on source datasets from the General, NIJ Twitter groups Columns correspond the same datasets, now being used as target datasets. Each value indicates standard deviation over seeds.

---

Dataset
Mean
Std
that finetuning only STS-B (rather than on 40 datasets) yiclded similar results. We note that un- like MNLI which is known to be a good source (Phang et al., 20) STS-B B was previously only considered as a target task only. The results sug- gest, it might be good souree.
MultiRC
61.07
2.01
QQP
90.92
0.29
WSC
63.46
0.00
MRPC
87.70
0.95
CoLA
83.11
1.34
WIC
63.55
2.32
We gathered the models by manually searching for 'RoBERTA-base' models, ignoring ones that were working on languages other than English. It is possible we have missed models that did not clearly state their architecture as part of the model name. We are already aware of such models, for which the PT is not deducible from their title, for example those lately released by Juneja et al. (2022b).
BoolQ
77.09
3.19
COPA
49.00
4.90
SST2
93.81
0.26
CB
70.36
3.11
QNLI
92.28
0.48
WNLI
56.34
0.00
RTE
72.42
0.93
ESNLI
91.05
0.18
ANLI
51.67
0.36
Rank by Average over Targets
MNLI
87.07
0.23
Twitter Hate
52.30
1.03
In $4.2 show training one Linear Probe is enough to rank base models. Although tested on a large number of target datasets, presumably, this method does not always achieve accurate predic- tions. For example. the target domain might be so different that MNLI would not be relevant. As more accurate alternative, one can use several datasets to provide more reliable picture. We show that an average of finetuning gains over dif- ferent datasets is reliable way for choosing base model. As in the simpler case of LP, this supports the decoupling.
Twitter Offensive
84.67
0.41
Twitter Irony
70.84
2.53
Twitter Sentiment
70.59
0.34
Twitter Emoji
46.32
0.56
Twitter Emotion
82.08
0.58
Table 3: Scores of finetuning RoBERTa without inter- training.
RoBERTa, but few were trained from scratch. They vary in hyperparameters and training details, just as one can expect training approaches vary between different researchers. We supply the full list of models in Tables 4,5 for RoBERTa and respectively.
We report in Table the lost gain when choosing the highest models, ranked by average gain over the General group. This ranking method generalize well; The or 3 best-ranked models are close the best possible model overall for each target. For example, only 2 targets lose more than 1 point by choosing the top 3 models.
E.2 Results and discussion
We report the full results in Fig. 6. T5 models are reported in $7 (off-the-sheld only, most of General and NLI were part of T5's pretraining, in-house use 15.1.1). Seemingly, some traits the training that we did not account for are important. It is exemplified by models that are associated with the same datasets but differ in their gains. For example cross-encoder imple- mentations outperform other models and sentence- transformers underperform them.
Twitter datasets
Practically, we suggest to rank either in this method or by LP. If some use one method and others choose another it might be hard to compare the two rankings. Thus, we report that in our exper- iments the best predictor of the average score by the LP score gm MNLII- 0.0822 0.940
ang
G Symmetry metric
Notably, most models are not useful for inter- training. Still, many models do.
To measure symmetry of matrix M, decom- pose its symmetrical and skew symmetrical parts: M = + V where = (M + MMT) and
The best model on average MUPPET (Agha- janyan et al., 2021) which massive multitask learning approach. However, our results show
= { - AMT). - S is symmetrical: = S and
is skew-symmetrical: V --" The measure

---

set
name
avg. gain over General
LP gain over MNLI
imdb
aychang/roberta-buaee imde-
-5.91
-12.62
sentence_
-5.8
2.59
models 1
rextatuck/robertt-bbsseee-
-5.73
-17.09
twitter_ 10
-5.71
-9.76
sentence
sentence-transfonee traemnce-traasfoomrssirheetttr -asiit----------s-------
-5.58
2.59
finance_ O
zhayunduo/roherta- -tstottkttassssskttttttts
-4.49
12.60
sentence_2
sentence monnsforrrrsmmmmaccooo.o roberta- base-v3
.4117
-8.82
twitter_8
cardiffnlp/twire cober-oobrtaaaae -bberta-base-mmotion
-4.17
1.00
sentence_
sentence sentence-lransformeess transformersroobertaa basaaa-aaaellliie maaesenniiema------ee mean.tto...
.4.07
5,47
qa_3
navteca/roberta. sattec//robett---bu--suuaddd base-squad2
4.04
4.03
quora scratch 0
roberta-bbet
3361
8.35
neoyipeng/wwitters wooberta-bus-ssenimeeee
3351
-3.13
models_5
necyipeng/twwi hetppenggtwitttrrrttoooo roherta-base-sentinne
33.1
-3.13
sentence
sentence-tranee tra.ntnoorrmmrrsll--oreeee..
3350
5,47
models_8
cointograted/roeettaaeeee haserrlllt
3329
4.96
15
legal_
naibo/legal-on
leger--obrrraa- roberto-bbaatt
3,20
-1.76
twitter_ 12
cardiffalptwittes cooeteer--obra-a--- crase-stance-abrtii
-295
-8.99
17
sst2 _
Bhumika/roberta base- finetuned-ssi.
2.77
-3.00
18
models_ 14
cestwc/roberta-bane wttp://rbbett-bas-uminrree bww.--nigraam-eenaaaeee
2273
-8.39
19
models_11
maringrandurr/robette base- finetuned-smin sma-spam.......
-267
-,900
20
-2.57
3.67
21
twitter_ 13
hhotlooh/wiitteer-ohettt-----
-247
-3.12
22
models 0
-238
-2.47
23
twitter_6 6
carditfnl cardiffll/ttwiter-roo htwptitr-rrobbetraa---eeeeeeeeee
2231
-11.45
24
twitter_S
cardiffnlp// httpf//tp//ttttttr/et roberta mabera--bas--nne hase-mar2022
2.21
-5.38
25
quora
navteca/quora--e
hase
2.13
8.35
26
models_16
hoanhkhoa/roberth base finetuned-ne
2.12
6.82
27
twitter_3
cardiffnlp/wwitter roberra-bbtt---a Seeetta-aae--200 masee-221--222
-2.5
-5.00
28
twitter_11
arddfffnlp/tiittrr- roberta basttaa-bb-saaaeee base-stance stance-climate
-1.98
-6.29
29
legal 0
akdeniz27/roberza -aserraaaaaaaaddad
-1.90
-8.57
30
models 13
cardiffll//wiiter-rbbeee roberttt roberta-hase--tteen hase-stance-femiinst
-1.86
4.25
31
imdb
aypan cettoobberta-buu---ie
-1.86
-3,50
32
models 2
ghanashyammvattiiohhhttt waw.rrrrbrta----eeeewe-- fake-news
-1.81
-12447
33
scratch_1
neoyipeng/twitter htt////tttttrrttoeeeeeee roberta---aa---ao saae-basccssee
-1.71
-8.73
34
models_12
surrey- surrcy-nlp/robert. nlpproberta--eeee wwwebberttahhh--tiiiie hase-finctuned-ahbe
-1.65
40,66
35
twitter_4
pardiffnlptwwigg httpptttr-rtttttee roberta base-emoji
-1.63
-276
36
models_4
textattac
-1.61
3.73
37
legal_2
marshmellow www.melloo777hhberee base-aree cuad
-1.53
-8.57
38
legal 3
Rakib/roberta- anse-on-cuad
-1.33
-2.48
39
twitter
cardiffnlp/twiin httpp///tttttrbbbrttte woweert--bbs--eetnmeeee base-sentiment
-1.24
3.03
40
twitter 9
wewilffillpwwitterrbbbbeee base
-1.16
-1.49
41
models, 15
haw.raammbuuuuugoorrrtttteee saatttu-aae-iiit wikimovies wisi-iiimmviissss
-1.11
-1.76
finance_ I
vanadhi/roherta- bwwe-aaqa-aaaaaaaeeaaaaa-aaaaaae fiqa-fim-sq-ffff fim-sq-flit
-1.09
-1.00
43
models_3 3
allenai/reviews
-1.01
-1.17
44
models_7
princeton- princeton-nlp'sup nlp/sup-simese simese-roberta base
-0.99
4.26
mrpe _1
ji iixxi/roberrta
-0.44
-167
twitter
seddffflp/wwitterrobeeeee p base-offensive
-0.91
-279
47
sst2
textattac kroberta- Wroberta-base-SS... base-SST-2
-0.44
3.83
48
sentence
sentence-transfine
-0.00
1.68
49
twitter 2
cardiffnl http:f/////tttttrrrree roweeeer-rrrra--aa -oberta-hase--rony
-0.99
0.46
imdb
textattackrroberaa -ibe
-0.40
51
models_6
VietorSanh/rohert bas--basee-nn masee-fnnttune calttttnddd-yl--papaiiii
-0.25
40.66
52
models_10
gargam//rberta base- crest
-0.12
4.21
53
twitter_0
bhadresh-
Seaddrsh-savani/robeee savani/roberta- httpi//obeetabaae--eooo base-emotion
-0.05
4.61
54
q
shahrukhx00 /rooertta boolq
0.25
10.42
55
mrpe
Exxtatacckroberrt-bbar beaeeerrrrrbeeeeeeeeeeeeeeeeeeee base-MRPC
0,39
7.27
56
textattack/roberta... base-RIT
0.42
14.82
57
nli
mujecnsuug/robertaa mnli be
0.48
23.43
58
0
deepset/roberta- www.bebrrrtaaaa-suuudaeeee hase-squad2-covi//
0.50
-1.11
59
csarron/roberta gaaroo//obertt--as---qqqu
0.99
3.38
60 stsb O
textattack/roherta www---aass--------
1.05
14.66
61 models_9
textattack/robee base-QNLI
1.10
6.49
62
nli_0
textattack/roherta Maceeertt--bse-MMee base-
2,09
34,39
63
!
cross cosss-nncodrrflii ceteoodde/nllrroorettee base
2.77
34.09
64
stsb 1
bross-encoder//tt.-bee rowestt--bbbbbbsseeeeee
2,82
30.19
65
multitask_0
facebook/mmpppe--oober robeett babbrrttaaaeee
3.00
31.58
Table 4: RoBERTa models used, collected from Hugging Face models hub. Models sorted by average gain the General targets.

---

2
16.11
610
M1
11
14:30
271H
16:16
6:09
16.1
-3433
2.00
-156
2.15
5.2
47.60
2:23
3.16
9.34
4.27
110
16:30
1:45
21:00
0.4
2744
1.7
90
410
3.2
0.46
14
25.2
053
Tedtor
4.95
10.00
4.1
20.00
2.92
0.09
280
6.0
0.11
4.11
3.0
30.66
6:99
-3.11
0/
6:14
6.66
3.11
-14
3.18
0/7
-197
25.6
4.12
0.4
006
0:08
4.05
Quara
-1.44
11
4.12
4.00
0.6
144
201
<.25
1:56
1.4
+:31
116
3.13
0.2
Medera
9.07
122
001
100
614
1.18
0.43
0:48
1753
4.29
381
4:3
0.3
5.11
13h
1:31
30.00
6/4
1.15
1738:
41.21
3.74
431
11:8
7:1
1.0
-107
2.13
010
3.65
21:3
6:99
4.15
6.00
4.51
46.77
2.3
000
233
6m
410
016
040
Thrber
8.0
2.00
-3.45
-14
4.33
4:72
0:09
2001
1:19
1:1
416
2.11
500
.154
19:22
07:41
2.15
3.5
4.06
200
1.00
6.00
0.00
-104
4.17
40.5
0.11
-111
-1.24
2.0
2:31
T
2:06
3.0
3:11
5.00
0.44
172
-385
1142
-127
7.34
857
-331
18.1
2:0
2.21
Ouara
-0.23
8.00
4.12
5.00
087
0.4
119
4.5
1.00
2.0
1.28
031
141
0.00
2.3
Modes
1.68
1.16
410
116
015
8.88
0.53
0.49
211
1.41
6.9
T
100
1.2
421
4.33
9.7
5.0
Tatter
1.55
400
123
18.2
2.54
Lxp40
415
4:00
204
-101
..73
412
Teltee
0/
310
400
016
2.40
4011
4.11
000
1.9
800
1.16
1630
1:80
604
9.99
6.7
1.8
4.95
116
004
2.09
1.91
Legar
850
178
364
4.0
0.00
0.0
130
004
2.44
0.30
5.23
006
2.00
1.83
Legal
0.0
179
371
2.00
8:0
144
180
248
4.95
41/4
0.11
6.7
0.00
2.0
4.33
314
0/0
301
500
0.15
1:75
314
615
141
1047
1.28
2.2
000
2.0
1.24
875
000
410
0722
0.5
006
471
8.64
5.55
0.2
0.1
5.0
1.90
12.00
C/4
3.31
3.90
5.99
Preeee
006
116
2012
410
4715
Sertewo
136
1.00
1.72
050
45.72
4.57
0.9
Twetr
400
144
2.11
4.30
116
0.9
1.7
3.1
100
816
204
1.19
4.7
4.17
4:14
6.0
2.4
0.6
0.99
0.40
Vodex
200
1.79
6:66
100
054
2.1
186
4.0
4.25
4.33
011
047
0.42
0.0
0.25
Modes
320
1/79
321
4.0
0.4
1.72
3.12
2.5
-.01
233
4334
3.
0.0
230
0.12
Teetor
016
1/00
9.2
410
0.98
02b
230
6.00
4.22
6.90
034
211
018
0030
1079
010
5/1
616
4:20
114
4/4
6:0
wecc
0.00
1.00
<MM
2.12
5.99
0.00
100
1:56
028
3.1
4.22
5.99
704
700
1.4
071
0.0
0.0
150
352
016
5.88
0.1
0.38
1291
1850
435
0.0
110
0.38
4.6
1.68
11:07
241
C1
638
0.34
138
0.09
Mulrascc
6.03
12.11
3.29
300
0.00
240
000
030
4.50
11.19
0.00
372
006
0.00
3.0
Figure 6: Results of the off off-the-shelf models/targets experiment. Rows correspond to off-the-shelf RoBERTa models obtained by downloading from HuggingFace model huh. Columns correspond to the General datasets group. Each value indicates intertraining gain w.r.t. using the PT model,

---

Targel Cottont
026
6:36
44
3.84
676
Sunnarr 303
-12
000
058
477
1:16
135
4.79
-M4
4.50
4.05
273
425
2
188
311
431
Senmatre 200
-112
4014
125
4.11
0.3
4.50
127
-10
0.00
4.40
540
4.0
-118
Sunriatioran and
4.50
0.77
161
028
2.73
0.1
140
-323
-337
4.05
540
053
0.6
4.5
4.43
4.05
233
Kemriato wik
7:6
601
115
2.33
1.40
086
1.0
1.38
1.18
1.00
116
048
Sounce Crteet
930
435
3.3
140
437
950
0.67
083
142
1.69
1.40
007
0.30
271
175
676
068
123
133
021
3.90
940
Semmutatt.o22
8/6
144
0.89
012
235
157
3.4
237
233
2333
0.7
1.43
0.70
912
13:11
8.64
1.28
1.9
3.67
227
2:00
5:14
Me
10.21
4338
1.51
3.57
330
6:11
12:44
0.28
1.07
-182
1.29
0.53
9400
13.44
1.88
0.38
6.56
1.07
0.8
175
680
Figure 7: T5 off-the-shelf base models and targets. The intertrain gain over the pretrained model for each source (row) and target (column) datasets.

---

set
name
avg. gain across the General largets
0
QA
allenai/t5-small--uuu
-1.28
1
QA
allenai/t5-smi Seeenai/t-ssmlll-spppe -sall--quadll
-0.77
2
Summarization
jazzisfuture/new summary small
-0.25
Classification
-0.22
Sentiment
-0.22
5
Summarization
-0.06
6
Classification
num88488/-smma samaal--inetuned-booo boolg
-0.03
Summarization_ 10
aseda/t5-sman wil--ffnnnn-suuumm
0.14
8
Summarization_3
mengsay/tS-smal manlt-finettuee gigaword
0.23
9
Paraphrasing
hetpandya/t5 -ataa//mall--tppaeeee
0.34
Summarization_
bhuvaneswariff ses-smllll-ee Sealll-exL_suumarrreeee summarization
0.36
Summarization
bochaowei/tS... httpt//--lllllttuuiiiie
0.40
QA_1
allenai/umificdquttt wtw.iiiiiii----maaaaaa 15--sall
0.43
QA
allenai/55-small--see squad2--uestion -enerrrtnnnnnoioonn
0.62
QG
allenai/tS-sman taa---mlll--sp Sanalllquuad---ae -uuustiiooooonnn--- cenn--eneration
0.62
15
QA_6
SenostS-mmal--iineee finetuned-squadz2
0.90
Summarization 6
stovhliu/t5- -mmall-finctuued- soluneddiilllum-ca callsum-ca_test
0.94
17
Summarization
furyhawk/tS smmllll-lll---- -mall--fnnttueed finetuned-ble
1.17
18
Summarization
bochaowei/tS -ssalllliii--sll-------aa caaaaal-inttune--mmme -en-ee--nn--eiO
1,36
19
Summarization
Frederick029 (15-small-fime finetuned -illlluum
1,51
20
Classification
1.54
21
Sentiment_J
1.54
22
Summarization 8
airKlizz/t5-small-miiit https/---mmm-mmmlltoooiiiiiiii paaatl--commii--wii--eeeeee
1.79
23
Paraphrasing/QA. O
mmm448//5-small- www.amallllntuue--uuuuaaaa gur.urrffo-parrphraiii..
2.03
24
Paraphrasing/QA__4
Sirpandya/t5-aae saall-mmllsee
2.05
Table 5: T5 models used, collected from Hugging Face models hub. Models sorted by average gain over the
General targets.
Models
Top
Avg.
Max
number of datasets
while intertraining often improves results for small data size, the effect decreasing with the size. Surprisingly, the decrease drops below zero and at some size increases again. This suggests an unexplained underlying behaviour, presumably two competing effects, one that decreases gains with size and one that improves them (in general or towards 0). We produce three examples of the U-shape in Figures 8, 9, 10.
with loss, >
Ios81
0.37
2.11
3/22
In-house
1os83
0.2
1.15
1/22
Iass]
1.41
12.0
3/14
Off-the-shelf
Iossg
0.29
1.44
2/14
Table Lost Gain per target is minimal when choosing the highest models, ranked by average intertraining gain on General datasets. Results are reported when selecting top rank model best top rank models (@ Top column). Columns represent the aggregation of the lost gain: average, max and the number target datasets that lose least one point. Rows represent experiments, in in-house (with 22 models and 22 target datasets) or off-the-shelf (with base modes and 14 target datasets).
sets of
ONLI
16
MNU
MUNRC
DCP
96 (Aop Gain)
of symmetry of the matrix M, considers the rela- tions between and V. s = 1],1]in if is close to -1 it means that M is almost skew-symmetric (or anti-symmetric), if is almost 1, it means that M is almost symmetric. If it around zero, it means that it neither symmetrical skew-symmetrie.
s E []
100
150
200
300
400
800
Full
Target
Figure 8: Gains of QNLI from intertraining with differ- ent amount of training data (X-axis) and different base models (lines).
U-shape
We analyse how the intertraining gains change when more target data is available. We find that

---

I Scores
We report the source and target scores of training on MNLI datasets with 20 seeds. Target scores are the average over the General datasets. In Fig. 11, we present the results. Evidently the two are not correlated.
SS12
CB MNLI MURRC
0n (Aop, Gai)
1.1 Forgetting
If PT's success comes from honing the parameters, shifting from them and forgefting the knowledge gained during pretraining is inadvisable in general (Chen et al., 2020) and possibly for intertraining (Pruksachatkun et al., 2020). With more training data, comes also more forgetting. This may also explain why most source models have a negative gain and intertraining hurts. Despite that, we ob- served in $6.1 that more source data empowers intertraining and improves gains. Following that observation, we analyze the importance of forget- ting to the choice of source model.
-0
30
120
150
200
300
400
800
FJll
6a13
Targat size
Figure 9: Gains of SST2 from intertraining with differ- ent amount of training data (X-axis) and different base models (lines).
One common practice that causes forgetting is weight decay (Hanson and Pratt, 1988; Loshchilov and Hutter, 2019) a regularization term added the model updates. The decay term penalizes large weights, shrinking PT's large weights that are not necessary for the target objective.
For this experiment use the following exper- imental setup: ADAMW (Loshchilov and Hutter, 2019) optimizer with weight decay for decay and 0 (ADAM; Kingma and Ba, 2014) otherwise. L. regularization 0.1, results with other rates showed similar tendencies with effect size corre- sponding to the rate.
WIC
MUHIC
05 (Aog. Gaim)
We find intermediate models trained with weight decay to worse, but only if the pretraining did not include weight decay. Specifically, RoBERTa had weight decay during pretraining and T5 had ang not. We consider 9MNLI the average gain when source is MNLI and targets General with and without weight decay. With RoBERTa as PT, the gain with decay was slightly better than without, by 0.02 points, while with T5 decay lost 3.3 points. These changes were not reflected on the source task performance.
-2
30
320
Target
Figure 10: Gains of WIC from intertraining with differ- ent amount of training data (X-axis) and different base models (lines).
Second, limit the forgetting by adding reg- ularization forcing the model not be far from the pretrained model. This can be seen as the comple- ment of the previous method, rather than update to-

---

78.0
77.5
General Accuracy
77.0
76.5
76.0
75.5
75.0
74.5
86.6
86.8
87.0
87.2
87.4
MNLI Accuracy
Figure Source (MNLI) score against average score on General datasets after intertraining. Each point represents different MNLI intermediate nodel trained on different seed.

---

ward zero update toward the pretrained model. We find this method did change much for MNLI (- 0.28), but for models that hurt overall performance they prevented some performance loss, e.g. QQP (4.3). We note that while this regularization did not improve the top base model's gain, it did hurt the original finetuning the source task (-5.2 points on MNLI). We further address the effect of source task score base model quality in $6.2.
We also followed Kumar et al. (2022) method that should reduce forgetting with LP, but it had little effect.
All of the above findings imply that what deter- mines base model's effectiveness may be hidden in training hyperparameters. For example training on the same data and achieving similar results on the souree, may still get quite different results the target, depending on whether weight decay was used.
J Architectures
Figs. 12, 13 depict the gains of Tn-house mod- cls trained on General datasets over the General datasets. In Fig. 7 report the gains from train- ing on off-the-shelf T5 models. Interestingly, QQP which is known as bad source for RoBERTa, is among the best intermediate models for T5. Pre- sumably, this is due to different training, where T5 generates the paraphrases rather than picks between several ones.
On a similar note, many of the top base models train on con-saassiiication tasks, such as paraphras- ing and question answering. This implies that the model weights converge to something quite general, learning linguistic traits that are not all discarded during finetuning. We say those are linguistic, in the sense that the language, and perhaps common knowledge are what makes the datasets similar, the tasks are quite different.

---

9
626
D38
024
221
2:36
-333
2.8
11.45
9.11
7:14
020
3.00
0.09
9.27
0.88
927
0,00
433
025
4.10
11/22
000
427
0.29
-234
-223
22.00
535
2:38
233
417
8/2
21:43
4.18
1.00
0.17
333
312
181
0.18
3.35
6.11
0.00
0.90
1.8
001
6.00
21:43
0.00
4.00
233
0.2
1.16
680
0.59
0.39
3.1
0.10
15.48
6.66
606
1.83
20000
6000
0.00
4.17
121
1770
0/00
1.54
343
1.10
6.6
1.0
2:30
12:14
2.18
6:00
0.38
2.23
2.72
0:13
0.00
-5.40
-257
0.1
7:34
0.0
121
.00
0.66
461
Figure 12: BERT General sources and targets. The intertrain gain over the pretrained model for each source (row) and target (column) datasets.

---

6-a
3.1
021
222
100
600
121
<114
000
11:00
0.78
2:23
000
920
0004
3.11
244
467
8.43
8.23
-123
1:18
4.00
4.43
144
2.1
4.7
4.43
Seace
8.93
3.88
1.00
0.89
656
231
0.19
0.36
1.50
034
0.31
6:93
8.85
002
2.42
178
0.00
8.00
0.52
0.3
285
615
1.47
-144
080
1.00
0:00
6:00
044
247
11:00
2.3
2.13
2:10
036
1.81
004
0.31
1.41
0.6
174
2:22
0.00
6:00
223
180
010
0.00
330
1:0
623
0.16
2.23
2:33
1.0
373
-1.00
233
172
-323
171
Figure 13: T5 General sources and targets. The intertrain gain over the pretrained model for each source (row) and target (column) datasets.


Reviews:
[{"review_id": "iOt6flPBeZ", "reviewer": "Reviewer_H4LS", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Thorough experiments and interesting findings.", "reasons_to_reject": "See my comment above.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper empirically examines underlying factors that dominate the performance gain of \"intertraining\", where the pre-trained models are trained on a source dataset first before further fine-tuning on a target downstream dataset. The authors conduct extensive experiments and present several interesting findings: the contributions to the performance gain of the source dataset and the target dataset are nearly independent; the target dataset sensitivity could be efficiently estimated by linear probing; the source and target dataset sizes play an important role in performance gain. I think this work is pretty impressive as it provides a principled guideline for fine-tuning model selection via comprehensive experiments and studies. But I also find that there are some issues in the main content: the authors rank models by their averaged performance gain, which I believe could be problematic as the mean conceal extreme values in the distribution and is less informative in some comparisons, e.g., base models from MultiRC seem to results fewer extremely low results (<-5) than those from QQP, yet base models from QQP have a larger averaged performance gain and thus rank higher than those from MultiRC. In Section 5, the authors adopt a linear regression model to justify the independency between the source and target datasets. The authors report an MSE of 4.2, but I have no idea whether this error is acceptable for the independency to hold. I think the authors need to clarify this part, e.g., under this MSE, how much performance gain (in percentage) could be attained. In the end, the writing of this paper could be improved, as I see some typos in the main text.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 94: ’source’ -> ‘source’\nLine 109: max should not be in italics"}, "extra_text": {}}, {"review_id": "EuGjENjwv1", "reviewer": "Reviewer_seM1", "summary": "", "questions": "see above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.  This paper presents an empirical evaluation of the inter-training scheme. The authors propose several viewpoints that have been overlooked in previous studies and validate them through experiments. These viewpoints provide valuable guidance for future research work.\n2. The authors provide a comprehensive discussion on the potential impacts of the model, source dataset, and target dataset, which is highly convincing.\n3.  The paper is well-written.", "reasons_to_reject": "1. There are some formatting issues in this paper, such as the tables, which the authors should carefully address. However, these formatting issues do not affect the technical contributions of the paper.\n\n2. The experiments in this paper mainly focus on common datasets like GLUE benchmark, NLI, and Twitter. Including a more diverse range of test datasets would enhance the persuasiveness of this work. The authors could also consider whether the proposed method can be generalized to the visual or multimodal domains.\n\n3. Adding a figure for regression  in Section 5 would provide better clarity and aid in understanding.", "questions_to_authors": "see above.", "paper_topic_and_main_contributions": "Existing works find a fact that fine-tuned models on targer domain may provide a better starting point for a new finetuning process. Such a phenomenon dubed inter-training schema.\nIn this paper, author perform a systematic analysis of such scheme over a wide range of  tasks and get insight from these empirical studies. The research is begin with two observations:\n\n(1) some target datasets are intertraining-sensitive; \n\n(2) some base models are of high quality, finetuning on them provides consistent improvements on target datasets.\n\nBased on aforementationed observations,     a preferable base model can be selected independently of the target dataset. They substantiates their observation of independence by conducting experiments on a comprehensive set of target datasets and base models.\n\nIn addition to above findings, they analyze attributes of the source and target datasets that affect gains.                                            \nThey propose a practical approach to efficiently select models in a real-world setting, and make best models currently found  publicly available.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9TMxk0QeXZ", "reviewer": "Reviewer_hzbj", "summary": "", "questions": "- Is Pearson Correlation a suitable metric? The weighted Kendall's tau coefficient seems to be more robust (see: LogME: Practical Assessment of Pre-trained Models for Transfer Learning)\n- What is the computational efficiency of the proposed model ranking method (e.g. compared to brute-force fine-tuning?)", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and relatively easy to follow.\n- The authors conducted extensive experiments.\n- Open-sourcing the ranking method and model rankings will be valuable to the community.", "reasons_to_reject": "- The authors do not use any standard baselines for comparing against the proposed method for ranking models (e.g. LogME: Practical Assessment of Pre-trained Models for Transfer Learning and MODEL SPIDER: Learning to Rank Pre-Trained Models Efficiently).\n- The paper's overall contribution in relation to existing research remains ambiguous (see missing references for some examples). The authors' assertion that only a few works have explored intertraining/STILTs (lines 513-514) is inaccurate.", "questions_to_authors": "- Is Pearson Correlation a suitable metric? The weighted Kendall's tau coefficient seems to be more robust (see: LogME: Practical Assessment of Pre-trained Models for Transfer Learning)\n- What is the computational efficiency of the proposed model ranking method (e.g. compared to brute-force fine-tuning?)", "paper_topic_and_main_contributions": "The authors conduct an empirical analysis of intermediate training over 22 English classification tasks. They find that a model may exhibit strong performance even when the initial dataset that it was fine-tuned on is not semantically relevant to the target task. Based on their findings, they propose a ranking method for inferring such models in advance.\n\n**Post rebuttal changes**\n\n1. Increased Soundness score (+1)\n2. Increased Excitement score (+1)", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Clifton Poth, Jonas Pfeiffer, Andreas Rücklé, and Iryna Gurevych. 2021. What to pre-train on? Efficient intermediate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021).\n\n- E. Bassignana, M. Mu ̈ller-Eberstein, M. Zhang, and B. Plank, “Evidence > intuition: Transferability estimation for encoder selection,” in EMNLP.\n\n- Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer. Exploring and predicting transferability across NLP tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP).\n\n- Orion Weller, Kevin Seppi, and Matt Gardner. When to use multi-task learning vs intermediate fine-tuning for pre-trained encoder transfer learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 272–282, Dublin, Ireland, May 2022."}}, {"review_id": "ezrAL2AFIb", "reviewer": "Reviewer_mPxL", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper studies an interesting, and relatively unexplored problem.\n2. Solid experiments are conducted on a wide range of datasets.\n3. Clear takeaways, together with the model rankings, are provided for practitioners.", "reasons_to_reject": "One potential concern about this work is we are gradually shifting away from specialized NLP models and the pretraining/finetuning paradigm. Both the model size and corpus size are scaled up hundreds of times nowadays, compared to single classification datasets and Roberta. I am not sure whether the conclusion in this paper still holds after scaling up. For example, can we find a single dataset like \"MNLI\" to rank models for more complex tasks/larger models?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work provides a comprehensive empirical study on intertraining, in which some source datasets might provide a better starting point for other datasets to start fine-tuning. The authors show that (i) the gain of intertraining can be analyzed independently of the target datasets. (2) Better intertraining models can be relatively easy to find by linear probing on MNLI. More discussions on the dataset size and model architecture are also provided.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "There is one related paper I can recall:\nExploring and Predicting Transferability across NLP Tasks, Vu et al 2020"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the value of an intermediate model in an inter-training setting, which corresponds to a certain style of transfer learning, where the pretrained language model is being trained on different tasks before the task of interest. Experimental results on a wide variety of benchmarks show the importance of the size of target and source datasets, as well as the value of the base model. The paper ranks different models for tasks by the average performance gain under this inter-training setting.

Overall, the reviewers seem compelled by the detailed empiricism of the work and present suggestions for future research such as extensions to other domains or replication in a scaled up setting. Some issues pointed out regarding interpretability of MSE scores and aggregation of scores seem to be addressed by the authors’ response.

We hope the authors include the suggested changes and clarifications in the next iteration of the paper.
Decision: Accept-Main