Venue: EMNLP 2023

# Where to start? Analyzing the potential value of intermediate models

Leshem Choshen\({}^{\dagger}\), Elad Venezian; Shachar Don-Yehiya\({}^{\ddagger}\), Noam Slonim, Yoav Katz

###### Abstract

Previous studies observed that finetuned models may be better base models than the vanilla pretrained model. Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset. Here, we perform a systematic analysis of this _intertraining_ scheme, over a wide range of English classification tasks. Surprisingly, our analysis suggests that the potential intertwining gain can be analyzed _independently_ for the target dataset under consideration, and for a base model being considered as a starting point. Hence, a performant model is generally strong, even if its training data was not aligned with the target dataset. Furthermore, we leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings. Last, we release an updating ranking of best models in the HuggingFace hub per architecture.1

Footnote 1: [https://ibm.github.io/model-recycling/](https://ibm.github.io/model-recycling/)

+
Footnote †: footnotemark: [noams@iul.]

+
Footnote †: footnotemark: [noams@iul.]

## 1 Introduction

Finetuning pretrained models Devlin et al. (2019), is currently the standard and best approach for adjusting such models to perform a downstream task Chen et al. (2022). The resulting finetuned models are typically used for inferring the labels of new examples that are reminiscent of the data used for finetuning. However, it was shown Phang et al. (2018a) that finetuned models, trained on some _source dataset_, may represent better _base models_, namely a better starting point for a new finetuning process on a desired _target dataset_. This scheme, often referred to as _intertraining_, is the focus of the present work.

Given a target dataset, one may wonder what could be the intertwining gain, to determine whether it is worthwhile spending resources on selecting a base model. Assuming the potential gain is high, the following natural question is which base models are most promising, out of countless options available through hubs HuggingFace (e.g. Wolf et al., 2020). We propose pragmatic methods to answer both questions, supported by extensive experiments.

We begin with two observations: (i) some target datasets are _intertraining-sensitive_, i.e., have the potential to gain significantly from intertraining, while others are not, and are typically indifferent to the base model selection. Furthermore, revealing this property of the target dataset can be done efficiently, by examining the gains obtained when using a _single_ representative base model as a starting point; (ii) some base models are of high _quality_, i.e. finetuning on them provides consistent improvements on target datasets, but most base models are inferior and degrade performance. Furthermore, ranking base models by quality can be done on one target task - and efficiently, via _linear probing_, namely training only the base model classification head, over a single representative dataset.

Thus, we argue that a preferable base model can be **selected independently** of the target dataset. This is in contrast to the common perception (c.f. SS7) that the alignment of the target dataset and the source dataset - used to generate the base model - is a major factor in determining intertwraining success. We substantiate our observation of independence by conducting experiments on a comprehensive set of target datasets and base models, comprising models obtained under controlled conditions as well as models from HuggingFace. In addition to these findings, we analyze attributes of the source and target datasets that affect gains (SS6).

As some models are just better, not due to the choice of a current dataset, it makes sense to **rank the models once** and pick the best ones. But even

ranking a thousand models is costly. In SS8, we rely on our analysis to propose a practical approach to efficiently select models in a real-world setting. Moreover, instead of expecting others to rank the models, we share an updating site site featuring the best models currently found. So far, we tested over 2.5K models.

## 2 Preliminaries

In this paper, we use the following terminology. A _dataset_ is a set of examples and labels. Our goal is to maximize accuracy on the test of the _target dataset_, "target" for short. We discuss the difference between domain, task, and dataset in App. A.

A _pretrained (PT) model_ is a self-supervised model, e.g., RoBERTa Liu et al. (2019). A _fine-tuned model_ is a PT model that was further trained over some _source dataset_, denoted henceforth as "source". We assume access to many such models, e.g., through HuggingFace. A _base model_ can be either a PT model or a finetuned model. When finetuning over the target train data, one can start from any base model. _Intertraining_ refers to starting from a finetuned model as a base model, and in this case, we refer to this base model as an _intermediate model_. We denote by \(S^{t}_{m}\), the accuracy score obtained over the target test set, \(t\), after finetuning some base model \(m\) over the target train set. The intertwining _gain_ of model \(m\) w.r.t. using the PT model, is thus defined via \(gain\left(m,t\right)=s^{t}_{m}-s^{t}_{PT}\). Note that the gain may be negative. Given a set of intermediate models, \(M=m_{1}\dots m_{n}\), the intertraining _max-gain_ is defined as \(\max_{m\in M}\left(s^{t}_{m}-s^{t}_{PT}\right)\). Thus, theoretically, max-gain is achieved by finetuning all the available intermediate models and picking the one best performing on the target test set. To avoid overfitting and reduce costs, our goal is to find an intermediate model with a gain that is as close as possible to the max-gain, without explicitly fine-tuning all the intermediate models.

## 3 Experimental Setup

Our experimental setup is described next. The parameters for reproducibility are detailed in App. B.

### Dataset Groups

Our experiments are based on 3 groups of English text classification datasets described next (App. C). We focus on text classification for ease of evaluation, but assume the tasks are diverse enough for our conclusions to extend to other settings.

General.Containing GLUE Wang et al. (2018) and SuperGLUE classification datasets Wang et al. (2019), excluding test only and regression datasets. The datasets cover a wide range of tasks, from sentiment analysis through linguistic acceptability to natural language inference. It is the most commonly used benchmark in related work (SS7).

NLI.Containing Natural Language Inference and Entailment tasks. Datasets of this group all share the same task. There is some overlap between NLI and General; in Fig. 1 and mean comparison we subtract the overlapping datasets from General.

Twitter.Containing \(6\) Twitter datasets collected by TweetEval Barbieri et al. (2020). The tasks range from irony detection to emoji prediction. Datasets of this group all share the same domain.

### Models

Unless stated otherwise, our PT model of choice is RoBERTA Liu et al. (2019). We acquire intermediate models in two ways:

In-house.Obtained by finetuning the PT model over General, NLI, and Twitter dataset groups as the source datasets, with \(5\) seeds. Since we control the datasets and have knowledge about their features, this enables us to find relations between the dataset properties and the intermediate models generated from these datasets.

Off-the-shelf.66 RoBERTa models downloaded from HuggingFace (see App. SSE for more details). Since these models do not carry information excluding their names, this set allows us to validate our claims on a "real-world" model distribution.

### Models/Targets experiments

We test many intermediate models on various target datasets. We finetune each intermediate model and the PT on the target train, and report the intertraining gain over the target test. In the _In-house models/targets experiment_, all \(22\) datasets from the General, NLI, and Twitter groups act as both source and target and gains are average of \(5\) seeds. In the 

_Off-the-shelf models/targets experiment_, we download the \(66\) source models from Huggingface and test on the \(14\) General datasets as targets.

## 4 Results

Most models are worse than PT and about 1 in 6 are better, providing positive intertwraining gain. The in-house models/targets results are depicted in Fig. 1 and STDs and reference results in App. SSD. App. SSE reports results with off-the-shelf RoBERTa and T5 intermediate models.

The rows and columns in Fig. 1 are similarly ordered - first the General datasets, then the NLI datasets, and last the Twitter datasets. Loosely speaking, we do not recognize an approximate green block structure across the diagonal; namely, we do not observe clear intertraining gain for similar tasks (NLI); nor for similar domains (Twitter). However, some columns and some rows depict higher intertwining gains, while for others, the impact is minor. Taken together, these observations suggest little dependence between the source used to generate the intermediate model and the perfor

Figure 1: Results of in-house models/targets experiment. Columns correspond to target datasets and Rows correspond to intermediate models generated based on same datasets as source. The \(22\) datasets come from the General, NLI and Twitter groups. Each value indicates intertwining gain w.r.t. the PT model, averaged over 5 seeds. Sorted by group and source average gain (bottom row). Positive significant cells (>2 STD) are italicized.

 

mance over the target. This is in contrast to the common assumption (SS7) that the source and target need to be similar for intertwining to work. Next, we delve deeper into these observations.

### Target Sensitivity to Intertraining

Considering Fig. 1 columns, we notice that for some target datasets (e.g., ESNLI) intertraining makes little difference, while for others (e.g., COPA) the impact is quite significant. We argue that this target property can be predicted via an efficient and straightforward method. Specifically, the gains of one strong intermediate model should resemble the max-gains of a group of models. Indeed, MNLI highly correlates both with the max-gain of in-house models tested on the \(22\) targets in Fig. 1 (Spearman: \(0.89\), Pearson \(0.99\)) and off-the-shelf models tested on the \(14\) General targets (Spearman: \(0.90\), Pearson: \(0.94\), \(p<0.01\) for all). The replication on off-the-shelf models shows that this is a general result and not a reflection of MNLI being top of the in-house group. Overall, we find sensitivity is a characteristic of the target dataset separated from the source factor.

### Ranking Intermediate Models

Considering Fig. 1 rows, we notice that some intermediate models - e.g., MNLI - provide relatively high gains for many targets. We observe that this is a general phenomenon - stronger models are typically stronger for many datasets.

Identifying such models in advance could be practically valuable, since for a new target, one would consider only the highly ranked intermediate models (see SS8). In the following, we propose a simple yet efficient method to obtain such a static ranking - which is made once, without accounting for the target. A more comprehensive ranking alternative is described in App. SSF.

Given an intermediate model \(m\), we train a linear probe (LP) - i.e., train only the classification head - over MNLI, and consider the gain, denoted \(LP(m,MNLI)^{2}\). Evidently, this gain is a good proxy for the quality of \(m\). Specifically, let \(g_{m}^{avg}\) be the average gain of \(m\) over a set of target datasets. As depicted in Fig. 2, we observe that \(LP(m,MNLI)\) and \(g_{m}^{avg}\) are highly correlated in the in-house models/targets experiment (Spearman: \(0.46\), Pearson: \(0.78\), \(p<0.01\)) and the off-the-shelf models/targets experiment (Spearman: \(0.51\), Pearson: \(0.66\), \(p<0.01\)). In other words, if available intermediate models are ranked by LP on one dataset \(LP(m,MNLI)\), highly ranked models represent the most promising starting points on average. The high correlation means this connection holds not only for the top-ranked models, but throughout. Moreover, the replication on off-the-shelf models shows this is robust not only across targets but across sources.

To further support this claim, we use this ranking to find the most promising intermediate models. For each target \(t\), we consider the gain obtained by the top-ranked model and the max-gain obtained by one of the three top-ranked models, denoted \(g_{(1)}^{t}\) and \(g_{(3)}^{t}\), respectively. In comparison, we consider the max-gain obtained while considering _all_ available models, denoted \(g_{(max)}^{t}\). We further denote \(loss_{1}^{t}\equiv g_{(max)}^{t}-g_{(1)}^{t}\) and \(loss_{3}^{t}\equiv g_{max}^{t}-g_{(3)}^{t}\). In other words, \(loss_{1}^{t}\) represents the potential gain loss when using the top statically ranked model versus using the best available model for the target under consideration. Similarly, \(loss_{3}^{t}\) represents the lost gain when using the best model out of the top \(3\) ranked models versus using the best available model. Note, that even in this latter case, finding the best model involves finetuning only \(3\) models over the target train data, which is far less demanding compared to finetuning all available models.

In Table 1, we report statistics for \(loss_{1}^{t}\) and \(loss_{3}^{t}\) over the in-house and off-the-shelf models/targets experiments. Although the ranking is target-independent, the top \(3\) ranked models typ

Figure 2: Linear probing MNLI (x) is enough to predict finetuning gains (y) averaged over 14 General datasets. Each point corresponds to one off-the-shelf base model.

 

ically provide most of the potential intertwining gain. For example, instead of checking all \(66\) available models, using this simple strategy of checking the top \(3\) ranked models, each of the \(14\) targets lost at most \(1.62\) points.

## 5 Source and Target Interaction Analysis

Next, we analyse the interaction between the source dataset and the target dataset. Obviously, such interaction may impact the gain. On one extreme, since finetuning on the same data twice does not improve performance, intertraining is not valuable when the source and target data are identical. On another extreme, consider partitions of the same data. Obviously, training over half the data as the source would be beneficial for the other half, the target. Thus, we do not question that interaction may exist, but rather investigate how common and strong it is.

Interaction between dataset groups.Our General dataset group consists of diverse datasets, while the other groups share a domain (Twitter) or a task (NLI). We analyze whether datasets that share a domain or task with the target represent better source datasets than others.

Table 2 depicts the average gain of each source group vs. each target group. Comparing targets (table columns), we find the models trained on similar tasks, as a group, have a distinct behavior (ANOVA \(p<0.01\)). On average, using NLI intermediate models on NLI targets, yields a gain of \(0.63\), compared to a strong _negative_ gain when using intermediate models from other groups. Similarly, there is a same-group gain of 0.5 on Twitter.

Comparing sources (table rows), while NLI is best improved by NLI models, NLI models improve General datasets even more than NLI ones. Possibly, NLIs are just good intermediate models. Twitter models, however, do seem to improve Twitter targets more (ANOVA \(p<0.01\)), by 1 point. Hence, it seems the effects are mixed.

In summary, as a group, a similar source tends to yield greater gains than an unrelated source. However, in the rest of this section, we find this effect is of secondary importance to predicting model gains. A similar source may be more beneficial than a random one, but a well chosen source produces larger benefits regardless of similarity.

Symmetry as a similarity bound.We consider dataset similarity from another perspective. Similarity is a symmetric relation. Hence, if source-target similarity was a main reason for gains, we would expect that when the source \(A\) helps the target \(B\), the source \(B\) would help \(A\) as well. We assess the symmetry of the in-house models/targets experiment. We find that gains are far from symmetric (details in App. SSG). Thus, (symmetric) similarity seems like a weak explanation of which source data would help which target.3

Footnote 3: Even if similarity was a strong indicator we would not expect a full symmetry, as other factors such as sizes do matter.

Regression.As additional support for the relatively small impact of the source-target interaction, we show that the interaction is hardly needed to predict the scores. Specifically, a simple regressor can approximate the gain without considering such interactions. The regression fits 3 sets of coefficients. For each target two coefficients \(t_{i},t^{\prime}_{i}\) - which one may think of as capturing average gain and sensitivity to inter-training, per target; and for each base model \(b_{j}\) - which one may think of as capturing average inter-training gain, per base model. We then define \(\widehat{gain}(base_{i},target_{j})=(b_{i}+t_{j})\,t^{\prime}_{j}\) where \(i\) and \(j\) are the base model and target indices, respectively. Note that by construction, this regressor

\begin{table}
\begin{tabular}{l|c c c c}  & @Top & Avg. & Max & \multicolumn{2}{c}{\# targets} \\ Models & & & & s.t. \(loss^{t}_{n}>1\) \\ \hline \multirow{2}{*}{In-house} & \(loss^{t}_{1}\) & 0.37 & 2.11 & 3/22 \\  & \(loss^{t}_{3}\) & 0.2 & 1.15 & 1/22 \\ \hline \multirow{2}{*}{Off-the-shelf} & \(loss^{t}_{1}\) & 2.33 & 12.0 & 8/14 \\  & \(loss^{t}_{3}\) & 0.34 & 1.62 & 2/14 \\ \end{tabular}
\end{table}
Table 1: Lost Gain (\(loss_{n}\)) is small when choosing the \(n\) top-ranked models. Columns present the aggregation across target datasets of the lost gain: average, max and the number of targets that lose at least one point. Rows present either in-house (\(22\) models and \(22\) targets) or off-the-shelf (\(66\) models and \(14\) targets) experiments.

\begin{table}
\begin{tabular}{l l r r}  & General & NLI & Twitter \\ General & -0.37 & -2.68 & -0.54 \\ NLI & **1.26** & **0.63** & -0.03 \\ Twitter & -0.4 & -2.39 & **0.53** \\ \end{tabular}
\end{table}
Table 2: Intermediate models trained on sources from the same domain (Twitter) or task (NLI) as the target, yield greater gain. Numbers represent the average gain of intermediate models of a source group (rows) on a given target group (columns) .

 

has \(2N+n\) parameters, while trying to model \(N\hat{n}\) interactions; thus, it does not have enough degrees of freedom to explicitly model all source/target interactions. Still, it obtains satisfactory performance, as shown next. As a baseline, we fit the same regressor after randomly shuffling all the gains in the experiment. This shuffling ensures no information comes from the specific source and target, while maintaining the gain value distribution. We minimize Mean Squared Error (MSE) and fit with SGD until convergence.

Before we present the results, it would be beneficial to give some intuition on MSE. As MSE is the square of the error, an average MSE of 4, means that the average prediction was accurate up to 2 (\(\sqrt{4}\)) points on average or more accurate than that (as outliers have more effect on MSE than on the average error rate).

We obtain an MSE of \(4.2\) on the in-house models/targets experiment (3.3), versus \(9.6,\sigma=0.9\) when the gains are shuffled. Thus, knowing the source id and the target id provides significant information about the expected gain, with no need of keeping a designated parameter for each source-target combination. We further compare this regressor with two other regressors, one that considers only base model information (\(\widehat{gain}(base_{i},target_{j})=b_{i}\)) and one that considers only target related information (\(\widehat{gain}(base_{i},target_{j})=t_{j}\)). The MSE fit is \(10.4\) and \(8.2\), respectively, compared to \(10.8,\sigma=0.4\) on shuffled gains. This suggests both the source and the target impact the intertraining gain, with potentially somewhat stronger influence by the target.

In conclusion, our observations suggest that when considering intertwraining potential, loosely speaking it is enough to consider two separate issues - (i) choosing a base model; and (ii) determining the expected effect on the target.

## 6 Factors Contributing to Gains

So far, we showed the effects of the intermediate models are largely separate from those of the target. We proceed to examine how specific factors in each contribute to intertwining gains.

### Source and target sizes

Following the above observations, we ask what makes a base model good, or a target sensitive?

Figure 4: The average gain across targets decreases as the target training size increases.

Figure 3: For ’good’ sources the average gain increase as the source training size increases, while for ’bad’ sources it decreases.

 

We examine the effects of architecture (SS6.3) and source score (SS6.2), but start by examining the data sizes effect on the gain: First, we identify effects of the datasets sizes in controlled experiments. Next, we assess the extent to which the effect of dataset size is evident in previous experiments. For more related analysis we refer to Phang et al. (2018a); Pruksachatkun et al. (2020).

Effect of dataset size.We first control the source dataset train size. We create intermediate models on \(4\) source datasets - the top \(2\) and lowest \(2\) in-house models, according to the static ranking (SS4.2). For each, we limit the training to \(50-3200\) samples and use the General group datasets as targets. Evidently, for good sources (ANLI, MNLI), more training data yields greater gains (Fig. 3). However, additional data decreases performance for bad sources (MultiRC, QQP). We conclude that the amount of source data amplifies the effect determined by the type of data.

We experiment on the effect of target size, using General sources and General targets with train data of more than \(1600\) examples. We limit the target train sizes to between \(50\) - namely, few-shot setting - and \(1600\). As depicted in Fig. 4, the gain decreases as the target training size increases, implying larger potential for intertwining when the target training data is limited. Interestingly, for \(3\) targets we see positive gains on small datasets, which drops to negative gain as training data increases, and then seem to rise again towards zero. This 'U-shape' effect hints at two competing effects that should be better examined in future work (c.f. App. H).

Training size effects in practice.We examine whether the effect above is strong in comparison to other unknown factors. Considering the in-house models/targets experiment, the Pearson Correlation between source training size and source average gain is \(0.75\). Out of the \(5\) largest sources (ESNLI, MNLI, QQP, ANLI, and QNLI), \(3\) are also the source tasks with the highest average gain (MNLI, ANLI and ESNLI) and QQP is the source dataset with the second-lowest gain (negative). This is in line with our previous observation that additional source training data magnifies the positive or the negative intertraining effect of the source data.

We observe no significant correlation between target training size and target average gain, where the average is taken across sources. Still, targets with small training data seem to be more sensitive to intertwining. Specifically, the \(5\) targets with the smallest training data (CB, CoPA, WSC, WNLI, RTE) are also those for which we observe the maximal gain difference across all sources.

### Similar Source Score, Different Gain

One can expect that two models with similar performance on a source dataset would also have similar intertwraining gains. Our results suggest otherwise. We finetune \(20\) models over MNLI source and use them as intermediate models on the General target group. We compare the scores on the source test data to the average score obtained on the target datasets. Source task scores vary between \(86.5\) and \(87.5\) while General target average varies between \(74.5\) and \(79\), without correlation (c.f. App. I).

McCoy et al. (2019) found that finetuned models that show similar performance on their test data, still have different performance on out-of-domain challenge sets, suggesting the models learnt different generalizations schemes. Juneja et al. (2022a) suggested that those models converged to different basins in the loss space. They tagged models from one basin that tend to generalize better as _good_, and the rest as _bad_. We check whether good models are also better intermediate models for other tasks. We took their BERT models finetuned on MNLI as intermediate models - \(12\) good and \(12\) bad models - and used the General datasets as targets, finding that good models are indeed better for intertwining (\(3.65\) avg. gain for good vs. \(2.16\) for bad).

The differences discussed above are due to using different seeds. In App. I.1 we show that hyper-parameter selection can also impact the quality of an intermediate model, regardless of the score observed on the source test data.

In summary, similar training and/or similar performance on the source test, do not translate to similar intertwining gains on new target tasks.

### Effect of different architectures

We validate our main conclusions across different architectures. To that end, we repeat the in-house models/targets experiment with BERT Devlin et al. (2019) and T5 Raffel et al. (2020) architectures. (see full tables in App. J).

We start by showing that the loose source-target 

coupling holds across architectures. We then show that different source datasets rank differently across architecture, but that target sensitivity is similar.

To show the source-target independence, we repeat the regression fit (SS5). As before, the fit explains each architecture's gains much better than when data is shuffled (BERT MSE \(10.5\), random \(30.1,\sigma=4.17\); T5 MSE \(8.11\), random \(13.51,\sigma=1.5\)). Neither the average gain of intermediate models - trained over various sources, nor the average gain for target tasks, correlate across different architectures. However, the target sensitivity, measured by max-gain, is correlated across all architectures (pairwise Pearson \(0.6-0.94\), \(p<0.05\)). Thus, although the source-target decoupling and the target sensitivity are shared across architectures, a source dataset that produces high gains in one architecture might not do so in another.

A notable exception is the MNLI source dataset which achieves the highest gain in all three architectures. Possibly, some data sources provide a strong intermediate model regardless of PT, with MNLI as a prominent example.

## 7 Related Work

Various works use intertraining, often following the assumption of task alignment necessity [22, 23], namely, that the source acts as weak labeled data [21, 22]. While we consider intertwining through finetuning, adaptation to the target [21] or the domain [18] was also suggested. Such adaptation may be applied to any base model, and is complementary to the choice among base models. The need for alignment was also previously debated in the context of pretraining tasks [16, 23, 24, 25].

The properties of intertwraining were studied in other contexts. Phang et al. (2018) suggested the intertwraining scheme. Pruksachatkun et al. (2020) probed the linguistic knowledge changes after intertwraining, noting correlations to some target tasks, and hypothesized why some source tasks are good for intertwraining. Mosbach et al. (2020); Chang and Lu (2021) replicated the existence of good sources, but rejected the hypothesis. Others tried to find which tasks have an affinity to each other [23, 24, 25], or when to prefer multitask [23]. We study a much larger number of sources and targets, aiming to describe their natural distribution (c.f. SS9) and also find that while specific choice may help, given enough models, it is safe to identify the models that just excel generally.

Recent work focuses on fusing multiple base models rather than picking just one [26, 27, 28]. We expect our understanding to aid in choosing a subset of models to fuse as well.

Multi-task learning is another related field. It studies finetuning on different tasks at once [1, 2] and recently also a way to recycle models to do so was proposed [23]. In contrast to our analysis, similarity between tasks aids multi-task learning [2].

Deciding which information should accompany publications is an active research field, covering datasets [10, 1], human annotation sheets [21, 22], and models [16, 24]. Our work proposes to report upon sharing a model the aspects shown to affect its quality, such as \(LP\left(m,MNLI\right)\). For datasets, we propose to report intertwining sensitivity.

## 8 Practical recommendations

Based on the observations (SS4) and analysis (SS5), we propose a methodology for efficiently utilizing intertwraining in real-world settings. We suggest to collaboratively rank all available base models for intertwraining and to utilize this list whenever intertwraining is applied to a new target.

New model.When a new model becomes available, we encourage practitioners to assess and share its quality. This can be done efficiently by linear probing on MNLI (SS4.2) or comprehensively (App. SSF) by finetuning on various datasets.

We created statically ranked lists for RoBERTbase and T5-small in App. SSE. Moreover, we apply our methods to many architectures and 36 test sets in an updating site , reporting the best models 

New target.When considering intertraining on a new task, we recommend checking the target dataset sensitivity, and then choosing the base model. Since the model's rank hardly depends on the target dataset, we suggest using the static ranking. Specifically, we propose to finetune the top-ranked model, and compare the results to those obtained when finetuning the respective PT model. Assuming the gain justifies it, one should consider a few additional intermediate models, in descending order, according to the allocated resources.

## 9 Discussion

In SS8, we highlighted our practical recommendations for intertwining. Those, together with a systematic analysis of what affects intertwraining, cover our main contributions. We hope this analysis would also aid future work on interactions between datasets; intertwraining practices; and reusing fine-tuned models.

Our experiments mainly characterize what is probable rather than what is possible. We do not create specific models or aim to improve a specific task. Instead, we investigate what is likely to be found in typical practical settings. Accordingly, our findings are probabilistic in nature: Most models are not beneficial as intermediate models, but there are enough that are. Mostly, beneficial models are beneficial for many targets.

As a side effect, we do identify specific strong source models. MNLI was already considered a beneficial source dataset (Phang et al., 2018), a finding which we corroborate in a comprehensive manner. Furthermore, when considering off-the-shelf models we find models that outperform it (e.g., STS-B based for RoBERTa, and Quora for T5). To facilitate finding additional and better base models we will continuously report in the website website the best models per architecture.

## 10 Limitations

One obvious limitation is that our work is empirical in nature. As such, we report observations, sampled or common, with no theoretical guarantees, and one should recognize the existence of exceptions. Specifically, even though we have not observed it - there might exist target tasks that benefit greatly from a certain type of intermediate models; or intermediate models that help greatly in many targets while degrading performance in others.

Moreover, while testing 22 source datasets, many of which previously untested, we did not find a new top source for intertwraining. The best one we found for RoBERTa was already known to be good (MNLI; Phang et al., 2018; Pruksachatkun et al., 2020). With that, by checking dozens of off-the-shelf models, we did uncover new intermediate models that seem to outperform MNLI (e.g., STS-B for RoBERTa and QQP for T5 - c.f. App. SSE). More work is needed to assess the potential intertwining gain of the available datasets and models.

We ran thousands of finetuning experiments, spanning a vast number of tasks and base models. Thus, while we believe it is unlikely that reproducing our experiments will result in different outcomes, the large scale of our experiments places a heavy burden on trying to replicate our results. Moreover, the off-the-shelf models used in our experiments might not be hosted publicly in the future.

Another limitation is that we could not upload all of the models to a shared location. This project was very computationally demanding, but more than that, it was demanding in terms of disk space, hence we had to delete many models along the way.

Finally, for practical reasons, our results are limited to classification tasks in English. We hope that future work will aim to test our conclusions beyond this scope. Overall, in the space of classification, we see our results as robust, testing on 22 datasets (about double the amount of previous works (Pruksachatkun et al., 2020)). We hope the diversity of targets brings large enough differences between datasets that the results would hold in other scopes.

 



## References

* Abnar et al. (2021) Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. 2021. Exploring the limits of large scale pre-training. _ArXiv_, abs/2110.02095.
* Aghajanyan et al. (2021) Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning. _ArXiv_, abs/2101.11038.
* Aribandi et al. (2021) Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al. 2021. Ext5: Towards extreme multi-task scaling for transfer learning. _arXiv preprint arXiv:2111.10952_.
* Awathy et al. (2020) Parul Awathy, Bishwaranjan Bhattacharjee, John Kender, and Radu Florian. 2020. Predictive model selection for transfer learning in sequence labeling tasks. In _Proceedings of SustainNLP: Workshop on Simple and Efficient Natural Language Processing_, pages 113-118, Online. Association for Computational Linguistics.
* Bar-Haim et al. (2006) Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, and Bernardo Magnini. 2006. The second pascal recognising textual entailment challenge.
* Barbieri et al. (2020) Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1644-1650, Online. Association for Computational Linguistics.
* Barbieri et al. (2018) Francesco Barbieri, Jose Camacho-Collados, Francesco Ronzano, Luis Espinosa-Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. 2018. SemEval-2018 Task 2: Multilingual Emoj Prediction. In _Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018)_, New Orleans, LA, United States. Association for Computational Linguistics.
* Basile et al. (2019) Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In _Proceedings of the 13th International Workshop on Semantic Evaluation_, pages 54-63, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
* Bassignana et al. (2022a) Elisa Bassignana, Max Muller-Eberstein, Mike Zhang, and Barbara Plank. 2022a. Evidence > intuition: Transferability estimation for encoder selection. _ArXiv_, abs/2210.11255.
* Bassignana et al. (2022b) Elisa Bassignana, Max Muller-Eberstein, Mike Zhang, and Barbara Plank. 2022b. Evidence > intuition: Transferability estimation for encoder selection. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 4218-4227, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Bentivogli et al. (2009) Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The sixth pascal recognizing textual entailment challenge. In _TAC_.
* Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktaschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snl: Natural language inference with natural language explanations. In _NeurIPS_.
* Chang and Lu (2021) Ting-Yun Chang and Chi-Jen Lu. 2021. Rethinking why intermediate-task fine-tuning works. In _EMNLP_.
* Chen et al. (2022) Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. 2022. Revisiting parameter-efficient tuning: Are we really there yet? _ArXiv_, abs/2202.07962.
* Chen et al. (2020) Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. _arXiv preprint arXiv:2004.12651_.
* Chosen et al. (2022) Leshem Chosen, Elad Venezian, Noam Slonim, and Yoav Katz. 2022. Fusing finetuned models for better pretraining. _ArXiv_, abs/2204.03044.
* Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2924-2936, Minneapolis. Association for Computational Linguistics.
* Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In _MLCW_.
* de Marneffe et al. (2019) Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The CommitmentBank: Investigating projection in naturally occurring discourse. To appear in _Proceedings of Sim and Bedeung 23_. Data can be found at [https://github.com/mcdm/CommitmentBank/](https://github.com/mcdm/CommitmentBank/).
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_. Association for Computational Linguistics.
* Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_

Shachar Don-Yehiya, Leshem Choshen, and Omri Abend. 2022a. PreQuEL: Quality estimation of machine translation outputs in advance. pages 11170-11183.
* Don-Yehiya et al. (2022b) Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. 2022b. Cold fusion: Collaborative descent for distributed multitask finetuning. _arXiv preprint arXiv:2212.01378_.
* Ein-Dor et al. (2022) Lati Ein-Dor, Ilya Shnayderman, Artem Spector, Lena Dankin, Ranit Aharonov, and Noam Slonim. 2022. Fortunately, discourse markers can enhance language models for sentiment analysis.

* Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B. Dolan. 2007. The third pascal recognizing textual entailment challenge. In _ACL-PASCAL@ACL_.
* Gururangan et al. (2020) Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. _ArXiv_, abs/2004.10964.
* Hanson and Pratt (1988) Stephen Hanson and Lorien Pratt. 1988. Comparing biases for minimal network construction with backpropagation. _Advances in neural information processing systems_, 1.
* Holland et al. (2018) Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2018. The dataset nutrition label: A framework to drive higher data quality standards. _ArXiv_, abs/1805.03677.
* Juneja et al. (2022a) Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, Joao Sedoc, and Naomi Saphra. 2022a. Linear connectivity reveals generalization strategies. _arXiv preprint arXiv:2205.12411_.
* Juneja et al. (2022b) Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, Joao Sedoc, and Naomi Saphra. 2022b. Linear connectivity reveals generalization strategies. _ArXiv_, abs/2205.12411.
* Khashabi et al. (2018) Daniel Khashabi, Snigdha Chaturyedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_. Association for Computational Linguistics.
* Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint 1412.6980_.
* Krishna et al. (2021) Kundan Krishna, Jeffrey Bigham, and Zachary C Lipton. 2021. Does pretraining for summarization require knowledge transfer? _arXiv preprint arXiv:2109.04953_.
* Kumar et al. (2022) Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. 2022. Fine-tuning can distort pretrained features and underperform out-of-distribution. _ArXiv_, abs/2202.10054.
* Levesque et al. (2012) Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd schema challenge. In _Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning_.
* Levesque et al. (2011) Hector J. Levesque, Ernest Davis, and L. Morgenstern. 2011. The winograd schema challenge. In _KR_.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _ArXiv_, abs/1907.11692.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _ICLR_.
* Matena and Raffel (2021) Michael Matena and Colin Raffel. 2021. Merging models with fisher-weighted averaging. _arXiv preprint arXiv:2111.09832_.
* McCoy et al. (2019) R Thomas McCoy, Junghyun Min, and Tal Linzen. 2019. Berts of a feather do not generalize together: Large variability in generalization across models with similar test set performance. _arXiv preprint arXiv:1911.02969_.
* McMillan-Major et al. (2021) Angelina McMillan-Major, Salomey Osei, Juan Diego Rodriguez, Pawan Sasanka Ammanamanchi, Sebastian Gehrmann, and Yacine Jernite. 2021. Reusable templates and guides for documenting datasets and models for natural language processing and generation: A case study of the HuggingFace and GEM data and model cards. In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_, pages 121-135, Online. Association for Computational Linguistics.
* Mitchell et al. (2019) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucv Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. _Proceedings of the Conference on Fairness, Accountability, and Transparency_.
* Mohammad and Bravo-Marquez (2017) Saif M. Mohammad and Felipe Bravo-Marquez. 2017. Emotion intensities in tweets. In _Proceedings of the sixth joint conference on lexical and computational semantics ("Sem)_, Vancouver, Canada.
* Mohammad et al. (2016) Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In _Proceedings of the International Workshop on Semantic Evaluation_, SemEval '16, San Diego, California 

Marius Mosbach, Anna Khokhlova, Michael A. Hedderich, and Dietrich Klakow. 2020. On the interplay between fine-tuning and sentence-level probing for linguistic knowledge in pre-trained transformers. In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_, pages 68-82, Online. Association for Computational Linguistics.
* Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4885-4901, Online. Association for Computational Linguistics.
* Phang et al. (2018a) Jason Phang, Thibault Fevry, and Samuel R Bowman. 2018a. Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks. _arXiv preprint 1811.01088_.
* Phang et al. (2018b) Jason Phang, Thibault Fevry, and Samuel R. Bowman. 2018b. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. _ArXiv_abs/1811.01088.
* Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: The word-in-context dataset for evaluating context-sensitive meaning representations. In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_. Association for Computational Linguistics.
* Roth et al. (2021) Clifton Poth, Jonas Pfeiffer, Andreas Ruckle, and Iryna Gurevych. 2021. What to pre-train on? Efficient intermediate task selection. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 10585-10605, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Pruksachatkun et al. (2020) Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5231-5247, Online. Association for Computational Linguistics.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21:1-67.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Ram et al. (2021) Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. 2021. Few-shot question answering by pretraining span selection. _arXiv preprint arXiv:2101.00438_.
* Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In _2011 AAAI Spring Symposium Series_.
* Rosenthal et al. (2017) Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_, pages 502-518, Vancouver, Canada. Association for Computational Linguistics.
* Rothe et al. (2021) Sascha Rothe, Joshua Maynez, and Shashi Narayan. 2021. A thorough evaluation of task-specific pretraining for summarization. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 140-145, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Shimorina and Belz (2022) Anastasia Shimorina and Anya Belz. 2022. The human evaluation datasheet: A template for recording details of human evaluation experiments in NLP. In _Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval)_, pages 54-75, Dublin, Ireland. Association for Computational Linguistics.
* Shnarch et al. (2018) Eyal Shnarch, Carlos Alzate, Lena Dankin, Martin Gleizee, Yufang Hou, Leshem Choshen, Ranit Aharonov, and Noam Slonim. 2018. Will it blend? blending weak and strong labeled data in a neural network for argumentation mining. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 599-605, Melbourne, Australia. Association for Computational Linguistics.
* Shnarch et al. (2022) Eyal Shnarch, Ariel Gera, Alon Halfon, Lena Dankin, Leshem Choshen, Ranit Aharonov, and Noam Slonim. 2022. Cluster & tune: Boost cold start performance in text classification. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7639-7653, Dublin, Ireland. Association for Computational Linguistics.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
* Wae et al. (2018) Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2018. SemEval-2018 task 3: Irony detection in English tweets. In _Proceedings of The 12th International Workshop on Semantic Evaluation_, pages 39-50, New Orleans, Louisiana. Association for Computational Linguistics 

* Vu et al. (2020) Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Moht Iyyer. 2020. Exploring and predicting transferability across NLP tasks. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7882-7926, Online. Association for Computational Linguistics.
* Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In _NeurIPS_.
* Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
* Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _International Conference on Learning Representations_.
* Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. _Transactions of the Association for Computational Linguistics_, 7:625-641.
* Weller et al. (2022) Orion Weller, Kevin Seppi, and Matt Gardner. 2022. When to use multi-task learning vs intermediate fine-tuning for pre-trained encoder transfer learning. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 272-282, Dublin, Ireland. Association for Computational Linguistics.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Phu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariana Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Worstman et al. (2022) Mitchell Worstman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.
* Yadav et al. (2023) Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. 2023. Resolving interference when merging models. _ArXiv_, abs/2306.01708.
* Yu et al. (2021) Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. 2021. Fine-tuning pre-trained language model with weak supervision: A contrastive-regularized self-training approach. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1063-1077, Online. Association for Computational Linguistics.
* Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the Type and Target of Offensive Posts in Social Media. In _Proceedings of NAACL_.
* Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In _International Conference on Machine Learning_, pages 11328-11339. PMLR.

 

Task, Domain and Dataset

A _task_ is defined by the input and the output. The input in our context is a text instance. The output could be, e.g., positive/negative/neutral for a sentiment analysis task, entailed/not-entailed for a textual entailment task, etc.

A _domain_ is the type of text found in the examples, regardless of the labels. For example, a domain may be financial or comments for twitter.

A _dataset_ for our purpose is a set of examples and their labels, divided into train, dev, and test folds. Being such, each dataset has a domain (characterizing its examples) and a task (for which its labels are annotated). Hence, we consider a subset of a dataset as an another dataset. Note that in the literature those terms are often not well defined and may even be interchangeable Wang et al. (2019b).

## Appendix B Hyperparameters

For RoBERTa, we train for 10 epochs with linear learning rate 5e-5 with warm-up of 0.6% of training, batch size of 256, early stop epsilon 0.001 accuracy points, patience of \(20\times 50\times 256\) examples, validate every \(50\times 256\) examples, optimizer ADAMWLoshchilov and Hutter (2019), with weight decay 0.01 or 0. For BERT-base-uncased we use 2e-5 learning rate and never use decay. For T5 we use 1e-4 learning rate and train until early stopping occurs. We used A100 and V100 GPUs. Finetuning times vary, but all end within a couple of hours, most in less than an hour, some up to 8 hours.

## Appendix C Datasets used

All datasets could be downloaded from huggingface datasets. As we used groups of datasets we report here the full list of datasets they contain.

GeneralGLUE: CoLA Warstadt et al. (2019), SST2 Socher et al. (2013), MRPC Dolan and Brockett (2005), QQP data.quora.com/FirstQuora-Dataset-Release-Question-Pairs), MNLI Williams et al. (2018), QNLI Rajpurkar et al. (2016), RTE Dagan et al. (2005); Bar-Haim et al. (2006); Giampiccolo et al. (2007); Bentivogli et al. (2009), WNLI Levesque et al. (2011)

SuperGLUE: BoolQ Clark et al. (2019), CB de Marneffe et al. (2019), CoPA Roemmele et al. (2011), MULTIRC Khashabi et al. (2018), WIC Pilehvar and Camacho-Collados (2019), WSC Levesque et al. (2012)

NLIMNLI Williams et al. (2018), QNLI Rajpurkar et al. (2016), RTE Dagan et al. (2005); Barrain et al. (2006); Giampiccolo et al. (2007); Bentivogili et al. (2009), WNLI Levesque et al. (2011), ESNLI Camburu et al. (2018), adversarial NLI Nie et al. (2020).

TwitterEmoLMohammad and Bravo-Marquez (2017), Emoji Barbieri et al. (2018), Irony Van Hee et al. (2018), OffenseEval Zampieri et al. (2019), HatEval Basille et al. (2019) , Sentiment Analysis Rosenthal et al. (2017)

Whenever the test set is held out (such as is GLUE and SuperGLUE), we extracted 1K or 10% of the training examples as test set, the smaller. We did not experiment with the small Stance Mohammad et al. (2016) Twitter dataset originally found in TwectEvalBarbieri et al. (2020) to reduce noise. In the T5 experiment (SSE) we used stance datasets as well, to have a large group. For MNLI we use the mismatched validation set as a test and the matched as a validation set.

## Appendix D In-house models/targets additional information

We report in Table 3 the score of finetuning RoBERTa without intertraining.

We also report the standard deviation for each cell in the experiment, i.e., taking into account differences due to finetuning the intermediate model and target dataset in figure 5. For each seed, we finetune the PT over the source dataset to produce the base model, and use it to finetune the target task. It means that each seed utilises a different base model. Note that SS6.2 suggest different seeds may create base models with different quality. Note that to assess the variability of the averages reported in the main text (SS4) the Standard Error of the Mean is required, this is the STD divided by the square of the number of seeds \(SE=STD/\sqrt{5}\).

## Appendix E Models in the wild

### Models used

We collected manually 66 RoBERTa-base models. From their names most were finetuned from vanilla 

Figure 5: Standard deviation of in-house models/targets experiment. Rows correspond to intermediate models, generated based on 22 source datasets from the General, NLI and Twitter groups. Columns correspond to the same datasets, now being used as target datasets. Each value indicates standard deviation over 5 seeds.

 

RoBERTa, but a few were trained from scratch. They vary in hyperparameters and training details, just as one can expect training approaches to vary between different researchers. We supply the full list of models in Tables 4,5 for RoBERTa and T5 respectively.

### Results and discussion

We report the full results in Fig. 6. T5 models are reported in SS7 (off-the-shelf are on Twitter datasets only, as most of General and NLI were part of T5's pretraining, in-house use t5.1.1). Seemingly, some traits of the training that we did not account for are important. It is exemplified by models that are associated with the same datasets but differ in their gains. For example cross-encoder implementations outperform other models and sentence-transformers underperform them.

Notably, most models are not useful for inter-training. Still, many models do.

The best model on average is MUPPET Aghajanyan et al. (2021) which is a massive multitask learning approach. However, our results show that finetuning only on STS-B (rather than on 40 datasets) yielded similar results. We note that unlike MNLI which is known to be a good source Phang et al. (2018), STS-B was previously only considered as a target task only. The results suggest, it might be a good source.

We gathered the models by manually searching for 'RoBERTA-base' models, ignoring ones that were working on languages other than English. It is possible we have missed models that did not clearly state their architecture as part of the model name. We are already aware of such models, for which the PT is not deducible from their title, for example those lately released by Juneja et al. (2022b).

## Appendix F Rank by Average over Targets

In SS4.2 we show training one Linear Probe is enough to rank base models. Although tested on a large number of target datasets, presumably, this method does not always achieve accurate predictions. For example, the target domain might be so different that MNLI would not be relevant. As a more accurate alternative, one can use several datasets to provide a more reliable picture. We show that an average of finetuning gains over different datasets is a reliable way for choosing a base model. As in the simpler case of LP, this supports the decoupling.

We report in Table 6 the lost gain when choosing the highest models, ranked by average gain over the General group. This ranking method generalize well; The 1 or 3 best-ranked models are close to the best possible model overall for each target. For example, only 2 targets lose more than 1 point by choosing the top 3 models.

Practically, we suggest to rank either in this method or by LP. If some use one method and others choose another it might be hard to compare the two rankings. Thus, we report that in our experiments the best predictor of the average score by the LP score is \(g_{m}^{avg}=LP(m,MNLI)\cdot 0.0822-0.940\)

## Appendix G Symmetry metric

To measure symmetry of a matrix M, we decompose it to it its symmetrical and skew symmetrical parts: \(M=S+V\) where \(S=\frac{1}{2}(M+M^{T})\) and \(V=\frac{1}{2}(M-M^{T})\). S is symmetrical: \(S=S^{T}\) and V is skew-symmetrical: \(V=-V^{T}\). The measure

\begin{table}
\begin{tabular}{l c c} Dataset & Mean & Std \\ MultiRC & 61.07 & 2.01 \\ QQP & 90.92 & 0.29 \\ WSC & 63.46 & 0.00 \\ MRPC & 87.70 & 0.95 \\ CoLA & 83.11 & 1.34 \\ WIC & 65.55 & 2.32 \\ BoolQ & 77.09 & 3.19 \\ COPA & 49.00 & 4.90 \\ SST2 & 93.81 & 0.26 \\ CB & 70.36 & 3.11 \\ QNLI & 92.28 & 0.48 \\ WNLI & 56.34 & 0.00 \\ RTE & 72.42 & 0.93 \\ ESNLI & 91.05 & 0.18 \\ ANLI & 51.67 & 0.36 \\ MNLI & 87.07 & 0.23 \\ Twitter Hate & 52.30 & 1.03 \\ Twitter Offensive & 84.67 & 0.41 \\ Twitter Irony & 70.84 & 2.53 \\ Twitter Sentiment & 70.59 & 0.34 \\ Twitter Emoji & 46.32 & 0.56 \\ Twitter Emotion & 82.08 & 0.58 \\ \end{tabular}
\end{table}
Table 3: Scores of finetuning RoBERTa without intertwining.

 



\begin{table}
\begin{tabular}{l l l r r} \hline  & set & name & avg. gain over General & LP gain over MNLI \\ \hline
0 & imdb\_1 & aychang/roberta-base-imdb & -5.91 & -12.62 \\
1 & sentence\_4 & sentence-transformers/stbs-roberta-base & -5.84 & 2.59 \\
2 & models\_1 & textattack/roberta-base-ag-news & -5.73 & -17.09 \\
3 & twitter\_10 & lucoadromoneau/twitter-roberta-base-sentiment... & -5.71 & -9.76 \\
4 & sentence\_5 & sentence-transformers/robernka-base-mil-stsb-me... & -5.58 & 2.59 \\
5 & finance\_0 & hayunduo/roberta-base-stockwits-finetuned & -4.49 & -12.60 \\
6 & sentence\_2 & sentence-transformers/msmarco-roberta-base-v3 & -4.17 & -8.82 \\
7 & twitter\_8 & cardiffnlp/twitter-roberta-base-emotion & -4.17 & 1.00 \\
8 & sentence\_1 & sentence-transformers/roberta-base-mi-mean-to... & -4.07 & 5.47 \\
9 & qa\_3 & navracoberta-base-squad2 & -4.04 & 4.03 \\
10 & quora\_0 & cross-encoder/quora-roberta-base & -3.61 & 8.35 \\
11 & scratch\_0 & neoyipeng/twitter-roberta-base-sentiment-mlm+c... & -3.51 & -3.13 \\
12 & models\_5 & neojpersing/twitter-roberta-base-sentiment-mlm+c... & -3.51 & -3.13 \\
13 & sentence\_0 & sentence-transformers/nli-roberta-base & -3.50 & 5.47 \\
14 & models\_8 & cointegrated/roberta-base-formality & -3.29 & -4.96 \\
15 & legal\_1 & saibo/legal-roberta-base & -3.20 & -1.76 \\
16 & twitter\_12 & cardiffnlp/twitter-roberta-base-stanortion & -2.95 & -8.89 \\
17 & sslz\_0 & Bhumika/roberta-base-finetuned-sslz & -2.77 & -3.00 \\
18 & models\_14 & ccsrc/roberta-base-unigram-ternary & -2.73 & -8.39 \\
19 & models\_11 & mariagramdry/roberta-base-finetuned-sms-spam... & -2.67 & -5.90 \\
20 & qa\_1 & nlpoconnect/roberta-base-squad2-neq & -2.57 & 3.67 \\
21 & twitter\_13 & botido/twitter-roberta-base-finetuned-twitter.. & -2.47 & -3.12 \\
22 & models\_0 & textattack/roberta-base-ColA & -2.38 & -2.47 \\
23 & twitter\_6 & cardiffnlp/twitter-roberta-base-dec2021 & -2.31 & -11.45 \\
24 & twitter\_5 & cardiffnlp/twitter-roberta-base-mar2022 & -2.21 & -5.38 \\
25 & quora\_1 & navtezaqura-roberta-base-finetuned-ner & -2.13 & 8.35 \\
26 & models\_16 & hoanhko/noberta-base-finetuned-ner & -2.12 & -6.82 \\
27 & twitter\_3 & cardiffnlp/twitter-roberta-base-2021-124m & -2.05 & -5.30 \\
28 & twitter\_11 & cardiffnlp/twitter-roberta-base-stance-climate & -1.98 & -6.29 \\
29 & legal\_0 & akdeniz27/roberta-base-cuad & -1.90 & -8.57 \\
30 & models\_13 & cardinfnlp/twitter-roberta-base-steminist & -1.86 & -4.25 \\
31 & impl\_2 & aspmal?/roberta-base-imbtb & -1.86 & -3.50 \\
32 & models\_2 & ghanashymvattri/roberta-fake-news & -1.81 & -12.47 \\
33 & scratch\_1 & neocyipeng/twitter-roberta-base-sentiment-mlm-skep & -1.71 & -8.73 \\
34 & models\_12 & surrey-nlp/roberta-base-finetuned-abbr & -1.65 & -0.66 \\
35 & twitter\_4 & cardiiffnlp/twitter-roberta-base-empioi & -1.63 & -2.76 \\
36 & models\_4 & textattack/roberta-base-protein-tomatoes & -1.61 & 3.73 \\
37 & legal\_2 & marshmellow77/roberta-base-cuad & -1.53 & -8.57 \\
38 & legal\_3 & Rakiboyle-roberate-base-on-cuad & -1.33 & -2.48 \\
39 & twitter\_7 & cardiffnlp/twitter-roberta-base-sentiment & -1.24 & 3.03 \\
40 & twitter\_9 & cardiffnlp/twitter-roberta-base & -1.16 & -1.49 \\
41 & models\_15 & thatdaramebaazguyo/roberta-base-wikimovies & -1.11 & -1.76 \\
42 & finance\_1 & vanandhi/roberta-base-fiqla-film-sq-flit & -1.09 & -1.00 \\
43 & models\_3 & alleinalreviews & -1.01 & -1.17 \\
44 & models\_7 & princeton-nlp/sup-simscse-roberta-base & -0.99 & 4.26 \\
45 & mrpc\_1 & ji-xin/roberta & -0.94 & -1.67 \\
46 & twitter\_1 & cardiffnlp/twitter-roberta-base-offensive & -0.91 & -2.79 \\
47 & sst2\_1 & textattack/roberta-base-SST-2 & -0.84 & 3.83 \\
48 & sentence\_3 & sentence\_transomers/stsb-roberta-base-v2 & -0.80 & 1.68 \\
49 & twitter\_2 & cardiffnlp/twitter-roberta-base-irony & -0.69 & 0.46 \\
50 & imdb\_0 & textattack/roberta-base-imdb & -0.40 & -3.64 \\
51 & models\_6 & VictorSanhy/roberta-base-finetuned-yelp-polarity & -0.25 & -0.66 \\
52 & models\_10 & gargan/roberta-base-crest & -0.12 & -4.21 \\
53 & twitter\_0 & bhadensk-swami/roberta-base-emotion & -0.05 & -4.61 \\
54 & qq\_4 & shahrukk0/roberta-base-boolq & 0.25 & 10.42 \\
55 & mrpcc\_0 & textattack/roberta-base-MRPC & 0.39 & 7.27 \\
56 & nil\_3 & textattack/roberta-base-RTE & 0.42 & 14.82 \\
57 & nil\_2 & mujeensing/roberta-base\_mnli\_bc & 0.48 & 23.43 \\
58 & aa\_0 & deepest/roberta-base-squad2-covid & 0.50 & -1.11 \\
59 & aa\_2 & csarron/roberta-base-squad-v1 & 0.99 & 3.38 \\
60 & stsb\_0 & textattack/roberta-base-STS-B & 1.05 & 14.66 \\
61 & models\_9 & textattack/roberta-base-QNLI & 1.10 & 6.49 \\
62 & nil\_0 & textattack/roberta-base-MNLI & 2.09 & 34.39 \\
63 & ni\_1 & cross-encoder/nil-roberta-base & 2.77 & 34.09 \\
64 & stsb\_1 & cross-encoder/stsb-roberta-base & 2.82 & 30.19 \\
65 & multitask\_0 & facebook/mpuppet-roberta-base & 3.00 & 31.58 \\ \hline \end{tabular}
\end{table}
Table 4: RoBERTa models we used, collected from Hugging Face models hub. Models sorted by average gain over the General targets.



* [105] M. C. Dickson, J. R. Friedman, and A. K. S. Leibler. The role of the global model for the estimation of the age-dependent mortality in a cohort. In _Proceedings of the 27th International Conference on Machine Learning Research_, pages 349-368. Springer, Berlin, Heidelberg, Germany, July 2007.
* [106] P. F. G. Wilson. _The age-independent mortality analysis: an application to the age-independent mortality analysis_. PhD thesis, University of California, Berkeley, CA, USA, June 2007.
* [107] T. W. H. Lee and B. L. Tibshirani. Regression models with variable coefficients. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [108] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [109] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [110] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [111] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [112] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [113] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [114] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [115] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [116] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [117] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [118] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [119] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [120] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [121] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [122] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [123] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [124] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [125] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [126] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [127] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [128] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [129] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [130] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [131] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [132] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [133] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [134] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [135] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [136] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [137] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [138] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [139] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [140] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [141] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [142] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [143] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [144] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [145] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [146] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [147] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age-independent mortality analysis. In _Proceedings of the 2nd international conference on machine learning research_, pages 113-126. ACM, New York, NY, USA, June 2007.
* [148] Y. Li and X. Z. Liu. An empirical study of the age-independent mortality analysis using the age- 

* [26] M. Cacciari, G. P. Salambi, and A. Zisserman. The automated computation of the jet energy distribution in the CMS detector at LHC", _JHEP_**07** (2014) 039, doi:10.1088/1126-6708/2014/07/039, arXiv:1405.0523.
* [27] J. Alwall et al., "MadGraph 5: going beyond", _Nucl. Instrum. Meth. A_**506** (2003) 250, doi:10.1016/S0168-9002(03)01368-8.
* [28] S. Alioli, F. Petriello, and R. Rattazzi. The PDF4LHC Working Group for Higgs boson production at hadron colliders", _Eur. Phys. J. C_**71** (2015) 155, doi:10.1140/epjc/s10052-015-3998-x, arXiv:1502.0018.
* [29] T. Sjostrand, S. Mrenna, and P. Skands. A brief introduction to PYTHIA 8.8. Monashr. Nucl. Inst. Meth. A_**506** (2006) 159, doi:10.1016/S0168-9002(06)0048-8.
* [30] T. Sjostrand, S. Mrenna, and P. Skands. A brief description of PYTHIA 8.8. Comput. Phys. Commun. **191** (2015) 159, doi:10.1016/j.cpc.2015.01.024, arXiv:1410.6020.
* [31] GEANT4 Collaboration, "GEANT4--a simulation toolkit", _Nucl. Instrum. Meth. A_**506** (2003) 250, doi:10.1016/S0168-9002(03)01368-8.
* [32] E. Barberio, I. D'Agostini, and A. Zisserman. The PDF4LHC Working Group for Higgs Boson Production at Hadron Colliders", _Phys. Rev. Lett._**101** (2010) 011801, doi:10.1103/PhysRevLett.101.011801, arXiv:1007.0701.
* [33] L. Randall and R. Sundrum. _Chiral symmetry breaking in extra dimensions_. PhD thesis, Stanford University, 1998.
* [34] W. Buchmuller, K. Lane, and H. Georgi. _Phenomenology of the Standard Model_. Cambridge University Press, 1999.
* [35] B. Holdom, R. Hempfling, and R. Sundrum. _The Anatomy of the Standard Model_. Cambridge University Press, 1999.

[MISSING_PAGE_POST]

 

of symmetry \(s\) of the matrix M, considers the relations between \(S\) and \(V\), \(s=(|S|-|V|)/(|S|+|V|)\), \(s\in[-1,1]\) if \(s\) is close to -1 it means that \(M\) is almost skew-symmetric (or anti-symmetric), if it is almost 1, it means that M is almost symmetric. If it around zero, it means that it neither symmetrical or skew-symmetric.

## Appendix H U-shape

We analyse how the intertwraining gains change when more target data is available. We find that while intertraining often improves results for small data size, the effect is decreasing with the size. Surprisingly, the decrease drops below zero and at some size increases again. This suggests an unexplained underlying behaviour, presumably of two competing effects, one that decreases gains with size and one that improves them (in general or towards 0). We produce three examples of the U-shape in Figures 8, 9, 10.

 



## Appendix I Scores

We report the source and target scores of training on MNLI datasets with 20 seeds. Target scores are the average over the General datasets. In Fig. 11, we present the results. Evidently the two are not correlated.

### Forgetting

If PT's success comes from honing the parameters, shifting from them and _forgetting_ the knowledge gained during pretraining is inadvisable in general Chen et al. (2020) and possibly for intertwining Pruksachatkun et al. (2020). With more training data, comes also more forgetting. This may also explain why most source models have a negative gain and intertraining hurts. Despite that, we observed in SS6.1 that more source data empowers intertraining and improves gains. Following that observation, we analyze the importance of forgetting to the choice of a source model.

One common practice that causes forgetting is weight decay Hanson and Pratt (1988); Loshchilov and Hutter (2019) - a regularization term added to the model updates. The decay term penalizes large weights, shrinking PT's large weights that are not necessary for the target objective.

For this experiment we use the following experimental setup: ADAMW Loshchilov and Hutter (2019) optimizer with weight decay 1 for decay and 0 (ADAM Kingma and Ba (2014) otherwise. L2 regularization is 0.1, results with other rates showed similar tendencies with effect size corresponding to the rate.

We find intermediate models trained with weight decay to be worse, but only if the pretraining did not include weight decay. Specifically, RoBERTa had weight decay during pretraining and T5 had not. We consider \(g^{avg}_{MNLI}\) the average gain when source is MNLI and targets are General with and without weight decay. With RoBERTa as PT, the gain with decay was slightly better than without, by 0.02 points, while with T5 decay lost 3.3 points. These changes were not reflected on the source task performance.

Second, we limit the forgetting by adding a regularization forcing the model not to be far from the pretrained model. This can be seen as the complement of the previous method, rather than update to

Figure 10: Gains of WIC from intertwraining with different amount of training data (X-axis) and different base models (lines).

Figure 9: Gains of SST2 from intertwaining with different amount of training data (X-axis) and different base models (lines).



Figure 11: Source (MNLI) score against average score on General datasets after intertraining. Each point represents a different MNLI intermediate model trained on a different seed.

 

ward zero update toward the pretrained model. We find this method did not change much for MNLI (-0.28), but for models that hurt overall performance they prevented some performance loss, e.g. QQP (4.3). We note that while this regularization did not improve the top base model's gain, it did hurt the original finetuning on the source task (-5.2 points on MNLI). We further address the effect of source task score on base model quality in SS6.2.

We also followed Kumar et al. (2022) method that should reduce forgetting with LP, but it had little effect.

All of the above findings imply that what determines a base model's effectiveness may be hidden in training hyperparameters. For example training on the same data and achieving similar results on the source, may still get quite different results on the target, depending on whether weight decay was used.

## Appendix J Architectures

Figs. 12, 13 depict the gains of In-house models trained on General datasets over the General datasets. In Fig. 7 we report the gains from training on off-the-shelf T5 models. Interestingly, QQP which is known as a bad source for RoBERTa, is among the best intermediate models for T5. Presumably, this is due to different training, where T5 generates the paraphrases rather than picks between several ones.

On a similar note, many of the top base models train on non-classification tasks, such as paraphrasing and question answering. This implies that the model weights converge to something quite general, learning linguistic traits that are not all discarded during finetuning. We say those are linguistic, in the sense that the language, and perhaps common knowledge are what makes the datasets similar, the tasks are quite different 

* [105] M. C. Bader, J. D. Friedman, and A. G. Hinton. Learning to predict the presence of a large number of clusters. In _Proceedings of the 23rd International Conference on Machine Learning_, pages 47-68, 2009.
* [106] R. H. Sohn, P. K. Lee, and T. L. Jordan. The role of missing data in the context of the model. _Journal of machine learning research_, 1:1-10, 2009.
* [107] E. W. Koltun. _The Elements of Statistical Learning Theory_. Springer Science & Business Media, 2009.
* [108] F. V. LeCun, Y. Bengio, and I. Zisserman. Visualizing features from tiny images. _arXiv preprint arXiv:1905.01880_, 2019.

[MISSING_PAGE_POST]

 

Figure 13: T5 General sources and targets. The intertrain gain over the pretrained model for each source (row) and target (column) datasets.