Recyclable Tuning for Continual Pre-training
Year: 2023

Abstract:
Continual pre-training is the paradigm where pre-trained language models (PLMs) continually acquire fresh knowledge from growing data and gradually get upgraded. Before an upgraded PLM is released, we may have tuned the original PLM for various tasks and stored the adapted weights. However, when tuning the upgraded PLM, these outdated adapted weights will typically be ignored and discarded, causing a potential waste of resources. We bring this issue to the forefront and contend that proper algorithms for recycling outdated adapted weights should be developed. To this end, we formulate the task of recyclable tuning for continual pre-training. In pilot studies, we find that after continual pre-training, the upgraded PLM remains compatible with the outdated adapted weights to some extent. Motivated by this finding, we analyze the connection between continually pre-trained PLMs from two novel aspects, i.e., mode connectivity, and functional similarity. Based on the corresponding findings, we propose both an initializationbased method and a distillation-based method for our task. We demonstrate their feasibility in improving the convergence and performance for tuning the upgraded PLM. We also show that both methods can be combined to achieve better performance. The source codes are publicly available at https://github.com/ thunlp/RecyclableTuning.

heading: 1 Introduction
text: The emergence of pre-trained language models (PLMs) has revolutionized the entire field of natural language processing (NLP) (Bommasani et al., 2021). Through downstream adaptation, PLMs effectively stimulate the knowledge acquired during pre-training and achieve remarkable success in various downstream tasks (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020). Such adaptation
∗Indicates equal contribution. †Corresponding author.
can be achieved by either full-parameter fine-tuning or parameter-efficient tuning (Houlsby et al., 2019), and the latter enables learning lightweight adapted modules for downstream tasks. Currently, a de facto paradigm for handling NLP tasks has been formed, dividing practitioners into two groups: (1) upstream suppliers, who pre-train PLMs on taskagnostic data and release them on public platforms, e.g., HuggingFace (Wolf et al., 2020), and (2) downstream consumers, who download the PLM and conduct personalized adaptation using taskspecific data. The corresponding adapted weights might then be shared with third parties via platforms such as AdapterHub (Pfeiffer et al., 2020).
In real-world scenarios, PLMs may constantly get upgraded and released by the supplier. Correspondingly, the customer-side compatible update of adapted weights becomes necessary. Continual pre-training (Qin et al., 2022c) is a typical scenario where PLMs continually acquire fresh knowledge from growing data and gradually get upgraded. Before an upgraded PLM is released, consumers may have tuned the original PLM for various tasks and stored the adapted weights. However, when tuning the upgraded PLM, these outdated adapted weights will typically be ignored and discarded. This can lead to a loss of knowledge about downstream tasks encapsulated in the outdated weights, as well as a potential waste of computational resources. In this
11403
paper, we bring this issue to the forefront and argue that proper algorithms for recycling outdated adapted weights should be developed. To this end, we formulate the task of recyclable tuning for continual pre-training, which is illustrated in Figure 1.
Due to the parameter change during continual pre-training, one potential concern for recycling outdated adapted weights is their mismatch with the upgraded PLM. However, our pilot studies reveal that directly applying the outdated weights to the upgraded PLM yields substantial performance improvements as compared to zero-shot inference of the PLM. This shows that the upgraded PLM remains compatible with the outdated weights to some extent, indicating a close connection between continually pre-trained PLMs. Intuitively, such a connection provides a strong basis for our assertion that outdated weights are recyclable and useful.
To uncover hints for solving our task, we further investigate such a connection from two aspects: (1) linear mode connectivity (Qin et al., 2022b). We demonstrate that after adapting both the upgraded PLM and the original PLM to the same task, linearly interpolating the parameters of both adapted models could produce a series of checkpoints with high task performance (low loss). Such a property indicates a close parametric connection of both PLMs in the loss landscape; (2) functional similarity. After adapting both PLMs to the same task, we observe that their corresponding attention heads exhibit similar patterns given the same input. Such representational proximity implies that both PLMs own similar functionalities during text processing.
Both analyses above demonstrate the close connections between continually pre-trained PLMs. Based on the corresponding findings, we propose two methods for recyclable tuning:
(1) Initialization-based method, which leverages the adapted weights of the original PLM as the initialization for the upgraded PLM. This method is motivated by their close parametric connection in the loss landscape. We demonstrate that for a target task, initializing the tunable parameters with the outdated weights from a similar source task could accelerate the convergence and improve the training efficiency, compared to using random initialization. In addition, after sufficient training, this method generally improves the final performance. We also observe that the benefits of this method in terms of convergence and performance are greater when the source and target tasks are more similar.
(2) Distillation-based method, which distills the knowledge stored in outdated weights for tuning the upgraded PLM. We demonstrate that knowledge distillation can effectively facilitate knowledge transfer between continually pre-trained PLMs. Using only a small number of labeled examples, the upgraded PLM can outperform the original PLM when trained with far more examples. We also show that both initialization-based and distillation-based methods can be combined to further improve the performance. This means knowledge transfer through parameter space and model outputs are complementary to each other.
In a nutshell, these results highlight the practical benefits of recyclable tuning and point to an important future direction in sustainable NLP.

heading: 2 Related Work
text: Continual Pre-training. Conventionally, PLMs are trained on static data, ignoring that streaming data from various sources could continually grow. Continual pre-training requires PLMs to accumulate new knowledge in a continual manner (Gururangan et al., 2020), meanwhile alleviating the catastrophic forgetting problem. Prior works in this field focus on building benchmarks and analyses (Jang et al., 2021, 2022). Later works explored the applicability of traditional continual learning algorithms under this setting (Jin et al., 2022; Wu et al., 2021). Recent efforts were also spent on continual pre-training in a computationally efficient way (Qin et al., 2022c).
Previous works focus on improving the capabilities of PLMs during pre-training from the standpoint of upstream suppliers. Instead, we shift the focus to downstream adaptation from the perspective of customers. We highlight a previously overlooked issue of the incompatibility between upgraded PLMs and the existing adapted weights. For the first time, we examine the connections between continually pre-trained models and demonstrate the potential benefits of recycling outdated weights.
Knowledge Transfer for PLMs. Transfer learning for PLMs has gained increasing attention recently. Some works study task-level transferability for an individual PLM and find that fine-tuning on certain source tasks conduces to the performance on similar target tasks (Vu et al., 2020; Poth et al., 2021; Aghajanyan et al., 2021). Differently, we also study cross-task knowledge transfer for two different PLMs under the continual pre-training
scenario (§ 5.1). Besides, researchers also investigate cross-model knowledge transfer. They try to recycle lightweight adapted weights of the same task between two independently pre-trained PLMs, e.g., PLMs with distinct data (Su et al., 2022). As we would show later, unlike independently trained PLMs, continually pre-trained PLMs are guaranteed close connections. This distinction determines our setting is unique to previous works and may require different solutions.

heading: 3 Problem Formulation
text: Continual Pre-training. Following Qin et al. (2022c), we simulate the scenario where new data from 4 domains is gathered sequentially, i.e., biomedical papers (BIO, D1) (Lo et al., 2020), amazon reviews (REV, D2) (He and McAuley, 2016), computer science papers (CS, D3) (Lo et al., 2020), and news articles (NS, D4) (Zellers et al., 2019). Starting from the official RoBERTaBASE (Liu et al., 2019) (denoted as M0), we continually pre-train M0 on 4 domains. For each domain, we set the pre-training steps to 12.5k and the batch size to 2048. Denote Mi as the PLM that finishes training on Di, and Mi(t) as the PLM that starts from Mi−1 and is trained on Di for t steps. We assume the suppliers only release the PLM that finishes training on each domain, i.e., {M1, · · · ,M4} are developed and released. The pre-training details are described in appendix D.1.
Downstream Adaptation. At the same time, we have a set of downstream tasks to handle. To adapt Mi (0 ≤ i ≤ 4) towards a task Tj , we conduct supervised training using the loss function LTj . Denote the pre-trained weights of Mi as θ0i , we obtain its adapted weights ∆Tji for Tj after training. By assembling both θ0i and ∆ Tj i , the resultant model θTji = θ 0 i ⊕∆ Tj i can be deployed to handle Tj . Throughout this paper, we consider two tuning methods: full-parameter fine-tuning and a representative parameter-efficient tuning method, adapter tuning (Houlsby et al., 2019) (see appendix A.1 for more backgrounds). For the former, we have |∆Tji | = |θ0i |; while for the latter, |∆ Tj i | ≪ |θ0i |, where | · | denotes the number of parameters.
Recyclable Tuning. Before the release of an upgraded PLM Mi′ (i<i′), we have obtained adapted weights ∆Tji of an old PLM Mi for task Tj . Recyclable tuning aims at transferring the knowledge of
∆ Tj i to assist tuning Mi′ (i.e., learning new weights ∆ Tj i′ ). We denote the above process as ∆ Tj i →∆ Tj i′ . Intuitively, ∆Tji encapsulates abundant knowledge about the task Tj , which should benefit learning ∆
Tj i′ if exploited properly. Such benefits may include improving training efficiency or performance. To gain insights of solving the task, we first conduct a series of empirical analyses in § 4 to understand the connections among Mi, Mi′ , ∆Tji , and ∆ Tj i′ .

heading: 4 Empirical Analysis
text: We first investigate the compatibility of outdated weights and the upgraded PLM (§ 4.1), then we explore the (1) parametric connections and (2) representational connections of continually pre-trained PLMs from two aspects: (1) linear mode connectivity (§ 4.2) and (2) functional similarity (§ 4.3). The implementation details are left in appendix D.2.

heading: 4.1 Model Compatibility Analysis
text: We explore to what extent the outdated weights are compatible with the upgraded PLM and how this compatibility changes during continual pretraining. Specifically, we directly apply outdated weights to the upgraded PLM and record the performance variation during continual pre-training.
Settings. We first investigate the process when upgrading M0 to M1 on the BIO domain (D1). For downstream evaluation, we choose two classification tasks: CHEMPROT (Kringelum et al., 2016), which is a relevant downstream task to the BIO domain, and MNLI (Williams et al., 2018). Denote the model continually pre-trained on D1 for t steps as M1(t), its pre-trained weights as θ01(t), and the adapted weights of M0 for the downstream task as ∆T0 . We directly apply ∆ T 0 to the upgraded PLM M1(t), i.e., θ01(t) ⊕ ∆T0 , and evaluate the performance on the test set of the downstream task. In experiments, t is selected from 1.25k to 12.5k with an interval of 1.25k. We also report M1(t)’s zero-shot inference performance by testing θ01(t).
Results. From the results in Figure 2 (a, b), we observe that for both adapter and fine-tuning: (1) with t increasing, the performance of θ01(t)⊕∆T0 drops quickly at first. This means that ∆T0 becomes outdated shortly after the backbone model M1(t) changes. (2) After sufficient pre-training steps, the performance converges to a plateau which is still much higher than the zero-shot inference performance of M1(t). This implies that continually
pre-trained PLMs are intrinsically connected with their “ancestors”, otherwise the ancestor’s adapted weights ∆T0 would not improve the performance of its offspring M1(t).
Extension to Multiple Domains. Next, we extend the above experiments to 4 sequentially released PLMs as mentioned in § 3 by directly applying ∆T0 to {M1, · · · ,M4}. We derive from Figure 2 (c, d) that: (1) applying outdated weights consistently performs better than zero-shot inference even if the backbone PLM is trained over multiple domains; (2) the performance of M4 is the best among {M1, · · · ,M4} though M4 is trained for the longest time. This may be because the NS domain (D4) is the most similar one to M0’s pre-training data (Gururangan et al., 2020), and continual pre-training on a similar domain of the original PLM mitigates the incompatibility.

heading: 4.2 Linear Mode Connectivity Analysis
text: Backgrounds. Linear mode connectivity measures whether two sets of model weights can be connected via a linear parametric path, along which the performance (loss) of the downstream task remains high (low) (Frankle et al., 2020). In other words, it tests whether linear interpolations of two model weights perform comparably to both endpoints. If this property holds, then both model weights probably lie in the same loss basin, which indicates a close connection between them in the parameter space (Qin et al., 2022b). For more de-
tailed backgrounds, please refer to appendix A.2.
Settings. Following most of the settings in § 4.1, we adapt both M0 and M1(t) towards the task CHEMPROT and obtain the weights θT0 and θ T 1 (t), where θT0 = θ 0 0 ⊕∆T0 and θT1 (t) = θ01(t)⊕∆T1 (t). Then we linearly interpolate both θT0 and θ T 1 (t) as:
θ(µ) = (1− µ)θT0 + µθT1 (t), (1)
where µ ∈ (0, 1). In experiments, we evaluate the performance of 25 evenly distributed interpolations and two endpoints (i.e., µ = 0 and µ = 1). If there does not exist a significant performance drop along the linear path, we deem both endpoints linearly mode connected. We choose M1(t) that is continually pre-trained for {2.5, 5.0, 7.5, 10.0, 12.5}k steps and evaluate mode connectivity for each M1(t) and M0. In addition, we pre-train a new RoBERTaBASE (dubbed as MIND) from scratch (details in appendix D.1) and test its connectivity with M0, i.e., θ(µ) = (1−µ)θT0 +µθTIND. In this way, we can compare the difference between continually pre-trained models (M0 and M1(t)) and independently pre-trained models (M0 and MIND).
Results. We illustrate the performance of the interpolations and two endpoints in Figure 3, from which we conclude that: (1) for continually pretrained PLMs, although there exists a small performance drop in the midpoint, the interpolations generally achieve comparable performance to endpoints; (2) the connectivity does not vary much with t increasing, which means within a reasonable range, the connectivity is not sensitive to longer pre-training; (3) while for independently trained PLMs, the performance drops significantly in the middle, which means the adapted weights of these PLMs cannot be linked by a high-performance lin-
ear path; (4) the above conclusions hold for both adapter and fine-tuning.
The above findings imply that when learning the same task, two continually pre-trained PLMs would probably be optimized into two minima lying in the same loss basin, or at least the optimal regions corresponding to both minima have a substantial intersection; otherwise, there should exist a significant performance drop in between.
Intuitively, the existence of a high-performance (low-loss) path between two optimal regions implies that model weights can be easily optimized from one optimal region to another without incurring a loss barrier. In this regard, it is promising to use outdated adapted weights as the initialization to find the optimal solution for the upgraded PLM, which would be explored in § 5.1. In this way, we explicitly facilitate cross-model knowledge transfer through the parameter space.
Extension to Multiple Domains. Next, we evaluate linear mode connectivity between the initial M0 and Mi (1≤ i≤4) using the task CHEMPROT. We derive from the results in Figure 4 that although the performance tends to drop slightly near the midpoint, the connectivity of all continually pretrained models is still far better than independent PLMs (i.e., MIND in Figure 3). We also observe that the performance drop between M0 and M2 is larger than M0 and M4, though M4 is trained for a longer time than M2. This means longer pretraining does not necessarily result in poorer connectivity; rather, the pre-training domain has a great impact.

heading: 4.3 Functional Similarity Analysis
text: The close parametric connection revealed by linear mode connectivity does not guarantee that continually pre-trained PLMs share similar functionalities
when processing the text information. Following Gong et al. (2019), we explore functional similarity through the lens of attention distribution. Specifically, we investigate three continually pre-trained models (M0, M1, and M2) and fine-tune them on CHEMPROT to obtain adapted models (θT0 , θ T 1 , and θT2 ). We feed the same input sampled from CHEMPROT to the three adapted models. Then we select attention heads from the same position (i.e., the h-th head in the l-th layer) in three models, and visualize their attention distribution. Note the selected head of Mi+1 is trained from that of Mi.
From Figure 5, it is found that the attention patterns of M1 and M2 are quite similar to those of their “ancestor” M0. Such representational proximity indicates that the corresponding modules of continually pre-trained PLMs own similar functionalities. Since adapted weights play a pivotal role in stimulating PLM’s abilities and functionalities (Ding et al., 2022), such functional similarity partially explains why the outdated adapted weights can be directly applied to the upgraded PLM and achieve non-trivial performance in § 4.1.
In a nutshell, all the analyses in this section validate the close connection between continually pretrained PLMs. Intuitively, such a connection im-
plies that the adaptation process of these PLMs towards downstream tasks should be closely related and transferable as well, which serves as the strong basis for our recyclable tuning.

heading: 5 Methods and Experiments
text: Based on the findings in § 4, we propose two ways to explore the practical benefits of recyclable tuning: initialization-based method (§ 5.1) and distillation-based method (§ 5.2). The training details of this section are discussed in appendix D.3.

heading: 5.1 Initialization-based Recyclable Tuning
text: We first investigate directly using outdated weights as the initialization for tuning the upgraded PLM.
Framework. Without loss of generality, we experiment when the initial PLM M0 is continually pre-trained on the BIO domain (D1) and upgraded to M1. Before the release of a new PLM M1, assume we have tuned M0 on N tasks {T0, · · · , TN} and obtained the corresponding adapted weights {∆T10 , · · · ,∆TN0 }. When tuning M1 on a target task Tt, instead of using the random initialization for tunable weights, we initialize them using M0’s
adapted weights ∆Ts0 trained on a source task Ts. Considering that in practice, it is possible that the outdated weights of exactly the same task are not available, i.e., Tt ̸= Ts. Thus we explore whether initialization from the outdated weights of a different task would suffice for our goal. Specifically, we consider three types of source tasks: (1) Tsame, which is the same task as the target one; (2) Tsim, which denotes a task similar to Tt, both Tsim and Tt typically belong to the same task type; (3) Tdiff, which belongs to a different task category from Tt.
Settings. We experiment with 6 target tasks of 3 types: (1) natural language inference: ANLI (Nie et al., 2020) and SICK (Marelli et al., 2014), (2) sentiment analysis: SST-2 (Socher et al., 2013) and Rotten Tomatoes (Pang and Lee, 2005), (3) emotion detection: Hate Speech (Davidson et al., 2017) and Tweet Eval-Offensive (Barbieri et al., 2020). The choices of Tsim and Tdiff for each target task are listed in Table 12 in the appendix.
We compare the proposed initialization strategies with random initialization and record (1) the test performance variation (w.r.t. training steps) during the early stage of downstream adaptation (Figure 6), and (2) the best test performance after the adaptation converges (Table 1). For adaptation, we mainly investigate adapter tuning and leave the experiments of fine-tuning in appendix C.3.
Results. The observations and corresponding conclusions are summarized as follows:
(1) Faster convergence: we observe from Figure 6 that compared with the random initialization baseline, our method significantly accelerates the convergence of downstream adaptation. This suggests that the outdated weights provide a more effective initialization, allowing the PLM to be more easily optimized to the desired local optima. In practice, this method could improve the training efficiency of tuning the upgraded PLM, which saves
the computations needed for adaptation. (2) Improved task performance: we also conclude from Table 1 that after sufficient training, initialization from the outdated weights of each type of source tasks (even for Tdiff) could improve the final performance (up to +1.9 average improvement). This demonstrates that initialization serves as a valid way for cross-model knowledge transfer.
(3) Similar source tasks benefit more: comparing the results of initialization from different source tasks, we find that the improvement in both convergence and performance can be generally ranked as Tsame>Tsim>Tdiff. This is because the knowledge required by more similar tasks has a greater overlap. Thus the knowledge transfer benefits more when the target task and source task are more similar. In practice, this finding expands the selection scope of source adapted weights, broadening the application scenarios for our initialization-based method.

heading: 5.2 Distillation-based Recyclable Tuning
text: According to Lin et al. (2021), model outputs often contain sufficient supervision that is complementary to the knowledge stored in parameters. Therefore, besides the initialization-based method, we also explore knowledge distillation (Hinton et al., 2015) to recycle the outdated weights.
Framework. Given a task Tj , assume we have optimized an outdated PLM Mi and obtained its adapted weights ∆Tji . Our goal is to distill the knowledge stored in ∆Tji to optimize an updated PLM Mi+1. We follow Sun et al. (2019) to construct our framework. For each data point x from Tj , denote P(x, θTji ) as the probability distribution the adapted Mi assigns over the label space, where θTji =θ 0 i ⊕∆ Tj i . We minimize the KL divergence between probabilities predicted by Mi and Mi+1. In addition, Mi+1 mimics Mi’s intermediate hidden representations of each layer. Specifically, given the same input x, denote hk(x, θ Tj i ) and hk(x, θ Tj i+1) as the normalized hidden states of the k-th layer of Mi and Mi+1, we minimize the mean-square loss of hidden states together with the KL divergence as follows:
LKD = KL(P(x, θTji )||P(x, θ Tj i+1))+ αΣk||hk(x, θTji )−hk(x, θ Tj i+1)||2,
(2)
where α denotes a hyper-parameter. During optimization, only ∆Tji+1 is tunable. Besides LKD,
we also introduce the original task loss LTj , which is calculated using supervised training examples from task Tj , with another hyper-parameter β:
Lfinal = βLTj + (1− β)LKD. (3)
Settings. We consider the sequentially released PLMs {M0, · · · ,M4} as mentioned in § 3. Following Gururangan et al. (2020), we choose three tasks T1: CHEMPROT, T2: IMDB (Maas et al., 2011) and T3: ACL-ARC (Jurgens et al., 2018), which are relevant to domain D1, D2 and D3, respectively. We mainly consider recyclable tuning between adjacent PLMs, i.e., Mi and Mi+1, and also evaluate non-adjacent PLMs (e.g., Mi and Mi+2) in appendix C.7. For each task Ti (i ∈ {1, 2, 3}), we consider two settings:
(a) First, we recycle Mi’s outdated weights to Mi+1, which is denoted as ∆Tii → ∆Tii+1. Here the evaluated task Ti is relevant to the pre-training domain Di of the original PLM Mi. During continual pre-training on Di+1, Mi+1 suffers from catastrophic forgetting of Di. Hence Mi+1 should perform worse on Ti than Mi. In experiments, both PLMs are adapted using the same 32-shot dataset.
(b) Second, we evaluate the recyclable tuning from Mi−1 to Mi, which is denoted as ∆Tii−1 → ∆Tii . Different from setting (a), here the evaluated task Ti is relevant to the pre-training domain Di of the newly released PLM Mi. Mi performs better
than Mi−1 since Mi has acquired more knowledge related to Ti when learning Di. In light of this, we explore whether Mi could achieve better performance than Mi−1 even when trained with fewer supervised examples. Specifically, the data size of Mi−1 is set to {32, 256, 32}-shot for {T1, T2, T3}, and the data size of Mi is set to {16, 32, 16}-shot, respectively. We also evaluate our method under the zero-shot setting in appendix C.4.
We compare our method with utilizing only the task loss (Lfinal-LKD) to validate the benefits of knowledge distillation. Further, we explore combining both distillation-based and initializationbased recyclable tuning (Lfinal+Init.). This is implemented by first using the outdated weights as the initialization then tuning with Lfinal. We also report teacher performance (Teacher) as a reference.
Results. It can be concluded from Table 2 that: (1) compared with optimizing only the task loss (Lfinal-LKD), distilling knowledge from the outdated weights (Lfinal) significantly improves the performance, which shows that knowledge distillation is an effective way for recyclable tuning. (2) In general, Lfinal+Init. leads to better performance than Lfinal. This finding reveals that both distillation-based and initialization-based methods are complementary to each other and can be further combined to fully exploit the knowledge in outdated weights. (3) In Table 2 setting (a), Mi+1 performs worse than Mi on task Ti, which is because Mi+1 forgets some knowledge of domain Di when learning Di+1. However, such forgetting can be mitigated by designing better continual pre-training algorithms (Qin et al., 2022c). (4) In Table 2 setting (b), Mi outperforms Mi−1 despite being trained with fewer examples. This shows that the newly acquired knowledge on domain Di conduces to Mi’s performance in Di’s relevant task Ti, and improves the data efficiency. We further discuss the difference between distillation-based and initialization-based methods in appendix F.

heading: 6 Discussion
text: Training-free Weight Recycling. Both methods proposed in § 5 necessitate tuning the upgraded PLM. Such a process often relies on abundant computational costs and may be infeasible practically. Given the close connections among continually pretrained PLMs, we contend that weight recycling can be realized without training. As a preliminary exploration, we show in appendix B that it
is possible to learn a cross-task generalizable projection to directly upgrade the outdated weights and make them compatible with the new PLM. Upgrading outdated weights using such a projection requires far fewer computations (< 0.002‰) and still achieves satisfactory performance.
Downstream-compatible Continual Pretraining. From another angle, recyclable tuning addresses the incompatibility between outdated adapted weights and the upgraded PLM from the customer perspective, analogous to the concept of forward compatibility in software engineering. In fact, the responsibility for maintaining compatibility can also be shifted to upstream suppliers during PLM upgrading (i.e., backward compatibility). Potential solutions include adding regularization terms during continual pre-training to maintain compatibility with existing adapted weights. In this way, we solve the incompatibility problem once and for all, which is more customer-friendly. However, modifying pre-training objectives may come at the cost of reduced model performance.
Broader Application Scenarios. Although we primarily focus on recyclable tuning for one specific scenario (i.e., continual pre-training), PLMs may be subject to various types of evolution in practice. For instance, the expansion of model size (e.g., from T5BASE (Raffel et al., 2020) to T5LARGE), the upgrading of model architecture (Chen et al., 2022; Lee-Thorp et al., 2022), the alteration of optimization objective (e.g., from T5 to T0 (Sanh et al., 2021) and UNIFIEDQA (Khashabi et al., 2020)), etc. Once the backbone infrastructure is upgraded, massive adapted weights would become outdated and potentially wasted. Hence we believe recyclable tuning in fact has broader application scenarios and we hope our findings and solutions could inspire more future research in this area.

heading: 7 Conclusion
text: In this paper, we formulate the task of recyclable tuning for continual pre-training. We conduct empirical analyses for this task through the lens of model compatibility, linear mode connectivity, and functional similarity. Inspired by the corresponding findings, we explore the practical benefits of recyclable tuning through parameter initialization and knowledge distillation. We also envision our setup to serve as the testbed for other topics, e.g., crossmodel knowledge transfer and continual learning.

heading: Acknowledgments
text: This work is supported by the National Key R&D Program of China (No. 2020AAA0106502, No. 2022ZD0116312), Institute Guo Qiang at Tsinghua University, Beijing Academy of Artificial Intelligence (BAAI).
Yujia Qin and Cheng Qian designed the methods. Yujia Qin wrote the paper. Cheng Qian conducted the experiments. Yankai Lin, Huadong Wang, Zhiyuan Liu, Maosong Sun, and Jie Zhou advised the project. All authors participated in the discussion.

heading: Limitations
text: We only experiment with two kinds of PLMs (RoBERTaBASE and RoBERTaLARGE (appendix C.3 and appendix C.8)), leaving more diverse kinds of PLMs unexplored. While this allows us to demonstrate the effectiveness of our approach on these specific PLMs, it is important for future work to extend our problem setup to a wider range of PLMs in order to fully understand the generalizability of our findings.

heading: Ethical Statement
text: In this research, we consider the following ethical issues: • Privacy. Outdated adapted weights may contain
information about the data and tasks they were trained on. Thus it is important to consider the potential privacy implications when recycling these weights. Efforts should be taken to ensure that personal or sensitive information is not disclosed during weight recycling.
• Fairness. It is crucial to guarantee that the recycling of adapted weights does not introduce biases or unfairly advantage certain tasks or domains. Thorough analysis and testing are needed to make sure that recyclable tuning does not perpetuate or amplify existing inequalities.
• Responsible AI. The responsible development and deployment of AI systems require considering the potential impacts on the environment. By improving the efficiency and sustainability of PLM adaptation, recyclable tuning contributes to the responsible development of AI systems.
• Transparency. To facilitate the responsible and ethical use of recyclable tuning, it is vital to be transparent about the methods and assumptions underlying them. We encourage future works
to clearly document the conditions under which recyclable tuning is effective, as well as the potential limitations or risks.

heading: Appendices

heading: A Additional Backgrounds

heading: A.1 Parameter-efficient Tuning
text: Conventional downstream adaptation of PLMs involves optimizing all parameters (i.e., fine-tuning), which may cause a heavy burden on the computational infrastructure and storage space. To efficiently utilize the knowledge contained in PLMs, parameter-efficient tuning (PET) is proposed, which optimizes only a few parameters and freezes the majority of parameters (Houlsby et al., 2019). Despite extensively reducing the tunable parameters, PET achieves comparable performance to fine-tuning. Besides, due to its lightweight nature, adapted weights produced by PET are easier to train, store, and share among consumers. Thus we deem PET as an essential component in our problem setup. Without loss of generality, we consider a representative PET algorithm, i.e., adapter (Houlsby et al., 2019) in this paper. Adapter inserts tunable modules into both the feedforward module and multi-head attention module of each Transformer (Vaswani et al., 2017) layer.

heading: A.2 Mode Connectivity
text: Mode connectivity measures whether two minima in the parameter space can be connected by a parametric path, where the loss (performance) remains low (high) (Garipov et al., 2018; Freeman and Bruna, 2017; Draxler et al., 2018). Such a property implies that different minima can potentially form a connected manifold in the loss landscape. For two connected minima, we can interpolate them to obtain a series of high-performance solutions. These solutions can be ensembled to achieve performance (Garipov et al., 2018) that is better than the endpoints.
Prior works in mode connectivity show that under most cases, in neural networks, there exists a non-linear low-loss path between different minima. However, only occasionally a linear low-loss path could connect different minima. Later works further contend that it is non-trivial if both minima can be connected by a linear path (Frankle et al., 2020; Mirzadeh et al., 2020). The linearity indicates that both minima may probably lie in the same loss basin (Qin et al., 2022b), which is a more favorable property and indicates a closer connection between both minima. In view of this, we focus on analyzing the linear mode connectivity in this paper.
Previous efforts were mainly spent on investigating mode connectivity for non-pre-trained models, until recently, Qin et al. (2022b) explore such property for PLMs. They focus on tuning one static base model with different adaptation strategies. Differently, we take the first step to explore mode connectivity for different backbone models (continually pre-trained PLMs) and reveal novel insights. Following Qin et al. (2022b), we present the results of task performance (e.g., accuracy) to evaluate the mode connectivity in the main paper and also report the results of task loss in appendix E.

heading: B Training-free Weight Recycling
text: Although we have shown that initialization-based recyclable tuning could accelerate the convergence and improve the training efficiency, tuning the upgraded PLM still requires abundant training computations. Especially considering the massive number of tasks to handle, conducting adaptation for all of them whenever the PLM is upgraded is computationally expensive.
In this section, we explore whether we could alleviate the burden of supervised training, and directly upgrade the outdated weights at a small cost. A desired algorithm should consume significantly lower computations than that of training the new PLM from scratch. Meanwhile, this algorithm should achieve satisfactory task performance.

heading: B.1 Framework
text: Inspired by Qin et al. (2021); Yi et al. (2022), we propose a training-free weight recycling method. Specifically, we learn a cross-task generalizable projection that could directly produce upgraded adapted weights for a specific task, omitting the labor of supervised training. We contend that although there exist massive downstream tasks, a large percentage of them are intrinsically similar and can be categorized into the same task type (e.g., sentiment analysis, question answering, etc.). Intuitively, the upgrading of a certain task T1 should provide a referential experience for that of a similar task T2. In view of this, we propose to make the upgrading process of T1 recyclable so that the upgrading of T2 can be achieved efficiently.
For two sequentially released PLMs Mi and Mi+1, assume we have the adapted weights of Mi for both T1 and T2. We aim to recycle these adapted weights for tuning Mi+1 on both tasks. As illustrated in Figure 7, our framework consists
&!) &!&") (
&(() = 1 − ( &!) + ( &!&")!
Tunable
⊕
∆!*!
∆!*%
∆!&" *!
∆!&"*%∗
ℳ!&"
ℳ! ℳ!&"
Frozen
Basic Framework
(1) Projection Learning
Weight Mixup
Frozen
(2) Projection Transferring
Compatible
* /+ /+ 0 0 ℒ'( ℒ#
Unlabeled
Figure 7: Illustration of our training-free weight recycling algorithm. We learn a cross-task generalizable projection (i.e., the projection learning stage) that could directly upgrade outdated adapted weights (i.e., the projection transferring stage).
of two stages: (1) projection learning and (2) projection transferring. We learn an upgrading projection using task T1 in the first stage, and then apply (transfer) the learned projection to task T2 in the second stage. Note the first stage requires training while the second stage is training-free. Next, we introduce the details of the two stages.
Projection Learning. Instead of directly optimizing the parameters in ∆T1i+1, we learn a low-rank decomposition ∆T1i+1 = Proj(∆ T1 i ) as follows:
Proj∗i→i+1 = argmin Proj L(Proj(∆T1i )),
where Proj=Proj↑×Proj↓. Denote d as a lowdimensional bottleneck dimension, Proj↓ projects the dimension of ∆T1i to d, i.e., Proj↓(∆ T1 i ) ∈ Rd. Then Proj↑ projects the dimension from d back to |∆T1i |, i.e., Proj↑(Proj↓(∆T1i ))∈R|∆ T1 i |. Either Proj↑ or Proj↓ is implemented by a 2- layer MLP. During training, ∆T1i is kept frozen and only the parameters in Proj are tuned. Note the dimensions of ∆T1i and ∆ T1 i+1 are the same, i.e., |∆T1i |= |∆T1i+1|. Proj(∆T1i ) is then applied to the upgraded PLM Mi+1 to compute the loss L.
Projection Transferring. When upgrading the outdated weights of a similar task T2, we directly apply the projection Proj∗i→i+1 learned on T1 to ∆T2i and obtain the approximated updated weights ∆T2∗i+1:
∆T2∗i+1 = Proj ∗ i→i+1(∆ T2 i ).
We formulate the downstream tuning as prompt learning (Schick and Schütze, 2021), instead of
introducing additional classification heads for different tasks. Hence the number of parameters in ∆T1i and ∆ T2 i is the same, i.e., |∆T1i | = |∆T2i |. Note that only applying the projection to compute the upgraded weights consumes very limited computations (see Figure 8), hence we significantly reduce the computations of learning ∆T2i+1, compared with the conventional tuning-based method.
Besides, since the projection Proj comprises an integral multiple (d×) of ∆T1i ’s parameters, our solution is only feasible for parameter-efficient tuning. While for fine-tuning, it is computationally intractable to train the projection due to the tremendous size of parameters in ∆T1i and Proj. Being the first attempt in this research direction, we leave corresponding explorations for fine-tuning as future work.

heading: B.2 Experiments
text: Settings. We mainly evaluate M1 and M2 as defined in § 3. We choose a series of NLP tasks and categorize them into 3 classes: (1) natural language inference: MNLI, SICK, ANLI, QNLI (Rajpurkar et al., 2016), and WNLI (Faruqui and Das, 2018), (2) sentiment analysis: SST-2, Amazon Polarity (McAuley and Leskovec, 2013), and Rotten Tomatoes, (3) emotion detection: Hate Speech, Tweet Eval-Offensive, Tweet Eval-Hate, Tweet Eval-Abortion, Tweet Eval-Feminist, and Tweet Eval-Atheism from Barbieri et al. (2020). We partition the tasks belonging to the same category into source task T1 and target task T2 (see Table 3), and learn the projection Proj on the source task.
We consider the zero-shot setting for the first stage (projection learning) and use the knowledge distillation loss function LKD. Here the teacher model weights are the adapted M1, and
the student model weights are obtained by applying Proj(∆T11 ) to the pre-trained weights of M2. For the unlabeled corpus used for distillation, we evaluate both the target task data (denoted as LTKD) and Wikipedia corpora (LwikiKD ). Note for the former, we only use the input x and discard the corresponding label y (i.e., the zero-shot setting). The former can be seen as the upper bound for the latter since the data format of the latter may not be compatible with the target task. After that, we directly utilize the learned projection to upgrade the outdated weights of similar target tasks.
Baselines. We consider demonstration learning (Brown et al., 2020) as the baseline, which integrates a few labeled examples into the input text as additional context. The PLM directly performs inference on the test set without incurring any training. For reference, we also report the performance when M2 is adapted using the full dataset (FD) and the 32-shot dataset (FS). Instead, our method requires no labeled data.
Efficiency Evaluation. We compare the computational costs needed for our training-free method and the conventional tuning-based method in Figure 8. For the former, we record the time needed in projection transferring (i.e., computing the upgraded weights ∆T2∗i+1). For the latter, we record the training time needed until an adaptation converges. It can be derived that our method requires significantly fewer computations, which demonstrates its efficiency. In practice, such a projection can be trained once and for all. As long as we have obtained the projection, we can directly upgrade
potentially massive outdated weights in an efficient manner, and the computations involved during projection learning can be neglected. Although currently we only support projection transferring for a similar target task that belongs to the same category of the source task, we expect future work to explore how to train a universal projection that could be applied to an arbitrary task.
Performance Evaluation. The results are shown in Table 3, from which we find that: (1) our method generally outperforms the demonstration baseline, and could surpass the supervised performance (FD and FS) under certain cases, despite not using any labeled data. Hence, besides being computationally efficient, our method achieves satisfactory performance in general. This also validates our intuition that for continually pre-trained PLMs, the upgrading of a specific task could provide referential experience for similar tasks; (2) using the task data (LTKD) for distillation generally performs better than using Wikipedia (LwikiKD ), showing the importance of proper data distribution used for knowledge distillation.

heading: C Additional Experiments and Analyses

heading: C.1 Euclidean Distance Analysis
text: We report the Euclidean distance of continually pretrained PLMs and the corresponding adapted models. We evaluate when the official RoBERTaBASE is adapted on the BIO domain for 12.5k steps following the settings in § 3. We save the checkpoint for every 2.5k steps. For each checkpoint M1(t), denote its weights as θ01(t), we fine-tune it on CHEMPROT to obtain its adapted weights ∆T1 (t), where |∆T1 (t)| = |θ01(t)|. The resultant model weights are θT1 (t) = θ 0 1(t)⊕∆T1 (t).
Given two continually pre-trained models M1(t) and M1(t′), where t′ = t + 2.5k, we flatten their pre-trained weights θ01(t) and θ 0 1(t
′), and calculate their L-2 norm1: ||θ01(t′) − θ01(t)||. In addition, we also calculate the L-2 norm of flattened adapted weights (||∆T1 (t)|| / ||∆T1 (t′)||) and distance between adapted PLMs (||θT1 (t′)− θT1 (t)||). We illustrate the results in Figure 10 and find that: (1) ||θ01(t′) − θ01(t)|| / ||θT1 (t′) − θT1 (t)|| gradually decreases with t increasing. This is mainly because the learning rates are warmed up for the first 6% steps, and the learning rate starts to decrease at the 0.75k-th step, which means the PLM gradually moves slower in the parameter space; (2) the parameter change caused by downstream adaptation (i.e., ||∆T1 (t)|| / ||∆T1 (t′)||) is far
1We use torch.dist function in PyTorch (Paszke et al., 2019) for implementation.
smaller than that brought by continual pre-training (||θ01(t′) − θ01(t)||). This is because downstream adaptation converges shortly. After convergence, the model parameters generally stay in a specific optimal region. While continual pre-training constantly pushes the model weights away from the previous checkpoints in the parameter space. Another reason is that continual pre-training uses a large batch 2048, while downstream adaptation often uses a much smaller batch size (e.g., 16).

heading: C.2 More Visualization for Functional Similarity Analysis
text: In the main paper (Figure 5), we visualize three different attention heads of M0, M1, and M2. In this section, we present more visualizations to further support our claim. We also visualize the attention pattern of an independently trained PLM MIND. The results in Figure 9 again demonstrate our claim that continually pre-trained PLMs exhibit similar attention patterns, which independently trained PLMs do not have.
C.3 Initialization-based Recyclable Tuning for Fine-tuning and RoBERTaLARGE
In § 5.1, we mainly evaluate initialization-based recyclable tuning using RoBERTaBASE and adapter tuning. Here we extend the experiments to either fine-tuning (Table 4) or RoBERTaLARGE (Table 5). We choose 3 tasks in Table 1 and follow most of the settings. From Table 4 and Table 5, we find that the main conclusions are generally consistent with those mentioned in the main paper. This implies that the initialization-based method can be applied to different tuning methods and PLMs.

heading: C.4 Distillation-based Recyclable Tuning under the Zero-shot Setting
text: We extend our distillation-based recyclable tuning to the zero-shot setting where there is no labeled data for tuning the upgraded PLM. We show that it
is able to utilize unlabeled raw corpora to distill the knowledge of outdated weights. Specifically, we remove the task loss LT in Lfinal and only retain LKD. Instead of using supervised examples, we sample unlabeled data x from Wikipedia to compute LKD. We evaluate recyclable tuning between M1 and M2 and choose 4 downstream tasks, i.e., CHEMPROT, IMDB, SST-2, and MNLI. For each task, the outdated weights of M1 are obtained with the full dataset, and our goal is to distill their knowledge and optimize M2’s weights.
Two training-free baselines are considered: (1) manual prompting (Schick and Schütze, 2021), which restructures the input into templates by inserting prompts, and (2) demonstration learning, which has been introduced in appendix B.2. For both baselines, the PLM directly performs inference on the test set without incurring any training. Moreover, we also evaluate the performance when knowledge distillation is combined with the initialization-based method.
We list the results in Table 6, from which it can be derived that: (1) our method surpasses manual prompting and demonstration learning by a large margin, which shows the benefits of recycling outdated adapted weights in the zero-shot setting; (2) initializing tunable weights with the outdated weights could further improve the performance of LKD, which again demonstrates that both
initialization-based and distillation-based methods are complementary to each other.
C.5 Interpolation Distillation
Traditional knowledge distillation frameworks have no assumptions about the parametric connection between the teacher and the student, and resort to pulling closer their predictions (P) or inner representations (h). As we have shown in the main paper, continually pre-trained PLMs are guaranteed with close parametric connections. Therefore, traditional knowledge distillation methods may fail to exploit the parametric knowledge contained in the teacher model’s parameters. Here we explore another way for more effective distillation-based recyclable tuning under our setting.
Framework. Inspired by MC-SGD (Mirzadeh et al., 2020), we propose an interpolation distillation technique to fully exploit the parametric knowledge contained in outdated adapted weights. Specifically, for recyclable tuning between Mi and Mi+1, instead of optimizing the overall loss function using the only endpoint checkpoint (θLji+1 = θ0i+1 ⊕ ∆ Tj i+1) for task Tj , we linearly interpolate θ Lj i and θ Lj i+1 to obtain a series of model checkpoints: θ(µ) = (1−µ)θLji +µθ Lj i+1. After that, we feed data into θ(µ) and minimize the corresponding
loss together with L(θLji+1):
LITP(∆Tji+1) = L(θ Lj i+1)+γ
∑
µ∈{ 1 Nµ ,··· ,Nµ−1 Nµ }
L(θ(µ)),
where γ is a hyper-parameter, and Nµ denotes a constant integer. In practice, we found a small Nµ (e.g., 2) already achieves satisfying performance. During optimization, only ∆Tji+1 is tuned by receiving gradients from both L(θji+1) and L(θ(µ)). Experiments. We follow most of the settings in § 5.2 and evaluate the performance of interpolation distillation. We compare it with the results of LfinalLKD, Lfinal, and Lfinal+Init.. All results are shown in Table 7, from which we observe that the interpolation distillation method (LITP) generally outperforms the vanilla distillation (Lfinal), and could surpass Lfinal+Init. in certain cases. This shows that interpolation distillation successfully exploits the parametric knowledge contained in the outdated adapted weights, and serves as an improved method for the distillation-based method.

heading: C.6 Effects of Teacher Model Capability for Distillation-based Recyclable Tuning
text: For experiments of setting (a) in distillation-based recyclable tuning (§ 5.2), the teacher model is trained with the same 32-shot dataset as the student model. Here we explore whether a teacher model with stronger capabilities would conduce to the student’s performance. Specifically, keeping all the other settings the same, we change the teacher model’s data to the full-data size. The new results are placed in Table 8, from which we conclude that: (1) our methods (Lfinal, LITP, and Lfinal+Init.) still outperform the baseline without knowledge distillation (Lfinal-LKD); (2) comparing the student’s performance in Table 8 and Table 2 setting (a), we
Method Lfinal-LKD Lfinal LITP Lfinal+Init. Few-shot teacher
∆T11 →∆T13 AP 60.5±2.1 66.3±1.5 67.5±1.7 67.2±1.5 FT 61.9±1.3 64.7±0.9 64.8±0.8 65.4±1.3
∆T11 →∆T14 AP 56.6±1.1 57.9±1.5 65.3±1.7 64.9±3.6 FT 59.7±2.3 62.9±2.4 64.4±0.5 65.1±2.2
Full-data teacher
find through learning from a more powerful teacher, the student’s performance is improved as well.

heading: C.7 Experiments on Non-adjacent PLMs
text: For most of the experiments, we mainly focus on recyclable tuning between adjacent PLMs. We contend that the proposed methods should also work for non-adjacent PLMs since they are still guaranteed with close connections. To demonstrate this, we take the distillation-based recyclable tuning as an example. Specifically, we evaluate the distillation-based recyclable tuning between (M1, M3) and (M1, M4) using T1, and largely follow the settings in § 5.2. We choose setting (a) in § 5.2, and the only difference is that the teacher model M1 is trained either using the 32- shot dataset (dubbed as few-shot teacher) or the full dataset (dubbed as full-data teacher). While the student model is trained using the 32-shot dataset. In this way, we could understand the role of the teacher model in knowledge distillation.
The results are placed in Table 9, from which we find that: (1) introducing knowledge distillation (Lfinal) improves the performance than only using task loss (Lfinal-LKD) and (2) introducing the parametric knowledge either through interpolation distillation (LITP) or weight initialization (Lfinal+Init.) could further improve the task performance. Both conclusions are aligned with those obtained on adjacent PLMs. This demonstrates our claim that our recyclable tuning is not limited to adjacent PLMs, but also non-adjacent ones. Finally, we observe that the student performance when the teacher is trained
using full data is much better, which shows the benefits of learning from a more advanced teacher.

heading: C.8 Distillation-based Recyclable Tuning
text: Experiments using RoBERTaLARGE
Previous experiments for distillation-based recyclable tuning are based on RoBERTaBASE, now we turn to RoBERTaLARGE to show that our proposed methods are model-agnostic. We experiment with M1 and M2 using the task CHEMPROT. Other settings are kept the same as those in appendix C.7. In Table 10, we show that the results are generally aligned with our conclusions before. These results also reflect that our proposed method is agnostic to the specific PLM chosen.

heading: C.9 Effects of Data Size for Distillation-based Recyclable Tuning
text: Taking a step further, we study the performance of our distillation-based recyclable tuning at different data scales. Specifically, we focus on T2 (IMDB) for recycling M1’s outdated weights to M2, where M1 is adapted using the full dataset, and M2 is trained with {8, 16, 32, 64}-shot dataset,
respectively. By comparing the method mentioned in appendix C.5 (LITP) with only the task loss LT , we visualize the performance variation in Figure 11, from which we observe that: LITP surpasses only the task loss (LT ) in general. However, with the data scale increasing, the improvement becomes smaller. This is because M2 is more adept at T2 than M1 due to the incremental knowledge acquisition of D2. When there are only a few examples to train M2, the teacher model has the advantage of more labeled data. However, with the data size of the student gradually approaching that of the teacher, learning from the teacher gradually becomes redundant. The student model could well master the downstream knowledge on its own.

heading: D Training Details
text: We ensure that all the artifacts used in this paper are consistent with their intended use.

heading: D.1 Pre-training
text: We conduct pre-training using 8 NVIDIA V100 GPUs based on fairseq2 (Ott et al., 2019). We choose Adam (Kingma and Ba, 2015) as the optimizer. The hyper-parameters (ϵ, β1, β2) for Adam are set to 1 × 10−6, 0.9, 0.98, respectively. The dropout rate and weight decay are set to 0.1 and 0.01, respectively. The total number of parameters of RoBERTaBASE and RoBERTaLARGE are 125M and 355M, respectively. We implement pretraining using the codes of Qin et al. (2022a).
Continual Pre-training. We start with the official RoBERTa model and sequentially pre-train the
2https://github.com/pytorch/fairseq
PLM on 4 domains. For each domain, we set the batch size to 2048, the training steps to 12.5k, and the max sequence length to 512.
Pre-training from Scratch. For MIND that is pre-trained from scratch, we follow the model structure of RoBERTaBASE, and pre-train the model on the concatenation of Wikipedia and BookCorpus (Zhu et al., 2015), which is the same as the pre-training corpus of BERT (Devlin et al., 2019). We pre-train the model for 125k steps, using a batch size of 2048 and a sequence length of 512. The total computations involved are roughly comparable to those of BERTBASE. MIND has totally different initialization and pre-training corpus than the official RoBERTaBASE, which helps us understand the property between independently trained PLMs.

heading: D.2 Empirical Analyses
text: Model Compatibility Analysis. We adapt the initial PLM M0 on two tasks CHEMPROT and MNLI. The training hyper-parameters conform to those listed in Table 11. All experiments are conducted 3 times with different random seeds, and we report the average results.
Linear Mode Connectivity Analysis. All the training hyper-parameters conform to those in Table 11. The endpoints are adapted three times using different random seeds. We test the performance of 25 evenly distributed points along the linear path and two endpoints. We report the average performance over three random seeds.
Functional Similarity Analysis. We adapt different PLMs on task CHEMPROT using the hyperparameters listed in Table 11. We randomly sample one instance3 from CHEMPROT and feed it into different PLMs to obtain the scores after the
3We find empirically that the results and conclusions are very consistent across different random samples.
self-attention computation. We draw the attention scores for the first 25 tokens of the sampled instance.

heading: D.3 Methods and Experiments
text: For the optimizer of all the experiments in § 5, we choose AdamW (Loshchilov and Hutter, 2019).
Initialization-based Recyclable Tuning. We adapt M0 on the source tasks using the hyperparameters listed in Table 11. The adapted weights are further used as target tasks’ initialization (except the Random setting). The target tasks’ training configurations also conform to Table 11. We conduct the experiments for 3 times with different random seeds and report the average performance. The choices of Tdiff and Tsim for different target tasks are shown in Table 12. The evaluation interval for each target task is also reported in Table 12.
Distillation-based Recyclable Tuning. We set the maximum training step for CHEMPROT and ACL-ARC to 100k and the maximum training step for IMDB to 50k. The learning rate and batch size
are set to 1× 10−4 and 2, respectively. We warm up the learning rate for the first 8% percentage of total training steps. We report the average results over 3 different random seeds. As for other hyper-parameters discussed in § 5.2, we perform grid search for β over {0.1, 0.3}, and α(1 − β) over {0, 1, 5, 10, 50, 100}. We also conduct a grid search for the temperature in knowledge distillation loss over {10, 20} when calculating KL(P(x,Mi)||P(x,Mi+1)). We select the bestperforming combination of these hyper-parameters and then report the performance. Our grid search is performed for our method and all the baseline methods for a fair comparison.

heading: E The Visualization of Loss for Linear Mode Connectivity Analysis
text: When conducting experiments for the mode connectivity analysis in the main paper, we mainly resort to performance as the evaluation protocol for the interpolations following Qin et al. (2022b). In this section, we show the corresponding visualization of loss for Figure 3 and Figure 4, see Figure 12 and Figure 13. From these figures, we conclude that a significant loss barrier generally indicates the existence of a large performance drop.

heading: F Comparison of Initialization-based and Distillation-based Recyclable Tuning
text: Both initialization-based and distillation-based methods serve as powerful ways for recyclable tuning under the continual pre-training scenario. Both methods have their own advantages, where the initialization-based method can bring faster convergence and performance improvement, while the distillation-based method can bring improvement in performance as well (but may be less efficient). In addition, both methods can be combined with each other to further improve performance.
In terms of practical application scenarios, both methods are slightly different. For one thing, the initialization-based method requires that the architectures of the new PLM and the old PLM are the same. This requirement may be infeasible for broader application scenarios, such as recyclable tuning between different PLMs as discussed in § 6. For another, the initialization-based method typically requires access to the parameters of the outdated adapted weights. This can be a practical issue due to model privacy concerns. While some customers are willing to share their
adapted weights on public platforms like AdapterHub (Pfeiffer et al., 2020), a majority of adapted weights are publicly unavailable. In contrast, the distillation-based method can be achieved without access to the model weights, but through receiving model inference from the owner (e.g., API-based online knowledge transfer (Krishna et al., 2019)). In this sense, the distillation-based method could protect the model privacy to a certain degree.
Broader Impacts
This research has the potential to have a broad impact in several ways. • First, recyclable tuning could improve the ef-
ficiency of adapting PLMs to new tasks. By recycling adapted weights from previous tasks, the need for costly retraining can be reduced, potentially making it more feasible to apply PLMs in a wider range of scenarios.
• Second, the results of this research could have implications for the sustainability of machine learning systems. Reusing adapted weights rather than discarding them can help us reduce the carbon footprint and resource consumption of PLM adaptation, making it more environmentally friendly.
• Third, this research has the potential to benefit a wide range of stakeholders, including researchers, developers, and users of PLMs. Researchers can use the proposed task and benchmark to develop and evaluate new techniques for recyclable tuning, while developers can apply these techniques to improve the efficiency and sustainability of PLM-based systems. Finally, users of PLMs can benefit from the reduced costs and improved performance made possible by recyclable tuning. Overall, this research on recyclable tuning for continual pre-training has the potential to have a wide-ranging impact on the efficiency, sustainability, and practicality of machine learning systems.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitations (page 9).
3 A2. Did you discuss any potential risks of your work? Section Ethical Statement (page 9).
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section Abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3, Section 4, Section 5, Section B, and Section C.
3 B1. Did you cite the creators of artifacts you used? Section 3, Section 4, Section 5, Section B, and Section C.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section D.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
7 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We use the original train/dev/set partition for all the used datasets.
C 3 Did you run computational experiments? Section 4, Section 5, Section B, and Section C.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section D.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section D.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4, Section 5, Section B, Section C, and Section D.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Although currently we only support projection transferring for a similar target task that belongs to the same category of the source task, we expect future work to explore how to train a universal projection that could be applied to an arbitrary task. Performance Evaluation. The results are shown in Table 3, from which we find that: (1) our method generally outperforms the demonstration baseline, and could surpass the supervised performance (FD and FS) under certain cases, despite not using any labeled data. Hence, besides being computationally efficient, our method achieves satisfactory performance in general. This also validates our intuition that for continually pre-trained PLMs, the upgrading of a specific task could provide referential experience for similar tasks; (2) using the task data (LTKD) for distillation generally performs better than using Wikipedia (LwikiKD ), showing the importance of proper data distribution used for knowledge distillation