Rehearsal-free Continual Language Learning via Efficient Parameter Isolation
Year: 2023

Abstract:
We study the problem of defying catastrophic forgetting when learning a series of language processing tasks. Compared with previous methods, we emphasize the importance of not caching history tasks’ data, which makes the problem more challenging. Our proposed method applies the parameter isolation strategy. For each task, it allocates a small portion of private parameters and learns them with a shared pre-trained model. To load correct parameters at testing time, we introduce a simple yet effective non-parametric method. Experiments on continual language learning benchmarks show that our method is significantly better than all existing no-data-cache methods, and is comparable (or even better) than those using historical data1.

heading: 1 Introduction
text: Deployment of NLP models could be dynamic in real-world scenarios: models need to evolve continually when coming in new tasks or new domains (updating an event detection model to handle new event types, for example). Recent studies on continual (language) learning (Biesialska et al., 2020; De Lange et al., 2021) show that, compared with sticking with one single unchanging task, incrementally updating a model for a series of tasks is challenging: learning a new task will make our model perform poorly on previously learned tasks. The so-called catastrophic forgetting phenomenon is the central research topic in continual learning (Kirkpatrick et al., 2017).
One method for mitigating forgetting is through data replay (rehearsal). The model caches some previous tasks’ training data along the sequential learning process. When learning a new task, all previous tasks are re-learned with the current task. The vanilla rehearsal-based approach performs well on
1Code is available at https://github.com/Dicer-Zz/ EPI
many continual learning problems (Rebuffi et al., 2017; Rolnick et al., 2019; de Masson d’Autume et al., 2019), but access to previous tasks’ data could conflict with the original intention of continual learning. In fact, if previous tasks’ data are given, it approaches multi-task learning where all tasks are given at once and forgetting is not a problem. Meanwhile, in some situations, previous datasets are not available due to regulation or privacy considerations. Therefore, it is essential to look into rehearsal-free methods.
In this work, we study parameter isolation strategies for rehearsal-free continual language learning. Basically, each time a new task comes in, such a strategy allocates a new set of model parameters for that task, which aims to prevent potential interference with already learned knowledge, so that forgetting could be alleviated (Rusu et al., 2016; Mallya and Lazebnik, 2018). Three cruxes of making the idea work in practice are,
• for a possibly large number of tasks, we need to control the storage budget for each task.
• given a set of learned models, we need to identify the right model to query for a test sample.
• besides alleviating forgetting, we also would like to facilitate information sharing among tasks.
We give a solution for the above cruxes under the framework of parameter efficient tuning (PET) of pre-trained language models (PLMs). For each task in the continual learning sequence, we build a model for it using two sets of parameters: one is from a frozen PLM, another is an additional small set of new parameters (namely, delta parameters (Ding et al., 2022)). PET methods (Li and Liang, 2021; Hu et al., 2022; Liu et al., 2022a) show that by only fine-tuning delta parameters, performances of downstream language learning tasks could be competitive with (or even better than) the full-scale fine-tuning of the PLM. Therefore, we can share
10933
the PLM among all tasks in continual learning, while keeping a private delta parameter for each task, which is usually negligible compared with the PLM (less than 0.3% of the PLM’s parameters in our experiments).
Second, to determine which model to query in testing time, we propose using a nonparametric task identification method. Specifically, for each task, we record the first and the second moment of its training samples, and approximate the task’s input distribution with a Gaussian. Given a sample, we test whether it belongs to a task using its Mahalanobis distance to the task’s Gaussian (Lee et al., 2018; Ren et al., 2021). We show that, compared with state-of-the-art parametric task identifiers (Wang et al., 2022b,a), the nonparametric method significantly boosts the accuracy of getting the correct model (and performances of continual learning) despite its simplicity.
Third, to enhance knowledge transfer among continually learned tasks, we investigate information sharing methods among delta parameters of each task. We show that, by simply initializing the current task’s delta parameters with those of previous tasks (either by directly copying from them or soft selecting via attention mechanisms), the transfer of knowledge from learned tasks could be improved, especially in few-shot situations.
We conduct extensive evaluations on standard continual language learning benchmarks. The results show that our algorithm not only outperforms all existing rehearsal-free methods by a large margin (40% F1), but also is competitive with (or even better than) state-of-the-art rehearsal-based methods with the standard setting of data cache size.

heading: 2 Problem Definition
text: Continual learning aims to solve problems in streams: a model no longer faces a single unchanging task but a series of tasks arrived sequentially. Following previous works, we focus on task streams containing text classification problems, while the algorithm could be extended to other language learning problems.
Denote T = {T1, T2, · · · , T|T |} as a sequence of tasks, and the training set of the t-th task Tt is {(xti, yti)}, where xti ∈ Xt is an input text, yti ∈ Yt is its class label. At timestamp t, a continual learning model learns a mapping fθ : ∪tτ=1Xτ 7→ ∪tτ=1Yτ . The key property is that the model can only query the training data of Tt, and aims to
update the model parameter θ to not only predict the t-th task, but also keep performances on all previous tasks (even without using their data). In the testing phase, the model predicts a sample’s label from all seen labels ∪tτ=1Yτ without knowing which task it belongs to (class incremental).
In this work, we will focus on the above rehearsal-free setting of continual learning. Several relaxations of this setting could be applied to upper-bound its performances,
• rehearsal-based continual learning, where some previous tasks’ data can be used when training the current task. If all previous data are given, the problem is reduced to multi-task learning.
• task incremental continual learning, where task labels are given in testing time. It provides additional information about samples and makes prediction easier than the class incremental setting.

heading: 3 Approaches
text: Our method is summarized in Figure 1. In the training stage, for a new task, we assign (and save) a new private delta parameter for it and train the parameter (jointly with a shared frozen PLM) on the task’s dataset. We also save some statistics about the training set to model the task as a Gaussian distribution (e.g., the averaged vector representations of samples). In the testing stage, for a given sample, we fetch a proper learned delta parameter for predicting its label by consulting the Mahalanobis distance between the sample and the distributions of all previously encountered tasks. In the following, we start by describing the parameter isolation strategy with PET methods (Section 3.1). Then, we illustrate the task identification method used in the testing stage (Section 3.2) and knowledge sharing methods (Section 3.3).

heading: 3.1 Parameter Isolation with PET
text: One reason that causes catastrophic forgetting in continual learning is the interference among tasks: different tasks may guide the model to move towards different directions. Therefore, a simple way to alleviate forgetting is to use different parameters for different tasks. On the other side, a sufficient model capacity for each task is crucial to get highperformance models. Directly separating models could make large storage costs. Here, we adopt the parameter-efficient tuning framework which is able
to take advantage of a fixed powerful pre-trained model while keeping individual tasks moderately separated. In the following, we briefly review prefix tuning (Li and Liang, 2021) (the main PET method used in our experiments). 2
Denote a Transformer-based pre-trained model with multi-head attention blocks to be MHA(X). It maps an input sentence X ∈ Rn×d to a hidden semantic space H ∈ Rn×d, where n is the sentence length, d is the dimension of token embeddings and hidden vectors. For the t-th task in continual learning, we prepare a set of task-specific parameters by prepending p soft virtual tokens to the input of multi-head attention blocks at each layer. Denote parameters in the prefix as P ∈ Rp×d×L, where L is the number of MHA layers in the pre-trained model. 3
Besides prefixes P, another set of task-specific parameters are classification heads (we apply softmax activation for the multi-class classification). 4 During the training process, parameters of the PLM are frozen (thus, shared among all tasks), and
2The same idea can be applied with other PET methods (e.g., prompt-tuning (Hu et al., 2022), LoRA (Liu et al., 2022a)).
3In the implementation, virtual tokens are added by concatenating two Rp×d matrices to key and value matrices.
4For simplicity, we assume tasks have disjoint label spaces, thus their classification heads are different. In practice, those heads could be shared.
only private prefixes and classification heads are trained.
Private parameters are saved after training. One key property of PET methods is that one can train well enough models for tasks with a small size of P and a powerful PLM. It facilitates reducing the storage cost for parameter isolation methods. while retaining high-performance models for individual tasks (e.g., decrease by 1.34% in a benchmark dataset). In our experiments, the size of P is 0.27% of the PLM. Hence, the same storage budget for two tasks with full PLM fine-tuning can support 370 tasks with prefix tuning.

heading: 3.2 Task Identification
text: As we use different prefixes for different tasks, when a test sample comes, it is necessary to determine which prefix should be applied. One approach is to try all learned prefixes, and choose one with the largest prediction probability on the sample. In practice, this method suffers from the overconfidence problem of softmax predictors: even for those unrelated tasks, a prefix might give a prediction with high probability.
Another approach is to compare the test sample with training samples and choose its nearest neighbor’s prefix (Mensink et al., 2013). It works around the over-confidence problem, but still depends on
the robustness of sample representation and distance metrics. Here, inspired by recent studies on out-of-distribution detection (Lee et al., 2018; Ren et al., 2021), we improve this method by comparing the test sample with full distributions of tasks’ training samples. In the following, we represent an input x with the average of the last Transformer block’s outputs of the PLM, denoted by h(x) (i.e., the “pre-logit” vector).
First, for the t-th task, we perform a Guassian discriminant analysis as suggested by the softmax classifier: given p(y = c|h(x)) in the form of softmax (c ∈ Yt is a class label), we can assume p(h(x)|y = c) is a Gaussian with estimated mean µct and a shared covariance Σt among classes,
µct = 1
Nc
∑
yi=c
h(xi), (1)
Σt = 1
N
∑
c
∑
yi=c
(h(xi)− µct) (h(xi)− µct)⊤(2)
where Nc is the number of training samples with label c, and the values are obtained by maximum likelihood estimation.
Next, for a test sample x, we compare it with all tasks’ Gaussian and choose the nearest task’s prefix as the prefix for predicting x’s label. Unlike computing distance between samples, we need to apply metrics measuring distance between samples and distributions. Here, we use Mahalanobis distance. Specifically, the distance between x and class c is
− (h(x)− µct)⊤ (Σt)−1 (h(x)− µct) . (3)
In practice, we find that directly ranking tasks with Σt makes the distances have large numeric deviation. To make the computation more stable, we further share the covariance among all tasks Σ = ∑ tΣt and change the computation of Mahalanobis distance accordingly. To implement above non-parametric task identification method, we need additional storage for store class means {µct} and a shared covariance Σ. Furthermore, though they are moments of distributions, there may be a chance to get information about individual samples (Dwork and Roth, 2014). Here, inspired by querying with randomized response in differential privacy, we propose to add randomly masking on sample representations h(x). Specifically, we assign a random mask (with q% entries 0, other entries 1) for all tasks. For each h(x), the masked dimensions are dropped during the computation of µct and Σ c t . The model saves the masked
vectors and the mask itself (or the random seed generating the mask) for testing time distance computation. Besides encrypting moments, the simple masking strategy also helps to reduce storage cost.

heading: 3.3 Knowledge Transfer
text: Separating parameters is effective in mitigating catastrophic forgetting, however, it also blocks knowledge transfer among tasks. Given a sequence of learned prefixes P = {P1, ...,Pt−1}, we try several ways to utilize the knowledge acquired from preceding tasks in the hope that they could improve and accelerate the current task learning,
• Prefix Fusion. A natural way to combine knowledge from previous tasks is through the use of the attention mechanism. This allows the model to automatically extract useful information from previous tasks and integrate it into the current learning process. To achieve this, we prepend the learned prefixes of previous tasks to the prefix of the current task. Knowledge transfer is automatically facilitated through the multi-head attention mechanism of the Transformer. During new prefix learning, we fix the prefixes of previous tasks to avoid parameter drifting and catastrophic forgetting.
• Prefix Initialization. Training starting from well-trained parameters is another way to promote knowledge transfer, thus we can initialize a prefix with previously learned prefixes instead of random initialization. A good initial point can also help to speed up the convergence of the training process of PET methods. We try two ways, namely initialized with the last prefix Pt ← Pt−1 and initialized with the mean prefix Pt ← 1t−1 ∑t−1 τ=1Pτ .

heading: 4 Experiments
text: Datasets To demonstrate the generalizability of our approach, we use two kinds of datasets, differentiated according to the domain relevance between tasks. Far-domain, where the domain boundary among tasks are clear. Following MbPA++ (de Masson d’Autume et al., 2019) and IDBR (Huang et al., 2021a), we use 5-datasets collected by Zhang et al. (2015) to evaluate our method. It consists AG News (news), Yelp (business reviews), Amazon (product reviews), Yahoo!Answer (Q&A), and DBPedia (encyclopedic articles). These datasets are categorized into two
text classification tasks: topic classification (AG News, Yahoo!Answers, and DBPedia) and sentiment classification (Yelp and Amazon). Neardomain where the tasks are more closely related. We use Web of Science (WOS) from Kowsari et al. (2017) and 20 Newsgroups from Lang (1995) to assess our method for datasets with high inter-task relevance. WOS contains seven parent classes and five sub-classes under each parent class which have close relations. We organize continual learning tasks according to parent classes. The 20 Newsgroups consists six topics of news. We rearranged it into four tasks based on the principle of maximizing inter-task correlation. The details of the two datasets are in Appendix B.
Metrics Let ai,j be the testing accuracy on the i-th task after training on j-th task, the metrics for evaluating are,
• Performance of Continual Learning (CL). The average accuracy of all tasks after training on the last task, 1|T | ∑|T | i=1 ai,|T |
• Forgetting (Fgt). The degree of forgetting of previous tasks after training on the last task,
1 |T |−1 ∑|T |−1 i=1 (max |T |−1 k=i ai,k − ai,|T |)
• Accuracy of Task Identification (TI). The accuracy of getting the correct prefix for testing samples after training on all tasks.
Baselines We use the following continual learning techniques as baselines:
• FT, fine-tuning a model for each task sequentially while catastrophic forgetting occurs. This method is the lower bound of continual learning.
• MTL, training a model on all tasks as multi-task learning. This method is the upper bound of continual learning.
• Replay, saving part of the previous tasks as memory, train a model one step on the memory after every β ( 10 for our experiments) steps of training on the new task.
• LwF (Li and Hoiem, 2017), a typical regularization-based approach. We also combine LwF with the replay method as an enhancement.
• IDBR (Huang et al., 2021b), disentangling information by two simple auxiliary tasks (next
sentence prediction and task-id prediction) for learning better generic and specific representation spaces. This approach applies both episodic memory replay and regularization techniques.
• L2P (Wang et al., 2022b), it first introduces a prompt-based framework to continual learning. The main difference between L2P and our method is that L2P uses a parametric task identifier, while our identifier is non-parametric. We also adopt L2P-R, which is L2P equipped with a rehearsal buffer.
To ensure a fair comparison, FT, MTL, Replay and LwF are all prefix-based. We re-implement L2P to make it support prefixes (instead of only prompts in the original code).
Details We use BERT (Devlin et al., 2019) from HuggingFace Transformers (Wolf et al., 2020) as the PLM of our model. We set the default prefixes length to 16. Because our methods progressively assign a prefix to new tasks, we assign the same number of parameters to all baseline methods to ensure a fair comparison. See Appendix C for other configurations.

heading: 4.1 Main Results
text: In Table 1, we present the results of our method and baselines on the sampled 5-datasets and WOS. On the sampled 5-datasets, the rehearsal-based approach stores 50 samples per class, equivalent to 2.5% of the entire training dataset (except for IDBR, which is 20 per class). From the results, we can find that,
• On sampled 5-datasets, our approach surpasses all rehearsal-free methods by a large margin. It is even better than rehearsal-based methods: compared with the previous SOTA method IDBR, our method improves the accuracy from 73.19% to 74.43%, reducing the gap to the upper bound (75.40%) by 56%.
• When changing task orders of the sampled 5- datasets, the standard deviation of our method is small, which indicates that it is insensitive to task orders, which is rare in approaches without replaying.
• On the more challenging WOS, our approach is still comparable to replay methods that use approximately 10% training data (20 samples per class).
Table 2 presents the performances on the full 5-datasets. The conclusions drawn from this table are generally consistent with Table 1, but it also showcases some new findings:
• Our rehearsal-free method, even without task ID during the testing phase (class incremental), can outperform previous SOTA methods such as LAMOL (Sun et al., 2020), which relies on rehearsal and has knowledge of the task ID during testing (task incremental).
• When provided with the task ID ("oracle"), our method’s performance can be further improved by 1.7%. This suggests that there is still potential for enhancing our approach.

heading: 4.2 Discussions
text: To further inspect the proposed methods, we investigate the following research questions.
How the task identifier performs? To show the validity of the Mahalanobis distance (MD), we
compare it with two methods: (i) Maximum over softmax probability (MSP): using all prefixes for inference (task by task) and then choosing the label with the highest probability; (ii) Euclidean distance (ED) which ignores the covariance in Equation 3. The results are in Figure 2. We find that,
• MSP performs poorly (especially on 5-datasets). It shows that, at least for the vanilla MSP metric, the over-confidence problem is still essential. Another drawback of MSP is that it has to perform forward passes with all prefixes while our method only needs to compare vectors.
• Mahalanobis distance is always better than Euclidean distance for task identification. Therefore, different dimensions of sample representations should have different importance for detecting tasks, and modeling tasks’ distribution with anisotropic Gaussians provides a better approximation of the actual distribution.
Does prefix tuning help continual language learning? We can directly use the nonparametric task identifier to infer class labels of samples. Hence, one question is whether the additional prefix parameters provide performance gains. We build two non-parametric classifiers which use Euclidean and Mahalanobis distance. The samples are encoded only with representations from the PLM (without prefixes). As shown in Table 3, we can find that,
• Regarding the performance of continual learning (CL), the two non-parametric classifiers perform quite well: the classifier with Euclidean distance performs much better (62.8%) than all rehearsal-free method in Table 1 (< 30%) on 5-datasets, and is competitive on WOS. The classifier with Mahalanobis distance is more effective (69.8%): it is even competitive with the primary reply method (Replay). The success of non-parametric classifiers implies that the powerful representation ability from PLM is crucial to perform continual learning.
• With the help of parametric prefix tuning, the performances of the two methods are largely boosted. It proves that task-specific information is also important. Regarding the performances
of direct fine-tuning in Table 1, using separate prefixes could alleviate catastrophic forgetting.
How do different PLMs influence performances? The above analyses on non-parametric classifiers suggest us to explore the influence of different PLMs (Figure 3 and Table 4). We find that,
• Owing to the presence of a larger number of parameters, the sentence representations generated by larger models enhance task identification. Moreover, previous studies also suggest that the larger the PLM, the more effective the PET (Lester et al., 2021; Liu et al., 2022b). Figure 3 presents the performance across model scales, which are consistent with the previous findings in continual language learning.
• The PLM pre-trained on the same domain of continual learning tasks is able to extract a more valuable representation. For WOS, we replace BERT with SciBERT (Beltagy et al., 2019), which is a BERT-like model pre-trained on the scientific (WOS domain) corpus. As shown in Table 4, the closer SciBERT performs even better than bertlarge though it is smaller (with the same size as bert-base). Therefore, if some unsupervised texts from the same domain of continual learning are given, one could fine-tune the PLM for a better performance.
How random masking influence the task identifier? To validate the effect of random masking, we use five masking ratios equally spaced from 0% to 80% and two high masking ratios (95% and 99%) The results are in Figure 4. Our method shows a slight performance degradation in the range of masking ratio from 0% to 80%. Compared with no mask (i.e., masking ratio equal to 0), a masking ratio of 80% still gives more than 90% of the performance, and it only needs about 4% of the storage space for all three datasets. It suggests that pre-logit vectors (h(x)) is highly redundant for distinguishing tasks. However, as the masking ratio increases further, the task identification performances decrease rapidly, which causes the end continual learning performances to decrease as well.
Does knowledge transferability of prefixes matter? Finally, we evaluate knowledge transfer methods (Section 3.3) in few-shot settings. As shown in Table 5, we observe knowledge transfer in all settings on WOS for all transfer methods but
only in the 10-shot setting on 5-datasets for Mean. This is because the tasks in the WOS dataset are related, and thus knowledge can be shared among them. Moreover, we do not observe forward transfer in the full-shot setting on any benchmark, because knowledge transfer is unnecessary when the data are sufficient.

heading: 5 Related work
text: Continual Learning We discuss three main categories of continual learning methods: rehearsalbased, regularization-based, and parameter isolation methods.
Rehearsal-based methods alleviate catastrophic forgetting by replaying stored examples (Rebuffi et al., 2017; Rolnick et al., 2019; de Masson d’Autume et al., 2019) or pseudo-generative examples (Shin et al., 2017; Su et al., 2020; Sun et al., 2020) of previous tasks. Unfortunately, all of them carry the risk of privacy leakage and need nontrivial storage space.
Regularization-based methods restrict the updating of the model weights by knowledge distillation (Li and Hoiem, 2017; Triki et al., 2017) or parameter importance (Kirkpatrick et al., 2016; Zenke et al., 2017; Aljundi et al., 2018) to preserve the knowledge of previous tasks. However, these methods emphasize the model’s stability to previous tasks while weakening its plasticity to the new task (Parisi et al., 2019).
Parameter isolation methods assign a taskspecific parameter to a new task by splitting (Fernando et al., 2017; Mallya and Lazebnik, 2018; Serrà et al., 2018) or extending (Rusu et al., 2016; Xu and Zhu, 2018) the current model to prevent interference between tasks. Because these methods require task-id to choose the proper model at testing time, they only apply to task incremental
continual learning. Although our method also belongs to this category, our method applies to class incremental by introducing task identification.

heading: 6 Conclusion
text: In this work, we propose a new rehearsal-free parameter isolation continual learning method that leverages the capabilities of a pre-trained language model (PLM). Extensive experiments show that our method surpasses all rehearsal-free methods by a significant margin and is comparable (or even better) than previous start-of-the-art rehearsal-based methods on two benchmarks, whether the tasks are near or far. Meanwhile, we introduce random static masking to reduce the storage required by our method to adapt it to more demanding scenarios.
Limitations
Although our proposed knowledge transfer methods work well on WOS in the few-shot setting, it is less effective on 5-datasets. Moreover, all methods fail in the full-shot setting. Based on our approach, a more general approach to knowledge transfer is expected in future works. In addition, our approach requires a well-trained language model for task identification and a Transformer-based model (well-trained also) for parameter efficient tuning. Therefore, it is challenging to cooperate our approach with a language model with random initialization or non-transformer architecture.

heading: Acknowledgement
text: The authors wish to thank all reviewers for their helpful comments and suggestions. The corresponding authors are Zhicheng Wang, Tao Ji and Yuanbin Wu. This research was (partially) supported by NSFC(62076097), National Key R&D Program of China (2021YFC3340700), and East China Normal University International Conference Grant Programme.

heading: A Potential Risks
text: Pre-trained language models (PLMs) may inherit biases from the training corpus, resulting in offensive behaviors. Combining our approach with these toxic language models and deploying them in realistic scenarios might cause negative social impacts.

heading: B Datasets Details
text: For 5-datasets, the sampled setting samples 2,000 training examples and 2,000 validation examples from the original training set. The full setting is the same as the dataset used by de Masson d’Autume et al. (2019). We show task orders of 5-datasets we used in Table 6. We did not merge the label space of Yelp and Amazon as MbPA++ and IDBR did to create a more challenging setup. Our label space can be mapped to the label space used by MbPA++ and IDBR and not vice versa.
We download WOS-11967 from Huggingface Datasets (Lhoest et al., 2021), and we show the statistics of it in Table 7. We split the whole dataset to train/val/test set in the ratio of 0.6:0.2:0.2.
We access 20 Newsgroups from Huggingface Datasets corresponding to the 20news-bydata version on the official site. We show the task separation in Table 8. We take one-sixth of the training set as a validation set for a train/val/test splitting ratio 0.5:0.1:0.4.

heading: C Experimental Details
text: We train all models using AdamW (Loshchilov and Hutter, 2017) with β1 = 0.9 and β2 = 0.99 cou-
Class Name Train Test
Task 1 comp.graphics 584 389 rec.autos 594 396 sci.crypt 595 396 misc.forsale 585 390 talk.politics.misc 465 310 talk.religion.misc 377 251
Task 2 comp.os.ms-windows.misc 591 394 rec.motorcycles 598 398 sci.electronics 591 393 talk.politics.guns 546 364 alt.atheism 480 319
Task 3 comp.sys.ibm.pc.hardware 590 392 rec.sport.baseball 597 397 sci.med 594 396 talk.politics.mideast 564 376 soc.religion.christian 599 398
Task 4 comp.sys.mac.hardware 578 385 rec.sport.hockey 600 399 sci.space 593 394 comp.windows.x 593 395
Table 8: The task separation and statistic of original 20 Newsgroups.
pled with a linear scheduler with a warm-up ratio of 0.1. For sampled 5-datasets, WOS, and 20 Newsgroups, we set the identical learning rate λ = 0.03. For the full 5-datasets, we do grid searching for the learning rate for each task respectively, and the final learning rates are 0.003, 0.009, 0.005, 0.007, 0.003 for AG News, Amazon, Yelp, Yahoo!Answer and DBpedia, respectively. All experiments are conducted on NVIDIA RTX 3090 with 24GB video memory with a batch size of 32 and the maximum length of a sentence is 256.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations (Line 558-570)

heading: 3 A2. Did you discuss any potential risks of your work?
text: Appendix A (Line 838-844)

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and section 1 (Line 1-112)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 (Line 297-323)

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 (Line 297-323)
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We do not find any license in original papers.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4 (Line 297-323)
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? All of datasets comes from publicly available news or published articles, so we don’t think there are such problems.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? That information can be found in original papers.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix B (Line 845-865)
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4 and append C (Line 876-879)
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4 and appendix C (Line 867-876)

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4 (Line 368-370)
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Based on our approach, a more general approach to knowledge transfer is expected in future works. In addition, our approach requires a well-trained language model for task identification and a Transformer-based model (well-trained also) for parameter efficient tuning. Therefore, it is challenging to cooperate our approach with a language model with random initialization or non-transformer architecture