HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks
Year: 2023

Abstract:
With the scale and capacity of pretrained models growing rapidly, parameter-efficient language model tuning has emerged as a popular paradigm for solving various NLP and Visionand-Language (V&L) tasks. In this paper, we design a unified parameter-efficient multitask learning framework that works effectively on both NLP and V&L tasks. In particular, we use a shared hypernetwork that takes trainable hyper-embeddings and visual modality as input, and outputs weights for different modules in a pretrained language model, such as the parameters inserted into multi-head attention blocks (i.e., prefix-tuning) and feed-forward blocks (i.e., adapter-tuning.). Our proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods. Empirical results on the GLUE benchmark and multiple V&L tasks confirm the effectiveness of our framework.

heading: 1 Introduction
text: Pretraining and fine-tuning are now the prevalent paradigm in natural language processing, yielding state-of-the-art performances on a variety of tasks (Devlin et al., 2019). With pre-trained language models (PLMs) growing rapidly in size, it becomes increasingly infeasible to perform conventional fine-tuning on the entire model parameters. There has recently been one line of research on Parameter-Efficient Language model Tuning (PELT)(Houlsby et al., 2019; Li and Liang, 2021; He et al., 2021; Mao et al., 2022). They only update a set of extra trainable task-specific parameters that are newly introduced to PLMs. Although the number of new parameters is much fewer than the
*Work is done at the internship of Noah’s Ark Lab, Huawei Technologies.
†Equal Contribution. ‡Corresponding authors.
original PLM, training these parameters per single task is still costly, especially when targeting a number of tasks, i.e., multi-tasking scenario.
Therefore, we are motivated to start with a unified parameter-efficient language model tuning framework (He et al., 2021) and explore on a shared hypernetwork (von Oswald et al., 2020; Mahabadi et al., 2021) that is able to take multi-task information as input, and generate weights for tuning different task-specific modules of PLMs, such as the parameters inserted into multi-head attention blocks (i.e., prefix-tuning) and feed-forward blocks (i.e., adapter-tuning.). We name it HyperPELT. Besides, we propose a novel perspective of adopting parameter-efficient multimodal fusion for PLMs via the hypernetwork. Thus we explore to use an additional separate hypernetwork handling visual input and generating visual-specific weights for multiple modules of PLMs.
Empirical results on 8 tasks of GLUE benchmark show that HyperPELT achieves superior performances (87.09 vs. 86.53) with a tenth of the parameters (0.24% vs. 2.96%) when compared to state-of-the-art alternatives. Study on the few-shot transfer learning indicates that HyperPELT is more stable and efficient than alternatives. It confirms the effectiveness of our unified parameter-efficient multitask learning framework. What’s more, we evaluate our framework on V&L multi-tasks (4 tasks). Results show the promising performance of our novel fusion method on extending V&L ability on top of PLMs via hypernetworks.
In summary, we make the following contributions: (1) propose a unified parameter-efficient multitask learning framework that is able to take multi-task and multi-modality information as input, and generate weights for tuning different taskspecific modules of PLMs; (2) present a novel perspective of using hypernetworks to achieve the parameter-efficient multimodal fusion on top of PLMs; (3) design various experiments to compre-
11442
hensively demonstrate the effectiveness of our proposed framework in multi-task learning and fewshot domain transfer scenarios.

heading: 2 Related Work
text: Existing research has explored a large amount of methods on parameter-efficient tuning, such as the widely used adapter-tuning (Houlsby et al., 2019), prefix-tuning (Li and Liang, 2021) and the mixed methods (He et al., 2021; Mao et al., 2022). However, it is time & space-consuming to deal with a set of tasks in multi-task learning if we simply update and save separate replicas of model parameters per single task. In this work, we explore a hypernetwork-based multi-task learning framework to generate weights for different PELT modules.
Besides, there has been a series of recent work (Cho et al., 2021; Tsimpoukelli et al., 2021; Sung et al., 2021; Alayrac et al., 2022) to equip a language model with the ability of handling visual input with a small number of trainable modules and parameters. Different from existing work, we propose a novel perspective of multimodal fusion via extending the proposed parameter-efficient multitask learning framework. We further review recent research on parameter-efficient tuning for pure language and V&L tasks, as well as the corresponding work for multi-task learning in Appendix A.

heading: 3 Proposed Method
text: We target a general multi-task learning problem, which is formulated in Appendix B. In this section, we describe the hyper-embedding I for hypernetworks to generate weights ∆θ and which modules of PLMs to insert these weights to achieve PELT. In our methods, the hyper-embedding I consists of two: task-specific hyper-embedding Iτ , and visualspecific hyper-embedding Iv. We will mostly introduce the hyper-embedding Iτ , and Iv is used in a similar parallel manner. A simple linear projection layer is employed as the hypernetwork, for example, hτP (.) and h v P (.) are used for prefix-tuning, while hτA(.) and h v A(.) are for adapter-tuning as shown in Figure 1. The hypernetwork takes the hyper-embedding I as input, and outputs weights for multiple modules of PLMs.

heading: 3.1 Hyper-Embedding for PELT
text: Considering a flexible parameterization of taskspecific parameters for L layers of transformer, we introduce a set of layer id embeddings I = {li}Li=1, and block type embeddings B = {bj}5j=1, which specify the position where the parameters ∆θ are inserted to. Then, we compute a hyper-embedding Iτ ∈ RdI for each individual task via a task projector network, which is a multi-layer perceptron consisting of two feed-forward layers and a ReLU nonlinearity: Iτ = MLP([zτ , li, bj ]). Thus, it learns a suitable compressed hyper-embedding from a concatenation of task embeddings zτ ∈ Rdτ , layer id
embeddings li ∈ Rdτ , and block type embeddings bj ∈ Rdτ . In this way, the hypernetwork is able to produce distinct weights for tuning each task, and each transformer block at each layer.

heading: 3.2 HyperPELT: Incorporate with Prefix-tuning and Adapter-tuning
text: To further capture knowledge across tasks and transfer to others, we follow the unified parameterefficient framework (He et al., 2021), and input the hyper-embedding to a hypernetwork for generating the weights in adapters as well as prefix vectors. We extend the dimension for different embeddings to match the prefix length N , i.e., z ∈ RN×dτ , li ∈ RN×dτ , bj ∈ RN×dτ , and then compute the hyper-embedding Iτ ∈ RN×dI . We finally employ a hypernetwork hτP (.) with trainable parameters θhτP , to project Iτ to prefix vectors Pτ ∈ R
N×d: Pτ = h
τ P (θhτP , Iτ ) .
Besides, as depicted in Figure 1, we introduce a hypernetwork-based adapter layer with a trainable scaled parameter λ, which is inserted parallelly with feed-forward blocks. We generate adapter weights (W τup,W τ down) through a hypernetwork hτA(.): (W τ up,W τ down) := h τ A(θhτA , Iτ ), where W τdown ∈ Rdmid×d and W τup ∈ Rd×dmid .

heading: 3.3 VL-HyperPELT: Incorporate with Visual Modality
text: As illustrated in Fig. 1, we use CLIP (Radford et al., 2021) with a trainable visual mapping layer, which projects the visual representation to the identical dimension of task embedding, i.e., zv ∈ RN×dv , dv = dτ . Then we feed this visual representation zv to a visual projector network. In this way, we learn the visual hyper-embedding Iv ∈ RdI . Finally, taking the visual-specific hyperembeddings as input, we use visual-specific hypernetworks to generate visual-specific parameters to different modules in PLMs. Similar to the Section 3.1 & 3.2, the incorporation of visual-specific parameters to PLMs are the same as task-specific ones, e.g., used as prefix vectors via a prefix hypernetwork hvP (.) and adapter weights via an adapter hypernetwork hvA(.). We name it VL-HyperPELT.

heading: 4 Results and Analysis
text: We conduct a series of experiments to verify the effectiveness of our proposed framework compared to existing ones.

heading: 4.1 Implementation Details
text: Our models are built on T5BASE (Raffel et al., 2020) 1, which contains 12 layers and 222M parameters, and use the tokenizer of T5 to tokenize text inputs. We set N = 49, d = dτ = 768, dI = 64 for all the experiments. Following the training strategies from Raffel et al. (2020), we fine-tune all models with a constant learning rate of 0.001, use 218 = 262144 steps in all experiments with batch size of 128 and sample tasks via the conventional temperature-based sampler with temperature T = 2, i.e., sample corresponding task proportional to p1/Tτ , where pτ = Nτ∑T
i=1 Nτ and
Nτ is the number of training samples for the τ -th task. We did not experiment with other complex sampling strategies or tuning of T . For the experiments under multi-task training settings, we save a checkpoint every 1000 steps and report results on a single checkpoint with the highest average validation performance across all tasks.
In terms of the vision-and-language scenarios, we convert V&L tasks to the text generation format as Cho et al. (2021). We use ResNet101 as our vision encoder, and initialize it with weights from pretrained CLIP (Radford et al., 2021). Input images are resized to 224 × 224 for memory efficiency. We extract the 7 × 7 grid features produced by the last convolutional layer. The percentage of updated parameters is also reported as one metric for approach efficiency, and we do not take visual encoder into account since it is frozen in our experiment.

heading: 4.2 Datasets
text: Our framework is evaluated on the GLUE benchmark (Wang et al., 2019b) in terms of natural language understanding. This benchmark covers multiple tasks of paraphrase detection (MRPC, QQP), sentiment classification (SST-2), natural language inference (MNLI, RTE, QNLI), and linguistic acceptability (CoLA). The original test sets are not publicly available, and following Zhang et al. (2021), for datasets fewer than 10K samples (RTE, MRPC, STS-B, CoLA), we split the original validation set into two halves, one for validation and the other for testing. For other datasets, we randomly split 1K samples from the training set for validation and test on the original validation set.
In addition, we evaluate the few-shot transfer performance on four tasks and datasets: 1) the
1https://huggingface.co/t5-base
natural language inference (NLI) datasets CB and 2) the question answering (QA) dataset BoolQ from SuperGLUE (Wang et al., 2019a); 3) the sentiment analysis datasets IMDB (Maas et al., 2011); and 4) the paraphrase detection dataset PAWS (Zhang et al., 2019). For CB and BoolQ, since the test set is not available, we split the validation set into two halves, one for validation and the other for testing. For IMDB, since the validation set is not available, we similarly split the test set to form validation. For PAWS, we report on the original test set.
To evaluate our framework on V&L tasks, we experiment on four datasets COCO (Lin et al., 2014), VQA (Goyal et al., 2017), VG-QA (Krishna et al., 2017) and GQA (Hudson and Manning, 2019). Following Cho et al. (2021), we use VQA Karpathy split, which splits the VQA dataset into 605,102 / 26,729 / 26,280 image and question pairs separately as the train/validation/test set to evaluate VQA tasks in a generative manner. We further evaluate our framework on two datasets for V&L few-shot transfer learning: OKVQA (Marino et al., 2019); SNLI-VE (Xie et al., 2018).

heading: 4.3 Results on the GLUE Benchmark
text: We conduct experiments on GLUE for both singleand multi-task settings, as shown in Table 1. Compared to the single-task Adapters that finetunes all newly introduced parameters in adapters, our method yields a significant improvement by 2.21% with much fewer trainable parameters. It illustrates the effectiveness of our proposed multi-task training framework. The comparison to MAMAdapter shows that using hypernetwork to tune each transformer module and thus learn the shared knowledge across multitasks, leads to an improvement in task performance (86.53 vs. 87.09) while training fewer parameters (2.96% vs. 0.24%). Overall, our HyperPELT obtains the best performance with less trainable parameters.

heading: 4.4 Few-shot Domain Transfer
text: We use the above models trained on GLUE as reported in Table 1, and evaluate them on the test set of four different tasks, i.e., PAWS, IMDB, BoolQ, and CB, after being few-shot finetuned on each target training data, as shown in Figure 2. For the
tasks of CB and BoolQ from SuperGLUE, even though the backbone T5 was previously trained on the train sets of these two, the performance of all methods differs a lot. The two baselines still do not work with very few samples, like 4 and 16 samples. Therefore, we assume that the two baselines suffer from catastrophic forgetting problems to some degree during multi-task training. In contrast, our proposed HyperPELT works effectively on these two tasks. We speculate that the reason might be the use of hypernetworks on both prefix-tuning and adapter-tuning modules of transformer. We leave this exploration to our future work.
Besides, we show the results of Prompttuning (Lester et al., 2021) and fine-tuning only the task embedding in our HyperPELT. Note that in this comparison, we keep the same trainable parameters between these two methods, i.e., RN×dτ , where N denotes the prompt length in Prompt-tuning method. Our HyperPELT TaskEmbed mostly achieves a comparable or even better performance than Prompt-tuning.

heading: 4.5 Results on Vision-and-Language Benchmarks
text: We compare the pre-trained and full fine-tuning VL-T5 (Cho et al., 2021), and other adapter-based methods built on top of T5, i.e., CLIP-T5 and VLAdapter (Sung et al., 2021) in the multi-task training setting. The results and the number of trainable parameters are reported in Table 2. Since the used dataset is slightly different from Sung et al. (2021) and their checkpoint is not avaliable at this time, we re-implement CLIP-T5 and VL-Adpater. Compared to which, our method achieves a comparable performance with a fewer number of trainable pa-
rameters (e.g., 7.16% of VL-Adapter vs. 6.62% of VL-HyperPELT).
We further evaluate our models on multimodal few shot learning tasks and show its superiority in appendix E.1. To our best knowledge, we are the first to employ the visual modality to tune the very few parameters of different transformer blocks, instead of normally inserting image patch tokens to the input sequence. Experimental results evidence the effectiveness of our novel approach, thus providing a new perspective on how to extend the multi-modality capability on top of PLMs.

heading: 5 Discussion and Conclusion
text: In this paper, we propose a unified parameterefficient tuning framework for multitasks. On the one hand, we use the hypernetwork to reduce the scale of trainable parameters of existing adaptertuning and prefix-tuning modules. On the other hand, for the V&L tasks, we directly integrate the image features into the prefix vectors as well as adapters, which further reduces the number of trainable parameters for processing visual input. Extensive experiments on pure language and V&L tasks demonstrate the superiority of our proposed framework in both multi-tasking and few-shot settings. In the future, we plan to explore more combination of methods across tuning task-specific and visualspecific parameters for different modules of PLMs.
Limitations
Our experiments are conducted based on the T5base pre-trained language model. Due to the computational resource constraints, we did not conduct experiments on other similar PLMs, such as BART, and T5 model with larger scale, such as T5-large
and T5-3B. Although we believe our conclusion can generalize to other backbones since T5 is a classical encoder-decoder model, we will conduct more experiments to confirm for future work.

heading: A Related Work
text: In this section, we review recent research on parameter-efficient tuning for pure language and V&L tasks, as well as the corresponding work for multi-task learning.
A.1 Parameter-Efficient Multi-task Learning
As recent models grow rapidly in size, how to finetune pretrained models with a small number of trainable parameters becomes more crucial. Existing research (Liu et al., 2021a; Ding et al., 2022) has explored a large amount of methods on parameter-efficient tuning. These methods generally include two categories according to whether new trainable parameters are introduced. One category is that only a subset of model parameters can be updated while freezing the remain (Liu et al., 2021b; Lee et al., 2019). The other is introducing a few task-specific new parameters to different parts of pretrained models, such as multi-head attention (Li and Liang, 2021) and feedforward layers (Houlsby et al., 2019). In this method, a small network (often named as hypernetwork with the input embedding named as hyper-embedding) is often used to generate weights for a main network.
On the other hand, learning a unified model to perform well on multiple tasks (i.e., multi-task learning) is a challenging problem. It has to address many challenges such as catastrophic forgetting, and model overfitting in low-resource tasks while underfitting in high-resource tasks (Aharoni et al., 2019). Radford et al. (2019) highlights the ability of language models to perform a wide range of multitasks in a zero-shot setting. Mahabadi et al. (2021) proposes to use a shared hypernetwork (von Oswald et al., 2020) to generate weights for a small number of parameters in adapter modules, thus to allow the model to adapt to each individual task in a parameter-efficient manner.
A range of recent work aims to unify parameterefficient tuning methods (He et al., 2021; Mao et al., 2022), to achieve better tuning performance. We explore a framework to generate weights for different PELT methods using the hypernetwork. Compared to only generating weights for adapters, empirical results indicate that generating weights for multiple modules of PLMs achieves superior performance
with fewer trainable parameters.
A.2 Parameter-Efficient Tuning towards Vision-and-Language
Building vision-and-language models on top of PLMs pretrained on pure large text corpora has led to a noticeable improvement to V&L tasks (Cho et al., 2021). There is a series of recent work that extends the ability of language models to handle multimodal input in a parameter-efficient manner. For example, Frozen (Tsimpoukelli et al., 2021) aligns the image representation to the text representation space of frozen GPT model which thus is able to generate captions for images. VL-Adapter (Sung et al., 2021) introduces a limited set of new trainable parameters to T5 via the adapter-tuning approach that can match the performance of finetuning the entire model. Flamingo (Alayrac et al., 2022) uses an extra cross-attention module, whose keys and values are generated via visual features, thus enabling language modeling conditioned on visual inputs. Different from existing work, we propose a novel perspective of parameter-efficient multimodal fusion. We introduce a seperate visualspecific hypernetwork for handling visual input and generating weights for PLMs.

heading: B Mutli-task Learning Problem Formulation
text: Our paper targets at a general multi-task learning problem, where we are given the data from a set of tasks {Dτ}Tτ=1. T is the total number of tasks and Dτ = {(xiτ , yiτ )}Nτi=1 is the training data of the τ -th task with Nτ samples. We are also given a large-scale pretrained language model, i.e., T5, parameterized by θ, which generates the output yiτ for input xiτ . The standard multi-task finetuning minimizes the following loss on the training set:
Ltotal = T∑
τ=1
∑
(xiτ ,y i τ )∈Dτ
Ltask(θ, xiτ , yiτ ), (1)
where Ltask is the loss function of the tasks that is usually defined as the cross-entropy loss. Our goal is to efficiently finetune the given model in this multi-task learning setting, allowing knowledge sharing across tasks, and at the same time, enabling the model to adapt to each individual task.
We aim to integrate a unified hypernetworkbased parameter-efficient transfer learning method into a multi-task transformer model. In other word, we insert the parameters generated by the hypernetworks ∆θ into the layer and attention blocks of PLMs. During training, we only update the hypernetwork parameters θh with hyper-embedding {Iτ}Tτ=1 and parameters in layer normalization, while the remaining model parameters in θ are fixed as in the Equation 2.
Ltotal = T∑
τ=1
∑
(xiτ ,y i τ )∈Dτ
Ltask(∆θ, θ, xiτ , yiτ )
= T∑
τ=1
∑
(xiτ ,y i τ )∈Dτ
Ltask(Iτ , θh, θ, xiτ , yiτ ) ,
(2)

heading: C Number of Tunable Parameters
text: Following He et al. (2021), to simplify the computation of tunable parameters, we compute the
sum of parameter used in one encoder layer and one decoder layer as the parameter overhead of one single layer of the pre-trained encoder-decoder model. T5 has an encoder-decoder structure that has L layers. Each layer has Battn blocks and Bffn blocks. For the encoder-decoder models like T5, Battn = 3: the encoder self-attention block, the decoder self-attention block and the decoder cross-attention block and Bffn = 2: encoder feedforward block and decoder feed-forward block. For modifications applied at the attention blocks, the number of tunable parameters is computed by θattn = θattnW × Battn × L, where θattnW denotes the number of parameters used for one attention sub-layer. Similarly, the number of tunable parameters for the FFN sub-layers is computed by θffn = θ ffn W × Bffn × L. Finally, the total number of tunable parameters for prefix tuning and adapter variants is θ = θattn + θffn as applicable. Using T5 as an example, we present the number of parameters used by several representative methods throughout our paper in Tab. 3.

heading: D Experimental Setup
text: D.1 Input-Output Formats
As shown in Tab. 4, we formulate the input text and labels from each task to the corresponding
target text, and we learn these different tasks by predicting target text with the language modeling objective in Eq. 2.

heading: E Additional Results and Analysis
text: E.1 Multimodal Few-shot Learning We further use the models trained on V&L tasks as reported in Figure 3 and evaluate them on the test set after few-shot fine-tuning on OKVQA (Marino et al., 2019) and SNLI-VE (Xie et al., 2018). For OKVQA, since there is no test set, we split its original validation set into two halves, one for validating and the other for testing. For SNLI-VE, we use its validation set for validating, and test-P set for testing and reporting results. We follow the methods in Section 4.4 to select samples, and report results in Figure 3.
Compared with the full parameter fine-tuning, i.e., CLIP-T5, and the previous parameter-efficient V&L method VL-Adapter, our method achieves the best performance. It is also worth noting that for the used five random seeds, the variance of our method is generally smaller than VL-Adapter, which indicates that our method is more robust in this few-shot learning scenario. We believe that our framework, though training less parameters, can still capture knowledge across tasks and transfer them in the multimodal few-shot setting.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 6
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Appendix D.2
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? All of the datasets used in this paper use open-source licenses, and we make no modifications to the datasets in this paper. We will mark the open source licenses of the datasets in the open-source repository.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix D.2
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? These problems have been discussed in the original paper or websites which published the datasets.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? These information have been stated in the original paper or websites which published the datasets. We cite the link of each datasets used andthe reviewer can find these information there.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix D.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4 and Appendix D.2

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix D.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix D.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In the future, we plan to explore more combination of methods across tuning task-specific and visualspecific parameters for different modules of PLMs. Limitations
Our experiments are conducted based on the T5base pre-trained language model. Due to the computational resource constraints, we did not conduct experiments on other similar PLMs, such as BART, and T5 model with larger scale, such as T5-large
and T5-3B