Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefix
Authors: Kuan-Hao Huang
Liang Tan
Rui Hou
Sinong Wang
Amjad Almahairi
Ruty Rinott
Venue: EMNLP 2023 Findings (2023)
Keywords: text representations
prefix tuning
TL;DR: We demonstrate that combining the prefixes independently learned from NLP tasks can lead to better generalizability of representations compared to multi-tasking training.

Abstract: Many real-world applications require making multiple predictions from the same text. Fine-tuning a large pre-trained language model for each downstream task causes computational burdens in the inference time due to several times of forward passes. To amortize the computational cost, freezing the language model and building lightweight models for downstream tasks based on fixed text representations are common solutions. Accordingly, how to learn fixed but general text representations that can generalize well to unseen downstream tasks becomes a challenge. Previous works have shown that the generalizability of representations can be improved by fine-tuning the pre-trained language model with some source tasks in a multi-tasking way. In this work, we propose a prefix-based method to learn the fixed text representations with source tasks. We learn a task-specific prefix for each source task independently and combine them to get the final representations. Our experimental results show that prefix-based training performs better than multi-tasking training and can update the text representations at a smaller computational cost than multi-tasking training.

Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefixes
Kuan-Hao Huang Liang Tan
Rui Hou
Sinong Wang Amjad Almahairi
Ruty Rinott
UUniversity of California, Los Angeles khhuangēcs ucla.cllacooooo edu (liangtan, rayhou, sinongwang, aalmah, ruty)@meta.com com
Meta Al
Abstract
which causes computational issues especially when the number of downstream tasks grows.
Many real-world applications require making multiple predictions from the same text. Fine- tuning large pre-trained language model each downstream task causes computational burdens in the inference time due to several times of forward passes. To amortize the com- putational cost, freezing the language model and building lightweight models for down- stream tasks based fixed text representations are common solutions. Accordingly, how to learn fixed but general text representations that generalize well to unseen downstream tasks becomes a challenge. Previous works have shown that the generalizability of representa- tions can be improved by fine-tuning the pre- trained language model with some source tasks in multi-tasking way. this work, we pro- pose prefix-based method to learn the fixed text representations with source tasks. We learn task-specift prefix for each source task in- dependently and combine them to get the final representations. Our experimental results show that prefix-basee training performs better than multi-tasking training and can update the text representations smaller computational cost than multi-tasking training.
To amortize the computational cost, several works (Peters et al., 2019; Du et al., 2020) con- sider freezing the language model the text en- coder. They use the frozen language model to ob- tain the fixed representations for a text and build a lightweight model for each downstream task on top of such fixed representations. They show that by fine-tuning a pre-trained language model with some source tasks in a multi-tasking way, the generated fixed representations capture general information and generalize well for unseen target tasks.
In this work, we consider the same goal and make the inference computationally efficient. We aim to learn fixed representations from some source tasks that can generalize to unseen target tasks. In- stead of multi-tasking training, we propose method based on the prefix tuning (Li and Liang, 2021: Liu al., 2022b) to learn the fixed repre- sentations. Specifically, we learn a task-specific prefix for each source task independently. During inference time, all the task-specific prefixes are combined together to produce the final fixed repre- sentations. Since those prefixes carry task-specifis information, the generated fixed representations capture enough information generalize to unseen target tasks.
1 Introduction
Fine-tuning large pre-trained language models for downstream tasks has become a popular solution in natural language processing (Devlin et al., 2019; Liu et al., 2019b). Although effective, fine-tuning the whole language model might cause some com- putational burdens, especially for those applica- tions that require making multiple predictions from the same text. For example, given a Facebook post, we may want to know its topic, predict its sentiment, extract events, and decide if it contains offensive words, etc. If we train separate model for each task, we may have a latency during infer- ence time due to several times of forward passes,
Compared to multi-tasking training, the advan- tage of prefix prefix-based training is that the fixed text representations
be casily updated computational cost. For example, if want add source tasks, we can simply train new prefixes for those tasks without re-training the whole model. Similarly, if we want to remove source tasks, we can directly disable the corresponding prefixes dur- ing inference time. In contrast, multi-tasking train- ing requires re-training the whole model, which is less flexible and computationally expensive.
small
Our experimental results show that prefix-bart based training performs better than multi- talti-tasking train-
Work done while interning at Meta AL.

---

Clasification Head v
Torget Task
Target Task
Representations
Fixed Representations
Task Profix <
Frozon
Task Pool
Profix
(a) Training stage.
(b) Inference stage.
Figure Overview of prefix-hased training. (a) We train task-specific prefix for each source task. (b) All task-specific prefixes are combined together to obtain the fixed text representations. The text representations can be easily updated by adding or removing task-specific prefixes. The snowflake symbol and the fire symbol indicate whether the module is frozen not.
ing in terms of transferring knowledge from source tasks to target tasks. In addition, we design two ex- periments to highlight the flexibility of prefix-based training to easily update the text representations. All the results suggest that prefix-basee based training can promising approach to this research direction.
Cao et al., 2023). Our work is motivated by the fact that those learnable parameters can be viewed as embeddings for transferring knowledge rep- resenting tasks (Vu al., 2022; Asai al., 2022; Zhou al., 2022).
Method
2
Related Work
3.1 Problem Setup
General purpose text representations. Large pre-trained language models are usually used for extracting text representations (Devlin et al., 2019; Liu et al., 2019b; Lewis et al.. 2020), Several ap- proaches consider self-supervised learning to im- prove the quality of representations (Yan et al., 2021; Liu et al., 2021a; Gao et al., 2021; Chuang al., 2022). To improve the generalization for un- seen tasks, some works consider additional source tasks to learn the representations, such as natural language inference corpus (Conneau et al., 2017; Cer et al., 2018; Reimers and Gurevych, 2019) and paraphrase pairs (Huang et al., 2021). Recently, frozen text representations have caught attention for amortizing the computational cost in real-world applications (Peters et al., 2019; Du al., 2020).
Our goal is to learn fixed text representations that perform well on unseen target tasks. During the training stage, we consider M source tasks Tř ,.., T5 to learn text representations. In the inference stage, train lighweiight classifier for each target task TV: 2, ..., Ty based on the learned fixed text representations. That is, only the lightweight classifier will be trained while the text representations remain the same to reduce the com- putational burden during the inference time. Addi- tionally, we expect the learned representations can be easily updated (e.g.. add/remove/updatt source tasks) small computational cost.
3.2 Training Stage
As illustrated by Figure 1a, we learn task-specific prefix (Li and Liang, 2021; Liu al., 2022b) for each source task. It is worth noticing that ev- ery task-specific prefix is learned independently. Specifically, we follow the implementation of P- Tuning v2 (Liu et al., 2022b) and consider the soft prompt tuning (Lester et al., 2021b) for each Trans- former layer. For each layer, we learn an addi- tional key matrix K, = (ki,..,,k) kj) and an ad- ditional value matrix V,, (vi,...,v)) viJ. where is the length of the prefix. When computing the attentions for each layer, we concatenate the addi- tionally learned key matrix K, and value matrix V, with the original key matrix K and value matrix V. That is, we use K' = K, + and V = V, @ V
Prefix tuning and prompt tuning. Recently, prefix tuning and prompt tuning become popular ways to learn parameter-efficient models. Early works usually consider discrete prompts, where the prompts consist of real words (Shin et al., 2020; Schick and Schütze, 2,21 Scao and Rush, 2021). Later works study soft prompts, where the prompt words are learnabee (Li and Liang, 2021; Liu al., 2021b: Lester al., 2021a; Qin and Eis- ner, 2021; Liu et al., 2022b). Several studies have shown that prefixes and prompts can effectively capture the key information about the tasks (Liu et al., 2022a; Hsu et al., 2023; Wan et al., 2023;

---

calculate the scaled dot-product attention. More training details can be found in Appendix A.I.
Dutaset
Task Type
Truin
Source Tasks
The final task-specific prefix P consists of those learned key matrices {Kp") ..... Kp K2") and value matrices (V7"), V for all layers, where L is the number of layers. Since the language model is frozen during training, we expect that all task- specific information captured by the prefix P.
MNLI
Natural Language Inference (NLD)
393K
QNLI
Natural Language Inference (NLI)
105K
QQP
Paraphrase Identification (PI)
364K
SST-2
Sentiment Analysis (SA)
66K
Yelp-2
Sentiment Analysis (SA)
540K
ReCoRD
Question Answering (QA)
101K
WinoGrande
Commonsense Reasoning (CR)
40K
Target Tasks
RTE MRPC
Natural Language Inference (NLI)
2.5K
3.3 Inference Stage
Paraphrase Identification (PI)
3.7K
CR
Sentiment Analysis (SA)
2.3K
Assuming the task-specific prefixes we learn for the source tasks Tf .... Ti are P1, Pa,,.,, ..., PMr we concatenate them to be large prefix P In other words, for each Transformers layer, use K K @ K2 @ Ka calculate the attention and compute the final text representations. We then train classifiers for target tasks on top of the same fixed text representations, as illustrated by Figure 1b. Appendix lists more details.
MR
Sentiment Analysis (SA)
6.4K
MPQA
Sentiment Analysis (SA)
6.4K
BoolQ
Question Answering (QA)
9.4K
MultiRC
Question Answering (QA)
27K
CosmosQA
Commonsense Reasoning (CR)
25K
and V V/.... Vis
Table Datasets.
bility for updating text representations. In addition. prefix-based is faster and easier. Since all task-specific prefixes are trained inde- pendently they can be trained in parallel, using different parameters (e.g. learning rates). This solves the difficulty of multi-task training where it is hard to find a good configuration for all source tasks, as different tasks have different properties, e.g. tasks have varying sizes of training examples, which causes some tasks to be dominated by others (Liang and Zhang, 2020; Mao et al., 2022).
As mentioned in Section 1, to reduce the com- putational cost, all the prefixes and the language model are frozen during the inference stage. How- ever, the task-specific prefixes can still pass the task-specific information via the learned key matri- ces and value matrices when calculating attention. Therefore, the final text representations contain nec- essary information about source tasks that can be transferred to unseen target tasks.
Experiments
3.4 Comparison to Multi-Tasking Training
We conduct experiments to show the potential of presix-based training and its flexibility for updating text representations.
Multi-tasking learning (Collobert and Weston, 2008; Zhang and Yang, 2017: Ahmad et al., 2018; Liu et al., 9a; Zhou et al., 2021) is a common way to incorporate the knowledge of several source tasks into language models. fine-tunes a language model with multiple objectives for source tasks at the same time. Compared to multi-tasking training, prefix-based training has several advantages for obtaining fixed text representations.
4.1 Tasks
We consider different types of NLP tasks, includ- ing natural language inference, paraphrase identi- fication, sentiment analysis, question answering, and commonsense reasoning, as listed in Table 1.
The source tasks include the following datasets with more than 40K annotations: MNLI (Williams et al., 2018), QNLI (Demszky et al., 2018), QQP (Wang et al., 2019), SST-2 (Socher et al., 2013), Yelp-2 (Charwad et al., 2015), ReCoRD (Zhang et al., 2018), and WinoGrande (Sakaguchi et al., 2020). The target tasks include the following 8 relatively small datasets: RTE (Giampiccolo al., 2007), MRPC (Dolan al., 2004), CR (Hu and Liu, 2004), MR (Pang and Lee, 2005), MPQA (Wiebe et al., 2005), BoolQ (Clark et al., 2019), MultiRC (Khashabi et al., 2018) 8), and CosmosQA (Huang
The biggest advantage of prefix-based training is that the text representations can be easily updated at a small computational cost. For example, if we want to add (update) some source tasks, we can sim- ply train new (update) prefixes for those tasks. Sim- source tasks,
ilarly, if we would like to remove we can directly disable the corresponding prefixes during the inference time. Compared to multi- tasking training, which needs to re-train the whole language model when adding/updating/reovinng source tasks, prefix- based training has great flexi-

---

Method
Freeze
RTH
MRPC
CR
MR
MPQA
BoolQ
MultRCC
CosmosQA
Avg.
Upper bound (computationally expensive)
Fine-tuning the whole language model
x
86.38
88.01
93.59
91.19
91.93
85.28
80.80
77.77
86.87
Lower bound, without source task pre-training
Fine-tuning with frozen language model
58.59
77.28
91.66
87.99
90.29
70.53
68.74
57.27
75.29
With source task pre-training
Multi-tasking training
82,16
84, 90
91.03
90.60
90.83
79,44 44
74.00
66.53 70.90
82, 44
Prefix-based training (ours)
84.86
85.37
92.03
90.08
91.05
80.84
75.66
83.85
Table 2: 5-run average results for transfer setting. Prefix-based training performs better than multi-tasking training.
Method
Freeze
RTF
MRPC
CR
MR
MPQA
BoniQ
MultiRC:
CosmosQA
Avg.
Multi talting training
82,16
84,90
91.03
90.60
90.83
79.44
74.00
66.53
82.44
Prefix-basse training
84.86
85.37
92.03
90.08
91.05
80.84
75.66
70.90
83.85
Removing at most task
86.76
86.21
92.05
90.64
91.28
81.65
76.12
71.11
84.48
Removing at most tasks
87.30
86.67
93.05
90.76
91.70
82.18
76.12
72.00
84.97
Table 3: Best results from all combinations of removal. Removing hurtful source tasks leads to improvements.
et al., 2019).
freeze the representations, with training on source tasks, we see an overall improvement for all target tasks, which shows the importance of knowledge transfer from source tasks to target tasks.
For those datasets with the standard train, dev, and test split, we follow the standard split for train- ing. For those datasets without the standard split (c.g., GLUE tasks), we randomly split 1/3 exam- ples from the dev set as the internal dev set and use the rest 2/3 examples as the testing set.
Prefix- based training consistently performs bet- ter than multi-tasking training, especially for those target tasks that require high-level understanding, such as natural language inference and common- sense reasoning, We hypothesize that those types of tasks are more difficult than others and might be dominated by other simpler tasks, such as senti- ment analysis, during multi-taskin training. There- fore, multi-tasking training cannot transfer knowl- edge from those types of tasks well. In contrast, prefix-base based training has promising performance for all types of tasks.
4.2 Baselines for Comparison
We compare our proposed prefix-based training with multi-tasking training (Collobert and Weston, 2008: Zhang and Yang. 2017; Ahmad et al., 2018; Liu al., 2019a; Zhou al., 2021). Both ap- proaches are trained on the same source tasks and use pre-trained RoBERTa-large (Liu et al., 201 90))... Then, we we frezze both models and get fixed text rep- resentations as the features to train classifiers for the target tasks. Please refer to Appendix A for more details. To analyze the influence of source tasks and fixed representations, we additionally consider two simple fine-tuning baselines: fine- tuning the whole language model and fine-tuning with the frozen language model. Note that they directly use RoBERTa-large without training on source tasks.
4.4 Flexibility for Updating Representations
As mentioned in Section 3, one advantage of prefix- based training is that the text representations can be easily updated with a small computational cost. We conduct two experiments to verify this merit.
Removing hurtful source tasks. Prefix-based training gives us an easy way to disable some source tasks during the inference stage - just removing the corresponding prefixes without re- training. Therefore, we can easily find out some hurtful source tasks for particular target task by removing different combinations of source tasks. Table 3 shows the best results from all combina- tions of removal. We observe improvements when removing hurtful source tasks. For example, re- moving SST-2 task is helpful for natural language
4.3 Results for Transfer Setting
From Table 2, we first observe that simple fine- tuning without freezing text representations per- forms much better than with freezing. This indi- cates that although fixed representations reduce the computational cost, they largely limit the power of pre-trained language models. However, we

---

Prefix-Based
Multi- Tulti-Tasking
Training Step per Round 2000
Training Step per Round 20000
84
Average Score
82
78
76
2
6
7
4 Round
5
Round
Figure 2: Results of sequentially adding source tasks.
inference and commonsense reasoning tasks, while removing ReCoRD task is helpful for sentiment analysis tasks. This experiment shows that we can use the flexibility of prefix-basee based training to find out hurtful source tasks and improve performance.
puted by combining all task-specific prefixes to- gether. Our experimental results show that prefix- based training performs better than multi-tasking training and can update representations at smaller computational cost than multi-tas multi-tasking training.
Adding source tasks. To mimic the situa- tion when new source tasks are introduced, instead of training on all source tasks together, we sequen- tially add one source task every round and update prefix-based both models of prefix based training and multi- tasking training. For prefix-based -baee training, we train prefix for the new task. For multi-tasking training, we re-train the whole model with all existing source tasks. To fairly compare the two approaches, fix the number of training steps per round. We repeats of experiments with different orders add tasks and report the average performance on target tasks for each round in Figure 2. Experimental details can be found in Appendix B.I.
Limitations
In this work, our goal is to prove the concept that prefix-tuning training is better than multi-tasking training in terms of transferring knowledge and up- dating fixed text representations. We try to include as many tasks as possible in our experiments. How- ever, we understand that there might some differ- ences between our experimental settings and real- world cases. For example, the current experiments are limited to text-understanding tasks. Some other types of tasks, such structure predictions and syntax-related tasks, not considered in the cur- rent version. Also, in the real-world case, the num- ber of source tasks and target tasks can be larger, In this work, we provide proof of concept and demonstratc the potential benefits of the prefix- based training method. Increasing the number of tasks is therefore considered as our future study.
Overall, prefix-based training is better than multi-tasking training for every round. When the number of training steps becomes smaller, prefix- based training still has similar performance while the performance of multi-tasking training drops lot. This is because prefix-based training can use all training steps for the new task while mulli-tasking training has to share the training steps over all ex- isting source tasks. This experiment shows the flcxibility of prefix based training for updating text representations at a smaller computational cost.
Broader Impacts
Our model is based on large pre-trained language models. It is known that the models trained with a large text corpus may capture the bias reflecting the training data. Therefore, it is possible that the pre- dictions produced by model inherit the bias of pre-trained language models. We suggest care- fully examining the potential bias before applying our method to any real-word applications.
5
Conclusion
We focus on learning text representations with source tasks that can generalize well unseen target tasks. We propose prefix-based training method that learns a task-specific prefix for each source task. The fixed text representations are com-

---

References
Dorottya Demszky, Kelvin Guu, and Percy Liang, 2018. Transforming question answering datasets into nat- ural language inference datasets. arXiv preprint arXiv:1809.0290
Wasi Uddin Ahmad, Kai-Wei Chang, and Hongning Wang. 2018. Multi-task learning for document rank- ing and query suggestion. In 6th International Con- ference on Learning Representations (ICLR).
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
Akari Asai, Mohammadreza Salehi, Matthew E. Peters, and Hannameh Hajishirzi. 2022. Attempt: Parameter- efficient multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing (EMNLP).
Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, and Jun Zhao. 2023. Zero-shot cross-lingual event argu- ment extraction with language -rriente prefix tuning. In Severty AAAI Conference on Artificial Intel- ligence, AAAI 2023, Thirty- Fifth Conference on Inno- vative Applications of Artificial Intelligence (AAAI).
Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Un- supervised construction of large paraphrase corpora: Exploiting massively parallel news sources, In Pro- ceedings of the 20th International Conference on Computational Linguistics (COLING).
Jingfei Du Myle Ott, Haoran Li, Xing Zhou, and Veselin Stoyanov. 2020. General purpose text embed- dings from pre-trained language models for scalable inference. In Findings of the Association for Compu- tational Linguistics: EMNLP 2020.
Daniel Cer, Yinfei Yang. Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni John, Noah Constant, Mario Suajardo-Cespedess Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for english. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Nayana Yashwant Charwad, Sri Laxmi Chin- tala, Renuka Deshmukh, Aniket Sambhaji Gaikwad, and Venkata Prudhvi Raj In- dana. 2015. Yelp dataset challenge 2015. https://github..o http:/gghhubcoom/eenaeeeeeeeee wwwtrrenkkaeetmmukk/eepp--aa Challenge-201s 2015.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recogniz- ing textual entailment challenge. In Proceedings of the ACL-PASCAL @ AACLLLACCLL 2007 Workshop on Textual Entailment and Paraphrasing.
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang- Wen Li. Scott Yih, Yoon Kim, and James R. Glass. 2022. Diffese: Difference-basee contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (NAACL).
1-Hung Hsu, Zhiyu Xie, Kuan-Hao Huang, Prem Natara- jan, and Nanyun Peng. 2023. AMPERE: amr-awar prefix for generation based event argument extraction model. Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (ACL),
Minqing Hu and Bing Liu. 2004. Mining and sum- marizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
Christopher Clark, Kenton Lee, Ming- Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019, Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
James Y. Huang, Kuan-Hao Huang, and Kai-Wei Chang. 2021. Disentangling semantics and syntax in sen- tence embeddings with pre-trained language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (NAACL-HLT).
Ronan Collobert and Jason Weston. 2008 A unified architecture for natural language processing: deep neural networks with multitask learning. In Machine Learning, Proceedings of the Twenty- Interna- tional Conference (ICML).
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: machine reading comprehension with contextual commonsense rea- soning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLPP
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing (EMNLP).

---

Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistie:: Human Language Technologies (NAACI-HHLTT)
Xiaodong Liu, Pengcheng He. Weizhu Chen, and Jian- feng Gao. 2019a. Multi-task task deep neural networks for natural language understanding. In Proceedings of the 57th Conference of the Association for Compu- tational Linguistics (ACL).
Yinhan Liu, Myle Ou,, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized BERT pretraining approach. arviv preprint 1xXiv:1907.11699
Brian Lester, Rami AI-Rfou, and Noah Constant. 202 la The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Yuren Mao, Zekai Wang. Weiwei Liu, Xuemin Lin, and Pengtao Xie. 2022. Metaweighting: Learning to weight tasks in multi-task learning. In Findings of the Association for Computational Linguistics: (ACL).
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021b. The power of scale for wrrameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Bo Pang and Lillian Lee. 2005. Sceing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In 43rd Annual Meeting of the Association for Computational Linguistics (ACL),
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Chazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. -o-eeueeee-e pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Anmual Meet- ing of the Association for Computational Linguistics (ACL).
Matthew E. Peters, Sehastian Ruder, and Noah Smith. 2019. To tune or not to tune? adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepLANLP@ACII
BART: denoising
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying Ims with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (NAACL-HLT).
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/ICNLP).
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Sicong Liang and Yu Zhang. 2020. A simple general approach to balance task difficulty in multi-tat tasli-ttskkaa learn- ing. Xiv preprint arXiv:2002.0477 04799.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2020. WinoGrande: An adver- sarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence (AAAI).
Fangyu Liu, Ivan Vulic, Anna Korhonen, and Nigel Collier. 2021a. Fast, effective, and supf-supervised: Transforming masked language models into universal lexical and sentence encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Teven Le Scao and Alexander M. Rush. 2021. How many data points is prompt worth? In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
Xiao Liu, Heyan Huang, Ge Shi, and Bo Wang. 2022a. Dynamic prefix-tuning for generative template-based event extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (ACL)).
Timo Schick and Hinrich Schütze. a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume (FACL.).
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022b. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers) (ACL).
Timo Schick and Hinrich Schütze. 2021b. It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (NAACL-HLLT)
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 202 GPT understands, too. arXiv preprint arXiv:2103. 10385.

---

Taylor Shin, Yasaman Razeghi, Robert L. Logan IV. Eric Wallace, and Sameer Singh. 2020. Autoprompt: Fliiiiingg knowledge from language models with au- tomatically generated prompts. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Fan Zhou, Brahim Chaib-draa, and Boyu Wang. 2021. Multi-tase learning by leveraging the semantic infor- mation. In Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI).
Wangchunshu Zhou, Canwen Xu, and Julian McAuley. 2022. Efficiently tuned parameters are task embed- dings. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing (EMNLP).
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep mod- els for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Tu Vu, Brian Lester, Noah Constant, Rami AI-R Al-Rfou', and Daniel Cer. 2022. Spot: Better frozen model adaptation through soft prompt transfer. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers) (ACL).
Yixin Wan, Kuan-Hao Huang, and Kai- Wei Chang. 2023. PIP: parse-instruue instrrinste prefix for syntactically controlled paraphrase generation. In Findings of the Association for Computational Linguistics: ACL.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-lask benchmark and analysis plat- form for natural language understanding. 7th In- lernational Conference on Learning Representations (ICLR).
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Lang. Resour. Evaluation, 39(2-3):165- 210.
Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (NAACL-HLT).
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. Consert: con- trastive framework for self-supervised sentence repre- sentation transfer. In Proceedings of the 59th Annual Meeting of Association for Computational Lin- guistics and the I 1th International Joint Conference on Natural Language Processing (ACL))CNLP)).
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and ma- chine commonsense reading comprehension. arXiv prepriat 18100
Yu Zhang and Qiang Yang. 2017. survey on multi- task learning. arXiv prepreint arXiv: 1 1707.08114.

---

A
Training Details
SST-2, QNLI, ReCoRD, MNLI, WinoGrande,
Yelp-2, QQP
All the models trained with NVIDIA Tesla
QNLI, WinoGrande, MNLI, QQP, SST-2, ReCoRD, Yelp-2
V100 GPU.
A.1 Details for Training Task-Specifie Prefix
WinoGrande, SST-2, Yelp-2, QQP, QNLI, ReCoRD, MNLI
We follow the implementation of P-tuning v2 (Liu et al., 2022b) and set the prompt length to 5. The Batch size is 16 for all source tasks and the number of epoch is 40. We use Adam optimizer with the learning rate being 5e-3 and the weight decay being le-5, The classification head is one layer of MLP.
QNLI, Yelp-2, QQP, ReCoRD, WinoGrande, MNLI, SST-2
We notice that in the original implementation of P-Tuning v2, they add positional encoding to pre- fix. However, as we will concatenate several task- specific prefix together during the inference time, we remove the positional encoding when training the prefix to avoiding position mismatch between training and inference.
A.2 Details for Training Target Tasks
Following previous work (Du al., 2020), we train one attention layer on top of the fixed text repre- sentations and train one layer MI. based on the CLS token representation for each target task. The batch size is 32 and the number of epoch is 80. We use Adam optimizer with the learning rate being le-4 and the weight decay being le-5.
A.3 Details for multi-tasking
We uniformly sample data from source tasks for every batch. The batch size is 16 and the number of training steps is 400000. We use Adam optimizer with the learning rate being le-5 and the weight decay being le-5. The classification head is one layer of MLP.
B Experimental Details
B.1 Sequence Order
We consider the following 8 sequence to add the source tasks sequetially.
MNLI, QQP, ReCoRD, QNLI, SST-2, Wino- Grande, Yelp-2
Yclp-2, MNLI, ReCoRD, SST-2, QNLI, Wino- Grande, QQP
QQP, ReCoRD, MNLI, SST-2, QNLI, Wino- Grande, Yelp-2
ReCoRD, Yelp-2, WinoGrande, QQP, MNLI, SST-2, QNLI


Reviews:
[{"review_id": "617O9PjHdr", "reviewer": "Reviewer_VDm4", "summary": "", "questions": "Is there any more recent work can be compared in the experiment section?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback. Here are our responses to your review.\n\n**While the paper provides a comparison to a baseline method of multi-tasking training, which was published in 2017, there are more recent and sophisticated methods for learning text representations that could serve as stronger baselines for comparison.**\n\n> We would like to point out that although there are some recent works studying different multi-tasking training paradigms, many papers still choose the original multi-tasking training method as the baseline or apply it to downstream applications [1,2,3,4,5,6,7]. In fact, in industry, it’s pretty common to apply the original multi-tasking training method for learning text representations since other methods may require large hyperparameter searching.\n1. General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference, EMNLP-Findings 2020\n2. ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts, EMNLP 2022\n3. P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks, NAACL 2022\n4. Multitask Pretraining with Structured Knowledge for Text-to-SQL Generation, ACL 2023\n5. Crosslingual Generalization through Multitask Finetuning, ACL 2023\n6. When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLP, ACL 2023\n7. Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning, ACL 2023\n\n**The related work section of the paper discusses more recent papers that propose methods for learning text representations, but the paper does not provide a comparison to these more recent methods, which could limit the ability of the paper to contextualize the proposed method within the current state of the art.**\n\n> In the related work section, we do mention several works regarding learning text representations. However, most of them are self-supervised or consider only one source task. Although they can have great performance if we fine-tune the entire model, when freezing their representations, their performance will be similar to freezing RoBERTa, as we reported in Table 1. Also, our goal is to provide a proof-of-concept that prefix-based learning is better than multi-tasking learning in terms of training cost and flexibility. Therefore, we choose RoBERTa, the commonly used pre-trained language model, as the base model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper include proposing a novel method for learning easily updated general purpose text representations with adaptable task-specific prefixes, which can reduce the computational burden of fine-tuning a large pre-trained language model for each downstream task. The paper provides experimental results that demonstrate the effectiveness of the proposed method on several downstream tasks, and compares the proposed method to other methods, such as fine-tuning and multi-tasking training, showing that the proposed method achieves comparable or better performance while requiring less computational resources.", "reasons_to_reject": "While the paper provides a comparison to a baseline method of multi-tasking training, which was published in 2017, there are more recent and sophisticated methods for learning text representations that could serve as stronger baselines for comparison. The use of a weak baseline method could limit the ability of the paper to demonstrate the effectiveness of the proposed method compared to state-of-the-art methods. Additionally, the related work section of the paper discusses more recent papers that propose methods for learning text representations, but the paper does not provide a comparison to these more recent methods, which could limit the ability of the paper to contextualize the proposed method within the current state of the art.", "questions_to_authors": "Is there any more recent work can be compared in the experiment section?", "paper_topic_and_main_contributions": "This paper is about a method for learning easily updated general purpose text representations with adaptable task-specific prefixes. The paper addresses the problem of the computational burden of fine-tuning a large pre-trained language model for each downstream task. The main contribution of this paper is a method that can update text representations at a smaller computational cost than multi-tasking training by learning fixed but general text representations that can generalize well to unseen downstream tasks. The paper proposes a prefix-based method that adds task-specific prefixes to the input text, which allows the model to adapt to the specific task while still using the same general text representation. The paper provides experimental results that demonstrate the effectiveness of the proposed method on several downstream tasks. The paper also compares the proposed method to other methods, such as fine-tuning and multi-tasking training, and shows that the proposed method achieves comparable or better performance while requiring less computational resources.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vUPdRzeddv", "reviewer": "Reviewer_tLRw", "summary": "", "questions": "1) Can you please point out the main contributions of this work? Is it the empirical investigation wrt Multi task learning? \n2) Can you point out how this work is different from the following three works?\n\nPfeiffer, J., Kamath, A., Rücklé, A., Cho, K., and Gurevych, I. (2020a). AdapterFusion: Non-destructive task composition for transfer learning. \n\nPfeiffer, J., Rücklé, A., Poth, C., Kamath, A., Vulić, I., Ruder, S., Cho, K., Gurevych, I. (2020b). AdapterHub: A Framework for Adapting Transformers. \n\nHoulsby, N., Giurgiu, A., Jastrzkebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-efficient transfer learning for NLP", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback. Here are our responses to your review.\n\n**Can you please point out the main contributions of this work? Is it the empirical investigation wrt Multi task learning?**\n\n> As we mention in the introduction, our contribution is pointing out that learning individual prefix for each source task and combining them together result in better *text representations* compared to traditional multi-tasking learning in terms of training cost and flexibility. Prefix-based learning can be trained for source tasks in parallel and is easier to update source tasks. In contrast, multi-tasking is very hard to choose hyper-parameters, balance different source tasks, as well add or remove source tasks.\n\n**Can you point out how this work is different from the following three works?**\n> First, we would like to point out that all of them are related to adapters, which is a little bit different from prefix-tuning. Also, those papers have different research focus from ours.\n\n> **Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., and Gurevych, I. (2020a). AdapterFusion: Non-destructive task composition for transfer learning.**\\\n> This paper mainly discusses the catastrophic forgetting problem, which is different from our goal of learning text representations.\n\n> **Pfeiffer, J., Rücklé, A., Poth, C., Kamath, A., Vulić, I., Ruder, S., Cho, K., Gurevych, I. (2020b). AdapterHub: A Framework for Adapting Transformers.**\\\n> This paper introduces a library of various adapters, which is totally different from our focus to learn text representations.\n\n\n> **Houlsby, N., Giurgiu, A., Jastrzkebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-efficient transfer learning for NLP**\\\n>This paper also studies transfer learning, but their definition of transfer learning is to transfer the knowledge from pre-trained language models to single target tasks, not transfer from multiple source tasks to multiple target tasks. Our experimental setting is clearly different.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Investigated the empirical performance of prefix tuning for multitask learning", "reasons_to_reject": "Lack of novelty and missed the whole field of adapter models for transformers", "questions_to_authors": "1) Can you please point out the main contributions of this work? Is it the empirical investigation wrt Multi task learning? \n2) Can you point out how this work is different from the following three works?\n\nPfeiffer, J., Kamath, A., Rücklé, A., Cho, K., and Gurevych, I. (2020a). AdapterFusion: Non-destructive task composition for transfer learning. \n\nPfeiffer, J., Rücklé, A., Poth, C., Kamath, A., Vulić, I., Ruder, S., Cho, K., Gurevych, I. (2020b). AdapterHub: A Framework for Adapting Transformers. \n\nHoulsby, N., Giurgiu, A., Jastrzkebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-efficient transfer learning for NLP", "paper_topic_and_main_contributions": "The paper talks about how prefix tuning, a kind of adapter technique, is better at multitask learning compared to finetuning several models for each downstream task. The authors benchmark their method against the multitask learning approach of Zhang and Yang, 2017. They show that the average task score improves using the prefix tuning approach.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {"Missing_References": "> The authors seems to be not familiar with the plethora other methods that talk about adapter methods in general in the context of multitask learning. Several recent works such as [1], [2] talk about comparison of adapter methods for multitask learning in detail. \n\n[1] - Parameter-Efficient Transfer Learning for NLP by Houslby et al\n[2] - On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation by Ruidan et al"}}, {"review_id": "pPmFbmwRg2", "reviewer": "Reviewer_hBJN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback. Here are our responses to your review.\n\n**The difference in the main results (Table 1) between prefix-based training and multi-task training are low. Hence, the tradeoffs between both methods should be discussed. For example, the use of many prefixes may result in high memory usage during inference. What situations would this potential memory cost override the cost of finetuning an entire LM for the same tasks?**\n\n> We agree that prefix-based learning may need higher memory cost during inference. However, it largely reduces the training cost and has higher flexibility to choose source tasks. We will discuss the trade-off in the final version. Thanks for the suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The method allows high flexibility in choosing which source tasks to contribute to target tasks. Empirical results in the paper show certain source tasks harmed performance on target tasks during inference, which further supports advantages in the proposed method. \n\n2) Their method highlights the importance of hyperparameter tuning for each task, as opposed to the multitasking setting. This can be beneficial for making optimal use of low-resource source tasks in multi-task settings.\n\n3) The paper is well written and the methodology is clear. Details for the experiment -- such as data and model setup  -- are nicely provided.", "reasons_to_reject": "1) The difference in the main results (Table 1) between prefix-based training and multi-task training are low. Hence, the tradeoffs between both methods should be discussed. For example, the use of many prefixes may result in high memory usage during inference. What situations would this potential memory cost override the cost of finetuning an entire LM for the same tasks?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a prefix-tuning method to learn fixed text representations in language models. The goal of their method is to reduce computational costs for multi-task inference. To accomplish this, a prefix is first trained independently for each source task using a frozen LM. During inference, each learned prefix is concatenated to a LM, and representations are obtained using weights from the learned prefixes. \n\nThe method aims to reduce computational costs through the following observations: \n\ni) Finetuning a LM for multitask training can be computationally expensive. Hence, prefix-training can lighten training costs.\n\nii) Learned prefixes can be easily concatenated or removed from a LM – this allows for flexibility in choosing which source tasks to contribute to target tasks.\n\nEmpirical results show that prefix-based training in general performs better than multitasking training on a set of 8 target task datasets. Further ablation studies note performance increases due to flexibility in being able to remove prefixes during inference. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) Define what a fixed representation to improve clarity in the paper.\n\n2) There can be a few additional columns added to Table 1 to improve clarity (although the table is already wide so this suggestion might not be very beneficial!). \n\n    a) Instead of writing \"$\\textit{without source task pre-training}$\" or  \"$\\textit{with source task pre-training}$\", a column can be added on whether the source tasks were used. \n\n    b) Adding another column on how the MLP is used can also improve presentation clarity. \n\n    c) The \"$\\textit{freeze}$\" column heading could be titled \"$\\textit{Freeze LM}$\" for additional clarity."}}, {"review_id": "WwD6ReNc3c", "reviewer": "Reviewer_asuG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback. Here are our responses to your review.\n\n**Solving multiple tasks by text representations seems less used in practice due to its relatively low performance on real-world complex tasks, language generation, and data-scarce scenarios, which limits the application of the method.**\n\n> We agree that for some applications, solving multiple tasks by text representations might not be that common. However, we would like to point out its practical values. Like we mentioned in the intro section, given a Facebook post or a tweet, several downstream tasks would be applied to the text input, e.g., topic prediction, toxicity classification, information extraction, etc. It’s very common that we will face new tasks when developing new features. Therefore, having good text representations that can generalize to those new tasks becomes an important thing. Our paper provides a solution to this.\n\n\n**The transferability of source prompts is only verified on simple text classification tasks in which the source and target tasks are similar. Experiments when source/target tasks are not similar (for example, the source tasks only contain sentiment analysis and the target tasks are natural language inference) are not included. However, it may be OK for a short paper.**\n\n> Since our goal is to provide a proof-of-concept that prefix-based learning is better and more flexible than multitasking learning, we start from text classification tasks. We will explore more complicated tasks such as structure predictions and question answering in the future. Thanks for your suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a framework that efficiently generates text representations for multiple tasks. The effectiveness is verified by the experiments.\n2. The paper is well-written and the method is easy to follow.", "reasons_to_reject": "1. Solving multiple tasks by text representations seems less used in practice due to its relatively low performance on real-world complex tasks, language generation, and data-scarce scenarios, which limits the application of the method.\n2. The transferability of source prompts is only verified on simple text classification tasks in which the source and target tasks are similar. Experiments when source/target tasks are not similar (for example, the source tasks only contain sentiment analysis and the target tasks are natural language inference) are not included. However, it may be OK for a short paper. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies how to learn a general-purpose text representation that can be used for multiple tasks. They generate text representations conditioning on multiple source task prefixes obtained by prefix-tuning and train lightweight models based on these representations on unseen tasks. Updating source tasks is convenient in this framework. They conduct experiments on multiple text classification datasets to show the effectiveness of the method. The contribution belongs to ``approaches for data and compute efficiency''.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a prefix-based method for efficiently updating general-purpose text representations, reducing computational costs compared to fine-tuning or multi-tasking. It uses task-specific prefixes to adapt the model for specific tasks while maintaining the same general text representation. Experimental results demonstrate its effectiveness across various tasks, outperforming or matching fine-tuning and multi-tasking while requiring fewer computational resources. 

The problem addressed in this paper is intriguing, and the utilization of prefixes appears to be effective on the datasets used in the experiments. However, a notable question raised by the paper is the extent of computational savings achieved by the proposed method. While the paper frequently emphasizes computational efficiency, it lacks clear measurements or comparisons in this regard.
Decision: Accept-Findings