Two-stage LLM Fine-tuning with Less Specialization and More Generalization
Authors: Yihan Wang
Si Si
Daliang Li
Michal Lukasik
Felix Yu
Cho-Jui Hsieh
Inderjit S Dhillon
Sanjiv Kumar
Venue: ICLR 2024 poster (2024)
Keywords: language model
Generalization
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Pretrained large language models (LLMs) are general purpose problem solvers applicable to a diverse set of tasks with prompts. They can be further improved towards a specific task by fine-tuning on a specialized dataset. However, fine-tuning usually makes the model narrowly specialized on this dataset with reduced general in-context learning performances, which is undesirable whenever the fine-tuned model needs to handle additional tasks where no fine-tuning data is available. 
In this work, we first demonstrate that fine-tuning on a single task indeed decreases LLMs' general in-context learning performance. We discover one important cause of such forgetting, format specialization, where the model overfits to the format of the fine-tuned task.We further show that format specialization happens at the very beginning of fine-tuning. To solve this problem, we propose Prompt Tuning with MOdel Tuning (ProMoT), a simple yet effective two-stage fine-tuning framework that reduces format specialization and improves generalization.ProMoT offloads task-specific format learning into additional and removable parameters by first doing prompt tuning and then fine-tuning the model itself with this soft prompt attached. 
With experiments on several fine-tuning tasks and 8 in-context evaluation tasks, we show that ProMoT achieves comparable performance on fine-tuned tasks to standard fine-tuning, but with much less loss of in-context learning performances across a board range of  out-of-domain evaluation tasks. More importantly, ProMoT can even enhance generalization on in-context learning tasks that are semantically related to the fine-tuned task, e.g. ProMoT on En-Fr translation significantly improves performance on other language pairs, and ProMoT on NLI improves performance on summarization.
Experiments also show that ProMoT can improve the generalization performance of  multi-task training.

Published as a conference paper at ICLR 2024
TWO-STAGE LLM FINE-TUNING WITH LESS SPECIAL- IZATION AND MORE GENERALIZATION
Yihan Wang'
Si Google
Department of Computer Science
UCLA wangyihan61 72 gma i
si siiiaisy@googlee
com
Daliang Li Google da Baliangli@google..o
Michal Lukasik
Felix Yu Google
Google ml mllkasik@@ooglee.o com
Eelixyu@googie.co
Cho-Jui Hsieh
Inderjit S Dhillon Google i godggoogle.co com
Sanjiv Kumar Google
Department of Computer Science UCLA chohsieh@er cs. ucla.ede
sanjivx@google.co
ABSTRACT
Pretrained large language models (LLMs) are general purpose problem solvers applicable to diverse set tasks with prompts. They can be further improved towards specific task by fine-tuning usually makes the model narrowly specialized on this dataset with reduced general in-context learning performances, which is undesirable whenever the fine-tuned model needs to handle additional tasks where no fine-tuning data is available. In this work, first demonstrate that fine-tuning LLMs' general in-context learning performance. We discover one important cause of such forgetting, format specialization, where the model overfits to the format of the fine-tuned task. We further show that format specialization happens at the very beginning of fine-tuning. To solve this problem, we propose Prompt Tuning with MOdel Tuning (ProMoT), simple yet effective two-stage fine-tuning framework that reduces format specialization and improves generalization. ProMoT offloads task-specific format learning into additional and removable parameters by first doing prompt tuning and then fine-tuning the model itself with this soft prompt attached. With experiments several fine-tuning tasks and 8 in-context evaluation tasks, we show that ProMoT achieves comparable performance on fine-tuned tasks standard fine-tuning, but with much less loss of in-context learning performances across a board range of out-of-domaal evaluation tasks. More importantly, Pro.MoT can even enhance generalization on in-context learning tasks that are semantically related to the fine-tuned task, e.g. ProMoT on En-Fr translation significantly improves performance on other language pairs, and ProMoT NLI improves performance on summarization. Experiments also show that ProMoT can improve the generalization performance of multi-task training.
specialized dataset. However, fine-tuning
single task indeed decreases
INTRODUCTION
Natural language processing (NLP) has recently been revolutionized by scaling up transformer based large language models (LLMss) together with large-scale pretraining (Vaswani al. 2017 Devlin
et al.
Raffel et al. 2020a Brown et al. [2000] Rac et al. 2021; Chowdhery et al. 22222] Smith 2023). In addition to improved downstream performances, these pretrained L..Ms can perform broad array of unforeseen tasks when provided with prompt. This in-context
2019
2022 Touvron
Work done while Google.

---

Published as a conference paper at ICLR 2024
learning capability allows users to flexibly re-purpose LLMs for specific tasks with minimum amount of supervised data, making it extremely convenient for fast prototyping and experimentation, especially in the low data regime.
However, even the largest and most advanced LLMs leave a lot to be improved. Grounding and eliminating hallucinations (Maynez al.] 2020), reasoning and logical clarity (Creswell & Shanahan 2022), mathematics (Brown et al. 2020 Noorbakhsh et al., (021) are just a few examples where LLMS still lag behind the best human performances, or in some cases, the fine-tuned performances of the same model.
The most common practice to improve pretrained model is to fine-tune it a specialized task or several tasks. However, fine-tuning on LLM usually causes over-specialization to the fine-tuning tasks, and harm the model's pre-existing generalization ability on unseen tasks via in-context learning. As we show later, an m'15 model finetuned on single task loses its few-shot performance on unseen tasks within one thousand steps of fine-tuning. When faced with hundreds of downstream tasks and even unknown tasks, we expect to have a single fine-tuned model that is both superior supervised fine-tuned tasks and general unseen tasks. Thus, becomes very important to develop new techniques for finetuning that prevent over-specialization of these fine-tuned models only to few tasks.
Ground-Truth Output
Mercedes' Lewis Hamilton took the outright chumpionship lead for the first time this season with dominant victory in the Italian Grand Prix.
Pretrained m'55
Hamilton won the British Grand Prix.
Fine-tuned m13 on RTE
True
Fine-tuned m' with PooMoT (Ours) on RTB
Lewis Hamilton won the Italian Grand Prix.
Table 1: Output comparison of pretrained and fine-tumed mT5 models vs. fine-tuned with ProMoT on the RTE binary classification NLI dataset, performing in-context 1-shot summarization.
In this work, we discover that the loss of general in-context learning abilities during fine-tuning is. to a large extent, caused by format specialization, which makes model overfitting to the specific task format. For example, an mT5 (Xue et al. 2020) model learns in the output space with only "True" and "False" if we fine-tune it on binary classification dataset, losing its ability to flexibly generate different output styles according to the in-context prompts of other tasks. We show that format specialization tends to happen at the very beginning of fine-tuning, before the model fully learns the semantic content of the task.
Based these observations, we propose simple solution to alleviate format specialization: PROmpt Tuning with MOdel Tuning (ProMoT), which lff-l format learning to a small amount of task- specific parameters that are external to the model. ProMoTT is two-stage fine-tuning process. At the first stage, we freeze the pretrained model and tune a small set of additional parameters, where we find adding soft prompt before the input (Lester al. 202 l) is good choice. At the second stage, we freeze the additional parameters and tune the main model. Since format information is learned first, it mostly enters the small set of additional parameters. At inference time, we can decide whether to remove the additional parameters depending on whether the incoming task share the same format as the fine-tuned task.
Our experiments show that ProMoT significantly alleviates specialization during fine-tuning, while boosting generalization on semantically related tasks with different formats. For example, fine-tuning the model only on an NLI binary classification dataset, a mT5 XXL model consistently obtains improved in-context learning performance on summarization compared with the pretrained model, possibly due to improved grounding learned from NLI. See Table [] for concrete example. With ProMoT, we can obtain models with both better supervised performance compared to pretrained models and better general in-context learning performance compared to standard finetuning.
To summarize, our contributions are 4-fold:
We show empirically that general in-context learning capabilities decrease during single-task fine-tuning for T5 models. We identify format specialization as one of the important causes which mostly happens at the beginning of fine-tuning.
We propose a novel 2-stage fine-tuning framework: PROmpt Tuning with MOdel Tuning (ProMoT) to reduce format specialization during fine-tuning.
2

---

Published as a conference paper at ICLR 2024
Experiments on 10+ NLP tasks show that ProMoT significantly reduces specialization of fine-tuned models compared to standard fine-tuning, while reaching similar supervised per- formance. The reduction in specialization opens up opportunities to enhance generalization across very dissimilar tasks when they share some semantic aspects.
ProMoT can be combined with many existing fine-tuning and parameter-efficient fine-tuning methods. We show examples where ProMoT is combined with multi-task fine-tuning and fine-tuning with 1-shot prompts to further boost the generalization on unseen tasks.
RELATED WORK
Pretrained LLMs are general problem solvers with in-context prompts (Raffel et al., 2020b; Xue
al.] 2020; Radford et al. 2018 Chowdhery et al., [022] Min et al., 2022: Touvron al. 2023). Zhai et al. (2023) ) evaluates the catastrophic forgetting in mu multimodal language model fine- tuning, which is limited to image classification tasks. Chan et al. (2022); Gao al. (2020) study the effect of pretraining data distribution on in-context learning image recognition tasks, where the tension between in-context learning tasks and fine-tuning tasks is discussed. They propose changing the data distribution to case such tension, which could be difficult for generative NLP tasks. ProMoT is an orthogonal method that does not require changes in data distribution.
In recent study, |Raaaeesh et al. 2022) found that model size increases, the model becomes less prone to catastrophic forgetting. However such studies are mostly focused on tasks of similar format, e.g. a sequence of different classification tasks. In this work we explore vastly different tasks, e.g. classification v.s. long form generation where the format itself is critical.
Different from full fine-tuning, prompt-tuning (Lester al. 2021 Zhang al. 2021) adapters and LoRA (Hu et al. [202]] He et al. 2021; Houllby et al. (019)) adapt pretrained model to a task with small set of tunable parameters. Parameter-efficient methods like these largely leave the pretrained model intact, which can preserve the pre-existing in-context learning abilities. However, they also miss the opportunity to further improve the pretrained model with a small, high quality dataset that generalizes beyond the fine-tuned task. Besides, these murameter-efficiee methods also underperform fine-tuning on the supervised task many cases, as shown in (Lester al. 2021 Liu et al. 2021
and in our results.
Another line of work uses multi-task fine-tuning to improve generalization on unseen in-context learning tasks. [Wii et al. (2021a); Chung et al. (2022) fine-tune PaLM and T5 on large-scale multitask datasets with diverse natural language prompts, improving the zero- and few-shot performance on unseen tasks. Min et al. (2021) incorporate the in-context learning objective into fine-tuning multitask datasets with few-shor prompts. This approach relies on multi-task training to generalize, while orthogonally, ProMoT improves the generalization of each single fine-tuning task, whether used in a multi-task setting or not. ProMoTT can indeed be combined with multi-task training to obtain better generalization as we demonstrate in Sec. 554 In addition, such approaches often require human engineered instructions or prompts for each task to partly alleviate format specialization, while ProMoT uses prompt tuning, which has two advantages: 1) ProMoT does not require the elaborate trial and error of prompt engineering as it optimizes the soft prompts with data. 2) Soft prompts are more effective at absorbing the format compared to natural language prompts, as shown Table 6
3 FORMAT SPECIALIZATIO IN FINE-TUNING CAUSES THE LOSS OF IN-CONTEXT LEARNING CAPABILITIES
In this section, we first show empirically with an mT5 XXL model that 1) in-context learning abilities are lost during fine-tuning; 2) format specialization is an important cause for such loss; 3) format specialization happens the very beginning of fine-tuning.
3.1 LOSS OF IN-CONTEXT LEARNING CAPABILITIES DURING FINE-TUNING
In this subsection, first show that the in-context learning performance usually drops significantly after standard fine-tuning.
In our experiments, fine-tune a pretrained mT5 XXL model (13B parameters) (Xue al. 2020 on the Recognizing Textual Entailment (RTE) dataset (Wang et al.) 2019). In RTE tasks, the model is required to predict "True" or "False" for whether the two given sentences are entailed. We fine-tune
3

---

Published as a conference paper at ICLR 2024
the mT5 model with default hyper-parameee and input/output template used in PaLM (Chowdhery ct al. 2022).
120
1.2
RTE
1-shot triviaGA
RTE
Ratio of TrueiFalse
100
1- web questtons
20
100
1.0
Accuracy
80
15
Accuracy
80
60
0.6
-10
0.4
20
20
01
0.0
50
100
150
400
450
500
400
450
500
Fine
Fine-tune
Figure I: Loss of in-context learning abili- ties during fine-tuning. We show the learn- ing curve of a model being fine-tuned on RTE dataset while being tested on 1-shot QA datasets. Left axis: Accuracy on RTE. Right axis: Exact match rate on 1-shot QA.
Format specialization in fine-tuning: showing the frequency of "True/False" style outputs when evaluated on I-shot TriviaQA. The model is being fine-tuned on RTE. Left axis: Accuracy on RTE Right axis: Ratio of True/False.
Figure 2:
We want to see whether the model lost its in-context learning abilities on unseen task during fine- tuning. Therefore, evaluate the fine-tuned model with two 1-shot QA tasks, TriviaQA (Johh et al., 2017) and web_questions (Berant et al. 2013). The results are illustrated in Figure | where we see that when the accuracy RTE dataset increases with fine-tuning, performance few-shot QA tasks drops drastically. This phenomenon is general and not result of specific fine-tuning or evaluation tasks (more results in Section 5.3)
3.2 FORMAT SPECIALIZATION
Why are the in-context learning abilities of an LLM so easily lost after few hundred steps of fine-tuning? natural hypothesis is that due to the homogeneity of output formats in fine-tuning datasets, the model quickly specializes to this task format and learns to follow it no matter what the input sequence is. This leads to the loss in-context learning abilities on other tasks that do not share the same format. here by "format" we refer to the common characteristics of the sequences in fine-tuning task as a subset of all possible sequences, such as the language used, typical input/output lengths and styles, special tokens or punctuation, upper/lower case styles etc. For example, the output format of RTE is a set of two labels, "True" or "False", among all possible sequences of tokens of various lengths. Since all data points share the same format in single-task fine-tuning, the model receives a strong gradient signal that the output should follow this format, thus its in-context learning performance on other tasks with different formats will drop, even when they share important semantic similarities with the fine-tuned task.
To verify this hypothesis, we evaluate the RTE fine-tuned mT5 model on 1-shot TriviaQA task and count the percentage of outputs which are "True" or "False".. Figure shows that as the fine-tuning proceeds, the model outputs more "True" or 'False' even with a 1-shot prompted input from TriviaQA. In particular, after 300 fine-tuning steps, 90% of the output becomes "True" or "False". The same phenomenon happens on other in-context learning tasks. With a 1-shot WMT16 En-De translation prompt, after 500 steps of RTE fine-tuning, more than 99% of the output becomes "True" or "False". This indicates that format specialization a possible reason for the loss general in-context learning capabilities during fine-tu mne-tuning.
3.3 FORMAT LEARNING HAPPENS FIRST DURING STANDARD FINE-TUNING
Next, we show experimental evidence that format learning happens first during standard fine- tinettuning. This is not surprising as the overwhelming majority of fine-tuning data points have very similar formats, causing a gradient signal that dominates over others, more nuanced elements such as the semantic content of the task.
More concretely, for the RTE dataset, the "format" refers to the fact that the output E False) while the semantic content refers to the correlation between the input sequence and the output label.
4

---

Published as a conference paper at ICLR 2024
Stage
Stage 2
Evaluation
Model tuning by
Model: frozen
gradient descent
Model: 2-stage finetuned
Input
Saft prompt luning
Solt prompt:
Input
Input
gradient descent
frozen
Trained promot
Input
Figure 4: Overview of ProMoTT, our two-stage fine-tuning strategy. We run prompt tuning at Stage 1 and model fine-tuning with the trained prompt at Stage 2. Green denotes trainable parameters and blue means frozen.
We isolate format learning from semantic learning by creating randomized RTE dataset where the output labels are randomly shuffled, thus are no longer correlated with the input sequences. The gradients of format learning, Ptormat are then given by the gradients the randomized RTE dataset. By comparing with the full gradient on the original RTE we can detect when format learning happens during fine-tuning. We compute the gradients
the same batches of inputs for the two different settings. Figure and Figure 7 in Appendix show that at the very beginning of fine-tuning (step 0), the full gradient is highly aligned with the format-only gradient Goormrr signified by cos( (go, Itrmout)) ) = 1. Since randomized RTE and original RTE share the format information only and contain totally different semantic content, this alignment implies that the model is mostly learning the format, After 400 fine-tuning steps, this alignment disappears where the cosine similarity drops to around 0 when the True/False ratio reaches nearly 100%.
PROPOSED METHOD: PROMPT TUNING WITH MODEL TUNING (PROMOT)
The observations from Section 3 inspire us to decouple format learning from fine-tuning, in order to alleviate spe- cialization to the fine-tuned task and preserve general in- context learning abilities. The key idea is to offload format learning to a separate small set of parameters during early fine-tuning, and allow the model's 's own parameter changes afterwards to focus more on the semantic content of the task. We propose two-stage fine-tuning strategy called ProMoT. illustrated in Figure 4 At the first stage, ProMoT trainable
cos So. 2h..9amann >} cosl < goon. Stsmar,
1.0
0.8
cool 5.g
0.4
uses prompt tuning to capture the format soft prompt while the model itself is frozen. At the sec- ond stage, ProMoT freezes the learned soft prompt and fine-tunes the model itself to focus on semantic skills that might be more transferable.
0.2
0.0
MLP kemel
Query Modules
Key
Value
Figure Format specialization happens at the beginning of fine-tuning: we show the cosine similarity between the full gra- dient y and the format gradient foormat on the MLP kernel, Query. Key and Value on the attention module. The g and Iormat are much better aligned at the start of training, compared to at 400 steps. Comparison between more steps can be found in Figure 7 (Appendix).
Stage 1: Prompt Tuning. Here we use a continu- trainable prompt (soft prompt) (Lester al. 2021) prepended before the embedded inputs as the separate small of tunable parameters. The soft prompt for given fine-tuned task P. E RRxx is a small set of free parameters taking the form of few trainable embeddings, where p is the prompt length and is the embedding size. Given an input sequence, prompt tuning first embeds it with the text embedding layer of the pretrained model, and then prepends it with the soft trainable prompt. The soft prompt is then optimized to reduce the loss while the pretrained model is frozen. As indicated Section fine-tuning first learns the format. We expect that by prompt tuning first, the soft prompt will learn the format. Although it is not guaranteed that
We compute the format gradient at 400 steps, Itoema, 400. by first fine-tuning the model on RTE for 400 steps, then computing the gradient the randomized RTE dataset with the same batch of input sequences.
5

---

Published as a conference paper at ICLR 2024
the soft prompt only learns the format, the small capacity can prevent the soft prompt from learning all semantic skills in most realistic NLP tasks, as demonstrated by the performance gap between prompt tuning and standard fine-tuning.
Stage 2:
Fine-tuning with trained prompt. After prompt-tuning, expect the trained prompt now storing most the format information. We then freeze the soft prompt and fine-tune the pretrained model. Importantly, as shown in Figure 4 the soft prompt is still prepended before the input during this stage, forcing the model to learn things not captured already by the soft prompt.
Other parameter-efficient and fine-tuning methods. ProMoT is a general framework that can be combined with different parameter-efficient tuning and fine-tuning techniques in respective stages. Conceptually, the prompt-tuning at the first stage can be replaced by other commonly used parameter- efficient methods such as LoRA Hu al. (202I). However, empirically we found prompt- is much better than LoRA on absorbing format information in early fine-tuning, More discussions can be found in Appendix C.7 For fine-tuning methods, we show examples combine ProMoT with multi-task fine-tuning (Section 5.4) and 1-shot in-context learning prompt (Section 5.3] Section 5.4).
Training with 1-shot prompt introduced by Min al. (202I) in a multi-task training setting.
Evaluation. After the two-stage fine-tuning, we obtain fine-tuned model checkpoint and a trained soft prompt for specific fine-tuning target task. We expect the soft prompt stores most of the format information, and we only use this prompt during inference when the inference task has the same format as the fine-tuned target task. Otherwise, we remove the learned prompt and simply feed the original input into the fine-tuned model.
EXPERIMENTS
5.1 SETTINGS
Datasets. We use RTE (Wang et al. 2019; Bentivogli et al. 2009) and WMT14 En-Fr (Bojar et al. 2014) as two fine-tuning tasks in our main experiments. They are selected as examples of classification (RTE) and generative tasks (WMT14 En-Fr translation). Experiments additional
fine-tuning tasks including SNLI (Bowman et al. 2015) and OpenbookQA (Mihaylov et al. 2018 can be found in Appendix C
We use 8 tasks unseen during fine-tuning to evaluate the model's generalization abilities. The evaluation tasks are chosen represent four types of tasks:
Natural language inference: CB (De Marneff et al. 2019) and WiC PPilehvar & Camacho-
Collados 2018) from superGLUE (Wang al. 2019)
Closed book QA: TriviaQA (Joshi et al. 2017). web questions Berant et al. 2013
Translation: WMT16 En-Ro, WMT16 En De (Bojar al., (0166)
Summarization: XSum (Narayan et al. 2018). Wikil.ingua (Ladhak et al. 2020
For each evaluation task, we use 1-shot and 4-shots prompts and task templates from PaLM Chowd. hery et al. 2022) described in the Appendix
Metrics. We report accuracy for classification tasks, exact match ratio for QA tasks, BLEU score (Papineni et al, 2002) for translation tasks and Rouge-2 score (Lin) 2004) for summarization tasks. We evaluate the model on development set for superGLUE sub-tasks (RTE, CB and WiC) and on test set for all other tasks. Besides per-task performance, we also report the normalized average (Norm. Avg.) performance all evaluation tasks by averaging the performances normalized to [0,100], following the "normalized preferred metric" in BIG-bench (Srivastava et al. 2022 and
Chung et al. (0022)
Models. We primarily use mT5 (uee al. 2020) XXL model (Raffel al. 2020b) in our main experiments, which is pretrained on multi multi-linuual corpus and contains 13B parameters. This is to accommodate multi-lingual scenarios among training and evaluation tasks. To show the effectiveness of our method on different pretraining corpus, model sizes and architectures, we also include experiments on mT5 XI., T5.1.1 XXL. and PaL.M 8b in Appendix C. T5 based models

---

Published as a conference paper at ICLR 2024
shown to have meaningful few-shot performance as shown in Chung et al. (2022) We do not consider FLAN-TS (Chung et al.) 2022) as a base model in our experiments because it has already been fine-tuned large amount of supervised datasets, including evaluation datasets. More experimental details can be found in Appendix
Comparing methods. We compare ProMoT with several different configurations, including
Pretrained model: We evaluate the pretrained model on all tasks without any fine-tuning. Fine-tune the pretrained model without trainable prompts. We also include a multi-task version in Section 5.4 which is commonly used to boost model generalization unseen tasks.
Standard fine-tuning:
Prompt tuning: Tune the trainable prompt with pretrained model frozen. As the model is fixed, prompt tuning will not change the pretrained model's performance on in-context learning tasks comparing when the prompt is removed.
Our proposed method: ProMoT:
Our proposed two-stage fine-tuning strategy. Our proposed method: ProMoT+1-shot: To further boost in-context learning performance, we prepend I-shot example to the input in Figure 4 during training.
5.2 SUPERVISED PERFORMANCE ON FINE-TUNING TASKS
We first show that ProMoT training can achieve similar or even better performance on fine-tuning tasks compared to standard finninuning We apply three different fine-tuning methods four different tasks and report the result in Table 2 We report the best performance within the same number of fine-tuning steps (See Appendix B for more details). ProMoT outperforms standard fine-tuning on supervised performance on 3 out of 4 fine-tuning target tasks and outperforms prompt-tuning on 4 out of tasks. Therefore the improved in-context learning performance on unseen tasks (better generalization ability), as will be demonstrated in the next few sections, comes without sacrificing the fine-tune task's ' performance.
Prompt tuning
Standard Fine-tuning
ProMoT (Ours)
RTE
91.34
92.06
92.78
WMTH En-Fr
39.28
41.80
41.30
SNLI
88.53
88.91 91
89.62
OpenbookQA
73.60
77.2
81.6
Table 2: Comparison of supervised performances of mT5 XXL model on fine-tuning target tasks. We use O-shot in fine-tuning tasks. We report accuracy for RTE, SNLI and OpenbookQA, and BLEU score for WMT14 En-Fr.
5.3
GENERALIZATION WITH SINGLE TASK FINE-TUNING
Pretrained
Standard Fine-tuning
PruMoT (Ours)
ProMeT I-shot (Ours)
Fine-tuning
RTE
47.653
92.06
92.78
93.86
1-sbot
4-sbots
1-shot
4-shots
1-shot
4-sbots
1-shot
4-abots
15.43
16.56
20.10
21.24
22.26
22.33
Norm. Avg.
17.52
18.75
(-2,10)
(-2.19)
(+2.58)
(+2.49)
(+4.74)
(+3.58)
CB
46.43
51.79
73.21
82,
66.07
67.86
83.93
82.
WIC
49.69
49.69
50.00
50.16
51.41
53.61
51.25
50.63
Evaluation
trivinQA
17.58
19.02
0.15
0.11
17.64
18.66
17.82
19.62
que__qussionss
0.05
13.19
12.11
WMT16 ende
3.97
8.83
0.00
0.00
2.02
3.69
2.26
4.89
WMT16, enro
1.82
3.92
0.00
0.00
0.70
0.96
0.87
1.87
XSum
6.41
2.35
0.00
0.00
7.02
7.01
3.93
WikiLinguseen
4.59
1,33
0.00
0.00
4.84
4,90
4.87
3.43
Table 3: Performances of a mT5 XXL. model finetuned on RTE and evaluated on 8 different tasks to verify the generalization ability. The accuracy on fine-tuned task (RTE) is in the first row. We compare the Norm. Avg. (normalized average performance) with pretrained model and report the relative difference, where red denotes decreased performance and blue denotes increased performance. CB and WiC are also NLI tasks, very similar to RTE.
In this section, we evaluate and compare the few-shot performance on unseen tasks after fine- tuning. We show the evaluation results of fine-tuning RTE and WMT14 En-Fr in Table 3 and Table
7

---

Published as a conference paper at ICLR 2024
respectively. Experiments on additional fine-tuning tasks SNLI/OpenbookQA and additional base models including mT5 XL, T5.1.1 XXL and PaLM 8b can be found in Appendix C
Pretrained
Stundard tuning
ProMoT (Ours)
ProMoT I-shot (Ours)
Fine turing
WMT14 En Pr
1.98
41.80
41.30
41.19
I-sbot
4-sbots
1-shot
4-shots
1 1-shot
4-sbots
I-shox
4-sbots
Norm, Avg.
17.52
18.75
9.15
11.67
18.87
20.64
19:91
21.99
(-8.37)
(-7,07)
(+1.35)
(+1.8)
(+2.39)
(+3.24)
CB
46,43
51.79
16.07
32. 14
41.07
57.14
41.07
53.57
WIC
49.69
50.63
49.06
50.16
50.31
49.84
50.63
Evaluation
triviaQQ
17.58
19.02
3.20
3.15
13.63
15.20
16.93
18.19
web_ queestuuss
9.70
13.04
0.89
6.15
9.40
7.92
10.14
12.01
WMT16 ende
3.97
8.83
0.81
0.18
15.52
15.55
16.14
15.63
WMT16 enro
1.82
3.92
1.53
0.42
18.54
17.80
17.57
16.81
XSum
2.35
0.05
1.86
0.65
4.36
Wigiuinn
4.59
1,33
0.03
0.43
1.14
0.52
4.22
4.73
Table 4: Performances of a mT5 XXL model finetuned on WMT14 En-Fr and evaluated on 8 few- shot tasks to verify the generalization ability. BLEU on the fine-tuned task is in the first row. We compare the Norm. Avg. (normalized average performance) with pretrained model and report the relative difference, where red denotes decreased performance and blue denotes increased performance. WMT16 En-De and En-Ro are translation tasks with different language pairs from WMT14 En-Fr.
From both tables, we first observe that the model's in-context learning performance drops significantly after standard fine-tuning. In particular, the few-shot learning performances drop to near zero for 6 over 8 tasks in Table B with the only exceptions being CB and WiC where they share the same format as the RTE fine-tuning task.
On the contrary, ProMoT reduces the loss of the in-context learning performance unseen few- shot evaluation tasks, and even boosts some evaluation tasks that are semantically related to the fine-tuning task but with totally different task formats, resulting in increasing in-context learning performance on average. In Table 3 ProMoTT on the binary NIJI dataset dataset consistently improves few-shot performances on two summarization tasks beyond the pretrained model. In Table ProMoT training on English-Frenn French translation substantially improves few-shot performance on other language translation pairs such as English to German and Romanian. This cross-task generalization across different task formats are infeasible with previous fine-tuning techniques. Text examples from standard fine-tuning and ProMoT can be found in Appendix C.9 The improvement with less specialization and more generalization can be further boosted when we combine ProMoT with 1-shot prompt to incorporate in-context learning objective during fine-tuning.
It is however not surprising that even ProMoT cannot completely eliminate specialization and may still negatively influence some unseen in-context learning tasks compared to the pretrained model, depending on the characteristics of the fine-tuning task. In the next section, we show that a multi-task setup further improves the already strong generalization of ProMoT.
5.4 MORE GENERALIZATION WITH MULTITASK TRAINING
Multi-task training is commonly used improve model's generalization ability (Wei et al. 2021bb Chung et al. (022). As a general fine-tuning framework, ProMoT can be combined with multi-tasking and achieves better generalization compared standard multi-task fine-tuning.
We apply multi-task ProMoT training on mixed RTE and 4MT14 En-Fi translation dataset. At the prompt-tuning stage, we train a soft prompt for each task. At the fine-tuning stage, we mix different tasks and prepend the corresponding soft task prompt to each training example. We keep other configurations the same as Section 5.3 and report the results in Table We compare multi-task ProMoT with standard multi-task fine-tunne tuning. The results show that Multi-task ProMoT significantly outperforms standard multi-task fine-tuning on enhancing generalization with larger improvement on average on unseen 1-shot evaluation tasks. Similar to the single task setting, adding 1-shot prompt before each training input in the fine-tuning stage further boosts the performance of both multi-task fine-tuning and multi-task ProMoT.
8

---

Published as a conference paper at ICLR 2024
Pretrained
multi-task FT
Multi ProMoT
multi-task FT 1-shoo
Multi-ProMoT
1-shht
Multi-task
RTE
47.65
90.25
91.34
91.70
93.14
Fine tunettunig
WMT14 En-Fr
1.982
41.34
40.73
40.87
40.55
Norm. Avg.
17.52
(+2.54)
(+8.35)
(+5.10)
(+8.65)
CB
46.43
80.36
83,93
87.30
85.71
WiC
$1.10
51.41
53,29
$2.04
Evaluation
TriviaQA
17.58
15,76
16,99
16.53
17.18
Web_questions
9.70
9.70
10.04
9.40
10.38
WMT16 En-De
18.83
WMT16 En-Ro
1.82
1.52
18.41
5.62
18.57
XSam
6.41
0.44
4,50
1.82
4.32
WikiLingua/en
0.72
2.89
4.33
3,56
Table 5: Comparison of multi-task training on a mixed dataset RTE and WMT14 En-Fr. We compare the evaluation results of pretrained mT5 model, standard multi-task fine-tuning (FT) and multitask (Multi) ProMoT training. We compare the Norm. Avg. (normalized average performance) with pretrained model and report the relative difference, where blue denotes increased performance.
Joint Fine-tuning
Fine-tuning 1-shot
Fine-tuning with random prompt
ProMoT I-shot (Ours)
Fine-tuning
RTE
90.97
90.97
92.06
93.86
CB
83,93
78.57
83,93
83,93
WiC
50.47
51.41
51.72
51.25
TriviaQA
Evaluation
web questions
0.64
0.00
0.30
10.14
WMT16 ende
0.00
0.00
0.00
2.26
WMT16 enro
XSum
0.00
0.00
0.00
6.94
WikiLingua/en
0.00
0.00
0.00
4.87
Table 6: The ablation study results. Joint fine-tuning: fine-tuning the soft prompt and the main model together. Fine-tuning + I-shot: standard fine-tuning with a 1-shot natural language prompt attached every input sequence. Fine-tuning with random prompt: fine-tuning with fixed soft prompt randomly initialized with uniform distribution. ProMoT + 1-shot: ProMoT is applied with an attached 1-shot natural language prompt before each training input.
5.5 ABLATION STUDY
We conduct several ablation studies in Table First, instead of fine-tuning in a two-stage process, we consider "jointly fine-tuning" both the soft prompt and the model parameters in one stage. As shown in Table O this method still results in specialization and severe loss of in-context learning abilities. Thus the benefit of ProMoTT comes from its two-stage nature instead of merely adding more learnable parameters (soft prompt). In addition, fine-tuning the models with fixed random soft prompt does not help- as it does not help to remove format specialization. Another important baseline is to fine-tune the model with natural language prompts place instead of soft prompts, which also capture the format to some extend. In a 1-shot scenario, this approach is still far worse compared to ProMoT, showing that learned soft prompts work better than natural language prompts in reducing format specialization in fine-tuning
6 CONCLUSIONS AND LIMITATIONS
In this paper, we identify format specialization as one important cause of the loss of general in-context learning abilities during LLM fine-tuning, which tends to happen at the beginning fine-tuning. We are motivated to develop ProMoT, a simple yet effective two-stage fine-tuning framework that utilizes soft trainable prompts to absorb rask-specific formats before model fine-tuning. Experiments on diverse set of NLP tasks show that ProMoT reduces format specialization and results in surprising generalization across very different tasks, making a promising method to build general- purppsee capabilities into LLMs with small fine-tuning datasets. Although we have shown the effectiveness of ProMoT in our main paper, there is no theoretical guarantee on how much format specialization can be absorbed by the soft prompt during the first stage of ProMoT. Besides, experiments done with models smaller than 15B due to limited computation resources. It can be interesting to test ProMoT larger models.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank the reviewers for their invaluable feedbacks. The work is supported in part by NSF 2008173, 2048280, 2331966, ONR N00014-23-123300000000, ARL 20230936 and Cisco.
REFERENCES
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answa -nnweer pairs. In Proceedings of the 2013 conference empirical methods in natural language processing, pp. 1533-1544, 2013.
Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation, pp. 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. URL
http:/ www. aclweb. org/ anthology/...oooooooooooootoooat W/ W1 -
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale s Tamchyna. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 12-58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. URL http:http///t /www aclweb. org/ htsp//tthooooogg / W1 20014/1144 3302
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Hlerbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in NeurIPS, volume 33, pp. 1877-1901. Cur- ran Associates, Inc., 2020. URL htsL///ttpss//ppcceeeiiii neur
neuri ps cc cctpccppppp/////////
Stephanie C. Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent in- context learning in transformers, 2022. URL //tppp//.....tt..ii......m wwwpw/wwwww...//2....//..0 05055.
http///xxvvv../
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayang Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy
Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL https://eras www.vorgggeeee /204 02311
10

---

Published as a conference paper at ICLR 2024
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction ceatuctioon-finttmeed language models. arXiv preprint urXiv:2210.111 1416, 2022.
Antonia Creswell and Murray Shanahan. Faithful reasoning using large language models, 2022. URL
httg/bb////gg https://axxiv org/ abs 2208. 14277
Marie-Catherine De Marneff, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. proceedings of Sinn und Bedeutung 23, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT:
Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, pp. 4171-4186, 2019. BRLLattts///aclathooooo
org/ N1 9-1433
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint mrXiv:2012.15723) 2020.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Kirk-airkat and Graham Neubig. Towards unified view of parameter-efffcient transfer learning. arXiv preprint 100.0366,, 2021.
Yun He. Steven Zheng. Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen, Donald Metzler, et al. Hyperprompt: Prompt-based conditiiniin of transformers. In International Conference on Machine Learning, pp. 8678-8690. PMLR, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficienn transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu. Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, arXiv preprint arXiv:1705.033: 100.03355,3 2017.
Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4034-4048, Online, November 2020. As- sociation for Computational Linguistics. doi: 10. 10.18853/v1/2020 /00.://///000..mnnnnneeeeeeem finddings--mmll..3600. URL
https: /ettps:/aaclatee http:///acantthhooggg.o org. 200.0 findings emn ww..p....... 360
Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language
Processing, pp. 3045-3059, Online and Punta Cana. Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/20/ w/t.://..222.....mmmoo.e www.mmmmminnn2444. URL
https: hittps://aclanthor woachhoooyy..o org/ 2021 emnlp-ma n. 243
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. Text summarization branches out, pp. 74-81, 2004.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1906-1919, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/11/0220.co wtwia///aam....i...11... main. URI. https: / ac www.hlonnooooy...oooooooooo org/ 2020. aww.aclmmain 173.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
11

---

Published as a conference paper at ICLR 2024
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannanch Hajishirzi. Metaicl: Learning to learn in context. ar preprint arXiv:2110.15943, 2021.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint MtXiv:2202.128371 2022.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint MrXiv:1808.08745 2018.
Kimia Noorbakhsh, Modar Sulaiman, Mahdi Sharifi, Kallol Roy, and Pooyan Jamshidi. Pretrained language models are symbolic mathematics solvers too! arXiv preprint
2021.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for auto- matic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 0,31111/1073308.0773333 URL htttss:/acclantholloyy wwwpoy/y.rg//0000000000
Mohammad Taher Pilehvar and José Gamacho-Colladee Wie: 10,000 example pairs for evaluating context-sensitive representations. arXiv preprint arXiv:1808.0912) 6, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training. 2018.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican. Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre. Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher, 2021. URL htttts://tttttte org abs 22122.11211111
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text- to-text transformer. JMLR, 21(140):1-6) 2020a. URL http:htt:///it http://www...ggpppr/////////////
20-074.hhm
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020b.
Vinay Venkatesh Ramasesh, Aitor Lewkowyez, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2022. URL httpp:ps////ppeeeeieeeeeee/////e httee//ffrrmmi----aaa___eeeeee
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song. Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model, 2022. URL https 11990
7/ rxxiv.
12

---

Published as a conference paper at ICLR 2024
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Gariga-Alonse et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in NeurIPS, 2017. URL hattp::/pproceeedinngg newrips wc/pppper//01//EEE..e 3f55ee243547deee 045.447777766110000000001005-50-
Guyon, U. Von
pddf
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances neural information processing systems, 32, 2019.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. CoRR, abs/2109.01652, la. URL
2b1. 01652
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.0 652,, 2021b.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.1 1934, 2020.
Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catastrophic forgetting in multimodal large language models. arXiv preprint arXiv:2309.10311
2023.
Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable prompt makes pre-trained language models better few-shot learners. arXiv preprint arXiv:2108.1311 2021.
13

---

Published as a conference paper at ICLR 2024
A BROADER IMPACTS
In our work, we propose method to improve general-purpos language models with fine-tuning datasets. The improved general-purpose language model may be used in malicious applications such as generating disinformation. To mitigate the potential negative impacts, we can add watermark or deploy Al-generated text classifiers before releasing the model.
B EXPERIMENT DETAILS
B. INPUT TEMPLATE USED IN EXPERIMENTS
In Table 7 we list the natural language input template used our experiments for each task The
Task
Template
RTE CB
[premise] question: [hypothesis] it true or false? answer: (True, False)
[premise] question: [hypothesis] Is true or false neither? answer: (True. False, Neither) [sentence [sentence2] question: The word [word] used in the same way the two sentences. Is true or False? answer: (True, False] Q: [question] A) loption A] loption Q: [question]]
WIC
OpenbookQA
[option D) [option DJ
QA
Translation
Translate [source language] [target language]: [sentence 1] Article: [article] One sentence summary:
Summarization
Table 7: Input template for each task
example shown in Table Dis from ID 41141109 in XSum dataset.
B.2 OUTPUT POST-PROCESSING
For each task, we first extract the text after <extra_id Lextrraid__os 0d> and before cextra_id_1>> then trim the text by locating and remove the text after the second prefix token (Q:. Translate, Article;), For classification tasks including RTE, CB and WiC, we check whether the first output token is True or False.
B.3 DATASET AND MODELS
We list the statistics of all datasets used in the paper in Table All the datasets and models can be used in research context.
B.4 HYPER-PARAMETERS
For all mT5 models, we fine-tune with learning rate 0.001, drop rate 0. and label smoothing 0.1, following the default settings for T5 models (Raffel al. 2020b). For all prompt tuning experiments, we use learning rate 0.2 and prompt length 100. For all tasks except summarization tasks, we choose the model input sequence length larger than the input length in datasets. For summarization,
Dataset RTE
Version
Training
Validation
Test
v102
2,490
277
3,000
v102
250
56
WiC WMT14 En-Fr
v102
5,428
638
1,400
v003
15,786,977
3,000
3,003
WMT16 En-De
v003
4,548.885
2,169
2,999
WMT16
v003
610,320
1,999
TriviaQA
1..0
138,384
18,669
17,210
Web Questions
1.0.0
3.778
2,032
XSum
1.1.0
203,577
11,305
11,301
WikiLingualen
gem/wiki semn/wiki_inguuu_emm www.iinnnnggisshseeeooe
99,020
13,823
28.614
Table Version number, sizes of training, validation, and testing splits for each dataset used.
14

---

Published as a conference paper at ICLR 2024
Fine-tuning Datasets
Prompt Tuning + 1-shot
ProMoT 1-shot
RTE WMT14 En-Fr
92.78
93.86
39.41
41.19
Table 9: Performances of prompt tuning 1-shot and ProMoT + 1-shot on fine-tuning tasks.
cut each input to 1024 tokens. We use Adafactor optimizer and batch size without data-packing across all experiments. In inference, we use beam search to decode the outputs with width 4. More experimental settings are provided in the appendix. For ProMoT tuning, at stage we run prompt tuning for 5000 steps and save checkpoint every 1000 steps, then select the prompt checkpoint with the best performance on target task. At stage 2, we freeze the trained prompt and fine-tune the model for 1000 steps, checkpointing every 100 steps. We pick the model checkpoint with highest performance on the fine-tuned task as our final checkpoint. For comparison, we run prompt tuning and standard fine-tuning for 5000 and 1000 training steps respectively and report the performance of the best checkpoint. We explore fine-tuning with more steps in Appendix C.3
In ablation study in Section 5.3 we include an experiment to jointly fine-tune soft prompt and pretrained model. In this experiment, we finetune the model and prompt for 1000 steps with the same learning rate 0.001, following the setting in (He et al., 2022).
B.5 HARDWARE AND IMPLEMENTATION
All the experiments are implemented based on the original prompt tuning and T5x code base All experiments are run on cluster of 64 parallel TPUs. Time cost for different experiments varies, however, all training experiments can be finished within I day.
C ADDITIONAL EXPERIMENT RESULTS
C.I ADDITIONAL RESULTS ON SINGLE TASK FINE-TUNING
As complementary results of Table 3 and we list and compare the performance of prompt tuning + I-shot in Table 9 We also provide experiments on SNLI and OpenbookQA datasets in Table TO Without fine-tuning, pretrained mT5 failed to output "A", "B", "C", "D" for multi-choice QA in O-shot openbookQA dataset, which results in zero accuracy. We can see that the additional experiments
are consistent with our main experiments that ProMoT can achieve similar supervised performance on fine-tuning tasks with less forgetting and even better performance on general in-context learning tasks.
Datasets
Pretrained
Prompt-tuning
Standard Fine-tuning on SNLI
ProMoT (Ours) on SNLI
Standard Fine-tuning on OpenbuokQA
ProMoT (Ours) un Openbook
SNLI OpenbookQA
88.53
88.91
89.62
0.00
73.60
77.2
81.6
Norm, Average
17.52
17.52
16.20
19.83
0.00
17.40
(-1.32)
(+2.31)
(-17.52)
(-0.12)
CB
46.43
46.43
69.64
62.5
0.00
41.07
WIC
49.69
49.69
53.29
$1.25
0.00
50.0
IriviaQA
17.58
17.58
4.54
20.56
0.05
21.10
weh questions
9.70
9.70
2.12
9.94
0.00
10.53
WMT166 ende
3.97
3,97
0.00
2.48
0.00
2,80
WMT16_care
1.82
1.82
0.00
0.90
0.00
1.04
XSum
0.00
Wikil.inguaeen
4.59
4,59
0.00
4,39
0.00
5.20
Table 10: Performance of a mT5 XXL model finetuned on SNLI and OpenbookQA and evaluated on 8 I-shot tasks. The accuracy fine-tuned tasks are in the first two rows. Prompt-tuning doesn't modify pretrained model parameters and has the same in-context performance as pretrained model.
15

---

Published as a conference paper at ICLR 2024
C.2 4- SHOT EVALUATION RESULTS OF MULTI-TAN TASK TRAINING
As an additional result to Table in Table II we provide the comparison between the pretrained model, multi-task standard fine-tuning and multi-task ProMoT.
Pretrained
multi-task FT
Multi ProMoT
Multi-task Finc-tuning
RTE 47.65
90.25
91.34
WMT14 En-Fr
1.982
41.34
40.73
20.81
26.31
Norm. Avg.
18.75
(+2.06)
(+7.56)
51.79
82.14
78.57
WiC
49.69
52.82
52.50
Evaluation
TriviaQA
19.02
17.75
22.26
Web_questions
13.04
12.25
12.50
WMT16 En-De
8.83
0.24
18.84
WMT16 En-Ro
3.92
0.54
18.81
XSum
2.35
0.34
5.04
WikiLingua/en
1.33
0.39
1.92
Table 11: Comparison of multi-tass training mixed dataset of RTE and WMT14 En-Fr. We compare the 4-shot evaluation results of pretrained mT5 model, standard multi-task fine-tuning (FT) and multitask (Multi) ProMoTT training. We compare the Norm. Avg. (normalized average performance) with pretrained model and report the relative difference, where blue denotes increased performance.
C.3 TRAINING MORE STEPS: TRADE-OFF BETWEEN FINE-TUNING TARGET TASK AND IN-CONTEXT LEARNING ABILITIES
In Section 5.3
we report the results of the best checkpoints within 1000 steps of fine-tuning. With longer training period, target task and the performance on in-context learning abilities. Here we show the long-term trade-off between fine-tuning target task and in-context learning evaluation tasks by scattering the performance of different checkpoints within 20000 steps fine-tuning. In Figure 3 and 6 we plot the trade-off on classification and translation tasks, respectively.
more clear trade-off between the performance fine-tuning
15.0
50
Traditional Fine-Tuning
Traditional Fine-Tuning
ProMoT
12.5
ProMoT
Avg. Accuracy
10.0
Avg. BLEU
30
7.5
5.0
2.5
10.
0.0
41
42
43
44
41
42
43
44
BLEU Score
BLEU Score
Figure 5: Trade-off between BLEU score of
Figure 6: Trade-off between BLEU score of En-Fr (horizontal axis) and average BLEU score on other language pairs (vertical axis) when fine-tuning model on En-Fr transla- tion.
En-Fr (horizontal axis) and average accuracy on classification tasks (vertical axis) when fine- tuning the model on En-Fr translation.
As from the figures, datapoints for ProMoT is higher than standard fine-tuning the figures, which implies that with the same performance on fine- tne-tuning target task, forgetting is alleviated with ProMoTT fine-tuning.
16

---

Published as a conference paper at ICLR 2024
C.4 ADDITIONAL EXPERIMENTS ON T5 XXL
To show the performance of our method on an English-based pretrained model, we did an additional experiment on T5 XXL with fine-tuning target task RTE. The result is shown in Table T2 are consistent with our main experiments on the m'55 XXL model.
The results
Datasets
Pretrained
Prompt-tuning
Standard Fine-tuning
ProMoTT (Ours)
RTE
91.7
93.5
93.14
14.07
22.23
Norm. Average
19.75
19.75
(-5.68)
(+2.49)
CB
55.36
55.36
62.50
73.21
WiC
49.84
49.84
50.00
50.78
triviaQA
34.15
34.15
0.02
33.86
web reb_qqestiins
16.04
16.04
0.00
15,95
WMT16 ende
0.13
0.00
WMT16 enro
0.06
0.00
XSum
1.26
0.00
1.79
WikiiLingua/en
1.12
2.25
Table 12: Performance of T5.1.1 XXL model finetuned on RTE and evaluated on 8 1-shot tasks. The accuracy fine-tuned task (RTH) in the first row. Prompt-tuning doesn't modify pretrained model parameters and has the same in-context performance as pretrained model.
C.5 ADDITIONAL EXPERIMENTS ON MT5 XL
To show the performance of our method on an smaller-sizz pretrained model, we did an additional experiment mT5 XL with The results are consistent with our main experiments on the mT55 XXL. model.
target task WMT14 En-Fr. The result is shown in Table I3
Datasets
Pretrained
Prompt-tuning
Standard Fine-tuning
ProMoT (Ours)
WMT14 En-Fr
32,47
35.84
36.46
8.61
14.40
Norm. Average
13.59
13.59
(-4.98)
(+0.81)
CB
26.79
26.79
21.43
28.57
WiC
50.0
50.0
44.20
51.10
triviaQA
12.13
12.13
0.83
8.81
web reb_qqustions
6.59
6.59
0.44
5.31
WMT16_en ende
2.56
2.56
0.63
7.69
WMT16_enme
1.52
1.52
1.20
10.40
XSum
4.26
4.26
0.05
1.37
WikiLinguafen 4.88
4.88
0.06
1.94
Table 13: Performance of a mT5 XL model finetuned on WMT14 En-Fr and evaluated on 1-shot tasks. The BLEU score fine-tuned task (WMT14 En-Fr) is in the first row. Prompt-tuning doesn't 't modify pretrained model parameters and has the same in-context performance as pretrained model.
C.6 ADDTIONAL EXPERIMENTS ON PALM 8B
To show the performance our method on decoder-only models, we did an additional experiment on PaLM 8b model with fine-tuning target task WMT14 En-Fr. We use prompt length 50 and learning rate 0.3 in prompt-tunin and default mne-ttnnng hyperparameters in fine-tuning. The result is shown in Table 14. The results consistent with our main experiments mT5, where ProMoT can achieve similar supervised performance on fine-tuning tasks with less forgetting on general in-context learning tasks.
C.7 USING LORA IN THE FIRST STAGE
As we have discussed in Section conceptually we can use any effameter-efficient method at the first ProMoT fine-tuning stage absorb the task format information. Here did experiments
17

---

Published as a conference paper at ICLR 2024
Datasets
Pretrained
Prompt-tuning
Standard Fine-tuning
ProMoT (Ours)
WMT14 En-Fr
13.62
33.04
32.02
17.80
22.37
Norm. Average
26.09
26.09
(-8.29)
(-3.72)
CB
46.43
46.43
32.14
WiC
49.69
49.69
49.06
triviaQA
44.69
37.09
42.11
web questions
13.02
13.02
11.91
WMT16 ende
23.85
3.77
19.33
WMT16 enro
19.89
4.02
13.18
XSum
5.57
5.57
2.29
2.9
Wikil.ingua/en
5.59
5.59
3.14
4.77
Table 14: Performance of a PaLM 8b model finetuned on WMT14 En-Fr and evaluated on 1-shot tasks. The BLEU score fine-tuned task (WMT14 En-Frr is in the first row. Prompt-tuning doesn't modify pretrained model parameters and has the same in-context performance as pretrained model.
1.0
0.8
cos( go grormat 0
cos( < go ghomat,
cos( 9100. Iformat 100
cos( 9200. Gformat,
0.4
cos( g300.
Glormat, 300 >
cos( g400. gformat, 400
0.2
kernel
Query
Key
Value
Layers
Figure 7: Cosine similarity between the full gradient and the format gradient Itormat on different parts of the last decoder layer. We collect and show the cosine value for gradients MLP kernel, Query, Key and Value on the attention module.
compare LoRA and prompt-tuning (used in our ProMoT main experiments) in the first dine-tuning stage. We report the results in [5] As we can see from the table, ProMoT with prompt-tuning significantly better than ProMoT with LoRA, in both supervised fine-tuning task and unseen 1-shot evaluation tasks. This might partially due to better alignment of soft prompt between format description natural language corpus.
Datasets
Pretrained
Standard Fine-tuming
ProMoT with LoRA (r=2)
ProMoT LoRA (r=4)
ProMoT
WMT14 En-Fr
41.80
39.09
39.97
41.19
CB
46.43
16.07
26.78
23.21
41.07
WiC
49.69
50.63
$3.13
53.25
50.16
triviaQA
17.58
0.03
4.98
5.19
13.63
web_qquestions
9,70
0.05
3.30
3.84
9.40
WMT16_enn'
3.97
0.67
1.23
1.87
15.52
WMT16_enm
1.82
0.91
2.06
18.54
XSum
6.41
0.030
0.17
0.35
1.49
WikiLingua/en 4.59
0.05
0.64
0.57
1.14
Table 15: Performance of mT5 XXL model finetuned WMT14 En-Fr and evaluated 8 1-shot tasks. In this experiment we use LoRA at the first ProMoT stage instead of prompt-tunen rank of LoRA's low-rank update matrices. The BLEU score fine-tuned task (WMT14 En-Fr) is in the first row. Prompt-tuning doesn't it modify pretrained model parameters and has the same in-context performance as pretrained model.
r the
18

---

Published as a conference paper at ICLR 2024
C.8 PLOTTING MORE STEPS FOR FIGURE
To further strengthen our conclusion in Figure 3 here we plot the gradient alignment from step 0 to step 400. As can see from the figure, gradient alignment drops significantly after 300 steps which matched with Figure 2 where the true and false ratio increases before 300 steps and then remains stable.
C.9 QUALITATIVE RESULTS ON FINE-TUNING WMT14 EN-FR TASK
In Table we show example from fine-tuning task RTE. Here show examples for fine-tuning task WMT14 En-Fr translation on different unseen few-shot tasks. We compare the outputs from ground-truth targets, pretrained mT5, fine-tuned mT5 WMT14 En-Fr and ProMoT mT5 WMT14 En-Fr. The outputs are generated with a 1-shot example. As we can see from the examples, standard fine-tuning French; and 2) model tends to repeat its input which is similar to translation task. ProMoT alleviates this specialization on fine-tuning task and has better generalization.
WMT14 14 En-Fr will 1) make the model overfit its format and tend to output
WMT16 En-De
Target: Danach der Mann, der sich nach Angaben seines Anwalts mittlerweile wieder auf freiem Fußbefindet, in eine größere Zelle verlegt worden.
Pretrained: Danach wurde der Mann in eine größere.
déclaration de avocat, depuis été libéré, ensuite été transféré dans une cellule plus grande.
Fine-tune: L'homme, qui, selon
ProMoT: Danach wurde der Mann, der mittlerweile freigelassen wurde, in eine gröBere Zelle verlegt.
WebQuestions
Target: Milwaukee
Pretrained: Milwaukee, Wisconsin
Fine-tune: Where harley davidson corporate headquarters? Milwaukee, Wisconsin Q: what movies has scarlett johansson in? A: Girl with a Pearl Earring Q: where is harley davidson corporate headquarters? A: Milwaukee, Wisconsin Q: where harley davidson corporate headquarters?
ProMoT: Milwaukee, Wisconsin
WikilLingua/en
Target:
Ensure that you have never activated free trial of Amazon Prime on your account in the past. Repeat the steps above to locate products that are fulfilled by Amazon. Click the button that says,
Pretrained:
Click "Start my Free Trial. "Sign up with Amazon Prime. Submit your
order.
Fine-tune:
Si vous avez besoin d' unu aide supplémentaire, n d'éésitez pas communiquer avec l'équipe Amazon Prime. Vous pouvez vous inscrire I' Amazon Prime vous inscrire à I' Amazon Prime. Vous pouvez vous inscrire
ProMoT: Click "Start my Free Trial." "Sign up with Amazon Prime. Enter your credit card details or use one your saved payment methods. Submit your order. you do return, you will be charged $99 for year membership Amazon Prime at the end of your trial period
19


Reviews:
[{"review_id": "uoVAExgGek", "reviewer": "Reviewer_ptkK", "summary": "The paper proposes to use a two-step finetuning approach to reduce forgetting when the finetuned model is applied to other tasks. This is done by prompt tuning the model for some number of steps while freezing the rest of the model and then freezing the prompt while tuning the rest of the model in the subsequent steps. The hope is that the initial prompt tuning learns the format and the rest of the tuning learns the task. The performance is evaluates on a set of NLP tasks where the authors show competitive on task performance and better out-of-task performance.", "questions": "See weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The idea to separate format learning and task learning is interesting.\n- The paper is easy to follow.\n- The experiments show improvements in out-of-task performance.", "weaknesses": "- I am not sure if there's practical utility to the proposed setting. Improving transfer to other tasks which are data limited could be a use case but I found the multi-task section of the paper very underwhelming. Multi-task training was done on only 2 tasks while it would be more informative to do it on a lot more tasks, results over multi-task 1-shot baseline are very mixed and is also missing 4-shot comparison.\n- Prefix + Parameter tuning baseline seems to be missing. What happens if you do both prefix and parameter tuning but don't use the prefix embedding for other tasks. This should be done in both single task and multi-task setting.\n- It would be good to include a prefix-tuning column. This would have some performance for the source task and same performance as the pre-trained model on other tasks.\n- It is hoped that the prefix-tuning stage learns the format. It would be good to empirically test this hypothesis. One simple experiment is to check if the prefix embedding outputs the source task format on other tasks.\n- Results on Palm 8B in the appendix don't look as promising and they should not be in the appendix but in the main paper. The approach seems to show worse performance than pre-trained model on other tasks after using the proposed approach. Would also be good to include 4-shot results here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M1hTONppWa", "reviewer": "Reviewer_qDAR", "summary": "This paper proposes an innovative fine-tuning method that preserves the model's general in-context learning ability. The method involves absorbing task-specific format knowledge into prompt tuning parameters and finetuning the main model. During evaluation, the prompt tuning parameters can be optionally dropped depending on dataset similarity.", "questions": "1. In Table 3,4, It may be worthwhile to put more highlights on the most relevant datasets, e.g. CB and WiC for RTE, WMT16 en>de, en>ro for WMT14 en>fr.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The method is \"strange\". It furthers our understanding of param-efficient fine-tuning methods by showing and leveraging a unique property of prompt tuning.\n2. Evaluated on a diverse set of datasets.\n3. Ablation experiments showing the necessity of some design details.", "weaknesses": "1. Only on one model.\n2; Finetuned on only two datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5DYRYxigPI", "reviewer": "Reviewer_5e4F", "summary": "The work focuses on solving the issue of performance decrease after applying single-task finetuning for T5 and PaLM models. The paper proposes a two-stage finetuning: 1) model frozen with soft prompt tuning 2) model tuning with soft prompt tuning frozen. They conduct experiments on 10+ NLP tasks, and the proposed approach achieves similar supervised performance. The paper explores a variety of experimental settings, single-task, and multi-task finetuning, and provides comprehensive insights showing the generalization ability of the model.", "questions": "- Do you have any difference trends on different base models? and also different model types (e.g., decoder-only models)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is shown to be very effective in preserving the model generalization with single-task fine-tuning while it slightly reduces the specialization, which is very beneficial for LLMs.\n- Comprehensive evaluation on multiple tasks, and the authors run multiple runs on the few-shot setting to better show the model performance variance. \n- The experimental setting comprises many interesting settings (single-task vs. multi-task).\n- It is a well-written paper with good clarity on the technical experimental details.", "weaknesses": "- The limitation of the paper is only evaluated on encoder-decoder models. It would be interesting to also evaluate the approach on decoder-only models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QvzOqmm7eI", "reviewer": "Reviewer_3Pj7", "summary": "This work introduces PROmpt Tuning with MOdel Tuning (ProMoT) to address the issue of “format specialization”. This problem occurs when Pre-trained Language Models (PLMs) lose their in-context learning abilities after being fine-tuned for a specific task. To elaborate, PLMs are initially fine-tuned for the task using soft prompt tuning, wherein only the soft prompt is tuned for that specific task. Then, all parameters, excluding the trained soft prompt, are fine-tuned for the same task. Experimental results show that ProMoT not only prevents the format specialziation but also enhances the performance on the target task.", "questions": "See Weaknesses; Why do the authors use only two datasets for fine-tuning?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- **Reasonable problem setup and simple approach;** The proposed method, ProMoT, is demonstrated to be effective in preventing format specialization using a straightforward two-stage fine-tuning approach. Additionally, this paper reveals the cause of format specialization by analyzing the changes in gradient similarity between the data from both the randomized and original RTE datasets.\n- **Notable Experimental Results;** The authors conducted extensive experiments across diverse types of tasks including Natural Language Inference (NLI), closed book QA, translation, and summarization. Furthermore, experimental results show that the proposed method is effective in alleviating the issue of format specialization after being fine-tuned on the classification (RTE) and translation (WMT14 En-Fr) tasks.\n- **Clear Presentation;** This paper is well-written. Therefore, it is easy to understand the problem setup and the proposed method.", "weaknesses": "- **Limited Evaluation;** While the experimental results include diverse evaluation tasks, the fine-tuning datasets are limited to just two tasks: NLI and Trnaslation. Why did the authors limit their experimentation to only these two tasks? Is it implied that format specialization does not occur in other tasks, such as question answering and summarization? I couldn’t find a satisfactory explanation in the paper regarding the absence of experimental results from fine-tuning on other tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)