Progressive Prompts: Continual Learning for Language Models
Authors: Anastasia Razdaibiedina
Yuning Mao
Rui Hou
Madian Khabsa
Mike Lewis
Amjad Almahairi
Venue: ICLR 2023 poster (2023)
Keywords: natural language processing
continual learning
prompt tuning

Abstract: We introduce Progressive Prompts – a simple and efficient approach for continual learning in language models. Our method allows forward transfer and resists catastrophic forgetting, without relying on data replay or a large number of task-specific parameters. Progressive Prompts learns a new soft prompt for each task and sequentially concatenates it with the previously learned prompts, while keeping the base model frozen. Experiments on standard continual learning benchmarks show that our approach outperforms state-of-the-art methods, with an improvement >20% in average test accuracy over the previous best-preforming method on T5 model. We also explore a more challenging continual learning setup with longer sequences of tasks and show that Progressive Prompts significantly outperforms prior methods.

Published as a conference paper at ICLR 2023
PROGRESSIVE PROMPTS: CONTINUAL LEARNING FOR LANGUAGE MODELS
Anastasia Razdaibiedina") Yuning Maoi, Rui Hou-,
Madian Khabsa*, Mike Lewis*, Amjad Almahairi
University of Toronto & Vector Institute
Meta AI
anastasia www.aaabeeeennaaaallll...m wtt.oaatooaai..otooooooooooooo
{yuningm, rayhou, mkhabsa, mikelewisee.. aalmah)emeta.com
ABSTRACT
We introduce Progressive Prompts simple and efficient approach for contin- ual learning in language models. Our method allows forward transfer and re- sists catastrophic forgetting. without relying on data replay or large number of task-specific parameters. Progressive Prompts learns a new soft prompt for each task and sequentially concatenates it with the previously learned prompts, while keeping the base model frozen. Experiments benchmarks show that our approach outperforms state-of-the -ht-hartt methods, with an improvement 20% in average test accuracy over the previous best-preforming method on T5 model. We also explore a more challenging continual learning setup with longer sequences of tasks and show that Progressive Prompts signifi- cantly outperforms prior methods.
standard continual learning
INTRODUCTION
Learning a long sequence of tasks while gaining experience and avoiding forgetting remains key feature of human-level intelligence. Although pretrained language models have largely succeeded in learning on single task, their performance degrades in scenarios where multiple tasks are en- countered sequentially, also known as continual learning (CL) (de Masson D' A'ttume et al.] 2019) Huang al. 2021). Two major challenges arise in CL: (1) avoiding catastrophic forgetting. i.e., loss of the knowledge acquired from previous tasks after learning new ones (McCloskey & Cohen, 1989 Ratclifi 1990), and (2) allowing forward transfer, i.e., leveraging the knowledge from past tasks for efficient learning of new tasks.
Typical CL. approaches for language models train model on all tasks, which ensures forward trans- for but also leads to forgetting. These methods use data replay or add regularization constraints (Huang et al. 2021 de Masson D' Autume et al. 2019 Sun al. 2019), but they still suffer from forgetting due to inevitable changes in parameters shared between tasks. Other approaches, such as progressive networks (Rusu et al., 2016)), can alleviate catastrophic forgetting completely while supporting forward transfer, but are computationally expensive because they add a new copy of the model for each task. This can especially intractable for large-scale language models with billions of parameters, which have become standard in the NLP field (Zhang et al. 2022).
In this paper, we introduce Progressive Prompts novel CL approach for language models that supports forward transfer without forgetting. Our method is inspired by progressive networks, but significantly more memory-efficient because it only learns a fixed number of tokens, or prompt, for each new task. Learning prompt to adapt language models on single downstream task was introduced in prompt tuning (Lester et al.] (021), and was shown to match the performance of full model finetuning while training <0.01% of the parameters. In Progressive Prompts, we learn a separate prompt for each incoming task and sequentially concatenate it with previously learned prompts. Importantly, we share input tokens across all tasks and progressively prepend new prompts while keeping previous prompts frozen (see Figure I.. Our method can: 1) alleviate catastrophic
Work done during Meta Al research internship.

---

Published as a conference paper at ICLR 2023
Progressive Network
Progressive Prompts
Task specifi inputt representations
Imput representations shared scross al taska
XL. K2 Inpur ambeddings PI prompt for task prompt task
attention block
causel masking
frozen paramatars
X1
X2
X1
X2
P1
X1
Frozen input embeddings
Frozen input emboddings
prompt
prompt
Trainable 2nd taok prompt
Trainable 2nd task prompt
Figure I: Illustrating proposed method Progressive Prompts and contrasting it with simple adaptation of progressive networks using prompt tuning. In the simple adaptation of progressive networks we learn separate prompt and repeat the frozen input embeddings for each new task. This setup requires repeating input tokens for each task. In Progressive Prompts we use the same input and progressively append new prompt for each new task. Prior task prompts are not modified by the addition of new prompts.
forgetting by preserving the knowledge acquired in previous prompts, and 2) transfer knowledge to future tasks by sequentially learning new prompts given previous ones. We also introduce a new technique for prompt embedding reparameterization (Li & Liang 202T). We show that by passing the prompt embeddings through residual MLP we can stabilize prompt tuning and improve its performance.
We run extensive experiments on standard CL benchmarks for text classification, and show that Progressive Prompts outperforms state-of-the-art approaches both BERT and T5 architectures (Devlin et al.) 2018; Raffel et al.) (2020). We show over 20% improvement over the current SOTA T5 model (Qin & Joty (021). Furthermore, we run experiments on more challenging CL setup with longer task sequences, and show that our method outperforms prior approaches for T5 and BERT architectures.
Our main contributions in this paper are as follows:
We propose a novel CL approach, Progressive Prompts, that alleviates catastrophic forget- ting and supports knowledge transfer to future tasks all while learning < 0.1% of the total parameters.
Progressive Prompts is suitable for any transformer-based architecture. We show that significantly outperforms prior SOTA methods on standard CL benchmarks for both BERT and T5 models.
We propose more challenging CL setup encompassing 15 text classification tasks and show that our method significantly outperforms prior methods.
BACKGROUND
2.1 FINETUNING
The predominant technique for adapting the language model to downstream task T is finetuning, when all parameters of the model (initialized from some pre-trained weights) are updated during task adaptation (Devlin et al. 2018 Zhang al. 2020).
Consider a classification task with input text æ, and output scalar label y. Here pe is a probability distribution of output classes parameterized by the weights, O of the language model. In finetuning.
2

---

Published as a conference paper at ICLR 2023
perform gradient updates according to the following log- ligeliie objective:
mAX
Σ
log (e
ZYET
Despite its simplicity and effectiveness, finetuning updates all model parameters, and when there are multiple downstream tasks requires storing separate finetuned model for each of them.
2.2 PROMPT TUNING
Recent works explored prompt taning (Lester al. 2021 lightweight alternative for finetuning, when instead of training the whole model, a series of virtual tokens or soft prompt is trained. In prompt tuning, soft prompt is prepended to the input text while keeping other parameters frozen. In this case, the model parameters θ are composed of two sets: 1) the pretrained language model parameters / which are fixed downstream tuning, and 2) prompt parameters Op which are fine-tuned on the given task. Since prompt P has its own dedicated parameters Op. we are not restricted by the original model parametrization 0, and can perform gradient updates on the prompt parameters independently.
Given task and a probability distribution po.a, of output classes parameterized by and 0p. the learning objective here is to maximize the log- likelihood of given the input text concatenated with the soft prompt P. or
max
Σ SyEET
log (v][P:
2.3 CONTINUAL LEARNING
In this work, we focus continual learning setup, where the language model is presented with a sequence of m text classification tasks (T,..., In each task we have a set if i.i.d training examples (Ia, ) where r; is input text and y label from a set of predefined labels Yz associated with the task Tg. We assume that the model is parameterized by e and has access to the task identity in both training and inference. Hence, the learning objective across all tasks becomes:
max
log (yllz)
The most straightforward approach for continual learning is finetuning, which sequentially optimizes loss for task k, & E 1..m by updating all model parameters: (+) Continual finetuning will support forward knowledge future tasks, but will result in catastrophic forgetting, where performance on the earlier tasks decreases after learning new tasks, and eventually leads to higher generalization loss (Kirkpatrick et al. 2017 de Masson A'Atume et al. 2019 McCloskey & Cohen 1989).
Ea.VET, log +).
3 METHOD
Progressive Prompts We propose Progressive Prompts, a continual learning approach which pro- gressively learns prompt Ps for each new task T& (Figure [T). With Progressive Prompts we learn a separate prompt Ps for task Tk. and concatenate it with all previously learned prompts P. i < k before prepending to the input embeddings. During training, parameters 0 of the language model are always frozen, and parameters Hr, corresponding to prompt P: are only trainable during learning Th. and frozen afterwards.
The training objective for task Tr (k E (1...m) is to find prompt parameters 0p, that minimize negative log probability of training examples under our progressive prompt and the frozen base model:
[(0R)= = Σ log log PI. .p.,0ppppp)))
T.ETT,
Such progressive setup allows to achieve two goals for efficient CL: (1) it successfully eliminates catastrophic forgetting, and (2) allows forward transfer for subsequent tasks. Since Progressive
3

---

Published as a conference paper at ICLR 2023
Prompts trains separate prompt for each encountered task without modifying its parameters when new tasks are learned, old tasks do not suffer from forgetting. In addition to that, prompts learned on previous tasks allow information re-use for future tasks. A similar phenomenon has been shown by Vu et al. 2022) prompts learned on informative source tasks served as a good initialization for other downstream tasks.
Embedding reparameterization| Li Liang 2021) have shown that directly optimizing prompt pa- rameters can lead to training instability, while reparameterizing prompt embeddings matrix through a multi-layer perceptron (MLP) can improve performance. At the same time, Liu et al. (2021) report that prompt embedding reparameterization can hinder performance on certain tasks and is highly sensitive to its initialization. To solve this issue, we propose adding residual connection to the MLP reparameterivation. This residual connection improves optimization, because it avoids learn- ing an identity mapping by the MLP. More specifically, we reparameterize prompt Pi: for task T3 as follows:
P MLP(Pp) + Pa
After training on task A is complete, the reparameterization parameters MLP can be discarded, and prompt embeddings Ps can be replaced by their corresponding projection Pj.-
4 EXPERIMENTAL SETUP
4.1 DATASETS
Continual Learning Benchmark We first evaluate our approach on the widely adopted CL bench- mark for language models, which includes five text classification datasets by /Zhang al. (2015): AG News (4 classes, news classification), Amazon reviews (5 classes, sentiment analysis). Yelp reviews (5 classes, sentiment analysis), DBpedia (14 classes, Wikipedia text classification) and Ya- hoo Answers (10 classes, Q&A classification). We provide the task details in Appendix A. and sequences of tasks used in our experiments in Appendix [A.2]
Following previous works on CL for BERT model, including IDBR (Huang et al., 2021 and
MBPA+ (de Masson D' Autume et al., 2019), for BERT-based experiments we use four differ- ent orders of these five tasks. We use the same train and test sets as IDBR (Huang et al. 2021) and MbPA++ (de Masson D' Autume et al., 2019), consisting of 115,000 training and 7,600 test exam- ples for each task. Following Huang et al. (2021), for every task we randomly hold out 500 samples per class from the training set for validation, and use early stopping according to the validation accuracy on all seen tasks.
Following CL setup for T5 model as in LFPT5 (Qin & Joty 2021). we use three different orders of the AG News, Amazon, Yahoo and DBpedia datasets. We replicate Qin & Joty (2021) few-shot CL setting and sample 16 examples per task for the training set and keep the test sets unchanged.
Large number of tasks While previous CL approaches mostly focused relatively short task sequences of 3-5 tasks (Huang et al., 2021 de Masson D' ''utume et al.] 2019 Qin & (oty) 1) a more realistic CL scenario would be long sequences encompassing more tasks. Hence, create benchmark of 15 text classification tasks to evaluate the performance of Progressive Prompts along with widely adopted CL approaches. Our benchmark consist of the aforementioned five datasets from CL benchmark, combined with four tasks from GLUE benchmark (MNLI, QQP, RTE, SST2) (Wang et al.) 2018), five tasks from SuperGLUE benchmark (Wang al. 2019 (WiC, CB, COPA, MultRC,, BoolQ), and IMDB movie reviews dataset (Maas et al.) 2011). To evaluate performance under different dataset sizes, we create three different versions each dataset, with 20, 200 and 1000 training samples sampled per class, and report test set performance for each of these settings. Again, we follow Huang et al. | (202T) practice and for every task we randomly hold out 500 samples per class from the training set for validation, and use early stopping according to the validation accuracy on all seen tasks. We provide the task details in Appendix A. and sequences of 15 tasks used in experiments in Appendix A.2
Transfer learning experiments We also run ablation experiments to understand the effect of for- ward transfer with Progressive Prompts. Firstly, we perform experiments on six pairs of transfer learning tasks from similar domains (IMDb & SST2, Amazon & Yelp. QQP & MRPC) following Hendrycks et al. 2020).. Secondly, we perform experiments SuperGLUUE benchmark (Wang et al.
4

---

Published as a conference paper at ICLR 2023
2019) with Progressive Prompts and compare the performance against original per-task prompt tun- ing by Lester et al. (202T). Following Lester et al. (2021) setup, we use full training sets of eight SuperGLUE tasks and report best validation performance.
4.2 BASELINES
We consider nine baseline methods for comparison with ProgressivePrompts:
Finetune (Wang et al.
2020 de Masson D' Autumm et al. 2019 Huang et al. 2021): train all
model parameters on sequence of tasks (without adding any regulanzation constraints or replay- ing samples from the previous tasks).
EWC (Kirkpatrick et al. 2017): finetune the whole model with a regularization loss that prevents updating parameters that could interfere with previously learned tasks.
A-GEM (Chaudhry al. 2018); save examples from the past tasks and restrict the gradients used to update the model on new tasks based on the retrieved examples.
Experience replay: finetune the whole model with a memory buffer, and replay samples from old tasks when learning new tasks to avoid forgetting.
2019); augment BERT with an episodic memory that saves all seen examples. Perform replay during training, and local adaptation during test time.
MBPA+ de Masson D' D'Autume
IDRR (Huang et al. 2021): BERT-specific approach which continuously trains the whole model using data replay and regularization loss, which applies sentence representation disentanglement into task-specific and task- mask-generic spaces. Current SOTA on CL benchmark with BERT.
Per-task prompts (Lester et al. 202T): train separate soft prompt for each task, while keep- ing the original model frozen. This setup will eliminate catastrophic forgetting, since per-task parameters do not change when new tasks are learned, but will not result in forward transfer.
PromptTuning (Lester et al. 2021 Qin & Joty 2021): I): train shared soft prompt sequentially on all tasks, while keeping the orgginal model parameters frozen.
LFPT5 (Qin & Joty 202I): continuously train a soft prompt that simultaneously learns solve the tasks and generate training samples, which are subsequently used in experience replay. Current SOTA on CI. benchmark with T5.
4.3 IMPLEMENTATION DETAILS
Progressive Prompts is a model-agnostic CL method that can be used with any transformer- model. In our experiments, we use two language models adopted by the previous lines of works in CL for NLP: encoder-only BERT model (Devlin al. 2018) and encoder-decoder T5 model (Raffel To compare Progressive Prompts to the recent CL. approaches implemented specifically BERT. we use the pre-trained BERT-base model as IDBR and MBPA+ methods (Huang et al.
et al.
2020
with
202 de Masson D' Autume al. 2019). For comparison with LFPT5 approach (Qin & Joty
2021
we use pre-trained Ta-large model.
BERT Following Devlin et al. 2018), to predict the class of input text use the representation of its first token hiccs (which encoded by a special beginning-of -f-----n-----eceee symbol, (CLS)) as sentence representation. Here / is the whole-input representation matrix from BERT encoder. We apply a linear transformation parametrized by w and a softmax function to obtain the classification probabilities over classes E (1..C):
exp(wchiCLS)
ply = c/h) Eyec exp(wxllccss
In BERT experiments we train separate linear head in addition to prompt embeddings for each task, using cross-entropy loss between the predicted classification probabilities and ground truth class labels.
Although the original prompt tuning approach reports better performance with T5 v1.1 compared to T5, several subsequent works find version v..1. less stable for prompt tuning compared to the original T5 and report worse performance (Karimi Mahabadi al. 202T) Therefore, in this work the original TS model.
5

---

Published as a conference paper at ICLR 2023
T5 Following Raffel et al. (2020) and Lester al. (2021), adopt text-to-text formulation for all TS experiments, where classification labels are mapped into words (c.g. 0/1 could be encoded "Trus"""Falle"). T5 model applies a nulti-headed self-attention over the input tokens followed by nosition-wise feed-forward layers to produce an output distribution over target tokens. We train prompt embeddings for T5 model using cross-entropy loss. We report the rest of the implementation details in Appendix A.3
Prompt length For all Progressive Prompt experiments BERT, set single-task prompt length to 20 tokens and apply prompt reparameterization with two-layer residual MLP. For T5 experi- ments, we use single-task prompt length to 10 tokens in case long-sequence experiments and 50 in case of T5 experiments (so that total Progressive Prompt length more closely matches LFPTS prompt length of 300). We report more experimental details Appendix A.4
EXPERIMENTAL RESULTS
For all CL experiments evaluate methods after training all tasks and report averaged test set scores across all tasks. Metrics used for different tasks are shown in Appendix [A.1] We report Pro- gressive Prompts performance with BERT-base and T5-large models, and compare the existing CL approaches used with these models.
Order
Order
Method
DR
4
5
avg
Method
DR
2
3
avg
Finetune
14.8
27.8
26.7
4.5
18.4
Finetune
18.9
24.9
41.7
28.5
Replay?
67.2
64.7
64.7
44.6
57.8
Replay
35.4
37.1
41.5
38.0
A-GEM!
70.6
65.9
67.5
66.9
EWC
39.0
38.0
44.8
40.6
MBPA++
70.8
70.9
70.2
70.7
70.6
*FPTT5
47.6
52.6
57.9
52.7
IDBR
75.9
76.2
76.4
76.7
76.3
ProgPrompt'
75.2
75.0
75.1
75.1
ProgPrompt"
78.0
77.7
77.9
77.9
77.9
Per-task Finetune
70.0
70.0
70.0
70.0
Per-task Finetune
73.9
73.9
73.9
73.9
73.9
(a) Results with T5-large.
(b) Results with BERT-base.
Table 1: Summary of the results on two standard CL benchmarks with T5 and BERT models. Av- eraged accuracy after training on the last task is reported. All results are averaged over runs. For T5 experiments we followed Qin & Joty (2021) protocol and used few-shot CL setting. Methods marked with only train soft prompt while keeping the model frozen, other methods train the entire model. DR denotes whether the method requires data replay. O, and denote results from
Qin & Joty (2021) de Masson D) Autume et al. (2019) and HHang et al. (202)) respectively.
5.1 RESULTS ON STANDARD CONTINUAL LEARNING BENCHMARKS
T5 benchmark. Table Ta compares Progressive Prompts performance on the few-shot CL bench- mark for T5 model with the existing CL approaches, including previous SOTA LFPTS (Qin & Joty 2021). Notably, Progressive Prompts achieve over 20% improvement compared to LFPT5. This drastic increase in accuracy explained by the fact that Progressive Prompts method does experience forgetting and allows forward transfer, which is especially useful in few-shot regime. Of note, Qin & Joty| (202T) report better performance of LFPTS continual learning setting than Adapter-Fusion (Pfeiffer et al. 2020) strong parameter-efficient CL approach based on adapters. Hence, we did not include adapter baselines in this study, but instead focused on more parameter- efficient and better-perfon eeterrperrfommig method - LFPT5.
BERT benchmark. Table Ib] compares performance of Progressive Prompts across four different task orders with existing CL approaches for BERT model, including previous -OTA- IDBR (Huang et al. 2021). We want to note that CL benchmark for BERT focuses on full dataset experiments, contrary to few-shot CL benchmark for T5. Since Progressive Prompts most efficient in few-shot setting, we observe a larger performance gap with LFPT5 compared to IDBR. Overall, Progressive Prompts further improve performance compared IDBR, reaching 77.9 average score across all

---

Published as a conference paper at ICLR 2023
ders. We also note that Progressive Prompts is model-agnostic approach, contrary to IDBR, which designed specifically for BERT. Additionally, our method does not require storing data from the previous tasks for future replay, in contrast to the previous commonly used CL approaches for BERT IDBR and MBPA ++. In all of our experiments we use the embedding reparameterization defined in Section 3. We find that this reparameterization helps in stabilizing and accelerating training. Due space limitations, we explore the effect of residual reparameterization in Appendix B.33)
5.2 PERFORMANCE WITH LARGE NUMBER OF TASKS
Table 2 compares common CL approaches, including SOTA methods for T5 and BERT models (Qin & Joty 2021 Huang et al. 2021 on sequence continual learning with 15 tasks. We report aver- aged results across three orders (8, and 10) obtained with T5-Lar and BERT-base models. We provide the full non-averaged results for each order in Appendix [..] To investigate the effect of limited data settings, we perform training on different dataset sizes with 20, 200 and 1000 samples class.
Method &
avg
Method
avg
Num. samples +
20
200
1000
Num. samples
20
200
1000
Ta-Large results
BERT-base results
Finetune
9.7
8.3
7.4
Finetune
31.3
42.4
41.7
Prompt tuning"
17.4
13.9
10.9
Prompt tuning"
47.6
57.2
59.5
Replay
43.6
44.2
54.4
Replay
36.4
47.0
49.6
Per-task Prompts*
69.8
75.2
77.0
Per-task Prompts*
50.6
62.4
67.2
LFPT5*
54.3
58.2
69.2
IDBR
36.8
47.9
52.2
ProgPrompt"
76.2
78.7
79.5
ProgPrompt*
53.5
66.9
69.3
MTL
70.7
72.5
76.3
MTL
56.9
67.7
69.9
Per-task Finetune
68.2
73.7
78.1
Per-task Finetune
53.2
58.4
63.1
Table 2: Progressive Prompts outperforms existing continual learning methods on T5-Large and BERT-base models long sequences of tasks (15 tasks in total). Average results across three different task orders with 20, 200 and 1000 samples per class are shown. MTT. denotes multi-task learning. Methods marked with only train soft prompt while keeping the model frozen, other methods train the entire model.
Our approach, Progressive Prompts, consistently outper- forms all other methods across different data limits, with the strongest improvement in the few-shot setting of 20 samples per class: +21.% and +-333% over previous SOTA approaches T5 and BERT models respectively. We also provide more fine-grained performance met- rics defined by Lopez-Paz & Ranzato (2017), including forward transfer (FWT) and backward transfer (BWT) scores, well as evolution of accuracy with learning new tasks in Appendix [.I
Average attention scores across prompts
boolg
(query)
wic
qap
Incominn
imdo
s:12
alapediaa
amazon
Attention between prompts To investigate which prompts from the progressive pool are helpful for learn- ing new tasks, we visualized perment-prompt attentions (Figure 2). We used Progressive Prompts trained on or- der 8 with T5 model. For each of the 15 consecutive tasks, we passed all test examples through T5 encoder, and ex- tracted attention matrices from its last layer (24th layer). We averaged attention scores across all prompt tokens and all heads, and shown the results Figure Interestingly, prompts learn ignore irrelevant tasks and attend the most similar informative tasks. For ample, prompt learned on Amazon reviews task heavily attends to Yelp reviews prompt, and SST2 prompt attends to the prompt learned IMDb another sentiment analysis task.
yahoo
multic
mill
de Siodi
ooueex
Prior tasks Progressive Prompe
Figure 2: Average attention scores be- tween prompts in Progressive Prompts.

---

Published as a conference paper at ICLR 2023
5.3 FORWARD TRANSFER EXPERIMENTS
In this section study in more detail the forward transfer phenomenon achieved with Progressive Prompts. In particular, can a subsequent task prompt exploit the knowledge from a previously learned prompt in learning
related task?
Transfer Learning on a pair of tasks We compare the performance of a single prompt of length 2M versus Progressive Prompt consisting of two M-length prompts, M = 50 (see Figure B We follow similar transfer learning experimental protocol by Hendrycks et al. (2020) and use pairs of six tasks from similar domains: IMDb and SST2 (2-class sentiment classification), Amazon and Yelp reviews (5-class sentiment classification), MRPC and QQP (paraphrase detection). We use T5- Ta-Large model, and study both settings: full-dataset and few-shot (2, 5 and 20 samples per class). Our results are summarized in Table 3 and Figure 4 Overall, we observe that Progressive Prompts achieve forward transfer and outperform the standard prompt tuning across all dataset sizes. The strongest improvements happen in few-shot setting +0..4% and + 12.9% relative improvement under 5-shot and 2-shot setup.
Prompt Tuning
Prompt Tuning (w/o transfer)
Progressive Prompt (w/ transfer)
Task
Relative Improv.
Tunable Target Prampt Inout Text (100 takens)
N-shot
Source
Target
Progressive Prompt
imdb +sst2
93.6#25
95.8+ 00.
+2.4%
sst2 + imdb
93.1
93.3
+0.2%
Tunable
Frozen
amazon + yelp
61.2+05
63.5
+3.8%
Target
All
Prompt
Prompt
yelp - amazon
58,2#00
59.0
+1.4%
(50 tokans)
(50 tokans)
mrpc qqp
91.12a0
90.920.4
-0.2%
qqp - mrpe
84.6#0.
8..0
+0.5%
Figure 3: Transfer learning experimen- tal setup. Prompt Tuning: single prompt of 100 tokens is trained on target task. Progressive Prompt: two prompts of 50 tokens are trained sequentially on source and target tasks.
Average 80.3
81.2
+1.3%
imdb >stt2
85.0+160
91.4
+7.5%
sst2 - imdh
93.4-10.
93.2.1
-0.2%
amazon + yelp
50.2#5.0
52.5
+4.6%
20/class
yelp + amazon
37.8
54.9
+45.2%
mrpc - qΩР
88.621.3
89.4
+0.9%
Forward dataset
qqp -> mrpc
84.8+00
85.2-104
+0.5%
Average 73.3
77.8
+9.8%
133
Prompt
imdb +st2
58,2417
84.4
+45.0%
sst2 - imdb
61.1
91.2#:25
+49.3%
70
amazon > yelp
26.2438
27.3
+4.2%
5/class
yclp + amazon
24.2:111
29.7
+22.7%
mrpc qqp
88.4+25
92.0.1.3
44.1%
qqp -> mrpe
86.9+06
84.3+40.
-3.0%
Average 57.5
68.2
+20.4%
Table 3: Transfer learning results on pairs of six tasks from similar domains (IMDb & SST2. Amazon & Yelp. MRPC & QQP). All results are averaged over 3 runs. We report average between Fl and accuracy for MRPC and QQP, and curacy for all other tasks.
Figure 4: Progressive Prompts achieve forward transfer and outperform Prompt Tuning under different dataset sizes. Average scores across six target tasks are reported.
GLUE + SuperGLUE transfer We perform additional experiments on SuperGLUE benchmark to compare the forward transfer effect of Progressive Prompts with standard prompt tuning. In the original prompt tuning paper, separate prompt Pr is trained for each SuperGL.E task T (Lester ctt al. 202I). ). Pr a trainable matrix of 100 x dimension, where 100 is the prompt length and e the embedding dimension. Pr is concatenated with the input text and then fed into the model as [Pr: For Progressive Prompt we perform training in two steps. First, we learn a progressive prompt GLUE benchmark POLUE- POLUE is composed of 6 per-task GLUE prompts with length of 10 on random task order, and the final dimension of PaLUE is 60 x e. After that, we learn length-40 prompt (Prn, for every SuperGLUE task T and concatenate it with the GLUE
8

---

Published as a conference paper at ICLR 2023
prompt before appending to the input z:: [Piru POLUB: x] (illustration of the experimental setup is in Appendix B.2)
Following Lester et al. (2021) we train separate soft prompt for each SuperGLUE task on its full train set with fixed number of epochs, and report the validation set performance. Similarly to [Lsster al. (202T), for each SuperGLUE task use metries recommended by Wang et al. 2019) and for tasks with two metrics we compute an average of the two. Our results are shown in Table 4 Progres- sive Prompt improves the standard prompt tuning performance by 2.7 points on SuperGI.UE bench- mark. This confirms that progressive prompt con- catenation allows knowledge reuse from the previ- ous prompts that were learned on GLUE benchmark.
Method
SuperGLUE avg.
Per-task PT
74.5#22.
ProgPrompt
77.2#1.9
Per-task FT
81.3++0.66
Table 4: Performance of the Prompt Tuning approach vs Progressive Prompts on Super- GLUE. Averaged validation set score across all tasks is shown.
RELATED WORK
Continual Learning Existing continual learning approaches can be broadly organized into three main categories: (i) replay-based, (ii) regularization--ussed, and (iii) architecture-based (de Mas- son D' Autume et al., 2019; Huang et al.] (02T). Replay-based methods store a subset of data from previous tasks for future rehearsal via experience replay, representation consolidation con- strained optimization. The data can be either stored directly or synthesized by generative mod- els. Regularizationnbbasee approaches restrict changes of model's parameters avoid inference with previously learned tasks (Li & Hoiem] 2017] Kirkpatrick et al., 2017). Architecture-based approaches learn different set of parameters dedicated for a separate task. Recently, replay- -based and regularization-bur basdd approaches have been successfully applied for continual learning in lan- guage models. While replay-basee approaches have shown strong results for classification, question answering and relation extraction tasks, they pose significant memory requirements due to storing large number of samples for rehearsal (Rolnick et al. 2019 Huang et al., (021]. Additionally, for many applications such storage would not be feasible due to privacy settings, when access to past data is not available. Regularization- bbaed approaches are more memory-efficied than replay-based approaches, but suffer from catastrophic forgetting and are often not suitable for long task sequences (Kirkpatrick et al.] [ Razdaibiedina et al., 2022). In contrast to regularization based and replay- based approaches, architectural CL. approaches are more efficient in resolving catastrophic forgetting and, hence, are suitable for long sequences of tasks.
Parameter-effiiient Learning Recent works parameter-efficient learning have shown that by training a subset of parameters, we can achieve a full model performance (Houlsby et al., 2019 Li & Liang 2021 Karimi Mahabadi al. 2021 Lester et al.) 2021). While this line of work has mostly focused on learning single task, there have been some attempts on using parameter-efficient
tuning for CL (Madotto et al. 2220 Qin & Joty [021] Wang al. 2022b a). For instance. MMdotto
et al. (2020) proposes AdapterCL which learns a separate adapter block for each task, and QQn &
Joty (2021) proposes LFPT5 that learns a large (length 300) soft prompt that is continuously trained
on all tasks. While AdapterCL. resolves catastrophic forgetting problem, it does not allow forward transfer. In contrast, LFPTS allows forward transfer but suffers from forgetting.
7 CONCLUSION
This paper presents Progressive Prompts - novel approach for CL that addresses catastrophic forgetting in pre-trained language models, while allowing knowledge reuse from previous tasks. In contrast to many existing CL methods for NLP, the proposed approach does not require saving examples from the previous tasks for data replay. Moreover, our method does not require storing a large number of task-specific parameters. Progressive Prompts is a model-agnostic approach and experiments with two commonly used language models demonstrate that Progressive Prompts outperforms baseline methods on a standard CL benchmark for text classification and our custom benchmark of longer CL. sequences that spans 15 tasks.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank Jimmy Ba for many fruitful discussions and brainstorming of the ideas, which helped to improve this project. We also thank Victoria Lin and the FAIR team for their helpful comments. Finally, we thank the anonymous reviewers whose feedback helped to improve this paper.
REFERENCES
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. arXiv preprint arxiv\/8/2000420, 2018.
Cyprien de Masson D' D'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. Advances in Neural Information Processing Systems, 32, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Dan Hendryeks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve ciat----disributioon robustness. arXiv preprint arXiv:2004006/00, 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficiee transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. Continual learning for text classification with information disentanglement based regularization. arXiv preprint arXiv:2104.05448 05489.0..499.... 2021.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances Neural Information Processing Systems, 34:1022- 1035, 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barnkka--aaiiiaaa..aama al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):32113356 2017.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.0869) 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 2021.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40 40(12))9935-2947, 2017.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint 10.06221 2021.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguisties: Human language technologies, pp. 142-150, 2011.
10

---

Published as a conference paper at ICLR 2023
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, and Zhiguang Wang. Continual learning in task-oriented dialogue systems. arXiv preprint arXhv:2012.15500 2020.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and molivation, volume 24, pp. 109-165. Elsevier, 1989.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances neural information processing systems, 32, 2019.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247 2020.
Chengwei Qin and Shafiq Joty. Lfpt5: unified framework for lifelong few-shot language learning based prompt tuning of t5. arXiv preprint arXi::2110.0729 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21 (140):1-67, 2020.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. Continual unsupervised representation learning. Advances in Neural Information Processing Sys- fems, 32, 2019.
Roger Ratcliff.
Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2)2285, 1990.
Anastasia Razdaibiedina, Vivek Madan, Zohar Karnin, Ashish Khetan, and Vishaal Kapoor. Im- proving language models fine-tuning with representation consistency targets. arXiv preprint arXiv:2205. /603, 2022.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016.
Fan-Keng Sun, Cheng- Hao Ho, and Hung- Yi Lee. Lamol: Language modeling for lifelong language learning. arXiv preprint 19Xiv:1909.033.0 00009, 2019.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer. arXiv preprint 1xXiv:2/10.07904 2021.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 074,0,,, 2018
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, pp. 631-648. Springer, 2022a.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning prompt for continual learning. In Pro- ceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 149, 2022b.
11

---

Published as a conference paper at ICLR 2023
Zirui Wang, Sanket Vaibhav Mehta, Barnabás Póczos, and Jaime Carbonell. Efficient meta lifelong- learning with limited memory. arXiv preprint arXiv:2010.02500, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Opt: Open pre-trained transformer language models. arXiv preprint
2022.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting few- sample bert fine-tuning, arXiv preprint arXiv:2006.05987, 2020.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
12

---

Published as a conference paper at ICLR 2023
Appendix
A FURTHER IMPLEMENTATION DETAILS
A.1 DATASETS
Table 5 shows details of the datasets we used for our CL. experiments, along with
their evaluation metrics. Overall, used datasets from CL benchmark (Zhang et al. 2015), GLUE (Wang et al. 2018) and SuperGLUE (Wang et al. 2019) benchmarks, and added IMDB movie reviews dataset. Following common practive, for tasks that have
two evaluation metrics we use the average of the two as the final performance metric.
Dataset name
Category
Task
Domain
Metric
Yelp
CL benchmark
sentiment analysis
Yelp reviews
accuracy
2. Amazon
CL benchmark
sentiment analysis
Amazon reviews
accuracy
DBpedia
CL benchmark
topic classification
Wikipedia
accuracy
4. Yahoo
CL benchmark
QA
Yahoo Q&A
accuracy
AG News
CL benchmark
topic classification
news
accuracy
6. MNLI
GLUE
NLI
various
accuracy
QQP
GLUE
paraphrase detection
Quora
accuracy & F1
8. RTE
GLUE
NLI
news, Wikipedia
accuracy
9. SST2
GLUE
sentiment analysis
movie reviews
accuracy
10. WIC
SuperGLUE
word sense disambiguation
lexical databases
accuracy
11. CB
SuperGLUE
NLI
various
accuracy
12. COPA
SuperGLUE
QA
blogs, encyclopedia
accuracy
13. BoolQ
SuperGI.UR
boolean QA
Wikipedia
accuracy F1 & EM accuracy
14. MultiRC
SuperGLUE
QA sentiment analysis
various
15. IMDB
Other
movie reviews
Table 5: The details of datasets used in our CL experiments. NLI denotes natural language inference, QA denotes questions and answers task, EM denotes exact match scoring. First five tasks correspond to the standard CL benchmark, all other tasks are used in our long-sequener experiments.
A.2 TASK SEQUENCE ORDERS
We report task orders used for our CL experiments across BERT and T5 models in Table 6 below:
Order
Model
Task Sequence
T5
db amazon yahoo ag db amazon ag + yahoo yahoo amazon ag db
2
T5
3
4
BERT
ag yelp amazon yahoo + db yelp yahoo amazon db ag
5
BERT
6
BERT
db yahoo ag amazon yelp yelp ag db amazon yahoo
7
BERT
mnli cb wie copa qqp boolq rte imdb yelp amazon sst2 dbpedia ag multirc yahoo multire boolq wic mnli cb copa qqp rte imdb sst2 dbpedia ag yelp amazon yahoo yelp amazon mnli eb copa qqp rte imdb sst2 dbpedia ag yahoo multire boolq wic
8
T5, BERT
9
T5, BERT
10
T5, BERT
Table 6: Ten different orders of task sequences used for continual learning experiments. Orders 1-7 correspond to the standard CL becnhmark adopted by prior works. Orders 8-10 are our custom long-sequence orders spanning 15 tasks.
13

---

Published as a conference paper at ICLR 2023
A.3 IMPLEMENTATION
We use py Tyrch Paszke et al. 2019) and HuggingFace Transformers library Wolff et al. 2019) for our implementation. For the standard CL benchmark, we use official datasets provided by Zhang et al. (2015) available at /ttp:::/ggo..1///// eet al. (2019); Zhang et al. (2015). We use Hugging datasets (https: /tttps::/gitee //tgtttbbbb..///gg/// com/ huugiingfcee/ dat asets) to download data for GLUE tasks (Wang et al. 2018), SuperGLUE tasks (Wang et al. 2019) tasks, and IMDB movic reviews dataset (Maas et al., 2011). which we use for long-scquence CL experiments and/or ablation studies. Following previous studies (Rao et al, 2019 de Masson D' Autume et al., 2019), for CL experiments, for each dataset we use the available validation set test set (since test data is not available), and hold out 500 samples from the train set to construct the validation set. For our ablation studies, since we compare Progressive Prompts with the original prompt tuning (Lester et al., 202T), we follow their set up and report maximal validation set performance.
following de Masson D' Autume
A.4 EXPERIMENT DETAILS
We use Adam optimizer (Kingma & Ba 2014) and set batch size to 8 for all the experiments, except for MTL runs with a batch size (due to memory limitations). We train each prompt between 10 and 300 epochs. depending
the number of data points. We the prompt checkpoints with the best validation set score as our final prompts. Prompts are initialized from randomly sampled tokens as in Lester et al. (202T), hyperparametes shown in the Table beloo:
Hyperparameter Num. samples +
CL benchmark
Long-sequence benchmark
1000
200
20
BERT
Epochs
40
40
150
300
Learning rate
le 4
le -
4
le -
le- 1 20
4
Prompt length
20
20
20
T5
Epochs
10
10
150
300
Learning rate
0.3
0.3
0.3
0.3
Prompt length
50
10
10
10
Table 7: Hyperparameters used for Progressive Prompts across different CL experiments.
early stopping in Huang al. [2021) save model checkpoint based on the best validation performance on the current task. We report test set performance after training on all tasks as our final metric. For SuperGLUE experiments. we report maximal validation set performance over the course of training as in Lester et al. (2021). We measure the validation performance after every epoch and use metrics described in Appendix A. We use 1 1% of samples per class for the replay approach (but no less than 1 sample per class), following We use the same hyperparameter setting for all prompt- based approaches (Progressive Prompts, prompt tuning, per-task prompts), except for prompt tuning we use longer shared prompt of 200 tokens. For all other approaches, we use hyperparameters provided in their corresponding papers.
For all CL experiments
et al. (2021)
B FURTHER EXPERIMENTAL RESULTS
B.1 LONG SEQUENCE EXPERIMENTS
Here we report results for orders 8, 9 and 10 of continual learning experiments with long task sequences of 15 tasks. We show test set perfor- mance averaged across all tasks for each method. Test scores are calculated after training has been completed. Results are shown Table 9 We also investigate improvement of Progressive Prompts compared to per-task prompt on the corresponding task in Figure 12 Clearly, some tasks benefit from knowledge sharing from the progressively added prompts. Additionally, assessed if initializing
Method
Few-shot
Full-shot
PT Prev. Init. ProgPrompt
48.2
50.0
53.5
69.3
Table 8: Comparison of Prompt Tuning and Progressive Prompts, when shared promot in Prompt Tuning is initialized from the previous task. Average test accuracy after observing all tasks is shown (averaged across three orders).
14

---

Published as a conference paper at ICLR 2023
new prompt from the previous task prompt would result in better performance of continual prompt tuning. We observe that Progressive Prompts outperform this setup both few-shot (20/class) and full-shot settings, Table 8
In addition average performance, compute more fine-grained performance metries de- fined by Lopez-Paz & Ranzato (2017) for different approaches under long-sequence experiments.
Specifically, we compute backward transfer and forward transfer metrics, and evolution of average accuracy over learning new tasks. Our results on FWT for task orders 8, and 10 are shown in Figure 5 Figure 6 and Figure respectively. Our results on BWT for task orders 8, 9 and 10 Figure and Figure T0 respectively. Evolution of accuracies is shown in
are shown in Figure
Figure
Method
Order 8
Order
Order
avg 200
Num. samples
20
200
1000
20
200
1000
20
200
1000
20
1000
La-Lerge results
Finetune
9.3
8.9
7.4
9.5
8.1
7.4
10.4
7.9
7.5
9.7
8.3
7.4
Replay
46.0
45.0
55.2
50.3
43.7
54.8
34.6
43.8
53.3
43.6
44.2
54.4
PromptTuning
8.4
8.2
24.4
16.8
8.7
12.2
8.0
7.9
17.4
13.9
10.9
Per-task Prompts
69.9
75.2
77.0
69.9
75.2
77.0
69.9
75.2
77.0
69.8
75.2
77.0
LFPTS
54.7
61.6
70.4
54.1
54.3
68.2
54.2
58.8
69.1
54.3
58.2
69.2
ProgPrompt
75.4
79.1
79.5
78.2
79.1
76.7
78.9
79.8
76.2
78.7
MTL
70.7
72.5
76.3
70.7
72.5
76.3
70.7
72.5
76.3
70.7
72.5
BERT base results
Finetune
29.9
43.4
40.9
30.5
42.0
42.5
33.6
41.9
41.8
31.3
42.4
41.7
Replay
46.3
51.0
48.1
51.5
34.9
46.5
46.3
36.4
47.0
49.6
Per-task Prompts
50.6
62.4
67.2
50.6
62.4
67.2
50.6
62.4
67.2
50.6
62.4
67.2
IDBR
39.7
48.4
52.3
37.9
46.6
54.1
32.9
48.8
50.1
36.8
47.9
52.2
ProgPrompt
67.9
68.9
65.8
70.0
51.9
66.9
69.0
53.5
66.9
MTL.
56.9
67.7
69.9
56.9
67.7
69.9
56.9
67.7
69.9
56.9
67.7
69.9
Table Average test set performance of Progressive Prompts and common CL approaches on long- sequence experiments with 15 text classicication tasks (orders 8, 9 and 10). We report results for BERT and T5 models across different limits of data 20, 200 and 1000 samples per class. MTL denotes multi-task learning. All results averaged over runs.
B.2 SUPERGLUE EXPERIMENTS SETUP
Comparison of the original prompt tuning SuperGI.UF. (Lester al. 2021) and Progressive Prompt setup is shown in Figure
B.3 EFFECT OF PROMPT REPARAMETERIZATION
We find that residual reparameterization allows to reach performance close to finetuning, and is especially helpful for BERT model. Table 10 shows the results of regular prompt tuming, prompt tuning with MI. reparameterivation and prompt tuning with residual MLP reparameterization on BERT-base model. As in all our experiments,
2-layer MLP with the hidden layer dimension of 800. We show the best performance on four different tasks with prompts of length 5 and 30. Following al. (22II) report the maximal validation set performance for each dataset. Our results show that residual MLP reparameterization results in performance improvement over standard prompt tuning, reaching close to finetuning performance (Table IO). Notably, with length- 5 prompt, residual MLP improves accuracy by approximately 6% and 4% for IMDB and QQP datasets, matching full model tuning. Regular MLP reparameterization generally leads to either smaller improvement than residual MLP or even worse performance than prompt tuning.
15

---

Published as a conference paper at ICLR 2023
Order 120 sumplec cluss
20.300
C30011
31.003
$300.0
Orde 2200 sampley dees!
00:300
030010
633030
$300.00
Hacred
Pronut Snvg
Taa
Tec
Figure 5: Forward transfer score of different approaches on order 8. Different data limits are shown (20, 200 and 1000 samples per class).
16

---

Published as a conference paper at ICLR 2023
Turing
Poonett
Taring
dala Phongg
130033
130013
33A023
330010
330230
Hethed Nornot
Figure 6: Forward transfer score of different approaches order 9. Different data limits shown (20, 200 and 1000 samples per class).
17

---

Published as a conference paper at ICLR 2023
Order 10 10 wample cono)
Tedayy
Pogihoeet
20.0
1000.00
300000
$3.6
Tak
Rok
Tolk
Jecrip
Order camales: clesst
Tal
W6-33
Jernge
Order 11000 somples clees)
T148
$12.4
7a0h
900-22
7eb
Jesiae
Figure 7: Forward transfer score of different approaches on order 10. Different data limits are shown (20, 200 and 1000 samples per class).
18

---

Published as a conference paper at ICLR 2023
Pratet lanis
Prightent
sumplra
Ficta's
LeTT
loc Peorge
BB
B6.22
AVROO
Doler C1RE Largles cleed
dass Phonge
Aceage
Figure Backward transfer score of different approaches on order 8. Different data limits are shown (20, 200 and 1000 samples per class).
19

---

Published as a conference paper at ICLR 2023
9.41
BEGACS
Orter 10000 cempl
Tea
Searge
Figure Backward transfer score of different approaches on order 9. Different data limits shown (20, 200 and 1000 samples per class).
20

---

Published as a conference paper at ICLR 2023
Testun
100
60
KEC
0000AR
Orter 1O SATc WY Clavi
Fromial
3al
1100 camples
FC. 035044
PlccPic lgt
Figure 10: Backward transfer score different approaches on order 10. Different data limits are shown (20, 200 and 1000 samples per class).
Figure 11: Evolution of average accuracy after learning new tasks.
21

---

Published as a conference paper at ICLR 2023
Progressive Prompt vs Per-task prompts pperformance improvement on order B)
150
20 samples
100
Percentage improvement
samples
20
samples
multre #1
Wic #3
C3 25
000 86
116 88
592 #30
dbpedia #11
ag. DOWS #12
yel #13
Pa000 $35
mali
090
mndb
Figure 12: Per-task improvement of Progressive Prompts verus per-task prompts in CL experiment
with order across different data limits (20, 200 and 1000 samples per class). X-axis shows the quence of tasks, Y-axis shows percentage improvement of Progressive Prompts test score compared to per-task prompt on the corresponding task.
GLUE prefix 10 tokens x tasks
SuperGLUE ProgPrompt 40 tokens
CoLA
WIC
Lester's SuperGLUE prompt 100 tokens
Figure 13: Original prompt tuning versus Progressive Prompts on SuperGLUE datasets, For illus- tration, we show how SuperGLUE task WiC is leaned (we have similar scheme for other tasks), Prompt tuning trains single prompt of 100 tokens for WiC task. Progressive Prompts method leams a prompt of 40 tokens, which is progressively appended to the six frozen prompts of 10 to- kens learned on GLUE benchmark (with random task order). Total prompt length equal both approaches.
22

---

Published as a conference paper at ICLR 2023
Prompt len.
length
length 30
Task 4
PT
PT+MLP
PT+tesMPP
PT
PT+MLP
PT+resMLP
FT
IMDB
85.8o.7
83.4p.9
91.40.8
89.11.
90.21.5
91.21.4
92.90.4
QQP
72.9g.8
73.11.1
76.6 76.61.2
72.21.5
73.94.1
77. 77.30.9
78. 78.60.9
RTE
65. 15.80..
64.83.2
65.0g.s
67.8s.2
67
67.63.6
66.2a.1
66 26.0
MRPC
72.1o.8
71.51.4
75.80.9
73.40.8
76.93.4
79.01.3
86.31.6
Table 10: Effect of prompt embeddings reparametrization on prompt tuning performance with BERT model. Average performance across 3 runs is shown. PT: regular prompt tuning. PT+MLP: prompt tuning with prompt passed through 2-layer MLP, PT+resMLP (our approach): prompt tuning with prompt passed through 2-layer MLP with a skip connection (residual MLP). FT: full model finetun- ing. Best results for each prompt length (5 and 30) are highlighted in bold.
23


Reviews:
[{"review_id": "yujTj3p-tpH", "reviewer": "Reviewer_vg4X", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose the progressive prompts that add a new prompt for each task and fix all other parameters from both feature extractor e.g. BERT, T5 and the prompt from previous tasks.\n\nThe new method should be able to both transfer knowledge from the previous task to new task, as well as reduce the forgetting as the prompt for previous tasks are unchanged.", "summary_of_the_review": "The recommendation would be accept but marginally above the threshold. My reason for not a higher score results from the somewhat incremental on the technical novelty. My reason for not a lower level results from large gap on the performance compared to other method.", "strength_and_weaknesses": "Strength:\n1) The writing seems very good that makes paper easy to follow.\n2) The forgetting can be alleviated as the prompt for previous tasks are unchanged.\n3) Transfer of knowledge from the previous task to new task is achieved by the new approach.\n4) A lot better results on the experiments.\n\nQuestion:\n1) Should we compare the AdapterCL or other adapter model that could add a new part / parameters for each task, for the empirical results?\n2) For the eval results in Section 5.1 and 5.2, the progressive prompt on T5 has much larger gap than the other base lines. The BERT model with progressive prompt has not as large enhancement. Are there some root cause? Let me know if my understanding are not good.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "N5cBW-E0v5b", "reviewer": "Reviewer_KFhT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies continual learning in the NLP domain, focusing on pre-trained language models with transformers. The authors use various techniques to improve the continual learning performance, such as prompt tuning and embedding regularization. The proposed method, Progressive Prompt (ProgPrompt), learns a prompt for each new task. In addition, the base language model is kept frozen while a small number of parameters is added for each task. Since prompts and per-task expanded parameters are separate, this allows for reducing catastrophic forgetting. The paper then empirically shows the effectiveness of ProgPrompt for BERT and T5 models on various NLP benchmarks.", "summary_of_the_review": "**Initial**  \nOverall, while I find the proposed method intuitive with strong results, I believe many details are missing from the paper. In addition, I find the assumption of accessing task-identifiers very relaxed. While I mentioned from the novelty perspective, the work heavily relies on previous ideas, I do not find this a major limitation.\n\n**Update**  \nI would like to thank the authors for their response. While the proposed method still has a limitation regarding the task-boundary assumption, given the clarifications and new results, I increased the score, and I encourage the authors to add the full result to the paper. \n\n\n", "strength_and_weaknesses": "**Strengths** \n\n\n- The paper is well-written and easy to follow. The idea of prompt tuning is also intuitive.\n\n\n- The proposed method shows strong performance.\n\n\n\n\n**Weaknesses**  \n\n\n- One major of the work is the assumption of accessing task identifiers (for separating prompts and task parameters). I believe this assumption (similar to the majority of task-incremental works), is very relaxed for practical scenarios. For example, task boundaries and identifiers do not always exist in real-world streaming setups.\n\n\n\n- In addition, although very small, the computational costs grow as more and more tasks arrive. This is also the case for classical expansion-based CL algorithms and, at some point (e.g., hundreds of tasks), not negligible. \n\n\n\n- The main focus of the paper is on the final average performance (e.g., accuracy, F1) of the models/algorithms. However, it is important to compare continual learning methods from different perspectives that measure the evolution of the accuracies and forward and backward transfers.  \n\n\n\n\n**Questions** \n\n\n- Could the authors provide a more fine-grained analysis of the methods on at least one benchmark (e.g., with many tasks)? For instance, how the average accuracy evolves as each task is learned, how much forgetting (if any) we have, etc. ?\n\n\n- Does ProgPrompt require task identifiers at the test time?\n\n\n- Could the author provide more fine-grained performance measures \n\n\n- As explained in Section 3, ProgPrompt allocates different prompts and parameters for each task. Could the authors provide the total parameters/time/computation comparison of ProgPrompt vs. other methods for the experiments with a large number of tasks?\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HPXMwrkJ8ES", "reviewer": "Reviewer_Dpx6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new prompting based method for continual learning for language models. The method learns a sequnce of task specific prompts which allows it to overcome the forgetting problem for long sequences. The methods achieve state of the art performance on long range continual learning tasks.", "summary_of_the_review": "The paper proposes a new method for continual learning which is intereting. I believe some baseline comparisons are actually missing from their experiment setup which have been pointed out in the weakness section. ", "strength_and_weaknesses": "Strength\nThe method proposed in the paper is an efficient way to continually learn language tasks, requiring to train a very small number of parameters as compared to the overall model sizes (< 0.1%). The method is quite simple and intuitive, and is explained quite well in the paper. \n\nWeakness\nThere are a couple of questions that the paper raises:\n- Section 2.3 mentions the framework that the identity of each task is known a priori. Instead of concatenating the prompt vectors over time, how would the following baseline perform: the prompt has only one theta_p. update \\theta_p for each task starting from the previous one as initialization? If this alrternative is nearly as good, this might save a lot fof time at inference.\n- Another alternative is to directly train a prompt of size m \\times |\\theta_p| instead of one sequential vector per task. How would this perform as a baseline?\n- For the results in Section 5, can you also provide numbers for per task fine tuning as a baseline? This would help calibrate how much transfer can you actually get from other tasks.\n- In section 5.4, the training procedure to obtain the numbers in Table 4 is modified from the one described before. Why is this the case?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "37A_nI003F", "reviewer": "Reviewer_KEao", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper concerns language models in the pretrain-then-prompt-tune paradigm, and aims to introduce a prompting method to allow for  solving multiple tasks.  This can be described a continual learning setup, where the task ID is known to the model at both training and inference time.  The method, called Progressive Prompts, concatenates all prompts learned for previous tasks and learns a new task-specific prompt for each new task; thus the full prompt for task $N$ is the concatenation of the (previously-learned) prompts for tasks $1$ through $N-1$, concatenated with the newly-learned prompt for task $N$.  \n\nEvaluating their method on text classification tasks, the authors find favorable performance compared to their chosen set of baselines, which include both methods that fine-tune the full language model (with or without data replay/buffer) and other purely prompt-based approaches. \n\nAn additional finding of the paper is that parameterization of the prompt matters for performance; specifically, the authors parameterize the  (soft) prompt via a residual MLP and find via ablations that this parameterization improves performance.", "summary_of_the_review": "I do not currently recommend the paper for publishing because its observations are presented as a solution to catastrophic forgetting, which does not seem to be a problem in the prompt-engineering paradigm.  \n\n---\n\nUpdate post rebuttal: The authors have sufficiently addressed my concerns, and I have updated my score to an accept.", "strength_and_weaknesses": "Strengths:\n- The paper is well written and generally easy to follow.\n- The observation that parameterization of the prompt matters so much for performance is one that I had not appreciated before.\n- The novel observation in this paper, that Progressive Prompting enables forward transfer is interesting.\n\nWeaknesses:\nFor me, the main weakness of this paper---which prevents me from recommending acceptance---is that the connection to catastrophic forgetting (and thus, the motivation for many of the baselines reported in the paper) does not make sense.  Let me state this objection/confusion as clearly as possible so the authors may let me know if I have misunderstood, in which case I would be happy to amend my score.  As I see it, the problem of catastrophic forgetting in language models is a problem that only occurs in a fine-tuning setting.  In this work, the authors have specified the (common in continual-learning) setup in which the model knows the identity of the task both during training and during inference.  Since prompt tuning (either soft prompt-tuning or manual natural-language prompt tuning) has  now become standard practice for adapting pre-trained language models to a particular task, the standard thing to do for adapting a pretrained language model for multiple tasks is to learn a separate prompt for each of the tasks, and at inference time (since the task identity is assumed to be known) simply select the prompt corresponding to the given task and run the model using that prompt.  In such a setting, there is no forgetting since the model parameters themselves do not change.  Thus in some sense it seems like this paper is attempting to solve a problem which already does not exist in the prompt-engineering paradigm.  The baselines which involve retraining a full model (e.g. the baselines \"finetune\", \"EWC\", \"A-GEM\", \"Experience replay\", \"MBPA++\", \"IDBR\") don't seem relevant to me.\n\nHaving said the above, the \"standard thing\" that I mentioned is one of the benchmarks that the authors consider (Per-task prompts).  In my mind, the fact that the ProgressivePompts beats Per-task prompts is *the* interesting contribution of this paper, and (as the authors say) indicates some forward transfer between prompts.  This could be useful in many scenarios, especially in the low-data regime for subsequent tasks.  If the paper had been written with this as the main selling point (with all of the natural ablations and sanity-checks one would want), I would likely have been much more positive on it.\n\nI think there are two ways in which I would be convinced to change my score:\n1. The authors explain why my above argument is wrong, i.e. I have some misunderstanding and the problem of catastrophic forgetting is actually relevant in the prompt-engineering world; or, \n2. The authors reframe their paper as being about the forward transfer phenomenon, and conduct more thorough experiments to explore that phenomenon (e.g., how does the transfer vary with dataset size for the subsequent tasks, how does it compare to a single prompt of length 2M (where M is the length of each individual prompt), etc.).  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster