Task-Attentive Transformer Architecture for Continual Learning of Vision-and-Language Tasks Using Knowledge Distillation
Authors: Yuliang Cai
Jesse Thomason
Mohammad Rostami
Venue: EMNLP 2023 Findings (2023)
Keywords: Multimodal Learning
Continual Learning
Catastrophic forgetting

Abstract: The size and the computational load of fine-tuning large-scale pre-trained neural network are becoming two major obstacles in adopting machine learning in many applications. Continual learning (CL) can serve as a remedy through enabling knowledge-transfer across sequentially arriving tasks which relaxes the need to fine-tune all network weights from scratch. However, existing CL algorithms primarily consider learning unimodal vision-only or language-only tasks. We develop a transformer-based CL architecture for learning bimodal vision-and-language tasks based on increasing the number of the learnable parameters dynamically and using knowledge distillation. The new additional parameters are used to specialize the network for each task. Our approach enables sharing information between the tasks while addressing the challenge of catastrophic forgetting. Our approach is scalable learning to a large number of tasks because it requires little memory and time overhead. Our model reaches state-of-the-art performance on challenging vision-and-language tasks.

Task Transformer Architecture for Continual Learning of Vision-and-Language Tasks Using Knowledge Distillation
Yuliang Cai University of Southern California caiyulia@usc. edu
Jesse Thomason
University of Southern California
jessetho@usc. edu
Mohammad Rostami
University of Southern California
rostamim@usc.ed
Abstract
regularize fixed shared model learn each task through different information pathways, i.e., weights (Kirkpatrick et al., 2017; Aljundi et al., 2018). The core idea is to identify a subset of model parameters that are important to encode the learned knowledge about each task and then consolidate these parameters when updating the model to learn new tasks. A second approach is based model expansion (Rusu et al., 2016; Yoon al.). The idea to expand a base model via a small number of additional weights and specialize the network to learn new tasks through these weights. Finally, a group of algorithms use pseudo-rehearsal through experience replay (Rolnick et al., 2019; Mirtaheri et al., 2023). The idea is to store representative subset of training data for each task in mem- ory buffer, and replay them back along with the current task's data to maintain the encoded knowl- edge about the past task. Some methods relax the need for a memory buffer by enabling the model to generate seeudo-samples for the past learned tasks that are used to implement experience replay (Shin et al., 2017; Rostami and Galstyan, 2023). De- spite being effective, existing CL methods consider unimodal tasks, e.g., vision-only (Lin al., 2021; Douillard al., 2022) language-only tasks (Jin et al., 2021; 1; Yang et al., 2022), without considering the unique challenges of multimodal tasks, making them inapplicable on VaL tasks.
The size and computational load of fine- tuning large-scale pre-trained neural networks are becoming two major obstacles in adopting machine learning in many applications. Con- tinual learning (CL) can serve remedy through enabling knowledge-trane -rannsffrr across se- quentially arriving tasks. However, existing CL algorithms primarily consider learning uni- modal vision-only or language-only tasks. We develop transformer-based CL architecture for learning multimodal vision-and-languae (Val.) tasks based on dynamic model expan- sion and knowledge distillation. Additional parameters are used to specialize the network for each task. Our approach, Task Atten- tive Multimodal Continual Learning (TAM- CL), enables sharing information between the tasks while addressing catastrophic forgetting. Our approach is scalable. requiring little mem- ory and time overhead. TAM-CI. reaches challenging multi-
SOTA performance modal tasks. The code is publicly available on https://gthub..oom/ Yuliiaaaaaaaa22/TTTMM CL.git.
1 Introduction
Large-scale pre-trained transformer models are ap- plied in wide range of applications across modali- ties, including vision-and-language tasks (Dosovit- skiy al.; Kim et al., 2021; Xu al., 2023). These models are usually pretrained on large dataset and then fine-tuned to generalize on a downstream task. Such task-level fine-tuning compromises the model generalizability and necessitates storing a copy of the base model for each task. Continual learning (CL) algorithms (Jin et al., 2021; Yang et al., 2022; Wang et al., 2022; Pelosin et al., 2022; Ermis et al., 2022; Srinivasan et al., 2023) have explored mit- igating these challenges for transformers through using a shared model that benefits from cross-task knowledge transfer.
We develop a new algorithm for learning vision- and-language (VaL) tasks in a CL setting based on dynamic model expansion. To this end, we leverage the self-attention layers of a base bimodal trans- former as shared encoder across all tasks. We then equip the base model with task-attention layers (Douillard et al., 2022) that help to specialize the model for each task using task-specific token. Our approach requires small memory overhead and limited inference time overhead during testing. It also does not need extensive hyper- hyper-parameter parameters and remains robust when facing an unknown number
Catastrophic forgetting is the primary challenge in CL (French, 1999). A group CL algorithms

---

of tasks. Our specific contributions include:
Task 1
Task 2
Task 3
Task N
A dynamically expanding, efficient trans- former architecture for multimodal CL.
(3)
(2)
(4)
[B]
Memory Buffer
Teacher
A training algorithm to handle diverse, sequen- tially arriving ssioo-and-language tasks.
Replay
Model
Knowledde Disillation
Model
[5]
Extensive experiments to demonstrate that the proposed model achieves SOTA performance.
The proposed CL training procedure: (1) A small portion of the data for previous tasks are randomly selected and stored in memory buffer. (2) The current task arrives with 2 (3) The training data D' is used as input to the teacher model to compute the distillation loss. (4) The memory buffer samples
Figure 1:
2 Background and Related Work
Transformers for Vision and Language Tasks: Multimodal transformers are developed for pro- cessing saioo-and-languaae (VaL) tasks (Su et al., 2019; Tan and Bansal, 2019 Kim et al., 2021; Chochlakis al., 2022: Chen et al., 2020). The core idea is use modality-specific self-attention layers to extract suitable features from each of the vision and language inputs. These features then are integrated at higher layers to extract cross-modal contextualised representations of the multimodal inputs. The idea is that these global vectors can model the interaction between the vision and the language inputs which is helpful to perform Val. tasks well. The idea has also been adopted other modalities, including in the context of video pro- cessing to relate vision and speech inputs (Arnab et al., 2021; Sun et al., 2019). These transformers are usually trained on a large-scale dataset and have been found to be highly effective when fine-tuned downstream tasks. Due to significant perfor- mance improvement, transformers are increasingly replacing older architectures.
replayed along with the current task data to train the main model. (5) After learning the current task, the teacher model of the next task will be copy of the current model.
inter-task attention mechanism to integrate infor- mation from previous tasks and reduces the rate at which important attention weights shift away from old tasks towards the current task. Douillard et al. (Douillard et al., 2022) propose a CL architec- ture for vision tasks using the ViLT model (Kim et al., 2021). Pelosin et al. (Pelosin et al., 2022) extend this work to exemplar-free setting via dis- tilling the attention-level matrices of transformers to enable model plasticity and to mitigate forget- ting effects. Ermis al. (Ermis et al., 2022) use the idea of adapters in a vision context. To the best of our knowledge, no prior work has explored CL for multimodal tasks using transformer architectures.
Problem Description
Consider a set of sequentially arriving VaL tasks (TT) each with the annotated training dataset D' = [((] L.Y ut) 2 where I Hxxx represents the image input, L RL repre- sents the language input, while yf is the text-typed discrete label. The order of these tasks and T are not known a priori. The training data points for Ti assumed be drawn iid from task task-specific joint distribution -..-.))) Our goal in multimodal CL is to learn each task time-step and then move forward to learn the next tasks. The learned tasks can be encountered at any time during testing in the future and hence, we would like to main- tain the performance of previous learned tasks by preventing catastrophic forgetting.
Continual learning for transformers: Despite
the successful adoption of transformers in various benchmarks, fine-tuning naturally compromises their generalizability. Using an independent trans- former per task would lead to significant memory load on disk as the size of transformers are increas- ing dramatically to account for solving more ab- stract tasks. CL seems to be a natural solution for these challenges but works on CL. using transform- ers are limited. Xin et al. (Jin et al., 2021) use adapters in combination with hypernetwork to en- able CL for language tasks. Alternatively, Yang et al. (Yang al., 2022) propose transformer calibra- tion module make transformer adaptive. The calibration module considered be independent from the base pre-trained transformer and helps to specialize it on a downstream task. The Lifelong Vision Transformer (Wang et al., 2022) utilizes an
When learned in isolation, each of these VaL tasks T can be learned using supervised learn- ing conditioned on selecting the suitable predictive model fiy (: e.g., a transformer with trainable

---

Y:
Outpus
Task Attention Block
Norm
Lord
Self Attention Blacks #11
Toak Attention
Layer
Teacher
Self Attention Blocks #1
Layer
Linear Projection
Word Embedding
Embeddlings
The
Sky
Is
Blue
Figure 2: The proposed transformer-bast based architecture: (left) The VaL inputs are converted into two sequences and then fed into self-attention layers to generate fused global feature vector. The data feature vector is then concatenated with the learnable task-specific tokens and then fed into the task attention layer to generate the input for the task-specific classifier heads. The same Val. inputs are also fed into the teacher model's 's transformer architecture compute Knowledge Distillation. (right) The task attention block architecture.
parameters Oas, and the discrimination loss L(-). e.g.. cross-entropy. However, due to the storage limit, we assume a shared model should be used, training separate model per task is impracti- cal. Additionally, only a small portion of training data for each task T; can be stored after training for space-saving purpose, which makes multitask learning (Caruana, 1998) an impractical solution. The single task learning strategy using a shared model is not ideal in CL either. Because when the model is updated to learn the current tasks, its performance the past tasks will degrade due to catastrophic forgetting (Kirkpatrick et al., 2017).
sists of a shared pre-trained replaceable multimodal transformer, an independent task attention block, and MLP classification headers. The task attention block receives task-specific tokens that make the model adaptive. We provide details about these modules and the strategies that we use train it.
4.1 Sequential Feature Extraction Block
We follow the ViLT feature generation proce- dure (Kim et al., 2021). To build a sequence from the input images, we decompose a given image E Hxx xC into patches and then flatten these patches to generate 2D vectors E RNNx(PA.))
Figure visualizes the high-level presentation of the solution that we propose to address multimodal CL. To use shared model across all tasks and benefit from knowledge transfer across all tasks, we consider base transformer model fo(::-) and make it adaptive by adding a unique task at- tention layer after its final layer. Additionally, we modify the loss supervised learning loss by adding a knowledge distillation loss (Hinton et al., 2015) on the intermediate model layers. We consider the teacher model in the knowledge distillation formu- lation to be a copy of f ((,)) when training the main student model on zth task T. Additionally, we rely on pseudo-rehearsal through experience replay (Rolnick al., 2019) using small memory buffer to tackle catastrophic forgetting.
Here, C' is the number of channels, P x P is the size of each image patch, and N HW/P² p2 is the number of patches with size P x P. After gener- ating the set of vectors, we apply trainable linear projection, V (R(PPC)) and a position embed- ding, Vpos [ to transform v into the sequential representation E IR Nx/
(1) We also extract word vectors for I R4xIV| after applying word embedding matrix T E RVVH and using position embedding matrix Tros E [R(++))HH, we embed data into E RA*H
class 01 V
UN uN VPoS
ビ= (chsss L 1T; I.T + pos
(2)
We then sum the image and text embeddings with their independent model-type embedding vector ttype and v/upe E "R" and concatenate them to build a single sequence
4 The Proposed Architecture
Figure 2 visualizes our transformer- archi- tecture for multimodal CL. The architecture con-
0 = [2 + +U; type + +
(3)

---

The combined vector is passed through D classic self-attention layers of the base VaL transformer, i.e., ViLT, and then are fed into the task attention layer, with the output embedding from the trans- former attention layers denoted as s and output from the task attention layer as s D+1
Finally, the output of the task attention block. D+1 is then fed into task speccfic classifier layers:
yi = (of 1....,
5 Training Algorithm
The architecture in Figure 2 visualizes snapshot of model, TAM-CI, at given timestep. Note that the architecture is dynamically expanded as more tasks are learned sequentially. We describe the suitable CL loss functions that we use for training.
8d MSS(LL((' (8t (0)))))+00000)1))))) + 1,... .
8d MLP(LN((")) + L.....
(4)
4.2 Task Attention Block
5.1 Token Expansion
The core idea of our work lies adopting the idea of self-attention on task-level, each time a new task is learned. Different from the vanilla input data level self-attention layer, the misk-attention layer is task-focused attention layer that trainable task token is initialized for each new task, denoted Ti C IR Gx1 for task i C [1, 2,,, T), where G is the size of latent space of each self-attention layer. Similar to the self-attention block (SAB), the task attention block (TAB) is a module which con- sists of an attention layer, layer normalization, and MLP. The attention layer's is instantiated as a task- attention layer, rather than vanilla self-attention. Task attention block takes two inputs, the output of the self-attention blocks s and the task token т, note that the same task token T used for all the in- put instances in that task sask-specific train/test dataset. The two vectors are concatenated to generate an input for task attention:
During the training stage, the transformer's self attention blocks and the task attention block are shared among all the tasks. However, for each of the new tasks, we define a new task token with the same dimension, C G x 1. and initialize a new accumulative task-specific classifier, Clf((), for task i, which the output dimension of Cif((-) is expanded based on CIf,-1()). With more tasks added on, the output dimension of task- specifce classifier i would be accumulating in the way:
E, E + E. 1,1=1111 = L....,,
(8)
Where E, denotes the output dimension for 7th classifier, E is the output dimension for 2th task in its original design. For task 2th task token with the updated path token, sP from the last self-attention block of the transformer, and send it into the task attention block, as described Sec 4.2. At this stage, only the 2th task token and task-specific classifier would be trainable, while all the other task tokens and classifiers remain frozen. During the testing stage, the task number, i, of test data is explicitly given, and s is combined with the 8th learned task token feed into task attention block along with using its correspond- ing task-specific classifier, while all other the task tokens and classifiers remain unused.
combine the
D+1 [T,, s1 | E [[N+1)xG 11-1,... I,..,T
8 ;
8 D+1 TA )), ( I,.,,
D-1 -MP((LN(8P-L)) + 8P-1,
T
(5) The task attention block is placed after the last self-attention block of the transformer. While we can have more than one task attention block, our architecture uses single TAB. The operation of the task attention layer is given as follows:
5.2 Loss and Knowledge Distillation
Our objective function consists of three loss terms: (i) cross-entropy Loss Le, which is the original objective function for each task in single task learn- ing setting. Note it can vary from task to task, (ii) the knowledge distillation (KD) loss, which is computed from S of the main student model and the teacher model, Likd- (iii) the diverse loss Lutiv which compares the data distribution of task tokens and makes them more diverse. The final loss is:
Q: = Wg Ti
K; We x s
D11
V, = W, x $ D11
(6)
A, Softmaz Q KF ///h),
O, W.AV. + b, E RXC
where is the number of attention heads in the transformer (Vaswani et al., 2017).
L = (1 -- + Aa + B
(9)

---

where À is set to " Ta denotes the total num- ber of tasks we have seen so far, denotes con- stant which varies for different tasks, and B = min(C 0.1 x ((1 A) + )
Algorithm 1 TAM-CL Train
INPUT: Model M, MemBuffer B, ReplayFreq for epoch num_epoch do for step. batch in dataloader do
Loss < TrainStep(M. batch) if step % f= then
The application of intermediate knowledge dis- tillation is the core of TAM-CL, which aims to distill the knowledge from the teacher model into the main student model in order constrain the distribution shift and prevent catastrophic forget- ting. our architecture, the teacher model a copy of model fo -( when training on zth task. Different from most other methods which apply knowledge distillation by computing the loss from the last layer output, and Uteocher, we introduce an intermediate knowledge distillation loss, i.... the loss term Likd is computed between the last self-attention block of the transformer and the task attention block, s². Through experiments in Sec- tion 6.2, we find that compared with knowledge distillation computed from the last layer output, eg. Dytox. and Yteacher. such an intermediate knowl- edge distillation is helpful in the architecture of pre-trained model followed by non-pre-trained block, which could constrain the probability shift of pre-trained parameters and leave the rest of the layers flexible enough learn new tasks. To our best knowledge, TAM-CL is the first to introduce intermediate knowledge distillation objective func- tion in multimodal continual learning.
batchreplay < B. BegeBachh
Lossruplay <- TramnStep(M, batchseptoy)
Algorithm 2 TAM-CL TrainStep
INPUT: Model M, Teacher Model T. Batch, Target 1, Token
k
- Model(Batch)
loss + Crass Loss(p. t
ModeISabPre + M.SAB(Batch)
TeacherSabPre <- T.S.Bs(Batch)
lossad + Loosssk ky) 1,..,f f
TeacherSabPre)
lossæio
loss 4- (1 X)loss Aalossas + Blossave
lass. backwacc
Return loss
6.1 Experiment Setup
We use five VaL datasets to generate sequential tasks. We use SNLI-VE (Xie et al., 2019), an image-sentence pairs dataset whereby a premise is defined by an image, rather than a natural lan- guage sentence in traditional Textual Entail- ment tasks, COCOQA (Ren et al., 2015), vi- sual question answering dataset based on Microsoft COCO image dataset, GQA (Hudson and Man- ning, 2019), a compositional question-ansseringg and visual reasoning dataset leverages scene graph structure, NLVR2 (Suhr et al., 2018), a visual reasoning dataset which takes two images and determine the correctness of the given sentence, OKVQA (Marino et al., 2019), knowledge-base visual
5.3 Experience Replay
The above training procedure enables training a shared model across the tasks but still, it need datasets for all tasks which breaks primary as- sumption in CL. To address this issue, we use memory buffer during the training stage at each time-step which stores tiny percentage, e.g.. as 5%, of the training dataset for all the previous tasks. When learning the current task with a spe- cific batch number, the next batch will be randomly selected from the memory buffer to consolidate the parameter distribution on previous tasks. As result, forgetting effects will be mitigated.
dataset whhere the im- age content is not sufficient to answer questions. Due to the computational limits, we trained all the models on part of the whole dataset, where the maximum size of the training examples are 80000. Table provides statistics of these dataset.
We use ViLT model with pre-trained weights, in our experiments. To main-
We introduce the Task Attentive Multimodal Continual Learning (TAM-CL) training procedure, which enables stable and high-performancee while mitigating forgetting effects in Multimodal contin- ual learning in Algorithms 1 and 2.
Name
# Training Examples
# Labels
NLVR2
80000
2
SNLI-VE
80000
3
COCOQA
78736
430
GQA
80000
1842
6 Experimental Results
OKYQA
18032
2910
We evaluate TAM-CL using five VaL tasks.
Table I: Statistics of the Val. dataset.

---

COCOQA + NLVR2 - OKVQA + SNLI-VE GQA
COCOQA
NLVR2
OKVQA
SNLI-VE
GQA
TAM-CL
66.09 (13.15%)
66.07 (14.87%
21.24 (22.59%)
64.05 (19.13%)
50.86
Finetune
40.67 (47.09%)
53.85 (79.42%)
8.26 (74.37%)
53.83 (46.81%) %)
51.92
FDR
48.74 (32.12%)
55.91 (29.89%)
11.59 (55.44%)
59.30 (28.26%)
50.67
EWC
51.44 (33.10%)
60.87 (42.19%)
16.16 (49.40%)
57.93 (35.21%)
49.67
ER
56.30 (27.20%)
62.06 (34.27%)
15.62 (50.10%)
61.12 (27.91%)
50.12
Dytox
60.67 (20.52%)
65.56 (15.37%)
10.41 (25.26%)
62.26 (20.70%)
11.12
Avg.
53.98 (28.63%)
60.72 (36.00%)
13.88 (46.19%)
59.75 (29.67%)
44.06
Table 2: Comparative: the accuracy and forgetting rate for each task in two different task sequences. For each task sequence, each value means the final accuracy of that task after learning the last task, and (forgetting rate %) means the forgetting rate of final accuracy compared with the best accuracy of each task. The last represent the average accuracy and forgetting rate of each task.
OKVQA GQA - COCOQA - SNLI-VE + NLVR2
OKVQA
GQA
COCOQA
SNLI-VE
NLVR2
TAM-CL
15.09 (54.56%)
41.91 (27.59%)
60.86 (18.36%)
60.47 (29.34%)
65.86
ablation TAB
13.97 (57.22%)
39.95 (28.69%)
59.92 (19.65%)
57.35 (37.22%)
65.71
ablation Likd
11.53 (65.36%)
37.19 (34.07%)
58.62 (21.71%)
58.34 (35.30%)
65.21
ablation replay
2.11 (93.69%)
27.25 (51.84%)
41.59 (44.33%)
50.06 (56.40%)
65.64
Table 3: Ablation: the accuracy and forgetting rate for each task in ablation experiments.
tain consistency in our comparison, we use ViLT with the same pre-trained parameters for all the ex- periments. Hence, we have 11 self-attention blocks (SAB), with dimension of 768 and attention heads of 12. We then attach one task attention block (TAB) after the transformer encoder which also has 768 hidden dimensions and 12 attention heads. For all four tasks, we apply Adam optimizer with = le-2, e 1e-8, 31 = 0.9, and B3 = 0.98.
rank the difficulty level of the five tasks by a met- ric: # Training Ezamples which are presented in # Labels
Table 1. Roughly, the bigger this value is, the eas- ier the tasks is. In this case, we rank our five tasks from difficult to easy as: OKVQA, GQA, CO- COQA, SNLI-VE and NLVR2, of which corre- sponding scores are 6.19, 43.43, 183.10, 26666.68 66 and 40000. To eliminate the bias due to a specific task order, we perform experiments on several task orders. More results are included the Appendix. Most CL algorithms consider relatively homo- geneous tasks. To evaluate the algorithm capacity for preventing catastrophic forgetting on hetero- geneous tasks that we have, we use the following normalization-base metric (Srinivasan al.):
Since there is no prior method for multimodal CL. we use extensions of Dytox (Douillard et al., 2022), FDR (Titsias et al., 2020), EWC (Kirk- patrick et al., 2017), Experiment Replay (Rolnick al., 2019), and Direct Fine- Finennuning, five alter- natives for comparison. Note that direct fine-tuning serves as lowerbound to measure the effect of catastrophic forgetting and effectiveness of CL. After learning each task T, we evaluate the for- getting rate on previous tasks Tk, k E 1,.., 1. To study the effect of task order, we perform ex- periments with different task orders that reflect the difficulty of tasks. For each of the learned tasks, the final performance accuracy and the for- getting rates compared with its best accuracy are re- ported. To assist the analysis of task order's impact to the experiment's performance, we intuitively
TF -) 5 ---- S S
(10)
where Tr(j 1) stands for the forgetting rate of task j after learning task 5 denotes the accuracy of task j before Icarning new tasks, sot+ denotes the accuracy of task ý after learning task i, j. and S means the accuracy of task ý by randomly choosing the output label, which is calculated by # labels In other words, Eq. (10) enables compar- ing forgetting rates across tasks that are consider-

---

ably different because we are measuring how well the model is preforming compared bascline of total forgetting for that task. More details about experimental setup are included in the Appendix.
underfitting on current task. While Dytox is de- signed for unimodal tasks, we conclude that Dytox is not suitable for multi-modal tasks.
Finally, TAM-CL outperforms every other method in Table2. Especially, it is 8.93% higher than the second best method for COCOQA, 31.43% higher than the second best method for OKVQA. and only 2.08% lower than the best accuracy for the last task, GQA. In the cross-task, multimodal CL scenario, TAM-CL is capable for significantly reducing the forgetting rate of previous tasks while maintaining the high accuracy of current task. Ad- ditional experiments are included the Appendix.
6.2 Comparative Results
We report our performance results after training on all five tasks in Table 1. As expected, we observe that fine-tuning is the worst algorithm regarding the forgetting rates for all tasks before GQA. On the other hand, the accuracy of GQA is the high- est among all the methods. The high accuracy is also expected as there is a trade-off between ac- curacy and forgetting rate of a task, without any constrain to the forgetting rate, the accuracy of fine-tuning method is supposed to be relatively higher. This empirical observation demonstrates the significance of adopting CL algorithm for learn- ing multimodal tasks sequentially. We adopt two regularization-basee CL methods, EWC and Func- tion Distance Regularization (FDR), in our compar- ative experiments, and observe that most of results are below the average accuracy for all the previous tasks. EWC and FDR often is effective when used on smaller models but in line with prior observa- tions in the case of using transformers for unimodal tasks (Srinivasan et al.; Jin et al., 2021; Douillard et al., 2022), we conclude that regularization-bart based CL methods are not suitable methods for CL with large transformers. This result suggests that trans- formers are sensitive with respect to weight consoli- dation because their learning capacities are compro- mised significantly when many weights are frozen.
6.3 Ablation Results
To reflect the necessity of each component in our design, ablation experiments are performed on the effect of Lika loss function, the training strategy with experience replay, and using the task attention block, respectively. In ablation experiments, we use the task sequence order OKVQA -> GQA COCOQA - SNLI-VE + NLVR2.
In the ablative experiment, we choose the full TAM-CL pipeline as the baseline and compare the performance of TAM-CL to each ablation task. Ta- ble 3 presents results for our ablation experiments. We observe that the full pipeline for TAM-CL leads to the best score in terms of both the forgetting rate and the performance accuracy. These results validate the necessity of every component in our approach for an optimal performance. We observe that the effect of dropping the Likd loss leads to an average performance drop of 12.74% across the four tasks, which demonstrates the significance of intermediate knowledge distillation to CL.
In contrast, the accuracy and forgetting rate re- sults for experience replay method is more decent, In Table 2, all the accuracy results of experiment replay are above the average line, which indicates that experiment replay is stable and relatively ef- ficient CL method which not constrained only small models and unimodal data. We also adopt more -tate-of-the-art method, Dytox, as our base- line. However, we observe that although Dytox has some result close to those of our method in some cases, the rest of the accuracies are relatively lower compared with other methods. For exam- plc, Dytox's accuracy on OKVQA is 10.41, only higher than fine-tuning. Noticing that this accuracy is corresponded with low forgetting rate, 25.26%. which indicates that Dytox prevents forgetting by underfitting the current task. Similarly, the accu- racy of GQA is 11.12, which verifies that Dytox's 's
Meanwhile, we also observe that when ablating the whole task-attention block, which also includes the Likd loss, the forgetting rates and accuracy per- formances are slightly better than only ablating the Likd loss. This higher performance may look unin- tuitive but our hypothesis is that the loss term Likd is specifically important to train the task-attention layer which is the main continual learning compo- nent in the model, leading to being more impactful,
As expected, our training strategy using experi- ence replay is also contributing to the optimal per- formance by mitigating catastrophic forgetting of previous tasks. We observe that experience replay is more helpful maintaining the accuracy of the carly tasks in the sequence. For example, in Table 3, by ablating experience replay, the forgetting rate

---

GQA COCQA + OKVQA + SNLI-VE + NLVR2
COCOQA
OKVQA
SNLI-VE
NLVR2
G
G
C
G
C
0
G
C
0
S 14.98%
3.22%
6.81%
5.81%
4.60%
4.58%
8.51%
9.43%
9.72%
20.72%
COCOQA - NLVR2 - SNLI-VE OKVQA + GQA
NLVR2
SNLI-VE
OKVQA
GQA
C
C
N
C
N
S 19.23%
S
O
7.40%
8.45%
6.58%
11.08%
14.57%
19.30%
23.34%
24.98%
27.29%
Table 4: Task order: the forgetting rate for each task in two different task sequences. The letter on each task sequence represent the shorteut of the task name: S:SNLI-VE, N:NLVR2, C:COCOQ,, O:OKVQA, G:GQA. The second row of each task sequences represents the current task, while the third row of cach sequence represent the previous task. Eg. After trained on COCOQA, the forgetting rate of G(QA) is 15.42%.
of OKVQA after learning NLVR2 is 93.69%, while the forgetting rate of GQA after learning NLVR2 is 51.84%, and the forgetting rate of COCOQA after NLVR2 is 44.33%, which are significantly less than 93.69%. These results demonstrate that the optimal performance of our method stems from using all the three primary ideas that we proposed.
which requires more distribution shift from the pre- vious task OKVQA, to achieve higher performance. Such correlations between the forgetting rate and task difficulty also be verified in sequence 2. For example, the forgetting rate of COCOQA after training the two relatively easier tasks, NLVR2 and SNLI-VE, are 7.40% and 8.45%. In contrast, after training OKVQU and GQA, the forgetting rate of OKVQA is 11.08% and 19.30%, which are much higher than after NLVR2 and SNLI-VE. For addi- tional experiments, please refer to the Appendix.
6.4 Effect of Different Task Orders
To further analyze the performance of TAM-CL, we compare the performance of TAM-CL on two different task sequences and analyze the impact of the task order on catastrophic forgetting. Ideally we would like to develop an algorithm that works well on all task orders. Although in practice we don't 't control the task order and the tasks are encountered in order determined by the environment, study the effect of task order, assuming it is given.
We also observe that even though the NLVR2 is relatively easier task, in sequence 1, after trained on NLVR2, all the forgetting rate of previous tasks rise more than twice. Our hypothesis is that al- tough NLVR2 is easy to train, it has an essential difference from other tasks that NLVR2 takes two images as a single input and perform visual reason- ing while all the other tasks only take one image at time. Such unique property of NLVR2 task raises the extent of parameter distribution shift, but not as drastically as the difficulty of task does. Meanwhile, we surprisingly find that after trained on some specific tasks, the forgetting rate of previous tasks can even decrease. In sequence 1, af- ter trained on OKVQA, the forgetting rate of GQA and COCOQA are 6.81 % and 5.81% respectively. However, after trained on the next task, SNLI-VE. the forgetting rates of those tasks drop to 4.60% and 4.58%, which might indicate the potential forward transfer capacity that we can further explore.
Table 4 presents the forgetting rate results for every time step. In another word, we evaluate the forgetting rate of previous tasks after the training of every single task. Inspecting results from both sequence, we conclude that the task order is key factor the performance of certain task. For example, in sequence 1, OKVQA is the third task, and the forgetting rate of OKVQA 20.72% af- ter learning the last task. Meanwhile, in sequence 2. OKVQA is the fourth task, but its forgetting rate is 27.29%, which is higher than that of the previous sequence. By our intuitive metric of task difficulty described in 6. our hypothesis is that, in sequence SNLI-VE and NLVR2 are two easier tasks, which requires less parameter distribution shift to achieve high accuracy, thus the parameter distribution for OKVQA is less shifted. However, in sequence 2, GQA is the second difficult task,
Conclusions
We developed an algorithm for multimodal contin- ual learning for transformer architectures based on dynamic model expansion and knowledge distilla-

---

tion. We use task-attention block which special- izes the transformer architecture for particular tasks using special learnable task token. Knowl- edge distillation helps to benefit from knowledge transfer across the tasks for positive forward trans- fer. We mitigate catastrophic forgetting using ex- perience replay. Our experiments demonstrate that approach is effective and leads to state-of-the art performance in terms of both forward trans- fer and catastrophic forgetting. Our TAM-CL ar- chitecture is a first algorithm in studying CL in multimodal settings and demonstrates that more explorations in this direction is necessary.
Memory aware synapses: Learning what (not) to for- get. In Proceedings of the European conference on computer vision (ECCV), pages 139-154.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. 2021. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836 6846.
Rich Caruana. 1998. Mullitask learning. Springer.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In Computer Vision-EECC 2020: 16th European Conference, Glasgow, UK, Au- gust 23-28, 2020, Proceedings, Part XXX, pages 104- 120. Springer.
Limitations
Although TAM-CL reaches the state-of-the-art per- formance, has its own limitations and we antic- ipate several future research direction to address these limitation. More specifically:
Georgios Chochlakis, Tejas Srinivasan, Jesse Thoma- son, and Shrikanth Narayanan. 2022. Vault: Aug- menting the dision-and-language transformer with the propagation of deep language representations. arXiv preprint AXh::220800902.
As we have three visual question answering tasks, one visual reasoning task and one visual entailment task, the designed experiments are not only cross-task, which is desired, but also cross-domain. We will further explore the performance of TAM-CL in single domain cross-task settings and compare with the other state-of-the-are methods.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
Arthur Douillard, Alexandre Ramé, Guillaume Coua- iron, and Matthieu Cord. 2022. Dytox: Transformers for continual learning with dynamic token expan- sion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9285-9295
In multimodal learning scenario, the modal can not only take multimodal input, but also uni-modal input, by setting the input of other modalities to some constant number. We will further explore TAM-CL's capacity uni- modal tasks and compare the result with other state-of-the-art methods.
Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cédric Archambeau. 2022. Con- tinual learning with transformers for image classifi- cation. Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 3781.
Due to the computational limits, some of the task that we trained on are not in its full ver- sion. For example, we are training TAM- CL on 80000 training examples for SNLI-VE dataset, where the full SNLI-VE dataset con- tains 529527 training examples. We will fur- ther explore the performance of TAM-CL on the full size of training examples for the tasks.
Robert M French. 1999, Catastrophic forgetting in con- nectionist networks. Trends in cognitive sciences, 3(4):128-135.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in neural network. arXiv preprint arxiv:1503.02531.
Drew A. Hudson and Christopher D. Manning. 2019. Gqa: A new dataset for weal-world visual reasoning and compositional question answering.
Xisen Jin, Bill Yuchen Lin, Mohammad Rostami, and Xiang Ren. 2021. Learn continually, generalize rapidly: Lifelong knowledge accumulation for few- shot learning. In Findings of Empirical Methods Natural Language Processing ( (Findings of EMNLP).
References
Rahaf Aljundi, Francesca Babiloni, Mohamed Elho- seiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018.

---

Wonjae Kim, Bokyung Son, and Iidoo Kim. 2021. Vilt: Vision-and aan-language transformer without convolu- tion or region supervision. In International Con- ference on Machine Learning, pages 5583-5594, PMLR.
Tejas Srinivasan, Furong Jia, Mohammad Rostami, and Jesse Thomason. 2023. 12i: Initializing adapters with improvised knowledge. arXiv preprint arXiv:2304.02166
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li. Lowei Lu, Furu Wei, and Jifeng Dai. 2019. VI-bert: Pre-training of generic risual-linguistr representations. In Inter- national Conference on Learning Representations.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Barbink-aa et al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):352133222
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang. Huajun Bai, and Yoav Artzi. 2018. A corpus for reasoning about natural language grounded in pho- tographs.
Zhiqiu Lin. Jia Shi, Deepak Pathak, and Deva Ramanan. 2021. The CLEAR benchmark: Continual learning on real-world imagery. In Neural Information Pro- cessing Systems (NeurIPS).
Chen Sun, Austin Myers, Carl Vondrick, Kevin Mur- phy, and Cordelia Schmid. 2019. Videohert: A joint model for video and language representation learn- ing. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7464-7473.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross- mooss-m encoder representations from trans- formers. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 5100-5111,
Mehrnoosh Mirtaheri, Mohammad Rostami, and Aram Galstyan. 2023. History repeats: Overcoming catas- trophic forgetting for event-centric temporal knowl- edge graph completion. In Findings of ACL.
Francesco Pelosin, Saurav Jha, Andrea Torsello, Bogdan Raducamu, and Joost van de Weijer. 2022. Towards exemplar-free continual learning in vision transform- ers: account of attention, functional and weight regularization. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 3820 3829.
Michalis K. Titsias, Jonathan Schwarz, Alexander G. de G. Matthews, Razvan Pascanu, and Yee Whye Teh. 2020. Functional regularisation for continual learning with gaussian processes.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
Mengye Ren, Ryan Kiros, and Richard Zemel. 2015. Exploring models and data for image question an- swering.
Zhen Wang. Liu Liu, Yiqun Duan, Yajing Kong, and Dacheng Tao. 2022. Continual learning with lifelong vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 171-181.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timo- thy Lillicrap, and Gregory Wayne, 2019. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32.
Mohammad Rostami and Aram Galstyan. 2023, Cogni- tively inspired learning of incremental drifting con- cepts. In ICCAI.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2019. Visual entailment: A novel task for fine- grained image understanding.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Des- jardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016. Progressive neural networks. arXiv preprint arXiv:1606.0467 04671.
Peng Xu. Xiatian Zhu, and David A Clifton. 2023. Mul- timodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intel- ligence.
Peng Yang, Dingcheng Li, and Ping Li. 2022. Continual learning for natural language generations with trans- former calibration. In Proceedings of the 26th Con- ference on Computational Natural Language Learn- ing (CoNLL), pages 40-
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. 2017. Continual learning with deep genera- tive replay. Neural Information Processing Systems (NeurIPS).
Tejas Srinivasan, Ting- Yun Chang, Leticia Leonor Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. Climb: A continual learning benchmark for iision-and-languag tasks. In Thirty- sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Jaehong Yoon, Eunho Yang. Jeongtae Lee, and Sung Hwang. Lifelong learning with dynamically expand- able networks. In International Conference on Learn- ing Representattons.

---

A
Appendix
training strategy, we also set the sample size of memory size as 5%, and the sample frequency as 100.
A.1
Implemntation Details
A.1.1 Data Preprocessing
As the tasks, which the model trains on, have com- pletely different dataset, we compress the image from its original size to (384, 640) before sending to the linear projection embedding block. In the self-attention layers of the pre- pre-trained transformer and task-attention block, the image-text feature vec- tor has the shape of (batch, 768).
A.2 Results on Additional Task Orders
Due space limit, additional comparative result on different task sequence of all time steps are presented here and each table presents the accuracy of the task right after trained on it and the forgetting rate of previous tasks after trained on the current task. Through the four tables, we observe that TAM-CL has the leading forgetting rate in different time steps in most of the cases.
Different from tasks such as COCOQA, PathVqa and SNLI-VE which takes only 1 input image, NLVR2 takes 2 input images with a hypoth- esis as text input. During the training stage, the text input is combined with one image every time be- fore feeding into the transformer, and concatenate the output from two images as one. Consequently, the output from transformer will be (batch, 1536). However, due to the input size limitation of the task-attention block, we compress the vector, V. from 1536 to 768 by taking the average value of the adjacent element:
As discussed in Section 6.2, the two regularization-base method, EWC and FDR are not suitable for large-model and long sequence continual learning. We observe that in some cases, the forgetting rate of FDR and EWC are close or even higher than the finetuming method, which means in such condition, those methods are not capable for preventing catastrophic forgetting. For example, in Table 6, after trained OKVQA, the forgetting rate of NLVR2 for Finetune method is 59.13%, while that of EWC is 52.07%, which is close the performance of without CL algorithm. Meanwhile, after trained on GQA, the forgetting rate of OKVQA for Finetune method is 40.90% and that of EWC is 42.28%, which even higher than the non-CL method baseline. In table 8, after trained on NLVR2, the forgetting rate of SNLI-VE for Finetune is 50.07, and forgetting rate for FDR is 47. 47.51% 51%, for EWC is 54.04%, which are close to and above the non-CL baseline.
Vi = V + v
0,2, 4, ..., 1534
2
(11) where V' is the compressed feature vector. We then feed V' into the task-attention block.
A.1.2 Hyperparameters for the Experiments For all experiments we perform, we use single A100 GPU with batch size of 16.
For the training epochs, due to the limit of com- putational force, we only train each task with small epochs: 5 epochs for SNLI-VE, 10 epochs for CO- COQA, GQA and NLVR2, 20 epochs for OKVQA. For EWC method, we applied the fisher sample percentage as 0.4, which means 40% of dataset are collected to build the fisher matrix. During the training stage, we the EWC weight 0.1. For Experience Replay method, the sample size of data memory buffer is 5%, and the sample frequency is 100, which means for every tasks, we randomly extract 5% of data and store them into the memory buffer. During the training stage, after 100 batches of current dataset are trained, we randomly select a batch of data from memory buffer and train the model on that batch from a random previous task. For TAM-CL, as the knowledge distillation loss is extremely small compared with other loss, we set the weight for Likd as 5000 for all the four tasks. As in TAM-CL, we adopt experience replay as our
Regarding the experience replay method, it pro- vides relatively high accuracy and low forgetting rates compared to the two regularization methods. However, the capacity of ER for preventing catas- trophic forgetting in later tasks are not stable in early tasks. For example, in Table 5, after trained on the last task, GQA, the forgetting rate of OK OKQ for ER is 50.10%, which close to FDR and higher than EWC. Comparatively, after trained on SNLI-VE, the forgetting rate of OKVQA for ER is 26.57%, far greater than the forgetting rate for FDR, which is 52.28%. In contrast, Dytox method has the stable capacity to prevent catastrophic for- getting, which is not affected drastically by length of task sequence, however, it lost the competitive- ness due to its low performance on training the current task. We observe that in Table 5, the accu- racy of OKVQA for Dytox is 19.12, whereas the

---

COCOQA NLVR2 OKVQA SNLI-VE GQA
COCOQA
NLVR2
OKVQA
COCOQA
COCOQA
NLVR2
TAM-CL
76.09
68.88
7.89%
31.63
9.45%
3.72%
Finetune
75.88
68.71
70.21%
32.24
19.41%
45.75%
EWC
75.89
67.93
73.65%
31.39
20.95%
64.20%
FDR
71.81
58.43
28.99%
26.01
24.54%
59.07%
ER
76.77
68.35
12.08%
31.19
13.64%
34.04%
Dytox
76.68
68.41
10.37%
13.93
16.15%
23.35%
SNLI-VE
COCOQA
NLVR2
OKVQA
TAM-CL
71.32
10.25%
6.98%
15.19%
Finetune
71.6
31.71%
41.57%
71.71%
EWC
71.03
23.55%
43.49%
29.75%
FDR
68.56
71.13%
51.00%
52.28%
ER
71.37
12.05%
10.46%
26.57%
Dytox
69.82
14.30%
16.25%
2.15%
GQA
COCOQA
NLVR2
OKVQA
SNLI-VE
TAM-CL
50.86
13.15%
14.87 %
22.59%
19.13%
Finetune
$1.92
47.09%
79.42%
74.37%
46.81%
EWC
49.67
33.10%
42.19%
49.40%
20.45%
FDR
50.67
32.12%
29.89%
55.44%
28.26%
ER
50.12
27.20%
34.27%
50. 50%10%
27.19%
Dytox 11.12
20.52%
15.37%
25.26%
20.70%
Table 5: Accuracy and forgetting rate of task order: COCOQA NLVR2 OKVQA SNLI-VE GQA
accuracy for all the other methods are above 30. Meanwhile, Dytox only obtains 6.10 in GQA and the accuracy for all other methods are above 50.
Finally, although TAM-CL is not leading ery single forgetting rate, it outperforms the rest of methods on 87.5% of the total forgetting rates. For the rest of the exceptional cases, TAM-CL has the above-average performance which proves its capac- ity and stability to prevent catastrophic forgetting.
We also aware that the accuracy of TAM-CL is not always leading among the six methods. How- ever, all of the differences between the top accu- racy and TAM-CL's accuracy are below 5%, and as our main focus on the improvement forgetting rate, we consider the slightly accuracy difference is in an acceptable range.

---

COCOQA NLVR2 SNLI-VE OKVQA GQA
COCOQA
NLVR2
SNLI-VE
COCOQA
COCOQA
NLVR2
TAM-CL
76.73
69.99
7.40%
71.12
8.45 %
6,58 %
Finetune
76.95
67.83
74.75%
71.72
62.51%
38.12%
EWC
76.69
71.67
64.88%
70.84
58.61%
27.77%
FDR
72.43
57.99
31.31%
67.59
38.96%
65.08%
ER
76.38
70.35
8.15%
70.69
8.65%
8.52%
Dytox
76.63
68.89
8.50%
70.45
9.43%
7.94%
OKVQA
COCOQA
NLVR2
SNLI-VE
TAM-CL
30.47
11.08% %
14.57%
19.23%
Finetune
32.12
19.19%
59.13%
22.22%
EWC
31.68
18.83%
52.07%
15.44%
FDR
11.49
44.23%
45.52%
41.27%
ER
30.91
12.92%
28.12%
17.92%
Dytox
19.12
17.16%
32.55%
37.06%
GQA
COCOQA
NLVR2
SNLI-VE
OKVQA
TAM-CL
56.20
19.30%
23.34% %
24.98 %
27.29 %
Finetune
57.03
36.53%
66.16%
34.11%
40.90%
EWC
57.21
33.51%
60.60%
32.84%
42.28%
FDR
42.67
41.08%
46.41%
74.08%
50.17%
ER
57.04
26.04%
32.89%
27.97%
41.33%
Dytox
6.10
21.81% 1%
38.50%
10%
34.43%
Table 6: Accuracy and forgetting rate of task order: COCOQA - NLVR2 - SNLI-EE - OKVQA > GQA

---

GQA COCOQA OKVQA SNLI-VE NLVR2
GQA
COCOQA
OKVQA
GQA
GQA
COCOQA
TAM-CL
56.58
74.44
3.22%
30.44
6.81 %
5.81%
Finetune
58.11
75.95
10.60%
31.99
12.78%
10.65%
EWC
58.69
76.12
10.45%
31.15
13.46%
7.97%
FDR
55.50
71.42
12.70%
28.39
12.17%
6.21%
ER
57.61
75.17
4.17%
31.21
9.58%
4.26%
Dytox
56.83
75.22
1.93%
29.87
11.29%
7.87%
SNLI-VE
GQA
COCOQA
OKVQA
TAM-CL
71.49
4.60%
4.58%
8.51 %
Finetune
72.54
9.91%
7.35%
15.93%
EWC
72.10
10.82%
6.43%
11.46%
FDR
70.17
8.29%
7.11%
13.43%
ER
72.39
6.00%
5.03%
10.47%
Dytox
72.44
8.13%
5.40%
10.07%
NLVR2
GQA
COCOQA
OKVQA
SNLI-VE
TAM-CL
70.04
9.43%
9.72%
20.72%
14.98% %
Finetune
70.32
55.09%
63.90%
77.80%
53.18%
EWC
67.57
57.47%
63.77%
80.89%
51.90%
FDR
53.60
33.54%
56.20%
63.64%
36.42%
ER
70.38
13.19%
9.90%
23.43%
19.93%
Dytox
67.34
15.86%
13.90%
41.45%
27.61% %
Table 7: Accuracy forgetting rate of task order: GQA
COCOQA - OKVQA SNLI-E NLVR2

---

COCOQA GQA SNLI-VE OKVQA NLVR2
COCOQA
GQA
SNLI-VE
COCOQA
COCOQA
GQA
TAM-CL
76.59
57.83
3,86%
72.03
3.37 %
1.66% %
Finetune
76.84
58.72
18.86%
72.09
23.57%
5.64%
EWC
76.49
57.27
18.65%
72.11
21.18%
4.26%
FDR
72.00
53.42
14.66%
68.25
20.47%
2.74%
ER
76.88
58.07
17.52%
72.19
9.01%
1.71%
Dytox
76.77
25.41
4.51%
72.33
3.77%
2.14%
OKVQA
COCOQA
GQA
SNLI-VE
TAM-CL
30.74
6.64%
4.78%
5.66%
Finetune
32.77
24.37%
9.89%
9.27%
EWC
31.78
22.88%
5.88%
8.65%
FDR
26.57
26.88%
8.84%
4.77%
ER
33.22
11.64%
6.11%
5.74%
Dytox
13.74
10.61% %
4.98%
6.74%
NLVR2
COCOQA
GQA
SNLI-VE
OKVQA
TAM-CL
69.15
10.09
11.74%
14.60%
16.79 %
Finetune
67.27
76.18%
55.89%
50.07%
75.11%
EWC
68.43
78.72%
54.84%
54.04%
79.46%
FDR
58.73
38.38%
40.97%
47.51%
52.12%
ER
67.63
16.08%
12.26%
24.12%
19.80%
Dytox
69.01
18.20%
23.63%
26.89%
29.90%
Table 8: Accuracy forgetting rate of task order: COCOQA - GQA SNLI-VE + OKVQA NLVR2


Reviews:
[{"review_id": "dUpzMR4QEY", "reviewer": "Reviewer_c6E7", "summary": "", "questions": "Please respond to the concerns listed in the weaknesses.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and efforts. We are glad that you have found our solution to be effective. We hope you are open to reconsidering our work and giving us a second chance by providing a rating with higher confidence. We have reponed below:\n\nA.Please note that the classifier is task-specific which means that for each of the new tasks, there will be a completely new initialized classifier. However, as the number of tasks increases, the size of new initialized classifier is cumulated. For example, task one’s classifier has an output label size of 100, and task two’s output label size is 50, then task two’s classifier has an output label size = 100 + 50 = 150. This increase is because although we only use intermediate knowledge distillation in our loss objective, we still preserve the choice of adding knowledge distillation to the final prediction as our future work. Due to the special property of visual-language tasks, especially VQA tasks, the output label size varies a lot from task to task. To do knowledge distillation from final output, we should have the two predicted vectors in the same dimension. As a result, with the cumulative classifier, we can use the output of task one’s classifier, with the dimension of [100,1], and the first 100 values of task two’s classifier [:100, 1], to compute the knowledge distillation loss. We hope the above addresses your concern but please let us know if it is not clear.\n\nB. We acknowledge that the absolute performance of VQA tasks may not be state-of-the-art, which is mainly due to the fact that we apply ViLT as our backbone transformer. In TAM-CL, the proposed architecture is shown in the yellow and red blocks of the main diagram, it is a generalizable add-on architecture that can be embedded into most existing V-L transformers, such as the BLIP2. In our comparison experiment, we are showing that with exactly the same backbone transformer, ViLT in our case, we are embedding ViLT with different continual-learning methods, such as EWC, Experience Replay, FDR, Dytox, etc, and TAM-CL is the best among all the methods. We can also change ViLT in all the baseline methods to other transformers, and the absolute performance will definitely improve. However, our purpose is to show that TAM-CL is more helpful, as an add-on continual learning architecture, than other methods relatively. Adding experiments with BLIP2 can be helpful but unfortunately, our GPU resources are limited to do the relevant experiences in only a week. Our hope is that you base your evaluation on the experiments that we have provided.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Training a task-specific classifier and task token is a concise and intuitive solution for adapting various tasks. And the teacher model and memory replay effectively promote performance and alleviate catastrophic forgetting.", "reasons_to_reject": "Weaknesses:\n\nA. The explanation for the expansion of the task-specific classifier is a bit confusing, especially Eq. 8. All tasks use the same classifier with accumulated weights, or each task has its own task-specific classifier?\n\nB. The absolute performance on VQA tasks is weak compared to existing zero-shot vision-language models. It's helpful to validate the performance of TAM-CL on more recent base models, e.g. BLIP2.", "questions_to_authors": "Please respond to the concerns listed in the weaknesses.", "paper_topic_and_main_contributions": "This paper presents a novel method for continual learning of vision-language tasks. The components proposed in this approach result in remarkable continual learning performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a86Rrv8KS7", "reviewer": "Reviewer_inPu", "summary": "", "questions": "See above.\n\nIf I have any misconceptions about the contributions, please raise them in the rebuttal briefly. Thank you!", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort. We are glad that you have identified our SOTA performance and the pioneer nature of our work. Given these important aspects, we hope you engage in post-rebuttal discussions and in case we can address your concerns, you reconsider the final rating for our work. We also appreciate that you have mentioned that you are open to discussions during the post-rebuttal period.\n\n1.1 We appreciate if you can  list the recent proposed CL methods that you have in your mind  in detail so we can add them to our comparisons. We agree that there are many recent methods but due to the fact that most of them are developed for computer vision task, their backbone models are convolutional neural networks instead of transformers. As a result, it is not trivial to extend them to handle multimodal tasks. .\n(Working on adding X-DER into baseline.)\n\n1.2 Could you please explain how to “showcase” the performance of GPT-4 in our setting by adding more detaisl? Actually, LLMs are not a primary consideration in CL. If you check the CL literature, you can see most continual learning methods (e.g., iCARL, ER, Dytox, GEM, DER, Adapters, etc)  are not comparing their performance with Large-scale models, but only with other CL methods, which use CNN layers, BERT, ViT, and other similar architectures as backbone. As a result, we are unsure how to include GPT-4 in our baseline when it is a model for language tasks. Please also note that GPT-4 is an API-based model, updated frequently from OpenAI’s side, and we do not have the freedom to change it to enable it to handle multimodal tasks. \n\n1.3 We acknowledge that there are alternatives for ViLT to do our experiments and we could base our exploration on other models. However, the main idea of the experiment is to show that using exactly the same base model, i.e., ViLT, TAM-CL generates better results than other continual learning methods to handle multimodal tasks. We also agree with you that if we change the base mode from ViLT to others such as CLIP, the performance may increase but please note that our computational resources are limited to do the experiments with CLIP in a week. We are running results for CLIP in the background and will update the results if we could generate new results during the post-rebuttal period. However, we respectfully ask the reviewer to focus on the broad message of our experiments that demonstrate TAM-CL outperforms alternative methods.\n\n1.4 We’ve done the experiment of using adapters as a baseline, as one of the methods mentioned in your comment. Here is the comparison result: \n\nAdapter:  77.50 (0%), 68.89 (1.49%), 30.17 (0.57%), 69.11 (1.28%), 53.88\n\nTAM-CL: 66.09 (13.15%), 66.07 (14.87%), 21.24 (22.59%), 64.05 (19.13%), 50.86\n\nNote that these two lines of comparison follow the same format as Table 2 in the paper. We acknowledge that the Adapter has a leading performance compared with our methods, and its forgetting rate of about 1% is also impressive. However, we don't take it as a regular type of continual learning method due to its high space complexity. For adapters, it inserts 2 adapter modules in each attention layer, l, for every new task, t, along with task-specific head, the space complexity of it is O(l*t) + O(t), while the first term is for the space growth of adapter module and the second term is for the space growth of heads. Such a high space complexity is, to some extent, atypical in the CL method community, so it can easily defeat most of the SOTA CL methods, not only ours, with its high accuracy and low forgetting rate. On the other hand, our method's space complexity is O(1) + O(t), as we have a single-layer task-sharing TAB, and only the heads are task-specific. As a result, we consider methods such as adapters are not comparable.\n\n2 \nPlease note that the choice of adding intermediate Knowledge Distillation into the loss objective function is a design specific for multi-modal to fuse the modalities well.. For other similar dynamic-architecture-based methods, such as DyTox, a vision-only approach, the knowledge distillation is applied to the end of the classifiers, not in the middle. We also tried to apply knowledge distillation loss to the output of the classifier when designing the architecture, but the result was much worse, which implies that such an arrangement of knowledge distillation is not suitable for multi-modal situations, and our choice of using intermediate knowledge distillation is the one that fits.\n\nWhile we agree that adding new experiments based on the reviewer’s feedback will strengthen our empirical results, but we respectfully ask the reviewer to consider that, as stated by the reviewer, this is a “first work” of its own kind and the novelty of the setting and idea can be outweighed when evaluating the work compared to well-established settings where extensive experiments are necessary to warrant a worthy contribution. We hope the reviewer is open to increasing the rating if we address the concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The model achieved state-of-the-art performance in the experiments.\n- The paper introduces a continual learning solution for multimodal data, which is the first work in this field.", "reasons_to_reject": "1. Since the authors claim to present a solution for the CL problem in multimodal data for the first time, they should provide a comprehensive comparison with baselines:\n\n1.1 The comparison with previous work is limited. The compared approach Dytox was proposed in 2022, but the rest approaches are older. There are many CL algorithms proposed recently that should be compared instead of the older ones.\n\n1.2 Apart from the compared methods, the effects of large-scale multimodal models like GPT-4 should be showcased. I know that due to the different baselines, the author's approach might not be directly comparable to large models like GPT-4. However, showcasing such results would help readers grasp the true value of this paper.\n\n1.3 The choice of ViLT model with pre-trained weights (BERT-base-uncased) as a baseline is questionable. Why not combine the CLIP image encoder and language model (like Frozen, MAGMA, etc)? This would provide a clearer assessment of whether the proposed CL method should be applied to the latest architectures and help the readers to judge the values of this approach.\n\n1.4 In the present day, numerous technological paradigms have undergone transformations. Approaches such as LoRa, adapters, and others are evidently capable of addressing the challenges posed by the continual learning (CL) problem. It would be beneficial for the authors to compare their approach against these methods. I recognize that making such a request might appear rigorous, and the authors might face constraints in fulfilling it. It seems that they focus on the CL problem. However, considering that the paper also introduces innovative concepts in model architecture, the contributions might appear somewhat scattered. Consequently, I recommend that the authors consider emphasizing the primary contributions of their work and focusing on a specific aspect (an algorithm for CL problem from the ML perspective).\n\n2. As stated by the authors in the limitation section, their CL approach can also address tasks that are vision-only or language-only. In my view, this indeed seems to be the case. The authors claim that their method is tailored for multi-modal applications, but I did not observe too much aspects of the design specifically intended for multi-modal tasks. If the method can effectively be applied to single-modal data, the authors should produce a solid paper focusing on the CL approach, rather than presenting an engineering submission with limited contributions for several fields and insufficient comparisons.", "questions_to_authors": "See above.\n\nIf I have any misconceptions about the contributions, please raise them in the rebuttal briefly. Thank you!", "paper_topic_and_main_contributions": "This paper aims to address the issue of  continual learning in multimodal data (vision and language) tasks, and the authors claim to be the first to propose a solution for the continual learning problem in multimodal data. To tackle this problem, the authors introduce a teacher-student-based training approach. Experimental results demonstrate that their approach achieves lower forgetting rates.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "Btw, the summary of contributions in the introduction fails to highlight the innovation. Additionally, there is room for improvement in writing to enhance clarity."}, "extra_text": {}}, {"review_id": "Cp3cMABF3d", "reviewer": "Reviewer_94xT", "summary": "", "questions": "1. Does the task attention block include the task-focused attention layer or not? In line 254, the paper says the task attention block takes the task token as input, but in Figure 2 on the right hand side, the task-attention block architecture includes the task attention layer.\n2. Task token is generated from task attention layer, how to ensure the same task token is always generated for the same task after training multiple tasks? Is there an evaluation on the consistency of the task tokens?\n3. Why to change the task sequence order in ablation study? Results in Table 3 shows that the experience replay module plays the most important role while the task attention block is the least, it makes one wonder whether when evaluating in the same order as Table 2, without TAB, performance can still outperform state-of-the-art methods.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort. We are glad that you have confirmed that our results support SOTA performance of our work. We are hopeful that the reviewer is open to engage post-rebuttal discussions so we can address the concern about the novelty of our work. \n\n\nReasons to reject\n\nWe agree that knowledge distillation and dynamic expanding architecture have been explored before. We are not making any contribution claim regarding these ideas either. However, please note that most works on CL are limited to uni-modal scenarios and specifically vision modality. Extending most uni-model CL methods to handle multi-modal tasks is not trivial because most of them use CNN architectures that are not known to be good models for NLP domain. The nice aspect of focusing on a transformer architecutre is that it allows handling multimodal inputs. On the other hand, if we change the base model of DyTox to ViLT and run it to V-L tasks in the comparative experiments, the results are quite unstable. The degraded performance implies that using “dynamic model expantion” and “knowledge distillation” in multi-modal learning is not trivial and further exploration is necessary. TAM-CL is a novel architecture for CL that focuses on learning multi-modal tasks, with the unique combination of dynamic token expansion + intermediate knowledge distillation, which is the innovation of our contribution. This is the main reason that as mentioned by the reviewer leads to SOTA perofrmnace of our method.\n\nResponse to the Questions:\n\n1. Yes, task attention block includes the task attention layer, as shown on the right side of Figure 2. The task token is associated with a specific task, and it is concatenated with the output from previous layer, so it is included in the “embedding” of right side of Figure 2.\n\n2. When a new task comes in, a task token will be initialized and stored in the task token list. During the training, for the same task, the same task token will be used and modified. After the training of that task, the corresponding task token will be stored. In other words, the task token for a single task is only generated once at the very beginning when we learn that task and will never be regenerated. The same task token, after training and storing, will be applied to that task during the test stage. \n\n3. This is a great question and touches on one of the less explored aspect of CL in the literature. The reason we changed the task order is expressed in  line 408. We do not have any control on the task order in a CL setting but the tasks are not equally difficult. However, most CL methods only study a single task order which ignores the effect of task order but for different task orders, the performance of CL algorithms may not be equally good. To demonstrate that irrespective of the task order, our methods leads to SOTA performance, we offered experiments on different tasks orders. Hence, we think that these experiments indeed is a strength for our empirical exploration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Instead of designing a task attention block for each new task, the paper proposes a single task attention block that takes input of task-specific token from a task attention layer to learn task specific knowledge. \n2. The paper addresses multimodal vision-and-language tasks.\n3. Experimental results show the proposed method outperforms some state-of-the-art methods.", "reasons_to_reject": "The paper only discusses continual learning for transformers in related work, however, continue learning has been widely explored in the literature for several years, ideas go beyond a specific model architecture. And dynamic expanding architecture and knowledge distillation has been widely explored, the paper fails to claim its novelty compared to previous work. Please refer to reference [1] in the Missing References section, or any other survey paper on continue learning.", "questions_to_authors": "1. Does the task attention block include the task-focused attention layer or not? In line 254, the paper says the task attention block takes the task token as input, but in Figure 2 on the right hand side, the task-attention block architecture includes the task attention layer.\n2. Task token is generated from task attention layer, how to ensure the same task token is always generated for the same task after training multiple tasks? Is there an evaluation on the consistency of the task tokens?\n3. Why to change the task sequence order in ablation study? Results in Table 3 shows that the experience replay module plays the most important role while the task attention block is the least, it makes one wonder whether when evaluating in the same order as Table 2, without TAB, performance can still outperform state-of-the-art methods.", "paper_topic_and_main_contributions": "The paper proposes a dynamically expanding transformer-based continual learning architecture for learning multimodal vision-and-language tasks, in addition, the paper applies knowledge distillation and experience reply to further prevent catastrophic forgetting.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1]. De Lange, Matthias, et al. \"A continual learning survey: Defying forgetting in classification tasks.\" IEEE transactions on pattern analysis and machine intelligence 44.7 (2021): 3366-3385."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers provided feedback for this paper. There was a high degree of consensus amongst reviewers in the overall reviews, soundness and excitement. They found the method interesting and effective. They also found the work novel. One reviewer complained about missing baselines but failed to provide the specific ones they had in mind. This reviewer also asked the authors to compare their work to GPT-4 which is unfair in my opinion, given that the multimodal version of GPT-4 isnt even publicly available. Other complaints included a request for more experiments (with BLIP2) and missing related work. Overall the authors have done a good job at addressing these concerns and overall I am satisfied with the paper given the reviews and rebuttal.
Decision: Accept-Findings