C-CLIP: Multimodal Continual Learning for Vision-Language Model
Authors: Wenzhuo Liu
Fei Zhu
Longhui Wei
Qi Tian
Venue: ICLR 2025 Poster (2025)
Keywords: Vision-Language Models
Continual Representation Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Multimodal pre-trained models like CLIP need large image-text pairs for training but often struggle with domain-specific tasks. Since retraining with specialized and historical data incurs significant memory and time costs, it is important to continually learn new domains in the open world while preserving original performance. However, current continual learning research mainly focuses on single-modal scenarios, and the evaluation criteria are insufficient without considering image-text matching performance and the forgetting of zero-shot performance. This work introduces image-caption datasets from various domains and establishes a multimodal vision-language continual learning benchmark. Then, a novel framework named C-CLIP is proposed, which not only prevents forgetting but also enhances new task learning impressively. Comprehensive experiments demonstrate that our method has strong continual learning ability across different domain image-text datasets, and has little forgetting of the original capabilities of zero-shot prediction,  significantly outperforming existing methods.

Published as a conference paper at ICLR 2025
C-CLIP: MULTIMODAL CONTINUAL LEARNING FOR VISION-LANGUAGE MODEL
Wenzhuo Liu Fei Zhu Longhui Weist
Qi Tian
School of Artificial Intelligence, UCAS State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA Centtee for Artificial Intelligence and Robotics, HKISI-CAS "Huawei Inc {11 1wwenzhuu2020, 2hufei2018)@ia. ac......... weilh2558ggmii..com
ABSTRACT
Multimodal pre-trained models like CLIP need large image-text pairs for training but often struggle with domain-specific tasks. Since retraining with specialized and historical data incurs significant memory and time costs, it is important continually learn new domains in the open world while preserving original perfor- mance. However, current continual learning research mainly focuses on unimodal scenarios, and the evaluation criteria are insufficient without considering image- text matching performance and the forgetting of zero-shot performance. This work introduces
datasets from various domains and establishes multi- modal vision-language continual learning benchmark. Then, a novel framework named C-CLIP proposed, which not only prevents forgetting but also enhances new task learning impressively. Comprehensive experiments demonstrate that our method has strong continual learning ability across diverse image-text datasets, maintaining zero-shot prediction capabilities with minimal forgetting and signifi- cantly outperforming existing methods.
INTRODUCTION
Multimodal pre-trained models like CLIP (Radford et al., (021) have recently gained widespread attention for providing general visual-language representations on downstream tasks such image question answering, classification, semantic segmentation, and object detection. Although CLIP is trained on large number of image-text pairs, it still struggles to handle image-text pairs from unseen domains (Zhang et al. 2024). limiting real-world applicability. A straightforward way is to fine-tune the pre- pre-trained CLIP model on the domain-speciff dataset. However, this often leads to catastrophic forgetting (French) 1999) of existing knowledge, including both the CLIP's original general repre- sentations (i.e., zero-shot generalization) and the knowledge other learned tasks. Besides, the high memory and training cost makes retraining CLIP infeasible. Therefore, a natural, fundamental yet underexplored question is how to maintain general representations of visual-language model while adapting to new domains continually?
Continual learning (CL) (Liu al. 2025 Hou al. 2019 Yan et 2021 Gao al., 2023) has
explored the problem of retaining old knowledge while learning new tasks. However, current CL studies have several key challenges: Firstly, compared to standard unimodal continual learning scenarios, multimodal settings present greater complexity. Although CLIP can handle both image classification and purely multimodal tasks, the challenges of addressing forgetting and adaptability differ significantly between these two tasks. For image classification, pre-trained models often ex- hibit strong zero-sho shot performance. As a result, many previous approaches Wang et al. (2022): Tang et al. (2023): D' Alessandro et al. (2023): ) Qiao al. (2023); Li al.) (2024); Wang al. (2024) rely on prompt-based designs, keeping parameters fixed without fine-tuning the model itself. In contrast, multimodal tasks are far more challenging, as poor performance in specific domain often necessitates simultaneous fine-tuning of both the vision and text encoders.
"This work was done during an internship at Huawei Inc.
Corresponding author. Code available httppss: : / thub. com/ Sma1 11 ppppaa/--CC. httpt/ocooooaaaaaaaeaaaaaa///

---

Published as a conference paper at ICLR 2025
Flickr30K (task 0)
COCO (task 1)
Pet (task 2)
70-
wio fine-tune
65
60-
Zero Shot Acc
fine-tune
Zero-Shot
50-
ZSCL
ModiX
53
40-
MOF-CI
DER
20
40 60 80
20
40
10 20
mage-Text R@1
Image-Text R@1
Image-Text R@]
Figure I: Performance comparison on downstream tasks and the general representation ability. The pre-trained CLIP (Radford et al. 2 has good ImageNet zero-shot accuracy but performs poorly on downstream tasks. Directly full fine-tuning remarkably enhances the image-text retrieval perfor- mance on downstream tasks, while leading to catastrophic forgetting of ImageNet zero-shot ability. Existing methods generally seek a trade-off between zero-shot generalization and downstream task performance. Differently, method (C-CLIP) impressively achieves strong downstream task per- formance (even outperforms full fine-tume teteintnnng and well preserves the general representation ability.
Secondly, the evaluation of vision-language models (VLMs) remains insufficient. While some recent works (Zheng al.] 2023 Yu et al. 2024 Srinivasan al. (022) have leveraged pre- trained vision-language models like CLIP in tasks such as few-shot class-incremental learning D'Alessandro al. (2023), visual muestion-answering Qian al. (2023), cross-modal retrieval Wang et al. (2021), their evaluations often focus on specific aspects, such as image classification performance or cross- cross-modal retrieval accuracy. We aim provide more comprehensive evalua- tion by assessing both the general zero-shot capabilities and their continual learning performance in downstream multimodal tasks.
Third, traditional CL methods (Kirkpatrick et al., 2017 Zenke et al. 2017 Li & Hoiem 2017 Hou
jtt al. 2019. Douillard et al. 2020) apply regularization to reduce forgetting, which unfortunately hinders the learning of new tasks. In other words, these methods forget less because they learn less, losing the plasticity the continual learning process gradually (Dohare et al., 2024). Therefore, seek leverage the properties of multimodal representation learning enable simultancous learning of new and old knowledge, overcoming past trade-offs.
In this work, establish a Vision-Language continual learning (VLCL) benchmark based on image-text datasets (e.g.. Flickr30K (Plummer et al. 2015) and COCO-caption (Chen et al. 2015) ) and propose the evaluation three aspects: image-text retrieval downstream tasks, retrieval in unseen domains, and the general performance of visual-language model. Then, we propose a multimodal continual learning approach named C-CLIP. Specifically, we demonstrate that reducing trainable parameters can yield similar results to the existing sophisticated CL method, and simplify the previously complex strategies with low-rank adaption (LoRA) (Hu al.) In addition. we exper- imentally find that existing methods (Kirkpatrick et al. 2017 Li & Hoiem 2017 Douillard et al.) 2020 Zheng et al. 2023 Yu et al., 2024) generally seek a trade-off between zero-shot generalization and downstream task performance. To overcome this limitation, we propose contrastive knowledge consolidation that not only reduces forgetting old tasks but also enhances learning on new tasks, even matching or exceeding full fine-tuning performance. The results in Figure [ show that our method significantly improves the performance on downstream tasks while generally preserving the ImageNet zero-shot accuracy.
Our main contributions are as follows:
We introduce a benchmark for continual multimodal representation learning, emphasizing that lanual-language model models should retain their original general performance and learn new image-text data simultaneously.
We propose the C-CLIP method that consists of multimodal low-rank adaptation and con- trastive knowledge consolidation, achieving the goal of learning more and forgetting less for the first time.
Extensive experiments demonstrate that our method significantly enhances performance across different domain image-text datasets without catastrophic forgetting.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 CONTINUAL LEARNING
Continual Learning (Kirkpatrick al., 2017), also known as incremental learning, has received extensive attention recently. Existing research mainly focuses on unimodal tasks like supervised
image classification, with class-incremental learning (CIL) (Masana et al. 2022) as the common benchmark. Regularization methods (Kirkpatrick et al. 2017 Zenke et al. 2017: Li & Hoiem 2017) mainly constrain the changes of parameters or feature spaces. Data replay methods (Rebuffi al. 2017 Hou et al. 2019) store and replay subset of old data, which leads to additional compu- tational, memory, and privacy costs. irchitecture-base methods (Schwarz et al.) 2018 Yan et al.)
2021 add new models for each task, but their parameters grow fast with more tasks, making them unsuitable for real-world applications. More recent rehearsal-frie approaches (Wang ct al. 2022
Tang et al. 2023 D Alessandro al. 2023 Guo et al. 2024 Li al., 2024; Wang al. explore parameter-ele -fficieet strategies (e.g., prompt tuning, prefix tuning) for continual fune-tuning of pre-trained models. However, the above methods mainly focus on unimodal scenarios, ignoring widely existing multimodal tasks (Ramachandram & Taylor 2017) in real-world applications.
2024
few recent studies (Zheng et al. 2023 Yu al. 2024) involve visual-language models continual learning. Specifically, Jin et al. (2020) studied the visually-grounded grounddd masked language prediction task by learning from streaming visual scenes. Zheng et al. (2023) and Yu et al. (2024) studied the multi-domain task incremental learning (MTIL) benchmark that continually fine-tunes CLIP on visual datasets and evaluates the zero-shot performance of new tasks. However, it is still far from the goal of continually updating multimodal pre-trained models. Mod-X (Ni et al., 2023) is most relevant to our work, which performs stage-wise wise data continual training on a fixed image- text dataset. However, each task shares the same data distribution, which differs from our goal of adapting CLIP to diverse domains. Additionally, none of these works considers CLIP's original (e.g., zero-shot generalization) performance during continual learning. In this paper, we study more realistic and challenging scenario named multimodal continual learning with the visual-language model (VLCL), and comparison of these settings shown in Table T
Table I: Comparison of CIL, MTIL, and VLCL benchmark.
Setting CIL. MTIL
Train Data
Eval Metric
Eval Origin Performance?
Image, Label
Classification Ace
X
Image, Label
Zero-shot Ace (only new tasks)
VLCL (ours)
Image, Caption
Texte-Text Recall & Zero-shot Acc
2.2
VISUAL-LANGUAGE REPRESENTATIONALL LEARNING
Vision-language representation learning (Radford et al.) 2021 Jia et al. 2021 Zhang et al., 2024 has gained widespread attention across various fields. Among them, CLIP (contrastive language-image pretraining) (Radford et al. 2021) performs exceptionally well across many downstream tasks such as VQA (Antol et al. 2015), classification (He et al. 2016), and detection (Ren et al.) 2016). CLIP consists of an image encoder and a text encoder. During the pre-training stage, paired image-text samples are treated as positives, while unpaired samples are used as negatives under the contrastive learning paradigm (Chen al. (020). Despite the impressive performance, training CLIP relies on large datasets like Laion-400M (Schuhmann et al.) (202T) and Conceptual Captions (Sharma et al.) 2018), making
Assource-intensigg In addition, although these large pre-training datasets cover diverse samples, well-trained vision-language models still struggle to match domain-specifii image- text pairs (Zhang et al., 2024). Therefore, continually fine-tuning CLIP without losing its original performance becomes key issue in real-world scenarios.
3 PROBLEM DEFINITION AND BENCHMARK
Notation and problem definition. Vision Linnua-Language continual learning (VLCL) involves learning from a sequence of T tasks. At each stage t E {I....,T), the model is fine-tuned on an image- caption dataset D {(v), ) where (v/, c() represents image-caption pair and n, is the
3

---

Published as a conference paper at ICLR 2025
number of pairs in this dataset. In this paper, focus vision- model like CLIP and represent model into two components: (1) a vision encoder fo : : + 2, parameterized by 0, that transforms an image into feature vector 2, fo(v) in high-dimensional space Z C R. (2) a text encoder 04 C + 2, with parameters y, which maps a caption e. to a feature vector
Ze 99(c))
VLCL aims to continually learn a function f VxC y that can assign the correct label to each image-caption pair from all seen tasks. Specifically, at stage the challenge minimize loss function & (e.g.. symmetric cross-entropy) on the new dataset D', while preserving knowledge from previous tasks and potentially improving on earlier learning (Aljundi) 2019) as follows:
min (e (L))))) E
0.ple
s.t. ((w.c).DDD ((0T((()))((()))((((--(00))))00)
& Sey (1, (1,2,,,,-11
The last term == {ey} is a slack variable that can represent the forgetting (cj > 0) backward knowledge transfer (e) & 0) on datasets Di of j-th old task.
VL.LL benchmark. To evaluate the continual learning performance of vision-language models, we establish novel benchmark that includes three evaluation tracks, summarized in Table 2 (1) Multimodal continual learning. Eight image-caption datasets are used in this track. Among them, Flickr30K (Plummer et al. 2015) and COCO (Chen al. 2015) are general real-world datasets. Other datasets, including Pets (Parkhi et al., 2012). Lexica (Shen et al. 2024), Simpsons, WikiArt (Salch & Elgammal 2015), Kream, and Sketch (Chowdhury et al. 2002) represent specific domains such as Al-generated images, art, clothing, and sketches. (2) Zero-shot retrieval. One held image- caption dataset, i.e., HAVG (Abdulmumin al.) 20022) is used to assess retrieval performance on unseen datasets, (3) Zero-shot classification. Previous CL work on CLIP has overlooked an im- portant aspect: the forgetting of general representations (i.e., zero-shot generalization). To evaluate this, we tested zero-shot performance on six image classification datasets: ImageNet (Deng et al.) 2009). CIFAR- 100 (Krizhevsky et al.] 2009). StanfordCars (Krause et al., 2013), Flowers (Nilsback) & Zisserman 2008) DTD (Cimpoi et al. 2014), and Food101 (Bossard et al. 2014).
Table 2: Evaluation setup of the VLCL benchmark datasets.
Evaluation Aspect
Dataset
Multimodal Continual I Learning
Flickr30K, COCO, Pets, Lexica, Simpsons, WikiArt, Kream, Sketch
Zero-shot Retrieval
HAVG
Zero-shot Classification
CIFAR-100, ImageNet, Flowers, DTD, Food101, StanfordCars
Evaluation metric. We focus on multimodal continual learning of VLMs and view each dataset as one task, similar to the -nnemmennneemettall learning in classical CL (Van de Ven et al. 2022). The task identity (task-ID) is not needed at inference time. (1) For multimodal continual learning, we evaluate each dataset after fine-tuning all eight datasets (similar to the last accuracy in CII.), and report Recall @1 1 for both Image-to-Tees Text (12T R@ 1) and Text-to- Image (T2I R R@) 1). The average performance on all datasets is also reported. (2) For zero-shot retrieval, we report 12T R@ 1. (3) For zero-shot classification, we report the performance during each stage of the continual learning process, as well as the final performance degradation (PD), which is calculated as the difference in accuracies between the original pre-trained CLIP model and the final fine-tuner model.
THE PROPOSED METHOD: C-CLIP
The core of continual learning is to preserve the performance of previous tasks and enhance the performance of new tasks. (1) For the first purpose, i.e., avoiding forgetting old tasks, without storing or replaying old data, existing studies (Kirkpatrick et al., 2017: Li & Hoiem| 2017 Zheng et al. 2023 Yu et al. 2024) have designed variety of CL strategies, and the general idea is to use the current data to constrain the input-output relationship in old tasks. We empirically and theoretically show that low-rank adaptation could achieve similar effect by reducing the number of trainable parameters. (2) For the second purpose, i.e., enhancing learning new tasks, we find that existing methods largely limit the plasticity of the model during continual learning, and we therefore
4

---

Published as a conference paper at ICLR 2025
contrastive knowledge consolidation
mm
Text encoder
D
A
man in blue wetauit surling
CLIP loss
Text or Image encoder
B.,
Image encoder
image-text pairs
Figure 2: Illustration of the C-CLIP model. We make adjustments to CLIP: (1) applying LoRA to reduce forgetting, though it slightly hampers new task learning; (2) introducing contrastive knowl- edge consolidation (CKC) improves learning on new tasks and reduces forgetting. This combination significantly enhances the continual learning ability of CLIP, achieving downstream task perfor- mance that matches or exceeds full fine-tuning and preserving its general zero-shot ability.
propose contrastive knowledge consolidation, which is designed for vision-language models. Our method is illustrated in Figure 2 and presented below in detail.
4.1 LORA INTEGRATION FOR FORGETTING MITIGATION
LoRA integration for VL.CL. To preserve the knowledge of previous tasks, we propose simply reducing the number of trainable parameters by leveraging parameter efficient tuning technique, i.c., low-rank adaptation (LoRA) (Hu et al.). As illustrated in Figure 2 LoRA composes of two rank decomposition matrices E R and A R=** where r E N is the rank and r < min(u, v). and u are the dimensionality of the input & I² for current layer and hidden features, respectively. In this work, we apply LoRA in both vision encoder fo and text encoder 9e of CLIP. At each continual stage the previous parameters (0'-11,²-11 remain frozen, while only the newly added A and B are trainable. However, since the model has no access to the task identity at inference, it is hard to decide which LoRA use. More importantly, storing all the LoRAs for previous tasks leads to increasing memory issues. Therefore, we propose to integrate the current (fLoRA (LLOA into the main backbone the end of each continual stage as follows:
{0 (-) (0'-1 + a OLORA "0111_1 + a (PLORA).
(2)
where a E [0, 1] is pre-defined coefficient to reduce the forgetting of knowledge after integration.
Theoretical analysis. the t-th continual stages, only the current dataset D' available, and an intuitive and general way to maintain the previous knowledge is to regulate the model by mimicking the input-output relation on previous (t 1) continual stages based on D follows:
(v,) ((fee))))
0.p
(3)
s.t. (( (If(v) -
E (o-
& 66030 0.
For an arbitrary N-layer MLP model, assume that the activation function is bounded and Lipschitz continuous, and the input and all weights have bounded norms, then the model is Lipschitz continu- ous with respect to the weights (Appendix A. provides detailed proof), i.e., (l - (o--(()) K (|0 11, then Eq. (3) can be rewritten as:
minE (m. A.P
((
(4)
s.t. 1 0t e/KK, - (/
where Ky and K2 are Lipschitz constant of vision encoder fo and text encoder 9p- Interestingly. LoRA freezes the old weights and introduces small set of parameters that achieve the goal of learning new tasks with ||0 8'-1|| & c/K,, |lp - & c/Kg. c 2 0. In other words, the proposed LoRA integration is simple way lear the objective of Eq. 44.
Empirical verification. In Figure 3 we compare the performance of LoRA integration with fine-
tuning, EWC (Kirkpatrick et al. 2017) ZSCL. (Zhang et al., 2024) and Mod-X (Ni et al. 2023

---

Published as a conference paper at ICLR 2025
Fine-tune
EWC
ZSCL
Mod-X
LaRA
C-CLIP
image-text recall
zero-shot accuracy
regularization loss
loss
80
Loss (Log Scale)
0.6
60-
10"
60-
10"
Log
0.4
55-
10"
50-
10-11
16"h
45
0.0
500
1000
500 1000
500
2000
0
500 1000
Global Step
Goal Step
Global Step
Global Step
Figure 3: Fine-tuning results on Flickr30K compare prior CL methods based on (a) image-caption recall, (b) ImageNet zero-shot accuracy, (c) regularization loss, and (d) CLIP loss. Similar to ex- isting regularization methods, LoRA reduces forgetting but sacrifices performance on new tasks. Regularization methods create a conflict between regularization loss and CLIP loss. In contrast, our method (C-CLIP) aligns the trends of these losses.
From the results, the performance on the current downstream dataset (Figure (a)), EWC, ZSCL and Mod-X ruin the zero-shot performance ImageNet-I1 IK (Figure (b)). (2) The proposed LoRA integration leads to less forgetting of zero-shot classification ability (Figure but the performance on the fine-tuned dataset is undesirable (Figure 3 (a)) Therefore, in the following subsection 4.2, propose novel contrastive knowledge consolidation strategy to improve both plasticity and stability.
draw the following key observations: (1) While all the methods improve
4.2 CONTRASTIVE KNOWLEDGE CONSOLIDATION FOR LEARNING ENHANCEMENT
In section 4.1, we have shown that LoRA integration can effectively reduce the forgetting of pre- vious knowledge. However, similar to other existing methods, the performance on new datasets is undesirable. This is because simply aligning the new model with the old feature space naturally performs poorly for new datasets and ignores the characteristics of multimodal tasks. Is it possible to reduce forgetting and improve new task performance simultaneously? To achieve this goal, our high-level idea optimize CLIP to learn better feature space from the old model, rather than just aligning with it. Technically, each image-text pair (of, c)) D' is mapped to deep feature space by both the old model {fo-1; : 9pp' } and new model {for-1; 9p then we propose contrastive knowledge consolidation (CKC) which consists two important points:
First, we introduce a projector hap Z - 2 after the vision and text encoders, optimizing the model in the projected space to keep the new and old feature spaces connected but not identical. This improves the plasticity for learning new tasks.
Second, for each image-text pair, its feature instances in the old, projected space is viewed as positive while that of others are viewed as negative. This remarkably increases positive and negative pairs. Training CLIP to align with the old projected features and away from the irrelevant features improves new task performance and mitigates forgetting.
Figure 4 presents a comparison of CKC and regularization loss, and the CKC loss for the t-th incremental task is follows:
1
2N
/- TT exp h2
2/-1/77
exp
-t-
14-1Th//+
Cxxc = 2N
log
+ log
E exp
TT (hi
29 2/-1/T)
2N E)=1 exp
12//7
(5)
Dhw (er (v:)), my (gpp (ct))
zィ [ (v:), Ii0- 1 (c:)
h; = I (fo (w;)), hyp (gp (c:)))l
[f (w:), - (c:
where [.] denotes the concatenation operation, N is the batch size, is the temperature parameter, and ||-I| denotes the Euclidean norm.
Total training objective. In addition, the original CLIP loss is used in the new feature space:
exp (21T2(,./7)) x(,//T Σn exp (2 ////7) Σn exp (26T 2 2/gg/)) (
exp (zt xt,e/T)
Ser = 2N log
Σ 1=1
+ log
(6)

---

Published as a conference paper at ICLR 2025
Foature confusion
Visian contrastive
Align with old space
Contrattion lramning after projection:
CLIP
500P
(a) Old Space
(b) CLIP Loss
(c) Regularization Loss
(d) CKC loss
Figure 4: Illustration of CLIP, CKC, and regularization losses: Previous methods align models old feature space, conflicting with CLIP optimization. CKC performs contrastive learning between old and new image-text samples in the projected space, aligning with CLIP loss.
(v)) and z (er(e:) (v))
where =
[e))( The final loss is L L'ke + CLL The total
training process can be summarized as follows: At each continual stage t, the model is augmented with LoRA and optimized with L"; At the end of stage t, the LoRA module is integrated into the backbone based on Eq. 2 (we simply use = 0.5 for all experiments of C-CLIP), Figure 3(c) method with other advanced CL methods. The results show that
compares the loss curve loss curve closely follows CLIP's trend, significantly differing from previous methods.
5 EXPERIMENTS
Table 3: Comparison results of image-texi retrieval trained datasets. We continually fine-tune
eight image-caption datasets, and then evaluate the performance after fine-tuning the final dataset.
Bickr3llk
simpsoos
COCO
lexica
wikiart
kream
sketch
gverage
per
Methods
C V11-
35.80
10.40
1.96
3.99
2.62
4.40
9.41
1.70
8.79
BWC
60.38
31.58
12.23
20.06
15.13
37.40
33.63
7.93
27.29
12T 12TR@T R@1
ZSCL
66.79
37.15
13.34
24.99
20.98
40.67
39.23
6,88
31.25
Mod-X
61.62
33.94
11.38
18.04
18.18
39.39
35.94
6.95
28.18
MOE-CL
63.63
34.11
12.45
23.03
17.03
41.09
38.67
7.27
29.66
DKR
63,60
28.98
1 11.04
21.74
19.05
38.02
37.92
7.06
28.43
C-CLIP
84.40
56.92
19.73
42.65
25.43
45.89
42.07
9.55
40.83 (99.58)
CLIP -iT-B/16
55.88
28.83
10.28
19.59
13.43
16.34
18,30
4.25
20.86
EWC
60.30
31.16
13.91
20.14
15.52
28.34
30.34
12.23
26.49
T21 R@L
65,52
39.85
14.22
25.37
19.91
34.76
34.99
11.98
30.85
Mod-X
61.16
35.37
13.01
21.84
17.24
30,98
28.62
10.66
27.36
MOE-CI.
63.17
40.31
15.45
24.09
16.36
35.78
34,04
11.02
30.03
DKR
62.72
37.73
14.94
23.60
20.00
31.31
30.91
10.26
28.93
C-CLIP
73.74
42.82
17.91
41.47
24.32
45.27
43.57
14.67
37.97
EWC
Mod-X
ZSCL
MOE-CL
DKR
C-CLIP
Finetune
Flickr30k (task1)
COCO (task2)
Pet (task3)
HaVG (unseen)
Image-te text R[1
84
60
Image-text R@1
21
24
18
36
12
15
54
(hage-k
30
Image-1
12
48
9
42
12
Index
Task Index
Task Index
Task Index
Figure 5: Continual fine-tuning performance on Flickr30k, COCO, Pets. HaVG is an unseen dataset.
Networks and comparison methods. We use CLIP (ViT-B/16) with pre-trained weights from
large-scale open-world datasets as the backbone. achieves a zero-shot classification accuracy of 67.73% on ImageNet-IK. The comparison methods include: (1) Classical CL methods like EWC KKirkpatrick et al. 2017); (2) MTIL methods like ZSCL (Zhang et al., 2024) and MOE-CI Yu
al. 2024), and (3) other recent methods like Mod-X (Ni al. 2023) and DKR (Cui et al.) 2024

---

Published as a conference paper at ICLR 2025
Table Zero-shot accuracy of ImageNet- IK (ZS-IIK) and CIFAR-100 (ZS-C100) during continual fine-tuning on eight datasets. Our method maintained high zero-shot performance.
Task ID
Method
PD (
1
EWC
67.73
$6.85
50.67
43.43
46,97
48.10 48.10
46.19
44,50
40.76
26.97
ImageNet
ZSCL
67.73
59.10
56.51
52.47
51.74
51.03
50.89
50.02
49.60
18.13
Mod-X
67.73
57.63
52.05
48.15
47.53
47.79
48.76
46,63
44.21
23.52
MOF-CL
67.73
58.82
54.29
49.08
49.75
50.33
50.61
48.88
47.05
20.68
DKR
67.73
58.54
53.69
49.55
47.80
47.55
49.67
45.46
45.88
21.85
C-CLIP
67.73
63.11
65.31
63.26
63.31
61.95
62.13
61.51
60.31
(.42(a10.71) 10.71)
EWC
66.87
55.85
53.88
46.08
42.73
47,58
47.63
43.93
42.79
24.08
CIFAR100
ZSCL
66.87
58.70
55.06
51.28
47.37
50.92
$2.65
53.02
52.19
14.68
Mod-X
66.87
56.91
55.46
48.50
44.04
43.43
49.82
49.49
44.34
22.53
MOF-CI.
66.87
58.06
58.49
50.13
45,29
48.42
50.75
$0.11
49.97
16.90
DKR
66.87
$7.06
56.72
49.00
44,49
42.15
48.09
48.44
46.45
20.42
C-CLIP
66.87
63.17
64.78
64.08
61.02
60.55
62.85
62.02
61.58
5.29 (+9.99)
ZSCL
Mod-X
MOE-CL
C-CUP
CLIP
Flowers
DTD
Food101
StanfordCars
80
60
40-
40
40
20
20
20
EWC
25CL
Mod-X
DKn
C-u
CLiP
EWC
z5a
Mod-f
OKR
CCCUP
CUP
EWC
ZSCL
Mod
DKA
CCu
CLIP
EWC
25c1
Mod-K
OkR
Ccu
CUP
MOE
NOE-C
MOE
MOE
Figure 6: sero-shot accuracy of Flowers, DTD, Food101, and Stanford Cars after continual fine-
tuning on eight image-caption datasets. C-CLIP significantly outperforms previous CL methods
Implementation details. C-CLIP is implemented in PyTorch lightning and trained on 8 NVIDIA 4090 GPUs with batch size of 1024. Input images are resized to 224 x 224, and we train for 40 epochs on each dataset. The initial learning rate is set to x 10-6 with a 5-epoch warm-up using a cosine-decay learning rate scheduler. The low-rank decomposition (R) of LoRA is set 16, with scaling factor of 2 x It and dropout of 0.1. We use the Adam W optimizer with 31 0.9, Bg = 0.99, and weight decay of 0.2. Learning rates are adjusted per dataset; for example, on COCO-caption (Chen et al. 2015)). the image encoder's learning rate is 5 x 10 10-7, and the text encoder's is 10 More details are provided in Appendix A.2
5.1 MAIN RESULTS
Comparison on trained datasets. As shown in Table the CLIP model pre-trained on large-scale datasets performs poorly unseen datasets. This aligns with findings in (Zhang al. 2024 Ni
et al.] 2023). However, full fine-tuning of these datasets may cause severe forgetting. For exam- ple, after fine-tuning twice on Flickr30K and COCO, the ImageNet zero-shot accuracy dropped from 67% to 25%. Previous CL methods show considerable gaps in new tasks when compared to full fine-tuning, sacrificing new task performance to mitigate forgetting. Our method significantly outperforms past approaches on new tasks. For instance, on datasets like Flickr30K and COCO, it greatly surpasses full fine-tuning in image2text retrieval performance. Moreover, As shown in Figure 5 C-CLIP improves performance on old tasks new tasks are learned, which is notable difference from previous CL methods. This demonstrates that our method can improve both new and old task performance during continual fine-tuning.
Comparison on unseen datasets. It is obvious that fine-tuning improves performance on the trained datasets. However, as shown in Figure 5 we observe that fine-tuning some task-specific datasets improves image-text retrieval performance on unseen datasets. Previous CL methods have shown a similar trend, but their training is unstable. For example, when fine-tuned on Al Al-generated datasets like Lexica, these methods suffer significant performance drops on real-world datasets like COCO and HaVG. This indicates that training on Al-generated datasets causes the model to forget its per-
8

---

Published as a conference paper at ICLR 2025
Table 5:
The effectiveness of each component in C-CLIP. The combination of LoRA and CKC results in better performance on new tasks with less forgetting.
flickr30k (task 0)
COCO (task
Method
12T R@
T21 R@
7S-TIK
7S-C.100
12T R@
T21 R@
7S-TIK
7S-C100
CLIP VIT B/16
35.80
55.88
67.73
66.87
10:40
28.32
67.73
66.87
Fine-tune
74.20
76.00
47.46
49.67
43.26
44.86
23.87
25.65
LoRA
66.90
73.10
62.18
62.53
38.17
40.55
61.49
61.03
CKC
81.39
76.16
51.24
53.08
45,63
43,95
45.67
49.72
LoRA & CKC
80.90
75.08
63.11
63.17
43.04
42,02
65.31
64.78
40
Snd, 46-4
50
Flickr30k
70%
COCO
Val12T R0)
30
105,
1e5, Band
46.4
45
165. 4e
S 4e
1c5. 8c
90%
2002 CKC
0001
LWF
Mod-X
60%
0001
Mod-X
CKC
20-
204, 4e-4
70%
50%
205, 20-5
25
2. 4e
10-
30
2e5, 2e
1000 2000 3000
Train Step
4000
1000 2000
Train Step
4000
50%
121
Z5-KK
30%
121
kg9
Z5-K
Figure 7: Impact of Learning rate: (left) image- to-text recall: (right) text-to-image recall.
Figure 8: Impact of Regularization losses on Flickr30K (left) and COCO-caption (right).
formance in real-world domains. Our method exhibits impressive performance, when trained on Al-generated datasets in Task 4, the model also improves unseen real-world datasets. This sug- gests that C-CLIP can continuously fine-tune using data from different domains.
Zero-shot Classification ability. Table A and Figure 6 verify that previous CL methods useful for maintaining zero-shot ability, showing that their strategy of minimizing updates to feature space helps reduce forgetting (though it may impact performance on new tasks). Our method demonstrates the best anti-forgetting performance. For instance, after two tasks fine-tuning from Flickr30K to COCO, the ImageNet zero-shot accuracy remained 65.1%, very close the original 67.7%. When continuously finettunig 8 datasets, C-CLIP still maintained high zero-shot performance.
5.2 ABLATION STUDY AND FURTHER ANALYSIS
Impact of components in C-CLIP. As shown in Table 5 we evaluated the impact of each com- ponent C-CLIP. LoRA significantly reduces forgetting but compromises new task performance, which is similar to previous CL methods. CKC improves both the learning ability for new tasks and reduces forgetting. However, CKC is not a strong constraint, so the forgetting remains rela- tively high. By combining CKC with LoRA, not only is forgetting further reduced, but new task performance matches or even exceeds that of full fine-tuning.
Impact of learning rate. Figure shows results on COCO with vary- ing learning rates. It can be seen that 1TTR@ 1 is significantly affected by the learning rate. When the learn- ing rates for the text encoder and im- age encoder are kept the same, CLIP training causes a degradation in the 12TR @1 1 metric. Therefore, we set the text encoder's learning rate to 80 times that of the image encoder to achieve optimal results. Further configuration details be found in Appendix A.2
Table 6: Comparison results of parameter count and model performance with different LoRA ranks (R).
Method Params
12T R@1
T21 R@1
ZS-11K
ZS-C100
Fine-tune
149M
74.20
76.00
47.46
49.67
R = 8
27.6M
80.42
74.81
63.05
63.13
R 16
29. IM
80.90
75.08
63.11
63.17
R 32
32.0M
80.95
75.01
63.03
63.15
64
37.9M
81.11
75.03
62.88
63,09
Impact of regularization losses. In Figure g we compare different regularization losses with CKC. Intuitively, these regularization methods, such as EWC, LWF, Mod-X, and LoRA, have similar effect by reducing changes in feature space. However, experimental results show that these losses not casily compatible with LoRA. When using LoRA, we reduced the coefficients of these reg- ularization losses, but their performance remained poor, leading to worse new task performance and more severe forgetting. In contrast, CKC integrates much better with LoRA.

---

Published as a conference paper at ICLR 2025
Table 7: The effectiveness of our method across various ViT backbones.
flickr30k (task0)
COCO (task
Backbone
Method
I2T R@I
R@
ZS-IIK
ZS-C100
R@
ZS-IIK
ZS-C100
Vanilla
31.10
52.94
62.59
61.24
8.30
25.53
62.59
61.24
VIT-B/32
C-CLIP
79.34
73.05
58.72
58.93
42.57
41.46
59.95
59.61
Vanilla
35.80
55.88
67.73
66.87
10.40
28.32
67.73
66.87
VIT-B/16
C-CLIP
80,90
75.08
63.11
63.17
43.04
42.02
65.31
64.78
Vanilla
59.10
61.66
75.01
76.78
25.86
33.14
75.01
76.78
VIT-L/14
C-CLIP
92.59
79.40
70.56
70.90
64.22
72.18
72.24
VIT-L/14@336px
Vanilla
59.30 93.15
63.38
75.76
75.95
25.70
33.60
75.76
75.95
C-CLIP
79.63
71.39
70.04
64.59
52.31
72.74
71.21
Table 8: Compare our method with other prompt-based continual learning approaches.
Method
Task
5
121:11 2S-11K
LeRA
62.18
61.49
58.47
57.74
55.03
$3.89
52.02
$1.02
L2P
$9.26
60.37
58.74
58.51
57.85
56.41
57.21
56.04
CPE-CLIP
57.36
59.87
59.13
58.58
56.61
56.11
55.45
55,40
C-CLIP
63.11
65.31
63.26
63.31
61.95
62.13
61.51
60.31
LoRA
66. 90
60.83
55.71
58.04
61.85
$9.85
62.29
63.28
L2P
58.82
52.88
47.61
48.01
46,29
46.93
48.54
47.74
CPE-CLIP
61.55
57.78
53.80
54.14
54,36
52.79
51.23
52.87
C-CLIP
80.91
82.07
83.62
83,29
83.13
85.05
85.58
84,40
12T @RI
LoRA
63,28
38.91
11,32
28.69
18.94
37.89
34.37
6.63
1.2
20.73
23.91
CPE-CLIP
52.87
23.72
8.19
17:42
13.96
28.99
27.15
5.44
C-CLIP
56.92
19.73
25.43
42.07
9.55
Parameters and LoRA setting. comparess the trainable parameters between full fine-tuning and various LoRA configurations. LoRA significantly reduces the number of trainable parameters, and most of the trainable parameters are embedding layers. As seen from the results of Task 0 and Task 1, different LoRA settings perform similarly in learning new tasks and preventing zero-shot forgetting. For simplicity, set the LoRA rank to 16 in our experiments.
Evaluation across ViT architectures. In our main experiments, we use ViT-B/16 with a hatch size of 1024. To demonstrate the effectiveness of C-CLIP across different architectures, we also test it on other backbones, including ViT-B/32, ViT-L/14, and NTT-L/14@336. Due to memory constraints, we reduce the batch size to 256 for the larger VYT-L models. As shown in Table 7 ViT-L significantly outperforms ViT-B, with substantial improvements in zero-shot performance on downstream datasets. Our method performs well across various VIT architectures, maintaining strong zero-shot capabilities and excelling in image-text retrieval on downstream tasks.
LoRA vs. Prompt-tuning. Prompt-based continual learning methods pre-trained models and only train prompts. We evaluate two representative methods, L2P (Wang et al., [022] and CPE- CLIP (D Alessandro et al. 2023). Table reveals two key findings: (I) Prompt-tume tuniig preserves general zero-shot performance
ImageNet-IKK whereas LoRA alone suffers from increased for- getting as tasks accumulate. (2) However, prompt-tuning does not prevent forgetting in downstream tasks, as updating prompts for new tasks leads to significant forgetting of prior ones ones-foo for instance, performance on the first task (I2T-T1) and all tasks (I2T@RI) R1) is notably worse than LoRA after learning all eight tasks. In summary, while prompt-tuning preserves the original model's knowl- edge, it struggles to learn and retain new tasks. In contrast, combining LoRA and CKC proves to be more effective in this scenario.
CONCLUSION
This work focuses on the continual learning of visual-language models. We establish a multimodal continual learning benchmark and call for evaluating the performance from three different aspects. Then, we propose C-CLIP that prevents forgetting and enhances new task learning impressively with LoRA integration and contrastive knowledge consolidation. Comprehensive experiments demon- strate that the proposed C-CLIP outperforms existing ofate-of-the-art methods and achieves strong multimodal continual learning performance across image-text datasets from various domains.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
This work has been supported by the InnoHK program.
REFERENCES
Idris Abdulmumin, Satya Ranjan Dash, Musa Abdullahi Dawud, Shantipriya Parida, Shamsud- deen Hassan Muhammad, Ibrahim Sa'id id Ahmad, Subhadarshi Panda, Ondřej Bojar, Bashir Shehu Galadanci, and Bello Shehu Bello. Hausa visual genome: A dataset for multi-modal english to hausa machine translation. arXiv preprint arXiv:2005.01133 2022.
Rahaf Aljundi. Continual learning in neural networks. ArXiv, abs/1910.02778, 2019.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433 2015.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. food-101-mining discriminative compo- nents with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings. part 13, pp. 446 446-461. Springer, 2014.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:/504.000 00325, 2015.
Pinaki Nath Chowdhury, Aneeshan Sain, Ayan Kumar Bhunia, Tao Xiang, Yulia Gryaditskaya, and Yi-Zhe Song. Fs-coco: Towards understanding of freehand sketches of common objects in text. In European conference on computer vision, pp. 253-270. Springer, 2022.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Zhenyu Cui, Yuxin Peng, Xun Wang, Manyu Zhu, and Jiahuan Zhou. Continual vision-language retrieval via dynamic knowledge rectification. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pp. 11704-11712, 2024.
Marco D' Alessandro, Alberto Alonso, Enrique Calabrés, and Mikel Galar. Multimodal parameter- efficient few-shot class incremental learning. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3393-3403, 2023.
Jia Deng. Wei Dong, Richard Socher, et al. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Shibhansh Dohare, Fernando Hermandez-Garvia, Qingfeng Lan, Parash Rahman, A Rupam Mah- mood, and Richard Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026): 768-774, 2024.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet:
Pooled outputs distillation for small-tasks incremental learning. In Computer vision vision-ECCCV ECCV 2020: 16th European conference, Glasgow, UK, August 23-28 28, 2020. proceedings, part XX 16, pp. 86- 102. Springer, 2020.
Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
11

---

Published as a conference paper at ICLR 2025
Xinyuan Gao, Yuhang He, Songlin Dong, Jie Cheng. Xing Wei, and Yihong Gong. Dkt: Diverse knowledge transfer transformer for class incremental learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattemn Recognition, pp.
2023.
Haiyang Guo, Fei Zhu, Wenzhuo Liu, Xu- Yao Zhang, and Cheng-Lin Liu. Pilora: Prototype guided incremental lora for federated inasemnnrem learning. In European Conference on Computer Vision, pp. 141-159. Springer, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVE conference on computer vision and patlern recognition, pp. 831-839, 2019.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and ussionllaguaae representation learning with noisy text supervision. International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Xisen Jin, Junyi Du, Arka Sadhu, Ram Nevatia, and Xiang Ren. Visually grounded continual learn- ing of compositional phrases. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 201 2029,, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A et al. Overcom-
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka ing catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-3522 2017.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13). IEEE, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947) 2017.
Zhuowei Li, Long Zhao, Zizhao Zhang, Han Zhang, Di Liu, Ting Liu, and Dimitris N Metaxas.
Steering prototypes with prompt-tuning for rehearsal free continual learning. In Proceedings of the IEEE/CVF Winter Conference on Applications Computer Vision, pp. 2523-2533 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024.
Wenzhuo Liu, Xin-Jian Wu, Fei Zhu, Ming-Ming Yu, Chuang Wang, and Cheng-Lin Liu. Class in- cremental learning with self-supervised pre-training and prototype learning. Pattern Recognition, 157:1, 2025.
Mare Masana, Xialei Liu, Bartlomiej Twardowski. Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image clas- sification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513-5533, 2022.
Zixuan Ni, Longhui Wei, Siliang Tang, Yueting Zhuang, and Qi Tian. Continual vision- language representation learning with off-diagonal information. In International Conference on Machine Learning, pp. 261 29149.. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large num- ber of classes. Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, pp. 722-729, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498-3505. IEEE, 2012.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to to--to--ppaa co-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference computer vision, pp- 2641-2649, 2015.
Zi Qian, Xin Wang, Xuguang Duan, Pengda Qin, Yuhong Li, and Wenwu Zhu. Decouple before interact: Multi-modar modal prompt learning for continual visual question answering. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2953-2962, 2023.
Jingyang Qiao, Xin Tan, Chengwei Chen, Yanyun Qu, Yong Peng, Yuan Xie, al. Prompt gra- dient projection for continual learning. In The Twelfth International Conference on Learning Representations, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Dhanesh Ramachandram and Graham W Taylor. Deep multimodal learning: A survey on recent advances and trends. IEEE signal processing magazine, 34(6):96-108, 2017.
Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39(6): 133-11199 2016.
Babak Saleh and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arXiv preprint arXiv:1505.0088: 2015.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Brraska--arwwine Yee Whye Tch. Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for contin- ual learning. In International conference on machine learning, pp. 4528 4537. PMLR, 2018.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemnymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1556-256s 2018
Xinyue Shen, Yiting Qu, Michael Backes, and Yang Zhang. Prompt Stealing Attacks Against Text- to Image Generation Models. In USENIX Security Symposium (USENIX Security). USENIX, 2024.
Tejas Srinivasan, Ting- Yun Chang, Leticia Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. Climb: A continual learning benchmark for aisoon-nnd-aaguuun tasks. Advances in Neural Information Processing Systems, 25:29440-29455 2022.
Yu-Ming Tang, Yi-Xing Peng, and Wei-Shi Zheng. When prompt-based incremental learning does not meet strong pretraining. In Proceedings of the IEEE/CVF International Conference on Com- pufer Vision pp. 1706-1716. 2023.
13

---

Published as a conference paper at ICLR 2025
Gido M Van de Ven, Tinne Tuytelaars, and Andreas Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12):1185-111 1197, 2022.
Kai Wang, Luis Herranz, and Joost van de Weijer. Continual learning in cross-modal retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3628- 3638, 2021.
Liyun Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu, Hierarchical de- composition of prompt-based based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems, 36, 2024.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 149, 2022.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3014-3022 2021.
Jiazuo Yu, Yunzhi Zhuge. Lu Zhang, Ping Hu, Dong Wang. Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 222330,, 2024.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pp. 3987-3995, PMLR, 2017.
Jingyi Zhang. Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-languaa models for vision tasks: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 19125-19136, 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PROOF OF LIPSCHITZ CONTINUOUS
Let the function
fw, x)) = (wn, a (wn-1 a (...o (w1 -x)))),
(7)
where w = {wi, W2, Wn.. , 2 is an input vector, wy are weight matrices, and a is an activation function. Assume that the activation function o is bounded and Lipschitz continuous; that is, there exist constants Mo. Lo > such that for all 2,
(x))) Mo. (or(u) ( S |u ju- u/.
(8)
Also, the input x and all weights wy have bounded norms; that is, there exist constants M2, Mi >> such that
||||||||||| < Ma.
||//|| S Mw.
(9)
where || || denotes the Frobenius norm. Under these conditions, we aim to prove that the function fww, x) is Lipschitz continuous with respect to the weights w; that is, there exists constant K > such that for any w and w',
[l x) f(w', x))l - K|[w w'll..
(10)
To prove this proposition, First, consider the base case n = 1. In this case, the function simplifies to 2 = (w).). For weight matrices to and w/j, we have
[[ x) - f(w(, = (l)(w - ow)
(11)
Since is Lipsehitz continuous,
[o((]æ) - =(w(x)]1 S Lg || - w/ w/
(12)
Moreover,
|| - w|=x| = [[( - w( >=||
$ ||l| - w/ ||||||]]] ||||||
(13)
s M |mm w/ |
therefore,
[l x) fw11 x)]l s L, k| - wille- This shows that when n = 1, f(w1,z) is Lipschitz continuous with respect to wi, with Lipschitz constant K1 L. Ma-
(14)
Next, assume that for n k, the function (w(wi x) is Lipschitz continuous with respect to wisk {Wi, w.,..., w J; that is,
(lfa( x) - fie (wi m) I f w/.ellr.
(15)
Now, we need to prove that the conclusion holds for + For ke + 1, the function is
[ww x) = a (wx+ (WInR
(16)
Considering two sets of weights w { [wi:k wx+1] and w/ {wise w we have
[] 2) - fw', m = ||o (wa-1sk (Wis, x)) - (wh+1 Using the triangle inequality and the Lipschitz continuity of o, obtain
(17)
xw - f(w', æ)]l < L. (/|wa fu (Wink x) w fu (004iff(w((.133 (w
+ +||We+1 (u(Wia,, whor fu x))
(18)
L., ([]wa+1 (fe(Wia, fa (wi x))
+ [1(++1 - wh+r) Sa (w
For the first term,
|| (fa(Wa,,z) - fa fu z))1| < ||lll>>|(aaa Ill [l (wF,, 1:0--fuul fa
(19)
< Ma Kr W - whulFF.
15

---

Published as a conference paper at ICLR 2025
For the second term,
[1 - w/4+1) Su(( fu(W x)|l $ ||0++1 Wh+1 1 |1
II (w x:)l
(20)
[0++1 wh+1ll M.
where M, is the boundedness constant of fa. Combining the results above, we have
Ilf(w, x) - f(w', (''' S Lo (M Kellwill w/allr r + My Myllea+1 wh+1llr).
(21)
Since
1w1 - w'= S I|w w'lFs 110+11 - whtil I 1|w - W'IFF
(22)
can let
Ke+1 Lo (M Kz + Mf).
(23)
thus obtaining
(w,( f(w',x)l K&+1 l|w w'll
(24)
This shows that when k+1, (w, x)) is Lipschitz continuous with respect to w. By mathematical induction, we conclude that for any n, the function f(w,z) x) Lipschitz continuous with respect to the weights
(a) Pets
(b) Kream
(c) Simpsons
SNACKS
"outer, Patagonia Classic Retro-X "a small dog with black Jacket Natural, photography of and white coat."
"A man dressed Santa Claus stand next to a woman.
tan jacket with blue pocket."
(d) Wikiart
(e) Sketch
(f) Lexica
"A dark landscape with bright moon and large central tree.*
'People are playing and walking in park."
'ultra realistic delorean dnc drifts road wzeckag..."
Figure 9: Examples of image-text data from different domains in the VLCL benchmark.
A.2 ADDITIONAL EXPERIMENTAL DETAILS
In Figure we present image-text data from different domains in the VLCL benchmark. Some image caption datasets have prede- fined splits, such as Flickr30K and COCO capticn, with test sets of IK and 5K, respectively. For Pet, Lexica, and HausaVG, we eval- uate their test sets. For other datasets like Simpsons, Sketch, and Wikiart, we randomly split 80% for training and 20% for testing. For Kream, the training and test sets are evenly divided. Figure 3 shows examples of tasks from different domains in the VLCL benchmark. During training, all images are resized to 224x224, and the maximum text length set to 77.
Table 9: Comparison of train- ing time (Epoch/s).
Method
flickr30k
COCO
Fine-tune LWF
13.15
44.58
15.72
53.20
Mod-X CKC
16.91
57.13
16.59
56.07
Regarding learning rates, we tested several values. For COCO-caption, we set the learning rate for the text encoder to 80 times that of the image encoder, while for other datasets, it was set to 10 times.
16

---

Published as a conference paper at ICLR 2025
The base learning rate was 5e-7 for COCO-caption le-5 for Flickr30K, and 3e-5 for other datasets, from Pet to Sketch. Our method's 's code will be open-sourced, and further details can be found the released code.
A.3 TRAINING EFFICIENCY COMPARISON
We assess the efficiency of CKC and other regularization losses by comparing their average per- epoch training times over 40 epochs. As shown Table the additional losses introduce some over- head. For instance, on COCO, LWF and Mod-X increased training time by 19.3% and 28.2% re- spectively. The computational overhead of our method is comparable these regularization losses, leading to 25.2% increase in training time.
A.4 LIMITATION
Our method is mainly designed for contrastive learning based version-language models like CLIP. More advanced multimodal large language models like LLaVA (Liu et al. 2024) typically use lan- guage modeling autoregressive loss. Nevertheless, considering the version encoder in LLaVA is from CLIP and plays an important role in LLaVA, we believe the proposed C-CLIP can also en- hance the vision ability of multimodal large language models. In the future, we will explore contin- ual learning for generative VLMs like LLaVA.
17


Reviews:
[{"review_id": "d8wmBguOPh", "reviewer": "Reviewer_v9wt", "summary": "Brief Summary: The paper tackles the task of continual learning for a VLM. In particular, the authors introduce new datasets from image captioning datasets (coco, flickr30k) and the model is evaluated on a range of vision-language datasets on retrieval and zero-shot classification tasks. \n\nThe key idea here is that the authors show that LoRA (low rank adaptation) achieves a similar effect as storing and replaying old data. Further, that contrastive learning loss reduces forgetting. \n\nExperiments on a number of datasets show that the proposed method (C-CLIP) trained can match or exceed full-finetuning results. Moreover, C-CLIP outperforms existing baselines by significant margins.", "questions": "Q1. It is a bit unclear to me if the model is provided the task id in the data-stream. For instance, during training does it know that the task is say pet-classification or is that information hidden.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Pros:\n\n1. The key idea of using LoRA as a replacement for memory bank and adding contrastive learning is easy to implement and is modular. As a result, it is promising that such method can be used to enable continual learning across a range of VLM models. \n\n2. The authors contribute dataset and benchmarks. The performance improvement over competitive baselines is very significant (in range of 7-10 points). \n\n3. The authors provide comprehensive set of experiments across multiple datasets and multiple baselines. The ablation study is also solid, particularly interesting is that Contrastive Loss (CKC) by itself provides significant improvement on its own (Table 4).\n\n4. The authors provide qualitative results visualizing their model (in appendix)", "weaknesses": "Cons:\n\n1. The authors should also show results with more multi-modal models and not just CLIP ViT-B/16, such as other CLIP variants with different backbones (say ViT-L/14). \n\n2. The authors could also explore more vision-language tasks, in particular that of captioning/description with generative VLMs which output a description (such as LLaVA). \n\n3.  (Minor) The authors have missed a close work [Ref1] which also explores the setting of multi-modal VLMs in continual learning setting. That said, the main contributions on modeling side are quite different.\n\n\n=======\n\n[Ref1]: Jin, Xisen, Junyi Du, Arka Sadhu, Ram Nevatia, and Xiang Ren. \"Visually grounded continual learning of compositional phrases.\" arXiv preprint arXiv:2005.00785 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CCD9wezl2g", "reviewer": "Reviewer_8g5w", "summary": "This paper presents C-CLIP, a new continual learning framework that aims to reduce the forgetting in Vision-Language Models (VLMs) while achieving high fine tuning accuracy on new datasets, specifically focused on CLIP. The authors also introduce a benchmark for Vision-Languge Continual Learning to evaluate CLIP’s performance in retaining zero-shot capabilities while acquiring new domain knowledge. To achieve that goal, C-CLIP combines Low-Rank Adaptation (LoRA) to reduce forgetting and Contrastive Knowledge Consolidation (CKC) to retain old knowledge. Experiments show that C-CLIP achieves high accuracy in downstream tasks while maintaining zero-shot classification performance across different domains.", "questions": "- Could the C-CLIP approach be generalized to VLMs beyond CLIP, or are specific adjustments required?\n- Could the authors further elaborate on why CKC avoids the trade-offs associated with other regularization methods?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper shows an innovative approach to continual learning within Vision-Language models, by focusing on the adaptation of new domains while retaining the zero-shot capabilities, with some key strengths:\n- The integration of LoRA adapters and CKC is a good approach to achieving robust continual learning in VLMs, especifically capable of retaining zero-shot capabilities\n- Experiments are performed with evaluations across a wide range of both seen and unseen domains, which highlight the method's effectiveness in preventing catastrophic forgetting.\n- C-CLIP advantages against other CL methods are clearly demonstrated through detailed figures and tables, facilitating the understanding of the experimental results.\n- A method that uses a Vision-Language model like CLIP in multimodal settings is highly relevant given the actual trend of focusing just on unimodal CL scenarios.", "weaknesses": "The paper also shows some weaknesses that could improve the overall result:\n- While LoRA reduces significantly trainable parameters, it would be interesting and beneficial to include an analysis of the computational costs involved in using CKC during the fine tuning process.\n- Although comparisons are made against baseline and some recent CL approaches, there is no comparison against other well known parameter efficient CL methods (e.g., DualPrompt, CPE-CLIP, Learning to Prompt, etc) that could strengthen the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6csZlMQSJC", "reviewer": "Reviewer_L3rn", "summary": "- This paper addresses continual learning scenarios for vision-language models, focusing on cross-modal retrieval across seen and unseen datasets.\n- The proposed approach combines two key components: (1) low-rank adaptation (LoRA) training with theoretical analysis of its connection to continual learning objectives, and (2) contrastive knowledge consolidation (CKC), which distills knowledge from previous task models as an auxiliary loss.\n- Experimental analysis is conducted with nine vision-language datasets, including one unseen dataset, where the proposed C-CLIP consistently outperforms competitive baselines. Additional experiments on zero-shot image classification and ablation studies further validate the method's effectiveness.", "questions": "Please refer to the weaknesses section for major questions. Here are some minor comments about the paper:\n- Eq. 23 is recursive with respect to K, could this bound serve as a meaningful bound? (could be considerably large for some nontrivial L and M?)\n- L96 - cite correct paper (pioneering works, not recent ones)\n- Is $h_\\psi$ in L277 used elsewhere after training? Why is an identical projector used for both vision and text embeddings in equation 5?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The application of continual learning to vision-language scenarios addresses important practical needs.\n- The proposed method effectively combines LoRA with knowledge distillation for vision-language continual learning, which sounds reasonable and is supported by theoretical analysis.\n- The paper presents strong experimental results across a comprehensive range of benchmarks, demonstrating consistent performance improvements over baselines and thorough evaluation across different experimental setups.", "weaknesses": "- Delivery should be significantly improved. Here is a non-exhaustive list I have identified while reading the paper, but I believe there are plenty of points that need to be improved:\n\t- What CIL and MTIL stands for? (I don't find Table 1 to be really necessary, though.)\n\t- Figure 2 does not effectively convey what CKC is about.\n\t- The range covered by the y-axis in Figure 3-(c) seems too wide, making it hard to see what is happening locally. Consider normalizing and clamping. Also, it would be interesting to see if such a behavior, i.e., alignment in main and auxiliary losses, is universally observed in other tasks.\n\t- What does Figure 3 in L284 refer to?\n\t- Please assign proper variables for equation 5 and 6, instead of using $z$ everywhere. This makes the equations somewhat confounding, e.g., substituting $t$ in  $\\tilde{z}_i^t$ with $t-1$ is not identical to $\\tilde{z}_i^{t-1}$, etc.\n\t- Many typos, e.g., space is view as (L280), loss is use in the (L294), etc.\n- While performance improvements in all major experiments are impressive (compared to performance gap among prior arts), it is hard to tell whether these results are reproducible and rigorously analyzed.\n\t- The codes are not provided, and the details in the paper may not be sufficient to fully reproduce the experiments reported.\n\t- Some important hyperparameters are not tested properly, e.g., $\\alpha$ for LoRA integration, ratio of CKC loss and CLIP loss, etc. The performance gaps in Table 4 for each row is quite large, making it hard to understand at which point these improvements emerge.\n\t- Only a subset of tasks are reported in major figures, e.g., Figure 3, 4, 6, 7 and Table 4, 5. It would be beneficial to see if a similar tendency holds for the rest of the tasks during training end-to-end (at least for some crucial setups like ablation studies).\n- Distillation in continual learning is not new, which was addressed in Contrastive Continual Learning (ICCV 2021) to the best of my knowledge. It would be necessary to provide some theoretical or empirical comparison between CKC and previous lines of work leveraging similar concepts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PG8zRce91W", "reviewer": "Reviewer_45wV", "summary": "The paper introduces a rich benchmark for general vision-language continual learning focused on evaluating performance degradation in zero-shot generalization as well as class-incremental performance. The paper also introduces a modified CLIP model to account for some limitations of similar models in the literature. The proposed model, C-CLIP, uses LoRA adaptation and Contrastive Knowledge Consolidation as the two main advancements of the standard CLIP backbone. The proposed model outperforms current SOTA models in preserving knowledge and zero-shot performance degradation.", "questions": "I think that LoRA can be compared with other parameter-efficient methods that keep the main backbone frozen like prompt-tuning (e.g. Khattak et al (2022). MaPLe: Multi-modal Prompt Learning), unless there is a particular reason with a mathematical justification for keeping LoRA. But in my opinion, eq. 4 holds with any other non-adapters parameter-efficient methods, and CKC is not constrained by the backbone adaptation. I understand that the authors might complain that LoRA is just a design choice, as any other else, but the paper lacks a clear justification for that since eq. 2 and eq. 4 hold for prompt-tuning too, and the intention here seems just preserving knowledge by freezing the backbone. CKC seems a clearer design choice on the other hand. \n\nDo you agree with this view, or am I missing something and maybe LoRA is already justified so there is no need to explore other parameter-efficient methods?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall, I positively evaluate the proposed manuscript. I like the idea of proposing a complete evaluation benchmark encompassing several strategies for measuring zero-shot generalization capabilities in continual learning settings. The benchmark is rich enough to increase the challenge that continual learning literature normally faces when proposing novel models evaluated on limited and strict benchmark datasets.", "weaknesses": "(1) The introduction needs a significant extension. Authors keep claiming that vision-language continual learning is under-explored, but they missed relevant research papers. Some examples are:\n\n- Qian et al (2023). Decouple before interact: Multi-modal prompt learning for continual visual question-answering\n- Wang et al (2021). Continual learning in cross-modal retrieval\n- D'Alessandro et al (2023). Multimodal Parameter-Efficient Few-Shot Class Incremental Learning\n- Srinivasan et al (2022). Climb: A continual learning benchmark for vision-and-language tasks\n\njust to name a few. I would also mention the general literature in FSCIL (Few-Shot Class-Incremental Learning) as well. All these works challenge vision-language continual learning from different perspectives and provide insightful feedback on specific problems encountered in multimodal settings.\nFurthermore, contrastive learning pre-training offers more nuances compared to a standard unimodal continual learning scenario, since it can solve image classification as well as pure multimodal tasks, as mentioned in the main paper. But it's clear that the burden of solving forgetting and adaptability problems is totally different in the two tasks, since in the former the language encoder needs to perform representational learning only on fixed text labels to pair with images. This consideration comes directly when considering the papers already mentioned in the current introduction and the ones I mentioned before. The proposed one is a comprehensive benchmark, but the introduction needs to clarify these concepts.\n\n(2) I think the overall benchmark definition is confused and badly explained. First of all, the authors want to clarify the role of each benchmark evaluation aspect. Some datasets are used in a continual learning fashion, with incremental experiences, others are used without the temporal factor, and you have to reach section 5.2 to figure it out. This is a benchmark that, I assume, the authors want to propose to unify evaluation criteria across VL CL models. I suggest putting a section in the main text (a table if it is necessary) clarifying how the benchmark is made, by referring to the continual learning datasets stats (e.g. number of splits, classes per split) and datasets role (e.g. continual learning, zero-shot generalization not in continual learning scenario). If there are space issues, I think there are sections that can be sacrificed since clarifying the benchmark is crucial (e.g. \"Theoretical Analysis\" section can be safely moved into the appendix).\n\n(3) Since the proposed model uses a parameter-efficient strategy, I think it's worth mentioning the whole corpus of rehearsal-free continual learning methods based on parameter-efficient tuning (e.g. prompt tuning, prefix tuning) other than standard methods like regularization-, architecture-, and replay-based ones.\n\n(4) Is the CLIP (first line) in Table 2 fine-tuned or just frozen? In Figure 3 there is a reference to \"Fine-Tune\" so I guess it's fine-tuned CLIP. If this is the case, why not add frozen CLIP in all the model comparisons as a baseline? It can be helpful to verify if pre-training alone is enough to ensure an acceptable generalization and performance degradation.\n\n(5) Line 232. \"the model has no access to the task identity\". It's ok but please state from the beginning that we are in a class-incremental learning scenario and thus that the benchmark is for CIL, as usual in any CL paper. \n\n(6) Line 255. \"leran\" -> \"learn\".\n\n(7) \"single-modal scenario\" is an unpopular term. I think the literature agrees with using the term \"unimodal\" instead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)