Stabilizing Zero-Shot Prediction: A Novel Antidote to Forgetting in Continual Vision-Language Tasks
Authors: Zijian Gao
Xingxing Zhang
Kele Xu
Xinjun Mao
Huaimin Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision-language Learning;  Continual Learning;
Primary area: online_learning
TL;DR: We thoroughly investigate the link between model stability in zero-shot predictions and anti-forgetting capabilities, and propose a novel replay-free method with the EMA-LoRA architecture to enhance continual learning.

Abstract: Continual learning (CL) empowers pre-trained vision-language (VL) models to efficiently adapt to a sequence of downstream tasks. However, these models often encounter challenges in retaining previously acquired skills due to parameter shifts and limited access to historical data. In response, recent efforts focus on devising specific frameworks and various replay strategies, striving for a typical learning-forgetting trade-off. Surprisingly, both our empirical research and theoretical analysis demonstrate that the stability of the model in consecutive zero-shot predictions serves as a reliable indicator of its anti-forgetting capabilities for previously learned tasks. 
Motivated by these insights, we develop a novel replay-free CL method named ZAF (Zero-shot Antidote to Forgetting), which preserves acquired knowledge through a zero-shot stability regularization applied to wild data in a plug-and-play manner. To enhance efficiency in adapting to new tasks and seamlessly access historical models, we introduce a parameter-efficient EMA-LoRA neural architecture based on the Exponential Moving Average (EMA). ZAF utilizes new data for low-rank adaptation (LoRA), complemented by a zero-shot antidote on wild data, effectively decoupling learning from forgetting. Our extensive experiments demonstrate ZAF's superior performance and robustness in pre-trained models across various continual VL concept learning tasks, achieving leads of up to 3.70\%, 4.82\%, and 4.38\%, along with at least a 10x acceleration in training speed on three benchmarks, respectively. Additionally, our zero-shot antidote significantly reduces forgetting in existing models by at least 6.37\%. Our code is available at https://github.com/Zi-Jian-Gao/Stabilizing-Zero-Shot-Prediction-ZAF.

Stabilizing Zero-Shot Prediction: A Novel Antidote to Forgetting in Continual Vision-Language Tasks
Zijian Gao Gaol:2 Xingxing Zhang" Kele Xu 12; Xinjun Mao Huaimin Wangl.2
1 School of Computer, National University of Defense Technology, Changsha, 410000, China. 2 State Key Laboratory of Complex & Critical Software Environment, Changsha, 410000, China. SScoool of Computer Science, Tsinghua University, Beijing, 100049, China. {gaozi jian19, xukelele, xjmao, hmwang) @nudt. edu. en, (rrhang1993@gmaiin com
Abstract
Continual learning (CL.) empowers pre-trained vision-language (VL. models efficiently adapt to sequence of downstream tasks. However, these models of- ten encounter challenges retaining previously acquired skills due to parameter shifts and limited access to historical data. In response, recent efforts focus on devising specific frameworks and various replay strategies, striving for typical corming-forgetiin trade-off. Surprisingly, both our empirical research and theoret- ical analysis demonstrate that the stability of the model in consecutive zero-shot predictions serves as reliable indicator of its anti-forgetting capabilities for previ- ously learned tasks. Motivated by these insights, we develop novel replay-free CL method named ZAF (Zero-shot Antidote to Forgetting), which preserves acquired knowledge through zero-shot stability regularization applied wild data plug- and-play manner. To enhance efficiency in adapting new tasks and seamlessly access historical models, we introduce a parameter-efficien EMA-LoRA neural ar- chitecture based the Exponential Moving Average (EMA). ZAF utilizes new data for low-rank adaptation (LoRA), complemented by a zero-shot antidote on wild data, effectively decoupling learning from forgetting. Our extensive experiments demonstrate ZAF's superior performance and robustness in pre-trained models across various continual VL concept learning tasks, achieving leads of up 3.70%, 4.82%, and 4.38%, along with at least 10x acceleration in training speed on three benchmarks, respectively. Additionally, our zero-shot antidote significantly reduces forgetting in existing models by at least 6.37%. Our code is available at https Jian //github. com/ http:aa--iiannaaaaaaaaaiiiiiiiia Gao/Stabilizing Zero- Shot- www.ieiiiniineeeeeeeeereeeeereee
1 Introduction
In the rapidly evolving landscape of artificial intelligence, pre-trained models have become funda- mental to achieving state-of-the-att results across a myriad of applications (55). Recently, large vision-language (VI.) models have provided remarkable predictions on downstream tasks without However, these models are typically trained on static datasets, which may not capture the continuously evolving variety and complexity of real-world data. As new concepts emerge and existing categories expand, the static nature of these pre-trained models can lead to diminished performance over time. For example, the CLIP model [30] achieves an accuracy of less than 60% on the MNIST dataset (i.e., a performance significantly lower than that of a conventionally trained CNN [17
any training examples [30 [3
2 [9]
49 To bridge this gap, continual learning (CL) has emerged as vital methodology, making learning new knowledge a lifelong process for the pre-trained VI. model.
Corresponding author. i Equal Contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, the adaptation process, whether involving comprehensive fine- tinettuning of the entire pre-
trained model 7 49 3 or parameter-efficient continual fine-tuning [33 45 29 inevitably undergo incremental parameter shifts. This presents a significant challenge in CI., where VI. models forget historical knowledge as they adapt to new tasks 41 45 Additionally, the high computational costs required to integrate new data with old data, combined with often limited access to complete previous datasets, further exacerbate the severity of forgetting [47 45). Thus far, most CL research has focused on mitigating the forgetting of class information from previously learned images, process known as class incremental learning (38]. Multimodal tasks such as visual question answering (VQA) and natural language visual reasoning (NLVR) have received less attention. Building upon the foundations laid by unimodal CL efforts, recent initiatives have aimed to mitigate forgetting from parameter shifts by developing robust continual VL learning frameworks, including Mixture-of- Experts [45], weight consolidation (49), and Layered-LoRR (3). Furthermore, a growing body of work is dedicated to developing angg-forg strategies, such as knowledge distillation (23 and
rectification [3] Meanwhile, latent replay mechanisms, including adversarial pseudo-replay [33
and prototype learning [47], have shown effectiveness in retaining historical knowledge, albeit with significant additional memory and computational costs. All these methods, however, struggle with
the inherent trade-off between acquiring new knowledge and preserving historical knowledge.
In this work, we aim to decouple the fooming-frggettig dynamic, striving for a win-win outcome enhancing anti forgettngg capabilities without interfering with the learning process. Our extensive empirical investigation reveals uniform patterns in the model's performance across learned and future VL tasks. Theoretically, identical. Motivated by these insights, particularly the observed stability of zero-shot predictions that reflects the stability required for old tasks (i.e., forgetting capabilities), we develop a novel CL approach. This method, named ZAF (Zero-shot Antidote to Forgetting), incorporates a zero-shot prediction stability regularization term within our EMA-LORA architecture. ZAF employs two low-rank adapters: one facilitates the learning of new tasks and ensures stable zero-shot predictions on wild data; the other, an Exponential Moving Average (EMA)-based adapter, preserves historical knowledge and utilizes it for zero-shot supervision. The synergistic interaction between these adapters effectively decouples learning from forgetting, thereby enhancing overall CL performance.
find that the generalization errors for old and new tasks are nearly
Our contributions include: (1) We conduct a comprehensive empirical and thcoretical study on continual learning for pre-trained VL models, establishing zero-shot stability as reliable indicator of anti-forgetting capabilities; (2) Inspired by this finding, we develop a plug-and-play zero-shot antidote that enhances models' anti-forgetting across various CL methods, through zero-shot prediction stability regularization on wild data; (3) We introduce an innovative replay-free CL method (termed ZAF), implemented within parameter-efficient EMA-LoRA architecture, effectively decoupling learning from forgetting; (4) Across various continual VL benchmarks and pre-trained models, our approach achieves state-of-the tf-the-rrrr- performance and significantly reduced complexity.
2 Related Work
Continual Learning (CL): Continual learning aims to train single model capable of incrementally updating its knowledge with a new sequence of tasks while preserving historical knowledge 41 39 However, due to parameter shift and limited access to historical data, the primary challenge in CL is forgetting previously learned tasks over time 41. A plethora of strategies have been explored to address this issue (32 18 6 12], including selective stabilization of network weights, replay of a few old training samples, construction of task-specific parameters, etc. Despite advancements the unimodal community, such as in image classification, multimodal tasks have received comparatively less attention. This gap highlights the complexity and emerging interest in multimodal settings [9] 35| 47). In this context, VQACL [47] introduced continual VQA task leveraging latent replay strategy. RATT 4 focused on continual image captioning through weight regularization and knowledge distillation. Distinctively, ConStruct-VII [3 pursued continual adaptation to fine-grained Structured VL Concepts reasoning skills without fixed task boundaries. This is particularly applicable to real-world scenarios, and thus, we choose this VL task as a case study in experiments.
4 8
24
is noteworthy that several studies closely related to ours discuss both zero-shot and continual learning capabílities 44
45 46 However, their objectives differ fundamentally from ours. These studies, including ZSCL,
[49 and MoF-Adapters 45), aim to preserve the zero-shot transfer ability inherent in pre-trained VL models during continual adaptation for sequentially arriving tasks, even

---

though these capabilities may be inherently limited. In contrast, our research is exclusively focused on enhancing the anti-forgetiigg capabilities in downstream CL tasks with our zero-shot antidote.
Vision-and-Languaaa (VL) Pre-training: VL pre-preeatee pre-training aims to enhance the performance of downstream VL tasks by pre-training models on large-scale image-text data. The effectiveness of these models largely depends their representational capacity, including factors like data quality and model architecture, as well as the similarity between pre-training tasks and downstream applications. Generally, higher data quality and greater task similarity lead stronger generalizability. However, due to the high costs of human annotation, most methods leverage noisy image-text pairs sourced from the web. which are a sub-optimal form of supervision (c.g. CLIP (30) and ALIGN [33). A novel dataset bootstrapping method, Captioning and Filtering (CapFilt), was introduced to utilize web datasets more effectively [9)] Furthermore, most existing pre-trained models excel either tasks like image-text retrieval [30] 20 or generation-based tasks such as image captioning [26 11. The primary challenge lies in designing model architectures capable of performing diverse tasks [44]. In response, the 'BLIP' framework was proposed, providing flexible transfer ability to both tasks [9]. Specifically, its image transformer is initialized from ViT pre-trained on ImageNet [37 (S and the text transformer originates from BERT [14]. An important variant of BLIP, enhanced with CapFilt to boost performance, is termed IBLIP w/ CapFilt-I. [9].
in understanding-baat
Additionally, to enable reasoning on downstream tasks, 'BLIP" was further fine-tuned on the NLVR2 dataset [36] with a more computationally efficient architecture, referred to as 'BLIP w/ NLVR' [19
In this work, we primarily focus on BLIP and its two variants, unlike the closely related studies 44 45 46] that utilize CLIP. Our choice is driven by the nature of our case study, which involves reasoning tasks as opposed to classification tasks typically associated with CLIP.
3
Preliminary Analysis
In this section, we first define the problem of continual VL learning and then examine the correlation between the model's torgetogg capabilities and cero-shot stability through an empirical study.
3.1 Formulation of Continual Vision-Language (VL) Learning
In this work, we focus adapting pre-trained VL models to sequence of newly arriving reasoning tasks, denoted as {T),... , "." }. Each task T involves a set of image-text pairs (I, T), with a binary ground-trutt function 1) E {(0, 1), (1, 0)} determining whether the text precisely describes the image content. The primary distinction among these tasks lies in the types of cognitive skills they aim to develop. Notably, traditional CL methods, which require identifying the specific task to which input belongs during inference, are impractical here. This is due to the challenge of discerning the required skills from free-form text and the likelihood that multiple skills may be simultaneously necessary. Given the constraints of privacy in practical applications, we adopt a strict policy of not retaining any task-specific data between training sessions. This leads us to a continual, replay-free setting without task identifiers, which is precisely the focus of our study.
3.2 Empirical Study of Anti-Forgetting and Zero-Shot Performance
To investigate the correlation between a model's anti-forgetting capabilities and its stability in zero- shot predictions, we conducted an empirical study using the Structured VL Concepts (SVLC) learning multimodal benchmark 333 This benchmark utilizes two widely recognized public VI. datasets:
Visual Genome (VG) [66] and Visual Attributes in the Wild (VAW) (28). adhering to protocols outlined by the VL-Checklist [48] The '7 Task VG+VAW benchmark, derived from both VG and VAW datasets, includes seven distinct concepts designed to assess a VL model's understanding of various relationships, including spatial and inter-object transitive actions, as well attributes such as size, color, material, and intransitive single-object actions. Examples of these concepts are depicted in Fig, 5 Additionally, the '7 Task VG' benchmark is solely based the VG dataset, and the '5 Task VAW" benchmark focuses on five attributes and object state concepts from the VAW dataset. We adhere strictly to the dataset splits outlined in ConStruct-VV 333
We rigorously evaluate the official implementations of all baseline models to ensure a fair comparison. Our evaluations largely adhere to the training regimes established by LoRA (10), MoE-edasters (5).
3

---

Continual Performance Matrix 7
MooE-Adapeers
Layered-LoRA
ConStruct-VL
71 7
T 7 7
T
96
22
(a)
MoE-Adapters
LoRA
Layered-LoRA
ConStruct-VL
Zero-shot Porformance
Learning Performance
Anti-Forgetling Performance
(b) Task VG
Figure Empirical study of anti-forgetting, learning, and zero-shot performance in CL methods.
Layered-LoRA 33 and ConStruct-VL. 33 Since MoE-Adapters [45] was originally designed for both pre-trained and downstream classification tasks requiring task-id inference, adapt its implementation to include a single task-dependen router for our needs. Given the focus on reasoning tasks. we opt for 'BLIP w/ NLVR' as our pre-trained model, chosen for its robust representation capabilities and adequate zero-shot transfer ability.
Fig. Tillustrates heatmap that depicts the performance of various CL methods across two bench- marks (see Appendix [.I for additional benchmarks). In this heatmap, rows represent training steps, and Aiy denotes the prediction accuracy on task T after training task T', is important that traditional CL methods typically focus only on the lower triangular matrix of results, which rep- resents performance predictions on future tasks, which are represented in the upper riaanularr matrix. By comparing these results, observe that larger average values and less fluctuation the red area, which represents strong zero-shot stability, typically correspond to similar patterns in the blue area, indicative of strong anti-forgetting capabilities, without adversely affecting the yellow area, where learning new tasks occurs. This observation suggests that model's stability in zero-shot predictions can reflect its antg-forget capabilities. Our findings further suggest that by systematically stabilizing zero-shot predictions during continual learning, can significantly enhance the model's 's ability retain historical knowledge without compromising the acquisition of new information.
previously learned tasks. However, our analysis extends include zero-shot
Theoretical Foundation and Our Approach
In this section, we first provide theoretical analysis to substantiate empirical findings. Subse- quently, we present an innovative CL approach specifically tailored for pre-trained VL models.
4.1 Continual Vision-Language Learning Objective
In the continual learning of sequentially arriving VL tasks = TT. the model is designed to align with the binary ground-trum function P(T, I) that evaluates whether the text T precisely describes the image content I. this case study. we employ BLIP-based encoders learn discriminative deep embeddings. Specifically, let fo represent the image encoder and fr the text encoder, while hh and hy serve as the cross-attention decoder and the final binary classifier, respectively. The predictive modeling process within the BLIP framework for learning task T is structured as follows:
PT( (f
where (T denotes the prediction probability made by the current model M ((*,T", B0,,T".t , (t)).
To sequentially fine-tune the BLIP model for downstream tasks, employ cross-entropy loss term, Lex (t (T, 1), P(T, I)). This loss measures the discrepancy between the current prediction, Pt and the ground truth, P(.), ensuring that the model progressively acquires new information.

---

(a) Traditional Procedures
(b) Our Procedures
Learning Procedure
Learning Procodure
Anti-forgatding Procedure
Anti- fotl-torggting Procedure
Zero shot aoooshhttAtiooo Antidete Procedure
Task
ArHA
Current Task 7
Inferring
Inforring
Cup sabll
Modell
B
Pre-trained
Dana
Data
VL model
Cup the tible
Pro-trained model
Lxp L7E
Lzs
Polygoral Limbrolaa
+
Leg
Sandy Gouple Round
Coupled Object
Decoupled Object
Figure 2: Comparison of training and inference procedures between traditional and our CL methods.
Let ELR denote the empirical errors on the observed tasks, and & represent the empirical error on the current task T For & E {t + I..... , and E {1...., t 1}. E and &, denote the generalization errors on the future task T and the old task T respectively. Inspired by the PAC-Bayes theory (27 and previous work in domain generalization 31 40 we present the upper bounds of these two errors under the continual VL scenario (see Appendix A for the detailed proof).
Proposition 1 For continual learning with pre-trained VI. models, let M' denote a solution of the continually learned tasks T T.. In particular, M = arg min) MMM M-MM (,M where - [IM- -MM-1|| I|2 < A represents the weight vectors for continual tasks only minor variations. For any 8 (0,1) with probability at least 5:
As f1, f1,--,,,-- 1--1 13. EAM') s SE,(M')+ + Div(T,, 7,) + dln(N N/J]] + In(1/5)
Σ
(2)
28
2N
Vk (1+,,...., Ea(M)) s ((a(M))
1
Div(T,
d[nn(V/d) [d) ln(1/8)
21
2N
i=1
where Div(To, Ty) = Ssuphen (PT, - PT, (I(h)) defines the H--divergecce between the distributions for tasks T; and Ty, with I(h) being the characteristic fuunctirrisiccfuncionn function. Here, N denotes the harmonic mean of the training example sizes for the observed tasks, and d represents the VC dimension of the parameter space.
Proposition demonstrates that the model M has consistent upper bounds on the generalization errors for both previously learned and future tasks. These abilities are significantly influenced by three key factors: empirical error of continual tasks, discrepancy between task distributions, and complexity of the parameter space. Such consistency underscores that the model's capabilities in zero-shot prediction can reliably indicate its anti-forgetting capabilities. In practice, maintaining this consistency necessitates the implementation of carefully designed regularization techniques.
Motivated by these insights, we develop a zero-shot regularization antidote to mitigate forgetting. Considering the unpredictable nature and broad array of future tasks, we initially introduce an unlabeled wild dataset, denoted as Dwild [Twild, [wid]. to assess the zero-shot capabilities of the continually learned VL model. It is erucial to note that both the text Twild and the images Iwild entirely unpaired and unlabeled? Additionally, the substantial number of wild examples in Dwita are distinct and separate from the actual downstream tasks {T*) }ー Fig. iillustrates some wild examples.
To stabilize zero-shot predictions, develop novel loss term, Lzs (P* Iwila), P (iTwaldd Iwaia)), ) where P( (-) is the prediction probability derived from the model the previous training step and used for zero-shot supervision. The formulation of our method's loss function is outlined below:
= LCE (T, P(T, + Lzs Pt (Tuas, Iwaa), (T Iwaa)),
(4)
Pleease refer to Appendix C.5 for more details about the construct of the wild dataset.
5

---

Algorithm Training Algorithm of ZAF
Input: A pre-trained VL model M number of epochs E, and hyperparameter a [0. Data: A sequence of training tasks 7 ....., T and a wild dataset Dwild {Twild, [wild}
Output: The continually learned model M for inference
1: Initialize the LoRA component (A, B) and EMA component W
2: for in 1 do
3: for in do
4: while not traverse over all current data T do
5:
Sample batch of current data B", and batch of wild data B
6:
W = + A B
7:
Predict the probabilities pt (B*) using W Calculate LCE with Pl (BB)) and ground truth P(B')
8:
9:
if then
10:
Optimize {A, with LcE
11:
else
12:
Predict the probabilities P'(Bwild) using W, and P1(wwild) using -- Calculate Lys with Pt and (*(2wwlld)
13:
14:
Optimize (A, B3 with loss function Eq.
15:
ift then
W + A. B, and M M + W
17:
else
Update the EMA component W and M with Eq. (
19:
return M
where Lrs =
Iwita)
Pt (Twild, Twita)ll || our experiments to restrain fluctuations.
4.2 EMA-LORA Architecture
To achieve the objective outlined above, rather than conducting comprehensive fine-tuning of the entire pre-trained model {vr, r², w vf² as defined in Eq. (I). we develop parameter-efficienn approach using an EMA-LoRA architecture based on BLIP. As depicted in Fig, [bb), for efficient adaptation new tasks, we first integrate LoRA adapters into all layers of the image encoder fu. text encoders fr, and the cross-attention decode hw, which are updated during the training of task T Specifically, each of fu. fr. and h. consists of a combination of non- con--aamettri functions referred to here as data norms, such as LayerNorm, which remain frozen, and two types of parametric functions, namely linear and embedding functions. Typically, linear and emhedding functions are parameterized by weight matrix W, which is optimized at every iteration by adding residuals to the weights from the initial pre- traited VL model M Formally, this is represented as W = M + A B, where A X and x respectively, with
and B are learnable low-rank matrices of dimensions the dimensions M During the training of task T only the current task's 's LoRA adapters {A, B} learned, and predictions Pt
being
made using W. This strategy not only enhances the adaptability of the model to new tasks but also maintains a balance between efficiency and performance.
As demonstrated Eq. (4, accessing previous models is essential for predicting wild data (i.c., P(-)). To facilitate this, implement the Exponential Moving Average (EMA) on the aforementioned LoRA adapters (see Fig bb)). This approach allows for memory-efficient access to previous models while learning new tasks. Specifically, the EMA process is conducted at the end of every epoch:
W -- aW + (1 a) B, and = W.
where the hyperparameter a E [0, 1] plays a crucial role in updating the model parameters (refer to Fig. for its sensitivity analysis). Importantly, M represents the final model used for inference after continual learning. The weights W are updated following the completion of the first downstream task, at which point there is no risk of forgetting, thereby eliminating the need for the zero-shot antidote Lzs. For subsequent tasks, predictions wild data using P((-) are made with M
Algorithm I provides detailed view of our training algorithm for the Zero-shot Antidote Forgetting (ZAF) framework. Key highlights of the proposed ZAF include: (1) Learning vs. Forgetting: As

---

illustrated in Fig. 2 a) and (b), traditional CL frameworks, such as those incorporating knowledge distillation, often struggle with a trade-off between learning and forgetting. This trade-off typically manifests in the balance of LCE and LXD. where the LoRA adapter is optimized to align with both old and new knowledge using current data: min LCB (CE(((((T)) (T*), )(T + LKD
(P'((P)))) (T'), p( Our approach introduces zero-shot antidote using wild data effectively separate the learning processes from the problem of forgetting: min Len( ( (T'), P(T)) + Lzs (Dwia) (D (2) Privacy Protections: Unlike conventional methods that require replaying old task data, our framework utilizes generated wild data to prevent forgetting, adhering to privacy constraints.
(3) Innovative Mixture Strategy: Deviating from the typical spatial Mixture of Experts (MoE) and Layered-LoRA architectures, our model employs a temporal mixture strategy through EMA. This approach not only conserves memory but also integrates downstream tasks incrementally and effectively. (4) Efficiency in Resource Usage: The EMA-LoRA architecture involves only two low-rank adapters, which are computationally efficient, and also ensure M| M I|2 I A, established in our Proposition
Experiment
We utilize the '7 Task VG+ '7 Task VG', and '5 Task VAW benchmarks for our empirical studies as discussed in Sec. 3.2 We use the object state - alfr. action - attr. size - rel. spatial attr. material rel. action altr. color 7-task sequence in the 7 Task VG+VAW benchmark and 7 Task VG benchmark, and the object state - attr. action + attr. sice + attr, material + attr. color 5-task sequence the 5 Task VAW benchmark. Given the reasoning task scenario, our primary focus is on the original 'BLIP' model and its two variants: 'BLIP w/ CapFilt-L' and 'BLIP w/ NLVR'. Notably, the latter variant, which is fine-tuned on the NLVR2 dataset, exhibits enhanced zero-shot transfer ability.
Benchmark:
Model Architecture: All experiments begin with the BLIP model architectures and its pre-trained weights. The image encoder, denoted as fu. utilizes ViTB/16, and the text encoder fr is BERT with a 12-layer encoder and 768 hidden size. The decoder f.2 extends fr. incorporating cross- cross-attention layers between every two self-attention layers, with each receiving encoded image tokens as additional input. Our binary classifier hy is a 2-layer MLP with a hidden size of 768. For all baselines, we adhere to the implementation protocols established in ConStruct-V VL 333 pioneering work in SVLC scenario that has demonstrated exceptional performance. We maintain a shared binary classifier heg across all task models. In practice, for the ILIP w/ NLVR model, hy is frozen across all tasks. For the BLIP and BLIP w/ CapFilt-L. models, h. is trained only for the first task T and then remains frozen for all subsequent tasks.
Baseline and Implementation: Due to the inadequate performance of unimodal CL methods in multimodal scenarios, as highlighted by [33], our analysis primarily focuses on frameworks and multi- modal CL methods designed for VI. scenarios. We include only one representative unimodal method, LwF 21. which mitigates forgetting through knowledge distillation for comparison. Detailed comparisons with other unimodal CL methods are provided in Appendix [C6] Among the methods compared, Continual-FT 7 and LoRA [10] train the VL model continuously without incorporating anti-forgetting procedures. Layered-LoRA (33 and MoB-Adapters (45) are viewed as representative continual VL learning frameworks. Furthermore, we compare our ZAF with ZSCL (4),, which aims to preserve the inherent zero-shot capability of the pre-trained model, and ConStruct- VL [33,, which employs expensive pscudo-replay strategy involving adversarial attacks every sample. Following the implementation from previous work [33], specifically, we adapt WD 0.05, an initial LR of 1. 1.25e=3, a cosine scheduler, and maximum of 12 training epochs in all experiments. For all low-rank adapters, the rank r is set to 16. The only hyperparameter, a, of our method is fixed at 0.85. Please refer to Appendix B for more implementation details.
Evaluation Metries: We utilize three reconnere evaluation metrics for continual learn- ing - Final Average Accuracy (FAA), Cumulative Average Accuracy (CAA), and Final Forgetting Measure (FFM) as detailed in 41 38 We define the accuracy on the task T after learning the E/=1 Ay- Upon completing all n tasks, we report FAA = AA,, CAA = A E. AA, and FFM = EE=l n- j=1 (1..6e(1..,-1)AAA Ang). The FAA is critical metric highlighting performance
task T as Aty The average accuracy after learning task T is denoted as AA,
7

---

Table 1: Overall performance (%) of CL methods across three benchmarks under various VL models.
VG+VAW
VG
VL model
Method
FAA (T)
CA) (L)
FAA (*)
CAA (f)
FPM (1)
FAA (f)
CAA (T)
FFM
Joint Learning
91.90
95.27
92.60
ContinualFFT
65.21
73.98
30.32
63.91
73.97
31.34
67.07
78.35
28.14
LoRA
10
75.39
76.59
20.73
69.16
75.89
28.20
71.54
79.07
22.48
Layered-LoRA
33
76.68
78.51
18.96
70.13
79.66
28.08
83.77
83.47
9.20
BLIP
LwF 21
70.93
73.62
26.26
77.05
29.05
80.07
84,32
14.93
ZSCL
66.87
19.08
75.65
27.45
25.13
Mol--Adapters
27.11
14.36
ConStruct-VL
87.27
86.98
6.14
89.01
91.87
5.80
83.73
86.34
6.47
7.AF (Ours)
90.03
89.45
3.32
92.49
92.39
1.97
89.13
90.03
3,93
Improvement
2.78
2.47
2.82
3.48
0.52
3.83
5.40
3.69
2.54
Joint I Learning
93.72
95.31
92,90
Continual-FT
67,20
74.85
28.02
70.05
75.17
23,99
71.95
79.31
22.18
LoRA 10
71,97
76.07
25.27
69.
77..52
28.49
13.78
Layerad-LoRA
76.66
76.27
19.20
78.00
27.16
81.89
11.18
BLIP w/
23
23.81
28.47
15.63
Caplilt-L
ZSCL 10
62.90
64.29
22.06
67.12
76.21
27.14
68.13
77.15
24.67
MoE--Adapters
45
69.76
73.29
27.34
63.99
76.19
35.34
80.01
84.10
14.43
ConStrue- VI.
33
85.16
87.61
8:75
88.95
90.69
5.22
83.33
85.57
6.28
ZAF (Our)
89.61
89:65
4.18
92.53
92.20
1.72
89.43
90.20
3.02
Improvement
4.45
2.04
4.57
3.58
1.51
3.50
6.10
4.63
3,26
Learning
95.07
92.36
Contimal-FT
27.96
78.60
20.55
20.69
27.25
Layered-LoRA 33
80.62
79.89
13.92
73.03
81.12
24.99
83.73
84.26
9.29
BLIP
LwF 21
73.00
77.26
23.12
71.11
79.39
27.09
82.10
84.69
11.24
NLVR
7SCL 4M
60.27
67.94
28.48
65.82
78.06
27.68
62.00
74.33
31.20
MoE-Adapter
15
72.50
74.81
23.74
67.09
76.54
31.83
79.05
84.21
15.58
ConStruct-VL
33
85,97
87.00
6.94
86.96
90.47
7.91
84,36
85.93
5.36
ZAF (Oun)
89.67
89.30
3.38
91.78
91.74
2,02
88.74
89.03
2.67
Improvement
3.70
2.30
3.56
4.82
1.27
5.89
4.38
3.70
2.69
discrepancies between CL methods and joint learning. The CAA provides a comprehensive view of overall historical performance, and the FFM quantifies the model's capability to mitigate forgetting.
ZAAF consistently achieves the highest FAA, CAA, and lowest FFM, significantly outperforming competitors across three benchmarks with different pre- trained models. 7SCL. 44), despite employing weight ensemble and knowledge distillation, exhibits inadequate learning on the challenging 7 Task VG+ VAW and 7 Task VG benchmarks, resulting in declined performance. Similarly, MoE-Adapters, like Continual-FT 7 and LoRA (IO), which lack anti-forgetting mechanisms, exhibit significant forgetting as the number of tasks increases. In contrast, both the Layered-LoRA architecture 333 with its task-specific parameter isolation strategy, and LwF (21], with only knowledge distillation strategy, show improved performance. Notably, only ConStruct-VL. [33] approaches the performance our method but requires computationally intensive training involving adversarial attacks and utilizing all historical models for pseudo-sampll sample generation. Despite this, ZAF shows substantial advantages, leading by 3.70%, 4.82%, and 4.38% in FAA under the BI.IP w/ NLVR model. Meanwhile, our method demonstrates a narrow performance gap compared Joint Learning the pinnacle of continual learning. The smaller this gap, the greater the challenge for CL methods to bridge it. Importantly, despite the inherent limitations in the zero-shot capabilities of some pre-trained VL models (e.g.. BLIP and BLIP w/ CapFilt-L), our zero-shot antidote still effectively mitigates forgetting, underscoring its utility across wide range of CL scenarios and pre-trained models.
Overall Performance: As shown in Table T
Complexity Analysis: Table 2 presents the model size, training parameter count, and training times various CL methods across three benchmarks. All comparisons are conducted on NVIDIA GeForce RTX 3090 GPUs using 'BLIP w/NLVR', which serves as the standard configuration for subsequent experiments. reflected in the table, ConStruct-V exhibits exceptionally high training times, especially in the 7 Task VG+VAW scenario, attributed to the utilization of all historical models

---

Table 2: Comparison of training complexity among various CL methods across three benchmarks.
Train Times (h)
Method
Model Size (M)
Train Params (M)
7 Task VG+VAW
Task
Task VAW
ContinualFFT
223, 94
223.94
5.57
2.25
2,90
LoRA 10
230.13
6.19
4.64
1.86
2.82
Layred-LoRA
33
230.13 267.29
6.19
11,10
4.04
5.60
LwF 21
230.13 236.33
6.19
8.13
5.19
6.34
MoE-Adaplers
45
251.04
27.10
6.01
2.82
4.19
ZSCL 49
223.94
223.94
11.83
6.44
7.83
ConStruct-L
35
230.13 267.29
6,19
247.78
102.59
73.24
ZAF (Ours)
236.33
6.19
8.35
5.81
6.67
Training Speed Acceleration
247.78/8.35 se 29.67
102.59/5.81
17.66
73.24/6.67 10.98
Table 3: Comparison of plugin performance for various CL methods across three benchmarks.
Task VG+NAW
7lisk VG
Task VAW
Method
CAA
FEM (U)
CAA (LM
FAA (f)
CAA
Joint Learning
93.37
95.07
92.36
LoRA [00
69.55
75.03
27.25
68.73
78.03
29.62
75.63
81.87
19.37
w/ Zero-shor Antidote
72.47
77.78
23.15
79.12
83.47
16.85
81,55
84.01
12.39
Layered-LoRA 33
80.62
79.89
13.92
73.03
81.12
24.99
83.73
84.26
9.29
w/ Zero-shot Antidote
83.81
85.11
10.37
84.10
88.75
11.26
86.66
87.02
4,98
MoE-Adapters
72.50
74,80 80
23.74
67.09
76.54
31.83
79.05
84.21
15.58
w/ Zero-shot Antidote
86.78
83.62
87.89
12.27
87.47
ConStruct-LL 33
85.97
87.00
6.94
86.96
90.47
7.91
84,36
85.93
5.36
w/ Zero-shot Antidote
89.60
88.13
1.26
92.05
92.06
0.88
86.94
86.72
1.22
EMA-LoRA
77.78
82.22
w/ Zero shot Antidote (ZAF)
89.67
89,30
3.38
91.78
91.74
2.02
88.74
89.03
2.67
Avernge Improvement
7.18
5.88
8.94
11.96
7.10
14.52
4.72
2.35
6.37
and complex adversarial attacks on every training sample. Although methods like MoB-Adapters and Layered-LoRA more efficient than ConStruct- VL, their performance is still found lacking. In contrast, our method, ZAF, maintains relatively modest training times and model size, comparable to LwF one of the simplest CL methods yet significantly enhances anti anti-forgetting capabilities. Compared to ConStruct-V VI., the currently best-performing method, ZAF achieves substantial training speed accelerations of 29.67, 17.66, and 10.98 the three benchmarks, respectively. These results underscore Z.AF's considerable advantages in terms of complexity, efficiency, and memory usage.
Plugin Analysis: Table presents a comprehensive plugin analysis, illustrating the significant reductions in FFM and improvements in FAA and CAA achieved by our zero-shot antidote when applied to various CL methods. In practical implementations, the model from the last task of the baselines is utilized to calculate the zero-shot regularization term. The results underscore the anti- forgetting capabilities of the zero-shot antidote, as evidenced by an average reduction of 14.52% in the FFM, and increases of 11.96% in FAA and 7.10% CAA, in the 7 Task VG benchmark. These observations further validate the universality and effectiveness of our proposed decoupled training objectives: enhancing anti-forgetting capabilities without compromising the learning process. For additional qualitative results, please refer to Appendix C.1 Furthermore, although our proposed EMA-LoRA architecture demonstrates anti-forgetting capabilities comparable to the Layered- framework, comparative analysis when both enhanced with the zero-shot antidote further underscores the superior performance of our approach (ZAF). This superiority arises from the synergistic effects of combining the EMA-LoRA neural architecture with zero-shot prediction stability, This configuration not only improves training efficiency but also significantly reduces memory usage. For additional results across various pre-treined models, please refer Appendix C.2
Hyperparameter Analysis: Fig. showcases the performance of our ZAF method across different hyperparameter values [0, I.. including comparative analyses with Joint Learning and our strongest competitor, ConStruct- VI.. Notably, when a ranges from 0.65 to 0. 0090, ZAFF consistently outperforms ConStruct- VL by clear margins in both FAA and CAA metrics across three distinct

---

Task VG+VAW
7 Task VG
Task VAW
Accuracy
Arr Leening
Lete CAL
Art Leaning
Durs Ca0ttai4L--AA
Durs CAd
Cord eucUTTRA
Controal PaA
Ryperparamster
Hyparparinnder
Hyperparameter
Figure 3: Comparison of FAA and CAA metries for ZAF across various a values against 3 baselines.
Finul Forgatio Mossure (N
25-
20
Cormutattve
73
76-
ContinusST LaRA LayeredkLotA Constructvtt ZAF
ContinuseT LoRA Layand-LLRAA Conatruci-VL ZAP
Continui-FF LoRt LaywessLORA Comflustivt. ZAr
Figure 4: Results of various CL. methods across three distinct task orders within Task VG+VAW.
benchmarks, underscoring its superior adaptability. However, setting a to 0.95 leads to a notable decline in performance, reflecting the model's compromised ability to integrate effectively. Overall, ZAF demonstrates remarkable robustness across a broad range of hyperparameter settings. Please refer to Appendix C.4 for ZAF's heatmaps across different values.
knowledge
Task Order Analysis: Fig Fig. presents statistical results of various CL methods across three distinct task orders within the challenging Task VG+VAW benchmark. Excluding the aforementioned task order within the Task VG VAW benchmark, the second and third orders are the rel. spatial > attr. size - aftr. material rel. action attr. color object state attr. action task sequence and the rel. spatial + attr. material + attr. state - attr. action + attr. size + rel. action attr. color task sequence, respectively. Our ZAF method exhibits remarkable robustness to variations in the task order, showing only minimal fluctuations across three performance metrics, and consistently outperforms existing CL methods by significant margin. For detailed task sequence and quantitative results, please refer to Appendix C33
Discussion and Conclusion
In this work, we analyze the challenges of learning and forgetting in continual learning for pre- trained VL models from both empirical and theoretical perspectives. A key finding is that zero-shot stability reliably indicates forgetting. We introduce a aad--aayy zero-shot antidote enhance anti-forgetting capabilities across various CL methods, integrating into our srrrmmtter-fffciien EMA- LoRA architecture. This facilitates efficient adaptation and memory-fite -free access to historical models. effectively circumventing the learning-forgetting trade-off prevalent in current CL. works 3 49 45
46). Our approach demonstrates superior performance across various continual VL benchmarks and pre-trained models. We anticipate future research will explore the anti-forgetting challenge from the perspective of zero-shot prediction stability, diverging from traditional mechanisms.
This work presents several potential limitations. The effectiveness of our plug-and-play zero-shot antidote presumes the existence of feasible CL framework, which may not be available in all contexts. Also, it is designed specifically for sequence of downstream tasks, limiting its applicability to the original training contexts of the pre-trained models. As fundamental research machine learning. the potential negative societal impacts are not immediately apparent at this stage.
10

---

Acknowledgements
This work was supported by the National Science and Technology Major Project (20232D0121101), and National Natural Science Foundation of China (No.62172426, 62106123).
References
[1] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Percira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 19(1):151-175, 2010.
[2] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104-120. Springer, 2020.
[3] Zhenyu Cui, Yuxin Peng, Xun Wang, Manyu Zhu, and Jiahuan Zhou. Continual vision-language retrieval via dynamic knowledge rectification. In Proceedings of the AAAI Conference Artificial Intelligence, pages 11704-11712, 2024.
[4] Riccardo Del Chiaro, Bartiomiej Twardowski, Andrew Bagdanov, and Joost Van de Weijer. Ratt: Recurrent attention to transient tasks for continual image captioning. Advances in Neural Information Processing Systems, 33:16736-16748, 2020.
[5] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale, arXiv preprint arXiv:2010. 1929,, 2020.
[6] Arthur Douillard, Yifu Chen, Arnaud Dapogny, and Matthicu Cord. Plop: Learning without forgetting for continual semantic segmentation. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 4040-4050, 2021.
[7] Alexandre Galashov, Jovana Mitrovic, Dhruva Tirumala. Yee Whye Teh, Timothy Nguyen, Arslan Chaudhry, and Razvan Pascanu. Continually learning representations scale. In Conference Lifelong Learning Agents, pages 534-547. PMLR, 2023.
[8] Claudio Greco, Barbara Plank, Raquel Fernández, and Raffaella Bernardi. Psycholinguistics meets continual learning: Measuring catastrophic forgetting in visual question answering. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 3601-3605, 2019.
[9] Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. Remind your neural network to prevent catastrophic forgetting. In European Conference on Computer Vision, pages 466-483. Springer, 2020.
[10] Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[11] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang. Zicheng Liu, Yumao Scaling up vision-language pre-training for image captioning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 17980-17989 2022.
Lijuan Wang.
[12] Ching-Y Yi Hung, Cheng-Har Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. Advances in Neural Information Processing Systems, 32, 2019.
[13] Chao Jia, Yinfci Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Lc, fun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling visual and pision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[14] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLLT, pages 4171-4186, 2019.
[15] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barnskaaaarwiska,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the notional academy of sciences, 114(13):3511-3526, 2017.
[16] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-75 73, 2017.
11

---

[17] Yann LeCun. Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-2222 1998.
[18] Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 312-321, 2019.
[19] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language image we-ttaining for unified vision-language understanding and generation. In International conference on machine learning, pages 12888- PMLR, 2022.
[20] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
[21] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, (D(11))2935-22 2947, 2017.
[22] David McAllester. Pac-bayesian model averaging. Proceedings of the Twelfth Annual Conference on Computational Learning Theory, pages 144-170, 1999.
[23] Zixuan Ni, Longhui Wei, Siliang Tang, Yucting Zhuang, and Qi Tian. Continual vision-language represen- tation learning with off-diagonal information. In International Conference on Machine Learning, pages 26129-26149. PMLR, 2023.
[24] Mavina Nikandrou, Lu Yu, Alessandro Suglia, loannis Konstas, and Verena Rieser. Task formulation mat- ters when learning continually: A case study in visual question answering. arXiv preprint arXiv:2210.00444 2022.
[25] OpenAL. Gpt-4 technical report. arxiv 2303.08774, 2023.
[26] Vicente Ordonez, Girish Kulkami, and Tamara Berg. Im2text: Describing images using million captioned photographs. Advances in neural information processing systems, 24, 2011.
[27] Anastasia Pentina and Christoph Lampert. pac-hayesian bound for lifelong learning. In International Conference on Machine Learning, pages 9999999. PMLR, 2014.
[28] Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes the wild, Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13018-13028, 2021.
[29] Zi Qian, Xin Wang. Xuguang Duan, Pengda Qin, Yuhong Li, Wenwu Zhu. Decouple before interact: Multi-modal prompt learning for continual visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 7748-8763. PMLR, 2021.
[31] Bernardino Romera-Parede and Philip Torr. An embarrassingly simple approach to zero-shot learning. In International conference machine learning, pages 2152-220200 2261. PMLR, 2015.
[32] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In Proceedings of the IEEE International Conference Computer Vision, pages 3400 3409, 2017.
[33] James Scale Smith, Paola Cascante-Bonilla, Assaf Arbelle, Donghyun Kim, Rameswar Panda, David Cox, Diyi Yang, Zsolt Kira, Rogerio Feris, and Leonid Karlinsky. Construct-vi: Data-free continual structured vi concepts learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14994-15004, 2023.
[34] James Seale Smith, Junjiao Tian, Shaunak Halbe, Yen-Chang Hsu, and 7solt Kira. A closer look at rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2409-2419, 2023.
[35] Tejas Srinivasan, Ting- Yun Chang. Leticia Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. Climb: A continual learning benchmark for vision-ane language tasks. Advances in Neural Information Processing Systems, 35:29440-200 29403, 2022.
12

---

[36] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang. Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. arXiv preprini arXi:1181100099) 2018.
[37] Hugo Touvron, Matthieu Cord, Mathijs Douze, Francisco Massa, Alexandre Sahlayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[38] Liyuan Wang. Jingyi Xie, Xingxing Zhang. Mingyi Huang. Hang Su, and Jun Zhu. Hierarchical decom- position prompt-based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems, 36, 2024.
[39] Liyuan Wang, Xingxing Zhang. Qian Li, Mingtian Zhang, Hang Su, Jun Zhu, and Yi Zhong, Incorporating neuro-inspired adaptability for continual learning in artificial intelligence. Nature Machine Intelligence. 5(12):1356- 1368, 2023.
[40] Liyuan Wang. Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong. Cosel: Cooperation of small continual learners is stronger than big one. In European Conference on Computer Vision, pages 254 254-271. Springer, 2022.
[41] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
[42] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi. Ruoxi Sun. Han Zhang, Chen-Yo Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehcarsal-free continual learning. In European Conference on Computer Vision, pages 63 631-648. Springer, 2022.
[43] Zifeng Wang. Zizhao Zhang. Chen- Yu Lee, Han Zhang. Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
[44] Zirui Wang. Jiahui Yu, Adams Wei Yo, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In International Conference on Learning Representations, 2021.
[45] Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-cxperts adapters. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
[46] Yu-Chu Yu, Chi-Pin Huang, Jr-Jen Chen, Kai-Po Chang, Yung-Hsuan Lai, Fu-En Yang, and Yu- Chiang Frank Wang. Select and distill: Selective dual-teacher knowledge transfer for continual learning on vision-language models. arXiv preprint 2rX2v:2403.09296, 2024.
[47] Xi Zhang, Feifci Zhang, and Changsheng Xu. Vqacl: A novel visual question answering continual learning setting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19102-19112, 2023.
[48] Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating tre-trained vision-language models with objects, attributes and relations. arXiv preprint rrXiv:2207.002211 2022. 2022.
[49] Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vasion-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19136, 2023.
13

---

A Theoretical Foundation
A.1 Preliminary
For the convenience of readers who are not familiar with the PAC-Bayesian framework, we introduce the most relevant concepts from the literature here (27). PAC-Bayesian theory studies the properties of randomized predictors, called Gibbs predictors. Formally, let X be an image-text input se y be an output set, and H C {h: X + }) a set of prediction functions (hypotheses), We assume that a distribution D is with X, V/ and a global labeling function h : X' - 3/ where h(x)) generates target label for all the imput, i.e., y/ = k(x)). Consider a bounded loss function L y x y - [0. for binary prediction, such that C(y1. 322) = 0 holds if and only if y1 = 3/2 Then, we define an expected
loss over the distribution ID by (M,, h) = (MM) = ((a)))= (z), 3)]. where (.)()) the prediction function parameterized on .M. Let D denote a training set following the distribution ID> with N data-label pairs. To minimize Ep (M), we can minimize an empirical risk over the training set D in parameter space, i.e. minM (MM) where (MM N
Σm ((PM Mo).
Below are one important definition and three critical lemmas for the proof of our Proposition
Definition (Based on Definition I of ( Given two distributions, T and S, let H be a hypothesis class on inpul space X and denote by I(h) the set for which h H is the characteristic function: that is, x E (h) + h(z)- = 1. The H-divergence between T and S is
Div(T S) sup (Pr(1(h)) PPs((h))) MEH
(6)
Lemma Let Set = ( and T be s source distributions and a target distribution, respectively, The H-divergence between {8 and T is bounded as follows:
Div(S, T)
Div(S,,T) T)-
Proof. By the definition of H-diveegence in Definition !
Div(S, T) = 2 sup (Ps(I(h)- - Pr(1(h))
ME H
= 2 sup NEH
Σ
(Ps, (I(h) - Pr(((H)))
< 2 sup NEW 1=1
- PPs (((h))))))((( -P)((
(8)
< 2
sup (Ps, (1()) (T(1)
$ NEW
1 Div(S, T),
where the first inequality is due to the triangle inequality (i.e., IE, a:l < Ex las)) and the second inequality is by the additivity of the sup function. This finishes the proof.
Lemma 2 Given two distributions, T and S. let Mi H and by E H be two prediction functions. The difference between the expected loss with and S is bounded by the divergence between T and S as follows:
(e, (MMM,,))) > )DiV(T,
where (iv : supher (Pr(((h)) Ps(I(h)) is the H-divergence for the distribution T and S (I)) is the characteristic function).
'We abbreviate the image-text input space as X instead of {X,I} for simplicity and ease of notation throughout this document. Likewise, instead of (1, 1).
14

---

Proof. By the definition of H --diiergeence in Definition !
Div(T,
sup
(P(((()) - Ps((()))
kCH
sup
(( (x) + h1 (x)] - ((a.)) PM (x) * (z))
Mr EM
2 sup
[ ((r.y)) (z), h (z)) - ((a (CP (z) (z)) (10)
MiK
sup
|&T(M1, k1) Es 41)
2 2 hn) - Es(M2,11))-
completes the proof.
Lemma 3 Let θ be cover of parameter space with VC dimension d. Then, for any d E (0,
with probability at least for any M O:
(M) - ED(M)) d[In(N/d] + mm(10)
(11)
2N
where (n(M) is an empirical risk with N samples in its training set D.
Proof. For the distribution D, according to 122. we have
(0) (0) 2 c) v 2me(N) where mN(N is the amount of all possible prediction results for N samples, which implies the model complexity in the parameter space θ We set (eN)) 2 ())* in our model, and assume
(12)
confidence bound e (lln(N/d)) (1(1)))) Then we get
2N
N Ō
P((E(00)- ED( 2 2) <
exp(-2Ne)) 8.
(13)
Hence, the inequality ( S holds with probability at least 8.
En)
It completes the proof.
A.2 Proof of Proposition
If continually learn t VL tasks T!....,77 T' that follow the distribution D1, Dr. solution
M can be obtained with their training data DI,... Dr. We define D1.- = {ID,.... ..... ........D [D. and Diat := {Di,... Dt}. Then, for a future task T where k E 1t ,n). its distribution is denoted DD with training set D3. Let & the zero-shot generalization error on future task T Here, we use E (M)) instead of (xM since assume there is no difference between labeling functions for each task for simplicity. Specifically, within the context of continual VL reasoning tasks, the labeling function is deñind
binary function. In fact, we cannot compute & since the distributions over the tasks (i.e., (Dx) and the tasks' data (i.e, D₂) are both unknown. However, we can approximate it by its empirical counterpart, based on the t observed tasks:
Ex(M) S (M)) Div 1Ba)
I (m)) + Div( + ddnnN [d)) + In(1/8)
2N
(14)
d[u((N/ /d] ]] In(1/8)
sEu(M))
De) +
21
2NV
where (M') - &D, (M') because | - M & A that represents the weight
vectors for continual tasks are only minor variations. The three inequalities hold due to Lemma 2 Lemma and Lemma I respectively. Du. = {Di) and we rewrite mixture of all the
distributions as D1a == Ei= Dj using convex combination. N 1==(tE (#E
is the harmonic
15

---

mean of the training example sizes (i.e., Nj) in each task. Of note, Div(II; Dx) is rewritten as Div(T, Tx) in the main body avoid using the distribution notation D, and Dz.
Furthermore, for previously learned tasks T where s {1...... t- 1), let e, denote the general- ization error on old task T We cannot compute &, since the distributions over the old tasks (i.e., D.) and the tasks' data (i.e, Dz) are both unavailable at current time t. However, we can approximate it by its empirical counterpart, based on l observed tasks:
E(M') s (aa(M)) + Div(DLn D2)
dlnn(N [dnN + In(1/6)
s ( Div(DIn. D.)+
2.V
(15)
{(u((M) 1 D,) +
d d[ln( /d]] In(1/0)
2f
2.V
where Div is rewritten as (iv T.) in the main body to avoid using the distribution notation Dj and DD,
Interestingly, the bound in both Eq. 14) and Eq. (5) contains two types of complexity terms that correspond two levels of our model: Div(D corresponds specifically the similarity between
target task with all t observed tasks, while / /dllin N (n(1/))((11))) belongs to the similarity among all tasks in general.
2N
Based on the definition of H-divergence in Definition we have
for future task T and {t+ 1,... nn.
Div(D, D2)
Pb, (1(h) (1()))
N
1m1
2
sup
Pe. (I(h)) (()N))
(16)
1=1 AEM
Σ Div(D,,
where the inequality holds due to s E {1..... t- 1), and current model Mr incorporates knowledge of the distribution from the old task T In fact, Div D2) can also be interpreted as the extent to which model Mr needs to be adjusted to learn task Th, Obviously, Div(D)... Dt) is generally larger than (Dv (D,) because T has already been learned previously, even though its data and model are no longer available.
can be concluded from the two bounds in Eq. 14) and Eq. 15) that a model M that has learned t tasks demonstrates consistent reasoning abilities on both future tasks and previously learned tasks. This consistency hinges on the model's ability zero-shot prediction and anti-forgetting. Additionally, the ability to maintain statistical consistency, i.e., showing stable performance across different tasks, is a key indicator of the success continual learning strategy, especially to anti-forge challenge. In practice, ensuring this consistency of trend requires carefully designed learning strategics and appropriate regularization measures to enhance anti-forgettin capabilities without interfering with the learning process.
B Implementation Details on Baselines
In the implementation of LwF (21). we conduct it based on LoRA, instead of fine-tuning the entire model. For each incremental task, we preserve the LoRA from the last task for the distillation procedure. Regarding ConStruct VL, the results are reproduced using the official codes and hyperpa- rameters. We use update steps nadt 10, the step size Àads 0.01 and the loss weight p p= 0.2. For ZSCL 49). which employs a KD loss to align with the original CLIP model, the hyperparameter dictates the weight ensemble frequency, as specified in the original papers. The KD weight is set to 1, and the hyperparameter / is set to 5 to enhance learning capability and overall performance, although it still performs inadequately in the challenging Task VG+VAW benchmark. For MoE- Adapters [45), originally proposed for image classification tasks requiring task-id for inference,
16

---

Attr.
POS: cloudless shy
POS: resting cat NEG: running cat
NEC: stormy sky
spacial
NEG: hair
NEG: food NEG: standing
POS:
POS: NEG:
NEG: crefl materall
drinking
Figure 5: Examples of Structured VL Concept reasoning task.
adapt its implementation to include single task-dependent router for our VL task where there is no task boundary. Following the original paper, the number of experts is set to 14, and the router chooses top-4 experts during training and inference.
C Extended Results
In this section, we provide some extended results for the main text.
C.1 Extended Plugin Analysis
Mot: Adapters
MoE. Adapters
MoE: Adapters
Mot: Adapters
McE-Adapters
MoE AdepArds
100
Layened-LRRA
Layened-LoRA
Layered-LofA
Layened-LLRA
Layered-LoRA
100
770
EMA
ZAF (Ours)
EMA
ZAF (Qurs)
EMA
ZAF (Ours)
(9)
ConSiruct-W
ConStruct-Vi VL
ConStruct-VL
ConStruct-VL
ConStruct-LL
ConStructVL
wi
wio
Zero-shot Antidote
Zero-shot Antidote
Zero-shot Antidote
Zero Antidote
Antidote
Task VG+VAW
Task VG
[c] Task VAW
Figure 6: Effect of our zero-shot antidote various CL methods across three benchmarks.
We present a comprehensive plugin analysis by detailing the performances various methods with and without our zero-shot antidote in Table Here, we offer visual comparisons in Fig, 6o illustrate the impact of our zero-shot antidote on enhancing both zero-shot prediction and anti-forgetting capabilities across different methods. As demonstrated in the figure, incorporation our zero-shot antidote significantly improves zero-shot prediction performance on unseen tasks (indicated by the upper triangular red area) for each method without compromising their learning abilities (represented
17

---

Table 4: Component analysis of our ZAF under three pre-trained VL models.
Task VG+VAW
Task
5 Task
VL models
Method
FAA (t)
CAA (1)
FFM (L)
FAA (1)
CAA (1)
FFM (1)
FAA (7)
CAA (f)
FFM ())
Joint Learning
91.90
95,27
92.60
EMA-LORA
76.24
20.11
64.50
78.96
81.92
86.28
12.69
BLIP
ZAF (Ours)
90.05
92.49
92.39
1.97
89.13
90.03
3,93
Improvement
13.81
9.56
16.79
27.99
13:43
30.66
7.21
3.75
8.76
Joint Learning
93.72
95.31
92.90
EMA-LoRA
73.77
CapFilt-L
ZAF (Ours)
89.61
92.53
92.20
1.72
89.43
90.20
3,02
Improvement
13.81
10.09
16.48
18.76
10.16
21.82
8.69
4.33
11.06
Joint Learning
93.37
95.07
92.36
77.78
NIVR
7AF (Ours)
89.67
89.30
3.38
91.78
91.74
2.02
88.74
89.03
2.67
Improvement 11.89
8.34
13.92
16.76
9.52
19.53
5.66
2.82
7.51
by the diagonal yellow area). Moreover, the anti-forgetting capability (shown in the lower triangular blue area) of each method is also substantially improved. Although ConStruct-Vi VI. already exhibits strong anti-forgttting capabilities, its performance is further enhanced when its zero-shot prediction ability is augmented with our antidote. These visual comparisons not only support our empirical findings that the stability consecutive zero-shot predictions can serve as reliable indicator of its anti-forgetiing capabilities but also verify that zero-shot antidote successfully decouples learning from forgetting.
C.2 Detailed Component Analysis
In the main text, present part results for EMA-LoRA architecture and ZAF, noting that EMA- LoRA demonstrates performance comparable to Layered-LoRA, and the integration of the zero-shot antidote significantly enhances its performance. Here, we perform an extended ablation study in Table 4 to delve into the interaction between the EMA-LoRA architecture and zero-shot antidote under three pre-trained VL models. The excellent performance improvement in the table can be attributed to the close cooperation between the EMA-LoRA architecture and the zero-shot antidote. As illustrated Fig. 2 (b). on the one hand, the EMA-LoRA architecture effectively ensembles historical knowledge to provide informative zero-shot supervision. On the other hand, the zero-shot antidote offers an effective solution that separates the learning processes from the problem of anti- forgetting. By incorporating them, ZAF significantly mitigates forgetting and provides overwhelming performance compared existing baselines, along with significantly improved efficiency.
C.3 Detailed Performance Analysis across Different Task Orders
Table 5: Overall performance (%) of various CL methods across different task orders.
Task Order CA)
Order
Task Order
Method
FAA (t)
FFM (L)
FAA (t)
CAA (t)
FFM (1)
FAA (1)
CAA
FFM (L)
Joint Leaming
93.37
93.37
93.37
Continual-FT
67.23
73.60
27.96
71.68
75.31
22.24
65.61
76.88
29.65
LoRA
74.13
21.99
Layered-LLode
3
80.62
79.89
13.92
75.92
83.47
19.66
78.23
76.96
24.86
77.26
75.38
20.58
81.32
ZSCL
49
60.27
67.94
28.48
60.27
67.94
28.48
55.46
70.95
35.68
MoE-Adapiers
45
72.50
74.81
23.74
71.42
76.96
24.86
72.61
78.73
23.27
ConStruct-VL
33
85.97
87.00
6.94
86.07
91.36
5.49
86.56
90.15
4.88
ZAF (Ours)
89.67
89.30
3.38
90.67
93.12
2.33
90.16
92.84
2.35
Improvement
3.70
2.30
3.56
4.60
1.76
3.16
3.60
2.69
2.53
illustrates the results of various methods across three different task orders, demonstrating the robustness of our approach. In Table further present the quantitative
In the main text, Fig.
18

---

performance of all these methods. It is evident that our ZAF method consistently outperforms all baselines by substantial margins. Moreover, our method exhibits minimal performance variation across different task orders, with the FAA consistently hovering around 90%. In contrast, none of the baselines come close reaching the FAA near 90%.
C.4 Extended Hyperparameter Analysis
.0
0.65
0.70
100
100
70
95
d= 0.90
a 0.85
0. 80
a 0.75
70
7 Task VG+VAW
0.50
0.55
a 0.60
0.65
a 0,70
50
0. 95
0,90
0.85
0.80
0.75
00
Task
0.50
æ= 0.55
a 0.60
0.65 65
0.70
102
= 0. 95
0. 0.
0.85
0,80
0.75
Task VAW
Figure Heatmap changes for ZAF across various values in three benchmarks.
The hyperparameter introduced by our method is important to control the rate of parameter updates. As depicted in Fig. 3 our ZAF method exhibits remarkable robustness across wide range of values compared to three baselines. Further insights into the qualitative impact of a the anti-forgetie norgefoggg capability are provided Fig. 7 which presents heatmaps our ZAF performance across various values. It can be observed from these visualizations that increase in the value significantly
19

---

improves the zero-shot prediction stability, thereby enhancing the model's s anti norgeforgg capability, This enhancement is attributed to preserving more historical knowledge higher values, which ensures more stable zero-shot predictions and less forgetting. However, it is important to note that setting a to 0.9 and 0.95 compromises the model's ability integrate new knowledge effectively, resulting in reduced learning ability (the brighter diagonal yellow area) and decline in overall performance. These observations underscore the effectiveness of our ZAF implementation and, in turn, also validate our empirical findings.
C.5
Wild Data Composition Analysis
caramic boy sandy
taupe finted grazing dog
rigid engine
polygonal umbrella
hanging handlc orlented beffigid waving sail
playing doughnut larracotla jeans
surrounding.ct
alim beadboard
Images
front left window on B
Texis
Sanin macaw
reating recliner against wall
board
textured horse eilver
slender sheep
skiing
fabric
serving bright trim
chrome sall
pointing horse
full biender
empty blender
plate chasing place mat
Figure 8: Randomly selected examples from our wild dataset.
The construction of the wild dataset is important for stabilizing zero-shot predictions across broad spectrum of downstream tasks. Like the SVLC multimodal benchmark, each sample in our wild dataset consists of triplets, including a single image and two texts. These texts do not correspond the image content; one text manipulated by altering specific concept found in the other. To ensure that images and texts remain unpaired, we shuffle them in the training batch, as done in the provided code. For our dataset, we first collect substantial collection of images from the VG and VAW datasets. We then identify the main objects within these images to form list of nouns. Next, we use ChatGPT4 (25) to generate list of concepts words covering wide range of structured VI. concepts, such shape (e.g., round, square, triangular), texture (e.g., smooth, coarse, sandy), and brightness (e.g., faint, light, bright). For each image, we then randomly combine words from the noun and concept lists in grammatical format create text. We create another text for the image by replacing the concept word in this text with another randomly selected concept from our concept list. It is important to note that the clarity of these texts (e.g.. round cat, sandy couple, faint bus) is not prioritized, as the main objective is to evaluate the model's 's zero-shot ability to handle image- text pairs. The generation process allows us create wild data efficiently with the help of ChatGPT4. In total, our dataset comprises /2,358 unique images and 30, /44 unique texts, resulting in 2 1,006 triplets. Fig. shows some randomly selected examples from wild dataset.
Task VG+VAW
Task VG
Task VAW
Final Average Accuracy Cumulative Average Accuracy
90.07
51.78
91.74
98
15.74
23:03
50
50
$0.01
38.55
Accunscy (N)
86
86
86
$3.02
75.75
74.02
80
Ungrammatical
Text
Text
Text
Text
Test
Figure 9: Performance of our ZAF using the wild data with different compositions.
As shown in Fig. 9 our comparative analysis evaluates the performance ZAF using different compositions of wild data. First, employing only current task data to calculate the zero-shot loss
20

---

yields notably limited results. This limitation stems from coupled training targets, similar to traditional knowledge distillation techniques, which rely on the same data source, as illustrated Subsection 4.2 Moreover, utilizing wild data with ungrammatical text leads to decreased performance, likely because the model tends to overlook image information and classify image-text pairs with ungrammatical texts as inherently negative. This limitation also hampers the effective evaluation of the model's ' zero-shot capabilities. In contrast, employing grammatically correct text achieves the best performance in our experiments. The figure clearly demonstrates that employing grammatically correct texts significantly enhances both FAA and CAA, thereby confirming the effectiveness of our process for creating wild data.
C.6 Extended Comparisons
Table 6: Extended comparison with unimodal CL methods across three benchmarks.
7 Task VG+VAW
7 Task VG
4 Task VAW
Method
FAA (f)
CAA (t)
FAA (T)
CAA (1)
FAA (1)
CAA
Joint Leaning
93.37
95.07
92.26
1.2 3
76.36
85.59
77.40
87.12
83.72
86.01
EWC 15
73.29
81.81
72.78
82.61
86.00
87.26
L2P 43
66.96
76.46
60.88
69.23
62.27
68.68
Dual Prompt
+2
58.59
68.64
65.55
66.94
60.07
71.98
ConStruct-VL 33
85.40
90.88
86.99
93.00
87.50
89.77
7AF (Ours)
90.77
93.23
91.71
94.73
89.26
90.95
Average Improvement
5.37
2.35
4.72
1.73
1.76
1.18
In Table we provide the overall performance of our ZAF method and other multimodal baselines. To conduct more comprehensive comparisons with CL methods, Table 6 presents the performances for widely-used regularization based unimodal CL methods such as L2 34 and Elastic Weight Consolidation (EWC) [15), alongside prompt-based baselines like Learning to Prompt (1.2P) [43) and Dual Prompt (DP) [42 Here, we use the rel. spatial + attr. size - attr. material - rel. action attr. color object state attr. action 7-task sequence in the '7 Task VG+VAW benchmark and '7 Task VG' benchmark, and the aftr. action -> object state -> attr. color > attr. size 4-task sequence in the '4 Task VAW" benchmark, using the 'BLIP w/ CapFilt-L' model. The task sequence settings and results are excerpted from (33
The table shows substantial performance gap between unimodal and multimodal methods, highlight- ing the insufficient adaptation ability of visual CL methods in multimodal scenarios. Additionally, our ZAF method consistently outperforms ConStruct- VL with notable margins in the FAA metric across three benchmarks, further proving its effectiveness. The more challenging the benchmark, the more remarkable the margins become.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer:[Yes Yes]
Justification: [NA]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuii wwwiicccguie//eeeeeeeeeis CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:::///nuriipee www.e/iw.......uu.ppeeee httpp///scccpp.tttttttttttteeeea
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
25

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
26

---

Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "ukXeYY5viR", "reviewer": "Reviewer_CbcE", "summary": "The paper presents a novel continual learning (CL) method named ZAF (Zero-shot Antidote to Forgetting) aimed at enhancing the performance of pre-trained vision-language (VL) models in zero-shot prediction tasks. The authors identify zero-shot stability as a key indicator of a model’s ability to retain previously learned information and propose a zero-shot stability regularization technique to maintain this stability. ZAF employs an EMA-LoRA architecture for parameter-efficient learning and applies zero-shot antidote regularization to wild data, decoupling the processes of learning and forgetting.", "questions": "please see weakness.", "limitations": "The authors discussed some limitations in the supplementary.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The introduction of zero-shot stability as a measure for anti-forgetting is a novel and promising concept. This approach allows for a plug-and-play solution that can be integrated with existing CL frameworks.\n\n2.The use of an Exponential Moving Average (EMA) combined with Low-Rank Adaptation (LoRA) for parameter efficiency is a significant contribution. This architecture balances the need for efficient adaptation to new tasks while maintaining access to historical models.\n\n3.The extensive experiments conducted on various VL benchmarks demonstrate the effectiveness of ZAF. ZAF’s approach to training significantly reduces computational costs compared to methods like ConStruct-VL.", "weaknesses": "1.\tFor the empirical study in Section 3.2 and Figure 1, the evaluations is the same as in [49] except for the datasets used, are there any differences? Additionally, could you provide a more detailed discussion on why “a model’s stability in zero-shot predictions can reflect its anti-forgetting capabilities” (line 152) can be observed in Figure 1?\n2.    The proposed method is restrictede to specific pre-trained model (BLIP). Can it be generalzied to more general multi-modal models, such as CLIP?  \n3.\tFor the Loss_ZS, have you performed ablations using other measurements such as cross-entropy, which aligns with the basic Loss_CE optimization of the model, or using KL divergence, etc.?\n4.\tThe idea of using EMA for updating offline LoRA has been proposed in a previous work, LAE [1]. Could you discuss the differences between the two approaches?\n5.\tThe experiment results in Table 1 are lower than those presented in ConStruct-VL [33], and even the proposed method shows lower results compared to [33]. Considering the experiment settings are identical to those in [33], could you explain the reasons behind this discrepancy?\n6.\tAdditionally, could you clarify why, in some cases, the FAA is actually higher than the CAA? Which mechanism in your method allows predictions for previous tasks to improve when learning a new task?\n\n[1] Gao, Q., Zhao, C., Sun, Y., Xi, T., Zhang, G., Ghanem, B., & Zhang, J. (2023). A unified continual learning framework with general parameter-efficient tuning. ICCV 2023.\n\nI will reconsider my rating upon the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review and valuable suggestions. We have addressed the common questions separately and will now respond to the unique concerns raised.\n\n1. **Differences between the evaluations in our Figure 1 and ZSCL[49]**\n\nOur approach diverges from ZSCL in both objectives and evaluation metrics. While ZSCL aims to optimize zero-shot and continual learning performances simultaneously, our research focuses on enhancing continual learning through zero-shot stability regularization. Evaluation Metrics Comparison:\n- **ZSCL**:\n  - Avg Metric: Average accuracy across all tasks and timestamps.\n  - Last Metric: Performance of all tasks after the full learning sequence.\n  - Transfer Metric: Zero-shot transfer ability, calculated from the upper-right matrix.\n- **Ours**:\n  - Final Average Accuracy(FAA): Similar to ZSCL's Last metric.\n  - Cumulative Average Accuracy(CAA): Measures average accuracy across learned tasks, derived from the lower-left matrix.\n  - Final Forgetting Measure(FFM): Assesses knowledge retention derived from the lower-left matrix.\n\nWe also introduce a **zero-shot stability metric (ZSS in Table R1)**, which evaluates the consistency of zero-shot performance across learning stages, computed in the upper-right triangle but **totally differnt from  ZSCL's Transfer Metric**. These tailored metrics underscore our specific focus on continual learning's unique challenges, demonstrating a clear methodological divergence from ZSCL's broader objectives.\n\n2. **Generalzied to more general multi-modal models, such as CLIP**\n\nWe evaluated our method using the BLIP model and its two variants. Here, we tested CLIP, a predecessor of BLIP known for its robust performance. Below is the performance comparison across three datasets:\n\n|Methods|7 Task VG+VAW |7 Task VG|5 Task VAW|\n|-|-|-|-|\n||FAA (↑) CAA (↑) FFM (↓)|FAA (↑) CAA (↑) FFM (↓)| FAA (↑) CAA (↑) FFM (↓)\n|LoRA|67.79 73.31 24.96|63.48 76.30 36.57|75.22 74.08 15.68\n|EMA-LoRA|69.24 73.16 22.76|64.35 77.34 33.87|76.25 76.61 14.41 \n|ZAF|75.04 75.79 16.80|68.57 78.223 28.77|77.64 77.05 13.30\n\nOur findings show that CLIP, though versatile, **underperforms compared to BLIP in our tests**. BLIP's optimized representational capabilities for image and text inputs make it better suited for the complex reasoning tasks we evaluate. This highlights the importance of **selecting the right pre-trained model for specific experimental challenges**.\n\n3. **Ablations using other measurements such as cross-entropy(CE) or KL divergence for the $\\mathcal{L}{ZS}$**\n\nThank you for the suggestion. Initially, we used the L1-norm for the $\\mathcal{L}_{ZS}$ loss term due to its computational efficiency. To further test our method's robustness, we experimented with CE and KL divergence as alternative loss measures. Below is a summary of the performance metrics for these loss variants across three datasets:\n\n|Model|Metric|7 Task VG+VAW |7 Task VG|5 Task VAW|\n|-|-|-|-|-|\n|||FAA (↑) CAA (↑) FFM (↓)|FAA (↑) CAA (↑) FFM (↓)|FAA (↑) CAA (↑) FFM (↓)\n|BLIP|$\\mathcal{l}_1$ norm|90.05  89.45  3.32|92.49 92.39 1.97|89.13 90.03 3.93\n||$KL$/$CE$|89.32 88.44 3.55|90.64 90.92 2.97|88.14 88.70 3.97\n|BLIP w/ NLVR|$\\mathcal{l}_1$ norm|89.67 89.30 3.38|91.78 91.74 2.02 |88.74 89.03 2.67\n||$KL$/$CE$|89.46 88.37 2.99|90.64 90.02 2.97|88.25 87.81 2.38\n\nPerformance is comparable across all three loss metrics, with a slight edge for L1-norm, showing **the robustness of our method to variations in loss design**. Notably, KL and CE produced identical results, reflecting their mathematical equivalence in binary prediction tasks, where predictions fit Bernoulli distributions. \n\n4. **Reasons for lower experimental results in Table 1 compared to ConStruct-VL[33]**\n\nDiscrepancies between our results and those reported for ConStruct-VL[33] in Table 1 arise from **different task orders** in our continual learning experiments. As detailed in Appendix C.3, **Task Order 2**, which aligns with ConStruct-VL's sequence, involves a reduced number of training samples per task. This effectively lowers the complexity and challenge of the task environment, leading to higher performance metrics:\n\n|Method|FAA (↑)|CAA (↑)|FFM (↓)\n|-|-|-|-|\n|Reported ConStruct-VL in[33]|85.4|90.88|-\n|Reprodeced ConStruct-VL|86.07|91.36|5.49\n|ZAF|90.67|93.12|2.33\n\nThe forgetting metric used by ConStruct-VL[33] averages forgetting across all tasks, different from our methodology, making direct comparisons somewhat skewed. Nevertheless, our data confirms that ZAF **not only replicates but significantly outperforms ConStruct-VL** under comparable conditions, demonstrating our approach's robustness and superior efficacy.\n\n5. **Why the FAA is higher than the CAA in some cases**\n\nThere is **no preset rule** that FAA should be higher or lower than CAA; this depends on the model's learning dynamics and how well it manages forgetting. For example, in our ZAF model using the 'BLIP w/ NVLR' framework on the 7-task VG+VAW benchmark, FAA occasionally exceeds CAA when newer tasks are learned effectively or earlier tasks are well-retained:\n\n|ZAF|$T_1$|$T_2$|$T_3$|$T_4$|$T_5$|$T_6$|$T_7$|AVG\n|-|-|-|-|-|-|-|-|-|\n|$T_1$|91.03|-|-|-|-|-|-|91.03\n|$T_2$|89.63|89.70|-|-|-|-|-|89.66\n|$T_3$|86.15|88.59|85.57|-|-|-|-|86.77\n|$T_4$|85.10|87.85|85.15|97.36|-|-|-|88.87\n|$T_5$|81.97|87.57|83.86|96.95|95.09|-|-|89.09\n|$T_6$|81.97|86.83|82.76|96.96|94.37|97.08|-|90.00\n|$T_7$|79.79|86.27|82.11|96.29|94.20|96.87|92.17|89.67\n\nHere, the FAA (89.67) slightly exceeds the CAA (89.30, the average of the AVG column) due to strong performance on the last task and minimal forgetting of earlier tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eCdls1sw2W", "reviewer": "Reviewer_DvN8", "summary": "This paper tackles continual pretraining / training for vision-language models by introducing both exponentially moving averaged LoRA, and more importantly, replay during training on additional unaligned text and image data. In doing so, the authors show that higher performance on standard continual VL tasks can be achieved. To motivate and support this approach, the authors also conduct a simple zero-shot generalization versus forgetting study, and highlight the connection between retained zero-shot generalization and reduced forgetting, using it as a (theoretically motivated) driver for the regularization / replay on additional data.", "questions": "I am currently advocating for rejection for reasons listed above. In particular, I currently fail to see how improvements don't just stem from conducting semi-supervised training and effectively introducing a replay buffer on data similar to that encountered during original pretraining, which gives an expected, yet unfair advantage to existing methods (and which is not an actual contribution unfortunately).\n\nI'm willing to raise my score, but would need to see this, as well as the smaller issues listed above, addressed.", "limitations": "The paper explicitly discusses limitations and societal impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper is sufficiently well written to make it easy to follow.\n\n* The performance of both EMA-LoRA and particular ZAF compared to other approaches is, on a purely numerical basis, very convincing and significant.", "weaknesses": "__[1]__ On the sensibility of zero-shot prediction as indicator for forgetting.\n\nSection 3.2 has effectively no actionability as insights are entirely of qualitative nature. It’s crucial to provide detailed numerical comparisons (L149-151). Moreover, it is quite expected that reduced changes in forward generalization / generalization on yet unseen data come with reduced forgetting, since it likely indicates that the model has forgotten loss / has experienced less catastrophic feature overwriting. For a model that zero-shot generalizes sufficiently well to all tasks before adaptation, retention of this zero-shot performance on remaining tasks naturally results in better performance on tasks encountered during training.\n\n> This has been explored before in e.g. Stojanovski et al. 2022 (https://arxiv.org/abs/2211.03186), which already show that an EMA-style objective helps when deploying pretrained models in a continual scenario. \n\nIn my eyes, the reasoning starts from the wrong premise - it is the choice of a more suitable method that results in higher zero-shot and seen-task / anti-forgetting performance. As a result, it is somewhat meaningless to use these metrics as a regularizer for a method, since these are the exact metrics methods want to optimize for anyway (e.g. also L187 “underscores that the model’s capabilities in zero-shot prediction can reliably indicate its anti-forgetting capabilities”).\n\n> However, I’d love to hear the authors perspective on this. How would the methods zero-shot performance cause the “anti-forgetting” capabilities - as opposed to a confounder (i.e. the choice of method) improving both?\n\n---\n\n__[2]__ Method and Experiments\n\n__2.1__ I would love to get more details on the motivation behind the loss term in Eq. 4.\n\n> What exactly is happening with each loss term, and what is the exact motivation behind these choices?\n\n__2.2__ More importantly however, if I understand correctly, the authors simply introduce additional data during tuning compared to existing approaches. Even if unlabelled, this is effectively semi-supervised training, which is a strict extension over supervised training on a same-sized supervised set. However, the authors do not discuss any semi-supervised learning works (nor do they compare against it).\n\n> Firstly, I might have missed it, but where is the additional data taken from? Secondly, how is the proposed approach not just replay on additional data similar to the original pretraining data - something that is an orthogonal improvement for any replay-free baseline method compared against?\n\n__2.3__ I’m not sure if the authors are claiming EMA training as a contribution (e.g. for EMA-LoRA).\n\n> If so, it would be great if the authors could relate this to existing works such as Stojanovski et al. 2022 linked above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and for recognizing the clarity and performance improvements of our manuscript. We have addressed the common questions separately and will now respond to the specific concerns you raised.\n\n**1. The relation of zero-shot prediction stability, absolute zero-shot performance, anti-forgetting, and model choice**\n\n- In our study, we concentrate on preserving acquired knowledge during the continual learning process by **stabilizing** the zero-shot prediction capabilities of a model as it learns. This stabilization is achieved **independently of any pre-existing generalization capabilities of a pre-trained model across tasks before adaptation**. Notably, the zero-shot data used are completely disjoint from both the original pretraining data and the downstream task data, ensuring a rigorous test of the model's adaptability and generalization capabilities.\n\n- Our primary concern is the **relative change** in zero-shot performance throughout the continual learning process, rather than the absolute zero-shot performance on downstream tasks. The rationale behind this approach includes: \n  - Even if a pre-trained model (such as BLIP) does not initially exhibit strong zero-shot abilities, it can still effectively preserve acquired knowledge by stabilizing zero-shot predictions. \n  - Conversely, a pre-trained model (e.g., BLIP w/ NLVR) with better initial zero-shot ability may still experience significant forgetting if zero-shot prediction stability is not maintained. \n\nThe detailed results are presented in the table below.\n\n|BLIP|7 Task VG+VAW | 7 Task VG | 5 Task VAW |\n|-|-|-|-\n|Zero-shot Accuracy|50.74|50.83|50.42\n|Final Forgetting w/o $L_{ZS}$|20.11|32.63|12.69\n|Final Forgetting w/ $L_{ZS}$|3.32|1.97|3.93\n\n|BLIP w/ CapFilt-L|7 Task VG+VAW | 7 Task VG | 5 Task VAW |\n|-|-|-|-\n|Zero-shot Accuracy|49.60|50.88|49.23\n|Final Forgetting w/o $L_{ZS}$|20.66|23.54|14.08\n|Final Forgetting w/ $L_{ZS}$|4.18|1.72|3.02\n\n|BLIP w/ NLVR|7 Task VG+VAW | 7 Task VG | 5 Task VAW |\n|-|-|-|-\n|Zero-shot Accuracy|67.89|68.82|70.39\n|Final Forgetting w/o $L_{ZS}$|17.30|21.55|10.18\n|Final Forgetting w/ $L_{ZS}$|3.38|2.02|2.67\n\n- It's crucial to understand that **zero-shot performance and anti-forgetting performance are not inherently linked** metrics in continual learning methods. Specifically, high zero-shot performance does not necessarily equate to enhanced seen-task or anti-forgetting performance. As detailed in Proposition 1, the generalization errors for both seen and unseen tasks are bounded by the discrepancy between task distributions (see term $Div$ and L185-187). This means that for continual tasks that closely resemble each other, such as classification tasks using CLIP noted in Stojanovski et al. 2022, there is a consistent correlation between high zero-shot and anti-forgetting performance when unseen tasks closely align with continual tasks. However, this correlation does not extend to dissimilar tasks, such as the reasoning tasks in our studies utilizing BLIP.\n\n**2. Where is the additional/wild data taken from?**\n\nAs mentioned in L194, further details about the construction of the wild dataset are provided in **Appendix C.5**. Importantly, these images/texts are entirely disjoint from both the original pretraining and downstream data, ensuring that our dataset provides a unique and independent testing environment. The wild dataset has been included in the supplementary materials. Additionally, Figure 9 presents a performance comparison of our method using the wild data with various compositions, further demonstrating its robustness.\n\n**3. Clarify the fairness of comparisons with existing methods**\n\n- As noted in L12, the proposed zero-shot stability regularization, implemented by the inclusion of additional data, facilitates the preservation of acquired knowledge **in a plug-and-play manner**. To ensure **fair comparisons**, we have also conducted **ablation studies** on existing continual learning methods with our zero-shot regularization and wild data, as detailed **in L297-311 and Table 3**. It can be observed that with the implementation of our zero-shot stability regularization, all existing methods consistently demonstrate significant improvements in their anti-forgetting capabilities, as indicated by reduced forgetting measurement (FFM).\n\n- Furthermore, as shown in Table 3, even though the robust baseline ConStruct-VL is augmented with a replay buffer for old samples, when enhanced with zero-shot stability regularization, our ZAF method still achieves comparable or slightly superior results. \n\n**4: Discuss some semi-supervised learning(SSL) works**\n\nOur method aligns with the inductive SSL framework, which utilizes both labeled and unlabeled data for training, as described in the survey by Van Engelen et al. [3]. However, our approach incorporates several distinct features:\n\n- **Purpose**: Unlike conventional SSL that enhances generalization within the labeled data space, our method uses unlabeled data to improve anti-forgetting capabilities across all previously learned tasks.\n\n- **Assumptions**: Our model doesn't rely on typical SSL assumptions like the manifold and low-density separation assumptions. Instead, it handles non-stationary task distributions, a common scenario in continual learning.\n\n- **Training Strategy**: Our approach diverges from standard SSL strategies like self-training or co-training by implementing stability regularization on predictions of unlabeled data, which aids in maintaining consistent task performance.\n\n- **Empirical Insights**: Empirical results (see Figure 9) demonstrate that leveraging unlabeled data from a diverse range of distributions is more effective than restricting to the labeled task distribution, challenging a core principle of traditional SSL where similarity between labeled and unlabeled data is preferable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QPafWG6S2e", "reviewer": "Reviewer_wYS7", "summary": "The paper introduces a novel continual learning (CL) method called ZAF (Zero-shot Antidote to Forgetting) designed to enhance the retention of previously learned skills in vision-language (VL) models without replaying historical data. The proposed approach leverages zero-shot stability as an indicator of anti-forgetting capabilities, incorporating a stability regularization term and an efficient EMA-LoRA neural architecture.", "questions": "How much wild data this method used in experiments.", "limitations": "As a fundamental research in machine learning, the potential negative societal impacts are not immediately apparent at this stage.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The introduction of zero-shot stability regularization and the EMA-LoRA architecture provides a novel way to tackle the forgetting problem in CL.\n\n2. Empirical and Theoretical Rigor: The combination of empirical studies and theoretical analysis strengthens the validity of the proposed method.\n\n3. Demonstrates significant improvements over existing methods across multiple benchmarks.", "weaknesses": "1. The paper evaluates the performance of ZAF using a relatively small number of subtasks. Conducting experiments with a larger variety of subtasks or across more extensive datasets could provide a more comprehensive validation of the method's effectiveness.\n\n2. Alongside the incremental learning performance on downstream tasks, it would be beneficial to report the model's zero-shot performance across a broad range of domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful summary and for acknowledging the contributions of our work with the ZAF model. We greatly appreciate your recognition of the paper's clear presentation, significant performance improvements, and its empirical and theoretical rigor. Your appreciation for the introduction of zero-shot stability regularization and the EMA-LoRA architecture is also highly valued by our team.\n\n1. **Conducting experiments with a larger variety of subtasks or across more extensive datasets**\n\nIn response to your suggestion, we expanded our evaluation framework to include two additional benchmarks, enhancing the diversity of our experiments. These benchmarks, namely the '5 Task VG+VAW' and the '5 Task VAW + 2 Task VG,' encompass a variety of distinct concepts from both the VG (Visual Genome) and VAW (Visual Attributes in the Wild) datasets. Below are the comparative results illustrating the performance of ConStruct-VL and our ZAF model:\n\n[5 Task VG+VAW Benchmark:]\n| Method  | 'BLIP' | 'BLIP w/ CapFilt-L' | 'BLIP w/ NVLR' |\n|-------|----------------|-----------|------------|\n|       | FAA (↑)  CAA (↑)  FFM (↓) | FAA (↑)  CAA (↑)  FFM (↓) | FAA (↑)  CAA (↑)  FFM (↓) \n|ConStruct-VL |84.02 86.38 6.17 | 82.69 85.43 6.61 | 83.63 84.97 5.01\n|ZAF| 89.26 90.05 3.61 | 89.28 89.91 3.39 | 89.05 89.10 2.40\n|**Improvement**| 5.24 3.67 2.56 | 6.59 4.48 3.22 | 5.42 4.13 2.61|\n\n[5 Task VAW + 2 Task VG Benchmark:]\n| Method  | 'BLIP' | 'BLIP w/ CapFilt-L' | 'BLIP w/ NVLR' |\n|-------|----------------|-----------|------------|\n|       | FAA (↑)  CAA (↑)  FFM (↓) | FAA (↑)  CAA (↑)  FFM (↓) | FAA (↑)  CAA (↑)  FFM (↓) \n|ConStruct-VL|84.11 86.69 6.45|83.13 85.97 6.94| 83.73 85.43 7.13\n|ZAF|90.13 90.96 2.12 | 89.87 90.63 2.97| 90.04 90.92 2.38\n|**Improvement**|6.02 4.27 4.33|6.74 4.66 3.97| 6.31 5.49 4.75 \n\nThese results affirm that ZAF not only sustains exceptional anti-forgetting performance across various task configurations but also consistently surpasses ConStruct-VL across all metrics. The improvements in FAA, CAA, and FFM are particularly notable, showcasing significant gains. This comprehensive evaluation substantiates ZAF's robustness and versatility, confirming its superior performance in more complex and varied learning environments.\n\n2. **Report the model's zero-shot performance across a broad range of domains**\n\nIn response to your suggestion, we have expanded our evaluation to include not only the continual learning performance but also the zero-shot performance of our ZAF model during the continual learning process. The results are tabulated as follows:\n\n[7 Task VG+VAW benchmark using 'BLIP':]\n|ZAF|$T_1$|$T_2$|$T_3$|$T_4$|$T_5$|$T_6$|$T_7$|AVG\n|---|---|---|---|---|---|---|---|---|\n|zero-shot performance|46.32|62.34|51.55|50.86|54.64|64.80|51.84|54.62\n|continual learning performance|90.59|90.91|85.94|97.36|95.64|97.34|92.50|92.90\n\n[7 Task VG+VAW benchmark using 'BLIP w/ NVLR':]\n|ZAF|$T_1$|$T_2$|$T_3$|$T_4$|$T_5$|$T_6$|$T_7$|AVG\n|---|---|---|---|---|---|---|---|---|\n|zero-shot performance|66.40|65.31|51.49|53.95|58.10|72.52|59.69|61.07\n|continual learning performance|91.03|89.70|85.57|97.36|95.09|97.08|92.17|92.57\n\n[7 Task VG benchmark using 'BLIP':]\n|ZAF|$T_1$|$T_2$|$T_3$|$T_4$|$T_5$|$T_6$|$T_7$|AVG\n|---|---|---|---|---|---|---|---|---|\n|zero-shot performance|47.00|55.91|50.00|50.03|50.54|60.60|50.16|52.03\n|continual learning performance|93.37|90.64|93.71|98.04|96.07|97.56|89.91|94.19\n\n[7 Task VG benchmark using 'BLIP w/ NVLR':]\n|ZAF|$T_1$|$T_2$|$T_3$|$T_4$|$T_5$|$T_6$|$T_7$|AVG\n|---|---|---|---|---|---|---|---|---|\n|zero-shot performance|68.00|66.50|50.51|50.16|59.11|63.27|61.06|59.80\n|continual learning performance|90.31|90.64|93.03|97.74|96.07|97.38|89.40|93.51\n\n[5 Task VAW benchmark using 'BLIP':]\n|ZAF|$T_1$|$T_2$|$T_3$|$T_4$|$T_5$|AVG\n|---|---|---|---|---|---|---|\n|zero-shot performance|46.08|71.72|58.42|67.86|65.39|61.89\n|continual learning performance|92.12|92.13|87.06|96.00|94.07|92.28\n\n[5 Task VAW benchmark using 'BLIP w/ NVLR':]\n|ZAF|$T_1$|$T_2$|$T_3$|$T_4$|$T_5$|AVG\n|---|---|---|---|---|---|---|\n|zero-shot performance|66.39|70.85|66.48|75.13|76.98|71.17\n|continual learning performance|91.07|88.63|86.32|94.94|93.44|90.88\n\nAs observed, our ZAF model delivers satisfactory continual learning performance across a sequence of downstream tasks. However, its absolute zero-shot performance across a broad range of unseen domains remains constrained, despite achieving the zero-shot stability we aim for. This outcome is expected, as utilizing current task data naturally bolsters continual learning performance at the expense of zero-shot prediction capabilities. Inspired by your feedback, we recognize the essential need to simultaneously optimize both continual learning and zero-shot learning capabilities to extend the practical applications of our model. Addressing this dual optimization challenge is the primary focus of our ongoing projects. We aim to significantly improve the model’s adaptability and generalization across unfamiliar tasks and domains, ensuring robust performance in a wider array of settings.\n\n3. **How much wild data this method used in experiments.**\n\nAs mentioned in Line 194, further details about the construction of the wild dataset are provided in Appendix C.5. In total, our wild dataset comprises 12,358 unique images and 30,144 texts, resulting in 21,006 unique triplets. Importantly, these images/texts are entirely disjoint from both the original pretraining and downstream data, ensuring that our dataset provides a unique and independent testing environment. The wild dataset has been included in the supplementary materials accompanying our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to all reviewers for your insightful feedback. We will first address common concerns about our zero-shot stability and EMA-LoRA architecture, followed by detailed responses to each reviewer’s specific comments.

**Q1: How zero-shot prediction stability indicates anti-forgetting capabilities**

**A1**: To elucidate how zero-shot stability serves as an indicator of anti-forgetting, we will explore this relationship from both empirical and theoretical perspectives. Supporting details are provided in Table R1 and Figure R1 of the newly submitted PDF file.
- **Empirical Evidence**: We included numerical values for all heatmap entries of Figure 1 within Figure R1. These metrics illustrate: Zero-shot Stability assessed through the ZSS metric in the performance matrix's upper-right triangle (red area); Anti-forgetting Ability evaluated using the FFM metric in the lower-left triangle (blue area); and Learning New Tasks measured by the average values along the diagonal (yellow area). These values show that higher zero-shot stability corresponds to improved anti-forgetting performance, demonstrating a direct correlation without negatively impacting new task learning abilities. Here are comparative metrics:
|7 Task VG+VAW|Zero-shot stability|Anti-forgetting|Learning|/|7 Task VG|Zero-shot stability|Anti-forgetting|Learning|
|-|-|-|-|-|-|-|-|-
|MoE-Adapters|5.21|23.74|92.85|/|MoE-Adapters|1.57|31.83|94.37|
|LoRA|6.03|27.25|92.91|/|LoRA|3.83|29.62|93.79|
|Layred-LoRA|4.95|13.92|92.83|/|Layred-LoRA|3.42|24.99|93.92|
|ConStruct-VL|3.39|6.94|92.16|/|ConStruct-VL|2.70|7.91|93.43|

- **Theoretical Support**: Per Proposition 1, the anti-forgetting ability ($\mathcal{E}\_{s}(\mathcal{M}^{t}) - \mathcal{E}\_{s}(\mathcal{M}^{s})  \leq Bound1$) and the stability of zero-shot predictions ($|\mathcal{E}{k} (\mathcal{M}^{t}) - \mathcal{E}{k} (\mathcal{M}^{s})|  \leq Bound2$) are tightly linked. The bounds provided in Table R1 show that the stability in zero-shot predictions effectively sets upper bounds for anti-forgetting ability, reinforcing the theoretical foundation that zero-shot prediction stability can predict anti-forgetting capabilities.
This refined explanation and empirical data support the thesis that zero-shot stability is an effective metric for evaluating a model's ability to maintain learned knowledge over sequential tasks.

**Q2: More details on the motivation behind the loss term in Eq.(4), and what causes predictions for previous tasks to improve when learning a new task**

**A2:** As theoretically analyzed in the bounds of Proposition 1, both zero-shot stability and anti-forgetting ability are influenced by **three key factors**: the empirical error of continual tasks $\hat{\mathcal{E}}$, the discrepancy between task distributions $\rm{Div}$, and the complexity of the parameter space $\sqrt{\cdot}$. These factors should be minimized as much as possible to maximize the consistency between zero-shot stability and anti-forgetting ability. 

In practice, this requires the implementation of carefully designed loss functions. As indicated in Eq.(4) of our manuscript:
  - The first term, $L_{CE}(P^t,\bar{P})$, aims to **minimize the empirical error** of continual tasks. 
  - The second term, $L\_{ZS}(P^t, \widehat{{P}^t})$, focuses on **minimizing the task discrepancy**, which is both crucial and intriguing. Specifically, to make the discrepancy $\sum\_{i=1}^{t} {\rm{Div}}(\mathcal{T}\_\textit{i}, \mathcal{T}\_\textit{k})$ approach zero, we encourage the distribution of task $\mathcal{T}^{k}$ to converge towards the tasks $\{\mathcal{T}^{i}\}\_{i=1}^{t}$ in the semantic space. Thus, we aim to align its predictions $P^t$ from the current LoRA model with the predictions $\widehat{{P}^t}$ derived from the EMA-LoRA model on tasks $\{\mathcal{T}^{i}\}\_{i=1}^{t-1}$.
  - To explicitly **minimize the model complexity**, we have implemented our method within a parameter-efficient EMA-LoRA architecture, as opposed to utilizing full-parameter fine-tuning.

Improving predictions for earlier tasks when introducing a new task is challenging due to the **low discrepancy/high similarity** required between task distributions. We attempt to demonstrate this under ideal conditions, but such similarity is rare. Thus, our primary focus is on maintaining performance on previous tasks (anti-forgetting), a more achievable goal in continual learning.

**Q3: Relation to existing EMA-style objective, such as[1,2]**

**A3:** Our approach differentiates from traditional EMA-style methods[1,2] by incorporating EMA-LoRA **not only during inference but also throughout training**. These weights are integrated into objective optimization as part of the $L_{ZS}(P^t, \widehat{{P}^t})$ equation, functioning as a zero-shot stability regularizer. This integration **enhances performance** as demonstrated in our ablation studies (Table 3), where EMA-LoRA with $L_{ZS}$ outperforms traditional configurations.

Additionally, using EMA weights during training **reduces sensitivity to momentum parameters** (L312-319). This was tested across various settings, showing robust performance with varying $\alpha$ values:
|$\alpha$|0.50|0.55|0.60|0.65|0.70|0.75|0.80|0.85|0.90|
|-|-|-|-|-|-|-|-|-|-
|EMA-LoRA |18.70|20.18|20.10|19.93|19.64|19.03|18.12|17.30|13.73|4.74| 
|ZAF|8.07|7.26|6.59|6.41|8.45|5.08|4.27|3.38|1.99|1.43 

In summary, our EMA-style objective, highlighted as one of our contributions, is indeed implemented effectively, as shown in the comparison table below:
|Metrics|Existing EMA-style Works|Our Work|
|-|-|-|
|Training|$\textcolor{green}{\times}$|$\textcolor{red}{\checkmark}$|
|Inference|$\textcolor{red}{\checkmark}$|$\textcolor{red}{\checkmark}$|
|Performance|worse|better|
|Momentum insensitivity|$\textcolor{green}{\times}$|$\textcolor{red}{\checkmark}$|

[1]Momentum-based Weight Interpolation of Strong Zero-Shot Models for Continual Learning

[2]A unified continual learning framework with general parameter-efficient tuning

Meta-review: The paper proposes a method to adapt pretrained vision language models to new tasks in a continual learning manner. 
The reviewers appreciated the contribution of the proposed replay free regularizer and the combination of EMA  and LoRA.
The main concerns of reviewers have been cleared during rebuttal and all reviewers have recommended acceptence.
Decision: Accept (poster)