Stabilizing Zero-Shot Prediction: A Novel Antidote to Forgetting in Continual Vision-Language Tasks

Authors: zijian  gao
xingxing  zhang
kele  xu
xinjun  mao
huaimin  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 10:22:08

Abstract: Continual learning (CL) empowers pre-trained vision-language (VL) models to efficiently adapt to a sequence of downstream tasks. However, these models often encounter challenges in retaining previously acquired skills due to parameter shifts and limited access to historical data. In response, recent efforts focus on devising specific frameworks and various replay strategies, striving for a typical learning-forgetting trade-off. Surprisingly, both our empirical research and theoretical analysis demonstrate that the stability of the model in consecutive zero-shot predictions serves as a reliable indicator of its anti-forgetting capabilities for previously learned tasks. 
Motivated by these insights, we develop a novel replay-free CL method named ZAF (Zero-shot Antidote to Forgetting), which preserves acquired knowledge through a zero-shot stability regularization applied to wild data in a plug-and-play manner. To enhance efficiency in adapting to new tasks and seamlessly access historical models, we introduce a parameter-efficient EMA-LoRA neural architecture based on the Exponential Moving Average (EMA). ZAF utilizes new data for low-rank adaptation (LoRA), complemented by a zero-shot antidote on wild data, effectively decoupling learning from forgetting. Our extensive experiments demonstrate ZAF's superior performance and robustness in pre-trained models across various continual VL concept learning tasks, achieving leads of up to 3.70\%, 4.82\%, and 4.38\%, along with at least a 10x acceleration in training speed on three benchmarks, respectively. Additionally, our zero-shot antidote significantly reduces forgetting in existing models by at least 6.37\%. Our code is available at https://github.com/Zi-Jian-Gao/Stabilizing-Zero-Shot-Prediction-ZAF.

Main paper: 
 introduction: 
 In the rapidly evolving landscape of artificial intelligence, pre-trained models have become fundamental to achieving state-of-the-art results across a myriad of applications [25] . Recently, large vision-language (VL) models have provided remarkable predictions on downstream tasks without any training examples [30, 13, 2, 19] . However, these models are typically trained on static datasets, which may not capture the continuously evolving variety and complexity of real-world data. As new concepts emerge and existing categories expand, the static nature of these pre-trained models can lead to diminished performance over time. For example, the CLIP model [30] achieves an accuracy of less than 60% on the MNIST dataset (i.e., a performance significantly lower than that of a conventionally trained CNN [17] ) [49] . To bridge this gap, continual learning (CL) has emerged as a vital methodology, making learning new knowledge a lifelong process for the pre-trained VL model. However, the adaptation process, whether involving comprehensive fine-tuning of the entire pretrained model [7, 49, 3] or parameter-efficient continual fine-tuning [33, 45, 29] , inevitably undergo incremental parameter shifts. This presents a significant challenge in CL, where VL models forget historical knowledge as they adapt to new tasks [41, 45] . Additionally, the high computational costs required to integrate new data with old data, combined with often limited access to complete previous datasets, further exacerbate the severity of forgetting [47, 45] . Thus far, most CL research has focused on mitigating the forgetting of class information from previously learned images, a process known as class incremental learning [38] . Multimodal tasks such as visual question answering (VQA) and natural language visual reasoning (NLVR) have received less attention. Building upon the foundations laid by unimodal CL efforts, recent initiatives have aimed to mitigate forgetting from parameter shifts by developing robust continual VL learning frameworks, including Mixture-of-Experts [45] , weight consolidation [49] , and Layered-LoRA [33] . Furthermore, a growing body of work is dedicated to developing anti-forgetting strategies, such as knowledge distillation [23] and rectification [3] . Meanwhile, latent replay mechanisms, including adversarial pseudo-replay [33] and prototype learning [47] , have shown effectiveness in retaining historical knowledge, albeit with significant additional memory and computational costs. All these methods, however, struggle with the inherent trade-off between acquiring new knowledge and preserving historical knowledge.In this work, we aim to decouple the learning-forgetting dynamic, striving for a win-win outcomeenhancing anti-forgetting capabilities without interfering with the learning process. Our extensive empirical investigation reveals uniform patterns in the model's performance across learned and future VL tasks. Theoretically, we find that the generalization errors for old and new tasks are nearly identical. Motivated by these insights, particularly the observed stability of zero-shot predictions that reflects the stability required for old tasks (i.e., anti-forgetting capabilities), we develop a novel CL approach. This method, named ZAF (Zero-shot Antidote to Forgetting), incorporates a zero-shot prediction stability regularization term within our EMA-LoRA architecture. ZAF employs two low-rank adapters: one facilitates the learning of new tasks and ensures stable zero-shot predictions on wild data; the other, an Exponential Moving Average (EMA)-based adapter, preserves historical knowledge and utilizes it for zero-shot supervision. The synergistic interaction between these adapters effectively decouples learning from forgetting, thereby enhancing overall CL performance.Our contributions include: (1) We conduct a comprehensive empirical and theoretical study on continual learning for pre-trained VL models, establishing zero-shot stability as a reliable indicator of anti-forgetting capabilities; (2) Inspired by this finding, we develop a plug-and-play zero-shot antidote that enhances models' anti-forgetting across various CL methods, through a zero-shot prediction stability regularization on wild data; (3) We introduce an innovative replay-free CL method (termed ZAF), implemented within a parameter-efficient EMA-LoRA architecture, effectively decoupling learning from forgetting; (4) Across various continual VL benchmarks and pre-trained models, our approach achieves state-of-the-art performance and significantly reduced complexity.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: [NA] Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Continual Learning (CL): Continual learning aims to train a single model capable of incrementally updating its knowledge with a new sequence of tasks while preserving historical knowledge [41, 39] . However, due to parameter shift and limited access to historical data, the primary challenge in CL is forgetting previously learned tasks over time [41] . A plethora of strategies have been explored to address this issue [32, 21, 18, 6, 12] , including selective stabilization of network weights, replay of a few old training samples, construction of task-specific parameters, etc. Despite advancements in the unimodal community, such as in image classification, multimodal tasks have received comparatively less attention. This gap highlights the complexity and emerging interest in multimodal settings [9, 4, 8, 24, 35, 47] . In this context, VQACL [47] introduced a continual VQA task leveraging a latent replay strategy. RATT [4] focused on continual image captioning through weight regularization and knowledge distillation. Distinctively, ConStruct-VL [33] pursued continual adaptation to fine-grained Structured VL Concepts reasoning skills without fixed task boundaries. This is particularly applicable to real-world scenarios, and thus, we choose this VL task as a case study in our experiments.It is noteworthy that several studies closely related to ours discuss both zero-shot and continual learning capabilities [49, 45, 46] . However, their objectives differ fundamentally from ours. These studies, including ZSCL [49] and MoE-Adapters [45] , aim to preserve the zero-shot transfer ability inherent in pre-trained VL models during continual adaptation for sequentially arriving tasks, even though these capabilities may be inherently limited. In contrast, our research is exclusively focused on enhancing the anti-forgetting capabilities in downstream CL tasks with our zero-shot antidote.Vision-and-Language (VL) Pre-training: VL pre-training aims to enhance the performance of downstream VL tasks by pre-training models on large-scale image-text data. The effectiveness of these models largely depends on their representational capacity, including factors like data quality and model architecture, as well as the similarity between pre-training tasks and downstream applications. Generally, higher data quality and greater task similarity lead to stronger generalizability. However, due to the high costs of human annotation, most methods leverage noisy image-text pairs sourced from the web, which are a sub-optimal form of supervision (e.g., CLIP [30] and ALIGN [13] ). A novel dataset bootstrapping method, Captioning and Filtering (CapFilt), was introduced to utilize web datasets more effectively [19] . Furthermore, most existing pre-trained models excel either in understanding-based tasks like image-text retrieval [30, 20] or generation-based tasks such as image captioning [26, 11] . The primary challenge lies in designing model architectures capable of performing diverse tasks [44] . In response, the 'BLIP' framework was proposed, providing flexible transfer ability to both tasks [19] . Specifically, its image transformer is initialized from ViT pre-trained on ImageNet [37, 5] , and the text transformer originates from BERT [14] . An important variant of BLIP, enhanced with CapFilt to boost performance, is termed 'BLIP w/ CapFilt-L' [19] . Additionally, to enable reasoning on downstream tasks, 'BLIP' was further fine-tuned on the NLVR2 dataset [36] with a more computationally efficient architecture, referred to as 'BLIP w/ NLVR' [19] .In this work, we primarily focus on BLIP and its two variants, unlike the closely related studies [49, 45, 46] that utilize CLIP. Our choice is driven by the nature of our case study, which involves reasoning tasks as opposed to classification tasks typically associated with CLIP.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: [NA] Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary analysis: 
 In this section, we first define the problem of continual VL learning and then examine the correlation between the model's anti-forgetting capabilities and zero-shot stability through an empirical study.In this work, we focus on adapting pre-trained VL models to a sequence of newly arriving reasoning tasks, denoted as {T 1 , • • • , T n }. Each task T t involves a set of image-text pairs (I, T ), with a binary ground-truth function P (T, I) ∈ {(0, 1), (1, 0)} determining whether the text precisely describes the image content. The primary distinction among these tasks lies in the types of cognitive skills they aim to develop. Notably, traditional CL methods, which require identifying the specific task to which input belongs during inference, are impractical here. This is due to the challenge of discerning the required skills from free-form text and the likelihood that multiple skills may be simultaneously necessary. Given the constraints of privacy in practical applications, we adopt a strict policy of not retaining any task-specific data between training sessions. This leads us to a continual, replay-free setting without task identifiers, which is precisely the focus of our study.To investigate the correlation between a model's anti-forgetting capabilities and its stability in zeroshot predictions, we conducted an empirical study using the Structured VL Concepts (SVLC) learning multimodal benchmark [33] . This benchmark utilizes two widely recognized public VL datasets: Visual Genome (VG) [16] and Visual Attributes in the Wild (VAW) [28] , adhering to protocols outlined by the VL-Checklist [48] . The '7 Task VG+VAW' benchmark, derived from both VG and VAW datasets, includes seven distinct concepts designed to assess a VL model's understanding of various relationships, including spatial and inter-object transitive actions, as well as attributes such as size, color, material, and intransitive single-object actions. Examples of these concepts are depicted in Fig. 5 . Additionally, the '7 Task VG' benchmark is solely based on the VG dataset, and the '5 Task VAW' benchmark focuses on five attributes and object state concepts from the VAW dataset. We adhere strictly to the dataset splits outlined in ConStruct-VL [33] .We rigorously evaluate the official implementations of all baseline models to ensure a fair comparison. Our evaluations largely adhere to the training regimes established by LoRA [10] , MoE-Adapters [45] ,Figure 1 : Empirical study of anti-forgetting, learning, and zero-shot performance in CL methods.Layered-LoRA [33] , and ConStruct-VL [33] . Since MoE-Adapters [45] was originally designed for both pre-trained and downstream classification tasks requiring task-id inference, we adapt its implementation to include a single task-dependent router for our needs. Given the focus on reasoning tasks, we opt for 'BLIP w/ NLVR' as our pre-trained model, chosen for its robust representation capabilities and adequate zero-shot transfer ability.Fig. 1 illustrates a heatmap that depicts the performance of various CL methods across two benchmarks (see Appendix C.1 for additional benchmarks). In this heatmap, rows represent training steps, and A ij denotes the prediction accuracy on task T j after training on task T i . It is important to note that traditional CL methods typically focus only on the lower triangular matrix of results, which represents performance on previously learned tasks. However, our analysis extends to include zero-shot predictions on future tasks, which are represented in the upper triangular matrix. By comparing these results, we observe that larger average values and less fluctuation in the red area, which represents strong zero-shot stability, typically correspond to similar patterns in the blue area, indicative of strong anti-forgetting capabilities, without adversely affecting the yellow area, where learning new tasks occurs. This observation suggests that a model's stability in zero-shot predictions can reflect its anti-forgetting capabilities. Our findings further suggest that by systematically stabilizing zero-shot predictions during continual learning, we can significantly enhance the model's ability to retain historical knowledge without compromising the acquisition of new information.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
theoretical foundation and our approach: 
 In this section, we first provide a theoretical analysis to substantiate our empirical findings. Subsequently, we present an innovative CL approach specifically tailored for pre-trained VL models.In the continual learning of sequentially arriving VL tasks T t = {T, I}, the model is designed to align with the binary ground-truth function P (T, I) that evaluates whether the text T precisely describes the image content I. In this case study, we employ BLIP-based encoders to learn discriminative deep embeddings. Specifically, let f ν represent the image encoder and f τ the text encoder, while h ω and h ψ serve as the cross-attention decoder and the final binary classifier, respectively. The predictive modeling process within the BLIP framework for learning task T t is structured as follows:EQUATIONwhere P t (T, I) denotes the prediction probability made by the current model M t = {ν t , τ t , ω t , ψ t }.To sequentially fine-tune the BLIP model for downstream tasks, we employ a cross-entropy loss term, L CE (P t (T, I), P (T, I)). This loss measures the discrepancy between the current prediction, P t (•), and the ground truth, P (•), ensuring that the model progressively acquires new information. Let Ê1:t denote the empirical errors on the t observed tasks, and Êt represent the empirical error on the current task T t . For k ∈ {t + 1, . . . , n} and s ∈ {1, . . . , t -1}, E k and E s denote the generalization errors on the future task T k and the old task T s , respectively. Inspired by the PAC-Bayes theory [27] and previous work in domain generalization [1, 31, 40] , we present the upper bounds of these two errors under the continual VL scenario (see Appendix A for the detailed proof).Proposition 1 For continual learning with pre-trained VL models, let M t denote a solution of the continually learned tasksT 1 , • • • , T t . In particular, M t = arg min ||M-M t-1 ||2≤∆ Êt (M)where ||M -M t-1 || 2 ≤ ∆ represents the weight vectors for continual tasks are only minor variations.For any δ ∈ (0, 1) with probability at least 1 -δ:EQUATIONEQUATIONwhere Div(T i , T j ) := 2 sup h∈H |P Ti (I(h)) -P Tj (I(h))| defines the H-divergence between the distributions for tasks T i and T j , with I(h) being the characteristic function. Here, N denotes the harmonic mean of the training example sizes for the t observed tasks, and d represents the VC dimension of the parameter space.Proposition 1 demonstrates that the model M t has consistent upper bounds on the generalization errors for both previously learned and future tasks. These abilities are significantly influenced by three key factors: empirical error of continual tasks, discrepancy between task distributions, and complexity of the parameter space. Such consistency underscores that the model's capabilities in zero-shot prediction can reliably indicate its anti-forgetting capabilities. In practice, maintaining this consistency necessitates the implementation of carefully designed regularization techniques.Motivated by these insights, we develop a zero-shot regularization antidote to mitigate forgetting. Considering the unpredictable nature and broad array of future tasks, we initially introduce an unlabeled wild dataset, denoted as D wild = {T wild , I wild }, to assess the zero-shot capabilities of the continually learned VL model. It is crucial to note that both the text T wild and the images I wild are entirely unpaired and unlabeled2 . Additionally, the substantial number of wild examples in D wild are distinct and separate from the actual downstream tasks {T i } n i=1 . Fig. 8 illustrates some wild examples. To stabilize zero-shot predictions, we develop a novel loss term, L ZS (P t (T wild , I wild ), P t (T wild , I wild )), where P t (•) is the prediction probability derived from the model in the previous training step and used for zero-shot supervision. The formulation of our method's loss function is outlined below: L = LCE(P t (T, I), P (T, I)) + LZS(P t (Twild, Iwild), P t (Twild, Iwild)), while not traverse over all current data T t do 5:Sample a batch of current data B i , and a batch of wild data B wild 6:W = M 0 + A • B 7:Predict the probabilities P t (B i ) using W 8:Calculate L CE with P t (B i ) and ground truth P (B i )9: if t = 1 then 10:Optimize {A, B} with L CE 11:else 12:Predict the probabilities P t (B wild ) using W, and P t (B wild ) using M t-113:Calculate L ZS with P t (B wild ) and P t (B wild )14:Optimize {A, B} with loss function Eq. ( 4)15:if t = 1 then 16:W ← A • B, andM 1 = M 0 + W 17: else 18:Update the EMA component W and M t with Eq. ( 5) 19: return M t where L ZS = ||P t (T wild , I wild ) -P t (T wild , I wild )|| 1 in our experiments to restrain its fluctuations.To achieve the objective outlined above, rather than conducting comprehensive fine-tuning of the entire pre-trained model {ν t , τ t , ω t , ψ t } as defined in Eq. ( 1), we develop a parameter-efficient approach using an EMA-LoRA architecture based on BLIP. As depicted in Fig. 2(b ), for efficient adaptation to new tasks, we first integrate LoRA adapters into all layers of the image encoder f ν , text encoders f τ , and the cross-attention decode h ω , which are updated during the training of task T t . Specifically, each of f ν , f τ , and h ω consists of a combination of non-parametric functions -referred to here as data norms, such as LayerNorm, which remain frozen, and two types of parametric functions, namely linear and embedding functions. Typically, linear and embedding functions are parameterized by a weight matrix W, which is optimized at every iteration by adding residuals to the weights from the initial pre-trained VL model M 0 . Formally, this is represented as W = M 0 + A • B, where A and B are learnable low-rank matrices of dimensions m × r and r × l respectively, with m × l being the dimensions of M 0 . During the training of task T t , only the current task's LoRA adapters {A, B} are learned, and predictions P t (•) are made using W. This strategy not only enhances the adaptability of the model to new tasks but also maintains a balance between efficiency and performance.As demonstrated in Eq. ( 4), accessing previous models is essential for predicting wild data (i.e., P (•)).To facilitate this, we implement the Exponential Moving Average (EMA) on the aforementioned LoRA adapters (see Fig. 2(b) ). This approach allows for memory-efficient access to previous models while learning new tasks. Specifically, the EMA process is conducted at the end of every epoch:EQUATIONwhere the hyperparameter α ∈ [0, 1] plays a crucial role in updating the model parameters (refer to Fig. 3 for its sensitivity analysis). Importantly, M t represents the final model used for inference after continual learning. The weights W are updated following the completion of the first downstream task, at which point there is no risk of forgetting, thereby eliminating the need for the zero-shot antidote L ZS . For subsequent tasks, predictions on wild data using P t (•) are made with M t-1 .Algorithm 1 provides a detailed view of our training algorithm for the Zero-shot Antidote to Forgetting (ZAF) framework. Key highlights of the proposed ZAF include: (1) Learning vs. Forgetting: As illustrated in Fig. 2 (a) and (b), traditional CL frameworks, such as those incorporating knowledge distillation, often struggle with a trade-off between learning and forgetting. This trade-off typically manifests in the balance of L CE and L KD , where the LoRA adapter is optimized to align with both old and new knowledge using current data: min L CE (P t (T t ), P (T t )) + L KD (P t (T t ), P t-1 (T t )). Our approach introduces a zero-shot antidote using wild data to effectively separate the learning processes from the problem of forgetting: min L CE (P t (T t ), P (T t )) + L ZS (P t (D wild ), P t (D wild )).(2) Privacy Protections: Unlike conventional methods that require replaying old task data, our framework utilizes generated wild data to prevent forgetting, adhering to privacy constraints. (3) Innovative Mixture Strategy: Deviating from the typical spatial Mixture of Experts (MoE) and Layered-LoRA architectures, our model employs a temporal mixture strategy through EMA. This approach not only conserves memory but also integrates downstream tasks incrementally and effectively. ( 4) Efficiency in Resource Usage: The EMA-LoRA architecture involves only two low-rank adapters, which are computationally efficient, and also ensure ||M -M t-1 || 2 ≤ ∆, as established in our Proposition 1. 
experiment: 
 Benchmark: We utilize the '7 Task VG+VAW', '7 Task VG', and '5 Task VAW' benchmarks for our empirical studies as discussed in Sec. 3.2. We use the object state → attr. action → attr. size → rel. spatial → attr. material → rel. action → attr. color 7-task sequence in the 7 Task VG+VAW benchmark and 7 Task VG benchmark, and the object state → attr. action → attr. size → attr. material → attr. color 5-task sequence in the 5 Task VAW benchmark. Given the reasoning task scenario, our primary focus is on the original 'BLIP' model and its two variants: 'BLIP w/ CapFilt-L' and 'BLIP w/ NLVR'. Notably, the latter variant, which is fine-tuned on the NLVR2 dataset, exhibits enhanced zero-shot transfer ability.Model Architecture: All experiments begin with the BLIP model architectures and its pre-trained weights. The image encoder, denoted as f ν , utilizes ViTB/16, and the text encoder f τ is a BERT with a 12-layer encoder and 768 hidden size. The decoder h ω extends f τ , incorporating cross-attention layers between every two self-attention layers, with each receiving encoded image tokens as additional input. Our binary classifier h ψ is a 2-layer MLP with a hidden size of 768. For all baselines, we adhere to the implementation protocols established in ConStruct-VL [33] , a pioneering work in SVLC scenario that has demonstrated exceptional performance. We maintain a shared binary classifier h ψ across all task models. In practice, for the BLIP w/ NLVR model, h ψ is frozen across all tasks. For the BLIP and BLIP w/ CapFilt-L models, h ψ is trained only for the first task T 1 and then remains frozen for all subsequent tasks.Baseline and Implementation: Due to the inadequate performance of unimodal CL methods in multimodal scenarios, as highlighted by [33] , our analysis primarily focuses on frameworks and multimodal CL methods designed for VL scenarios. We include only one representative unimodal method, LwF [21] , which mitigates forgetting through knowledge distillation for comparison. Detailed comparisons with other unimodal CL methods are provided in Appendix C.6. Among the methods compared, Continual-FT [7] and LoRA [10] train the VL model continuously without incorporating anti-forgetting procedures. Layered-LoRA [33] and MoE-Adapters [45] are viewed as representative continual VL learning frameworks. Furthermore, we compare our ZAF with ZSCL [49] , which aims to preserve the inherent zero-shot capability of the pre-trained model, and ConStruct-VL [33] , which employs an expensive pseudo-replay strategy involving adversarial attacks on every sample. Following the implementation from previous work [33] , specifically, we adapt WD 0.05, an initial LR of 1.25e-3, a cosine scheduler, and a maximum of 12 training epochs in all experiments. For all low-rank adapters, the rank r is set to 16. The only hyperparameter, α, of our method is fixed at 0.85. Please refer to Appendix B for more implementation details.Evaluation Metrics: We utilize three widely-recognized evaluation metrics for continual learning -Final Average Accuracy (FAA), Cumulative Average Accuracy (CAA), and Final Forgetting Measure (FFM) as detailed in [41, 38] . We define the accuracy on the task T j after learning the task T i as A ij . The average accuracy after learning task T i is denoted asAA i = 1 i i j=1 A ij . Upon completing all n tasks, we report FAA = AA n , CAA = 1 n n i=1 AA i , and FFM = 1 n-1 n-1 j=1 max t∈{1,...,n-1} (A tj -A nj ).The FAA is a critical metric highlighting performance discrepancies between CL methods and joint learning. The CAA provides a comprehensive view of overall historical performance, and the FFM quantifies the model's capability to mitigate forgetting.Overall Performance: As shown in Table 1 , our ZAF consistently achieves the highest FAA, CAA, and lowest FFM, significantly outperforming competitors across three benchmarks with different pretrained models. ZSCL [49] , despite employing weight ensemble and knowledge distillation, exhibits inadequate learning on the challenging 7 Task VG+VAW and 7 Task VG benchmarks, resulting in declined performance. Similarly, MoE-Adapters, like Continual-FT [7] and LoRA [10] , which lack anti-forgetting mechanisms, exhibit significant forgetting as the number of tasks increases. In contrast, both the Layered-LoRA architecture [33] , with its task-specific parameter isolation strategy, and LwF [21] , with only a knowledge distillation strategy, show improved performance. Notably, only ConStruct-VL [33] approaches the performance of our method but requires computationally intensive training involving adversarial attacks and utilizing all historical models for pseudo-sample generation. Despite this, ZAF shows substantial advantages, leading by 3.70%, 4.82%, and 4.38% in FAA under the BLIP w/ NLVR model. Meanwhile, our method demonstrates a narrow performance gap compared to Joint Learning -the pinnacle of continual learning. The smaller this gap, the greater the challenge for CL methods to bridge it. Importantly, despite the inherent limitations in the zero-shot capabilities of some pre-trained VL models (e.g., BLIP and BLIP w/ CapFilt-L), our zero-shot antidote still effectively mitigates forgetting, underscoring its utility across a wide range of CL scenarios and pre-trained models.Complexity Analysis: Plugin Analysis: Table 3 presents a comprehensive plugin analysis, illustrating the significant reductions in FFM and improvements in FAA and CAA achieved by our zero-shot antidote when applied to various CL methods. In practical implementations, the model from the last task of the baselines is utilized to calculate the zero-shot regularization term. The results underscore the antiforgetting capabilities of the zero-shot antidote, as evidenced by an average reduction of 14.52% in the FFM, and increases of 11.96% in FAA and 7.10% in CAA, in the 7 Task VG benchmark. These observations further validate the universality and effectiveness of our proposed decoupled training objectives: enhancing anti-forgetting capabilities without compromising the learning process. For additional qualitative results, please refer to Appendix C.1. Furthermore, although our proposed EMA-LoRA architecture demonstrates anti-forgetting capabilities comparable to the Layered-LoRA framework, a comparative analysis when both are enhanced with the zero-shot antidote further underscores the superior performance of our approach (ZAF). This superiority arises from the synergistic effects of combining the EMA-LoRA neural architecture with zero-shot prediction stability. This configuration not only improves training efficiency but also significantly reduces memory usage.For additional results across various pre-trained models, please refer to Appendix C.2.Hyperparameter Analysis: Fig. 3 showcases the performance of our ZAF method across different hyperparameter values α ∈ [0, 1], including comparative analyses with Joint Learning and our strongest competitor, ConStruct-VL. Notably, when α ranges from 0.65 to 0.90, ZAF consistently outperforms ConStruct-VL by clear margins in both FAA and CAA metrics across three distinct Task Order Analysis: Fig. 4 presents the statistical results of various CL methods across three distinct task orders within the challenging 7 Task VG+VAW benchmark. Excluding the aforementioned task order within the 7 Task VG + VAW benchmark, the second and third orders are the rel. spatial → attr. size → attr. material → rel. action → attr. color → object state → attr. action task sequence and the rel. spatial → attr. material → attr. state → attr. action → attr. size → rel. action → attr. color task sequence, respectively. Our ZAF method exhibits remarkable robustness to variations in the task order, showing only minimal fluctuations across three performance metrics, and consistently outperforms existing CL methods by a significant margin. For detailed task sequence and quantitative results, please refer to Appendix C.3. 
discussion and conclusion: 
 In this work, we analyze the challenges of learning and forgetting in continual learning for pretrained VL models from both empirical and theoretical perspectives. A key finding is that zero-shot stability reliably indicates forgetting. We introduce a plug-and-play zero-shot antidote to enhance anti-forgetting capabilities across various CL methods, integrating it into our parameter-efficient EMA-LoRA architecture. This facilitates efficient adaptation and memory-free access to historical models, effectively circumventing the learning-forgetting trade-off prevalent in current CL works [3, 49, 45, 46] . Our approach demonstrates superior performance across various continual VL benchmarks and pre-trained models. We anticipate future research will explore the anti-forgetting challenge from the perspective of zero-shot prediction stability, diverging from traditional mechanisms.This work presents several potential limitations. The effectiveness of our plug-and-play zero-shot antidote presumes the existence of a feasible CL framework, which may not be available in all contexts. Also, it is designed specifically for a sequence of downstream tasks, limiting its applicability to the original training contexts of the pre-trained models. As a fundamental research in machine learning, the potential negative societal impacts are not immediately apparent at this stage. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: [NA] Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: [NA] Guidelines:• answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Preliminary For the convenience of readers who are not familiar with the PAC-Bayesian framework, we introduce the most relevant concepts from the literature here [27] . PAC-Bayesian theory studies the properties of randomized predictors, called Gibbs predictors. Formally, let X be an image-text input set 3 , Y be an output set, and H ⊂ {h : X → Y} a set of prediction functions (hypotheses). We assume that a distribution D is with X , Y, and a global labeling function h : X → Y, where h(x) generates target label for all the input, i.e., y = h(x). Lemma 1 Let S = {S i } s i=1 and T be s source distributions and a target distribution, respectively. The H-divergence between {S i } s i=1 and T is bounded as follows: EQUATIONwhere the first inequality is due to the triangle inequality (i.e., | i a i | ≤ i |a i |) and the second inequality is by the additivity of the sup function. This finishes the proof.Lemma 2 Given two distributions, T and S, let M 1 ∈ H and h 1 ∈ H be two prediction functions. The difference between the expected loss with T and S is bounded by the divergence between T and S as follows:EQUATIONwhere EQUATIONIt completes the proof.Lemma 3 Let Θ be a cover of a parameter space with VC dimension d. Then, for any δ ∈ (0, 1) with probability at least 1 -δ, for any M ∈ Θ:EQUATIONwhere ÊD (M) is an empirical risk with N samples in its training set D.Proof. For the distribution D, according to [22] , we haveEQUATIONwhere m Θ (N ) is the amount of all possible prediction results for N samples, which implies the model complexity in the parameter space Θ. We set m Θ (N ) = 1 . Then we getEQUATIONHence, the inequality |E D (θ) -ÊD (θ)| ≤ ϵ holds with probability at least 1 -δ.It completes the proof.If we continually learn t VL tasks Here, we use E k (M t ) instead of E k (M t , h) since we assume there is no difference between labeling functions for each task for simplicity. Specifically, within the context of continual VL reasoning tasks, the labeling function h is defined as a binary function. In fact, we cannot compute E k since the distributions over the tasks (i.e., D k ) and the tasks' data (i.e, D k ) are both unknown. However, we can approximate it by its empirical counterpart, based on the t observed tasks:T 1 , • • • , T t that follow the distribution D 1 , • • • , D t ,EQUATIONwhereÊD1:t (M t ) = 1 t t i=1 ÊDi (M t ) because ||M i+1 -M i || 2 ≤∆ that represents the weight vectors for continual tasks are only minor variations. The three inequalities hold due to Lemma 2, Lemma 3 and Lemma 1, respectively. D 1:t := {D i } t i=1 and we rewrite a mixture of all the t distributions as D 1:t :=1 t t i=1 D i using convex combination. N = 1 t t i=1 1 Ni -1is the harmonic of the training example sizes (i.e., N i ) in each task. Of note, Div(D i , D k ) is rewritten as Div(T i , T k ) in the main body to avoid using the distribution notation D i and D k . Furthermore, for previously learned tasks T s where s ∈ {1, • • • , t -1}, let E s denote the generalization error on old task T s . We cannot compute E s since the distributions over the old tasks (i.e., D s ) and the tasks' data (i.e, D s ) are both unavailable at current time t. However, we can approximate it by its empirical counterpart, based on t observed tasks:Es(M t ) ≤ E1:t(M t ) + 1 2Div(D1:t, Ds)EQUATIONwhere Div(D i , D s ) is rewritten as Div(T i , T s ) in the main body to avoid using the distribution notation D i and D s . Interestingly, the bound in both Eq. ( 14) and Eq. ( 15) contains two types of complexity terms that correspond to two levels of our model: Div(D 1:t , •) corresponds specifically to the similarity between target task with all t observed tasks, whiled[ln( N /d)]+ln(1/δ) 2 Nbelongs to the similarity among all tasks in general.Based on the definition of H-divergence in Definition 1, for future taskT k and k ∈ {t + 1, • • • , n}, we have t i=1 Div(Di, Dk) = 2 t i=1 sup h∈H |P D i (I(h)) -P D k (I(h))| ≥ 2 t i=1 sup h∈H |P D i (I(h)) -P Ds (I(h))| = t i=1Div(Di, Ds), (16) where the inequality holds due to s ∈ {1, • • • , t -1}, and current model M t incorporates knowledge of the distribution from the old task T s . In fact, Div(D 1:t , D k ) can also be interpreted as the extent to which model M t needs to be adjusted to learn task T k . Obviously, Div(D 1:t , D k ) is generally larger than Div(D 1:t , D s ) because T s has already been learned previously, even though its data and model are no longer available.It can be concluded from the two bounds in Eq. ( 14) and Eq. ( 15) that a model M t that has learned t tasks demonstrates consistent reasoning abilities on both future tasks and previously learned tasks. This consistency hinges on the model's ability to zero-shot prediction and anti-forgetting. Additionally, the ability to maintain statistical consistency, i.e., showing stable performance across different tasks, is a key indicator of the success of a continual learning strategy, especially to the anti-forgetting challenge. In practice, ensuring this consistency of trend requires carefully designed learning strategies and appropriate regularization measures to enhance anti-forgetting capabilities without interfering with the learning process.In the implementation of LwF [21] , we conduct it based on LoRA, instead of fine-tuning the entire model. For each incremental task, we preserve the LoRA from the last task for the distillation procedure. Regarding ConStruct-VL, the results are reproduced using the official codes and hyperparameters. We use update steps n adv = 10, the step size λ adv = 0.01 , and the loss weight ρ = 0.2.For ZSCL [49] , which employs a KD loss to align with the original CLIP model, the hyperparameter I dictates the weight ensemble frequency, as specified in the original papers. The KD weight is set to 1, and the hyperparameter I is set to 5 to enhance learning capability and overall performance, although it still performs inadequately in the challenging 7 Task VG+VAW benchmark. For MoE-Adapters [45] , originally proposed for image classification tasks requiring task-id for inference, we adapt its implementation to include a single task-dependent router for our VL task where there is no task boundary. Following the original paper, the number of experts is set to 14, and the router chooses the top-4 experts during training and inference.In this section, we provide some extended results for the main text. We present a comprehensive plugin analysis by detailing the performances of various methods with and without our zero-shot antidote in Table 3 . Here, we offer visual comparisons in Fig. 6 to illustrate the impact of our zero-shot antidote on enhancing both zero-shot prediction and anti-forgetting capabilities across different methods. As demonstrated in the figure, incorporation our zero-shot antidote significantly improves zero-shot prediction performance on unseen tasks (indicated by the upper triangular red area) for each method without compromising their learning abilities (represented by the diagonal yellow area). Moreover, the anti-forgetting capability (shown in the lower triangular blue area) of each method is also substantially improved. Although ConStruct-VL already exhibits strong anti-forgetting capabilities, its performance is further enhanced when its zero-shot prediction ability is augmented with our antidote. These visual comparisons not only support our empirical findings that the stability of consecutive zero-shot predictions can serve as a reliable indicator of its anti-forgetting capabilities but also verify that our zero-shot antidote successfully decouples learning from forgetting.In the main text, we present part results for EMA-LoRA architecture and ZAF, noting that EMA-LoRA demonstrates performance comparable to Layered-LoRA, and the integration of the zero-shot antidote significantly enhances its performance. Here, we perform an extended ablation study in Table 4 to delve into the interaction between the EMA-LoRA architecture and zero-shot antidote under three pre-trained VL models. The excellent performance improvement in the table can be attributed to the close cooperation between the EMA-LoRA architecture and the zero-shot antidote.As illustrated in Fig. 2 (b ), on the one hand, the EMA-LoRA architecture effectively ensembles historical knowledge to provide informative zero-shot supervision. On the other hand, the zero-shot antidote offers an effective solution that separates the learning processes from the problem of antiforgetting. By incorporating them, ZAF significantly mitigates forgetting and provides overwhelming performance compared to existing baselines, along with significantly improved efficiency. In the main text, Fig. 4 illustrates the results of various CL methods across three different task orders, demonstrating the robustness of our approach. In Table 5 , we further present the quantitative performance of all these methods. It evident that our ZAF method consistently outperforms all baselines by substantial margins. Moreover, our method exhibits minimal performance variation across different task orders, with the FAA consistently hovering around 90%. In contrast, none of the baselines come close to reaching the FAA near 90%. The hyperparameter α introduced by our method is important to control the rate of parameter updates.As depicted in Fig. 3 , our ZAF method exhibits remarkable robustness across a wide range of α values compared to three baselines. Further insights into the qualitative impact of α on the anti-forgetting capability are provided in Fig. 7 , which presents heatmaps of our ZAF performance across various α values. It can be observed from these visualizations that an increase in the α value significantly the zero-shot prediction stability, thereby enhancing the model's anti-forgetting capability. This enhancement is attributed to preserving more historical knowledge at higher α values, which ensures more stable zero-shot predictions and less forgetting. However, it is important to note that setting α to 0.9 and 0.95 compromises the model's ability to integrate new knowledge effectively, resulting in reduced learning ability (the brighter diagonal yellow area) and a decline in overall performance. These observations underscore the effectiveness of our ZAF implementation and, in turn, also validate our empirical findings. The construction of the wild dataset is important for stabilizing zero-shot predictions across a broad spectrum of downstream tasks. Like the SVLC multimodal benchmark, each sample in our wild dataset consists of triplets, including a single image and two texts. These texts do not correspond to the image content; one text is manipulated by altering a specific concept found in the other. To ensure that images and texts remain unpaired, we shuffle them in the training batch, as done in the provided code. For our dataset, we first collect a substantial collection of images from the VG and VAW datasets. We then identify the main objects within these images to form a list of nouns. Next, we use ChatGPT4 [25] to generate a list of concepts words covering a wide range of structured VL concepts, such as shape (e.g., round, square, triangular), texture (e.g., smooth, coarse, sandy), and brightness (e.g., faint, light, bright). For each image, we then randomly combine words from the noun and concept lists in a grammatical format to create one text. We create another text for the image by replacing the concept word in this text with another randomly selected concept from our concept list. It is important to note that the clarity of these texts (e.g., round cat, sandy couple, faint bus) is not prioritized, as the main objective is to evaluate the model's zero-shot ability to handle image-text pairs. The generation process allows us to create wild data efficiently with the help of ChatGPT4. In total, our dataset comprises 12,358 unique images and 30,144 unique texts, resulting in 21,006 triplets. Fig. 8 shows some randomly selected examples from our wild dataset. As shown in Fig. 9 , our comparative analysis evaluates the performance of ZAF using different compositions of wild data. First, employing only current task data to calculate the zero-shot loss yields notably limited results. This limitation from coupled training targets, similar to traditional knowledge distillation techniques, which rely on the same data source, as illustrated in Subsection 4.2. Moreover, utilizing wild data with ungrammatical text leads to decreased performance, likely because the model tends to overlook image information and classify image-text pairs with ungrammatical texts as inherently negative. This limitation also hampers the effective evaluation of the model's zero-shot capabilities. In contrast, employing grammatically correct text achieves the best performance in our experiments. The figure clearly demonstrates that employing grammatically correct texts significantly enhances both FAA and CAA, thereby confirming the effectiveness of our process for creating wild data. In Table 1 , we provide the overall performance of our ZAF method and other multimodal baselines.To conduct more comprehensive comparisons with CL methods, Table 6 presents the performances for widely-used regularization-based unimodal CL methods such as L2 [34] and Elastic Weight Consolidation (EWC) [15] , alongside prompt-based baselines like Learning to Prompt (L2P) [43] and Dual Prompt (DP) [42] . Here, we use the rel. spatial → attr. size → attr. material → rel. action → attr. color → object state → attr. action 7-task sequence in the '7 Task VG+VAW' benchmark and '7 Task VG' benchmark, and the attr. action → object state → attr. color → attr. size 4-task sequence in the '4 Task VAW' benchmark, using the 'BLIP w/ CapFilt-L' model. The task sequence settings and results are excerpted from [33] .The table shows a substantial performance gap between unimodal and multimodal methods, highlighting the insufficient adaptation ability of visual CL methods in multimodal scenarios. Additionally, our ZAF method consistently outperforms ConStruct-VL with notable margins in the FAA metric across three benchmarks, further proving its effectiveness. The more challenging the benchmark, the more remarkable the margins become.Paper ChecklistPlease refer to Appendix C.5 for more details about the construct of the wild dataset.We abbreviate the image-text input space as X instead of {X , I} for simplicity and ease of notation throughout this document. Likewise, x instead of {T, I}. 


Decision: The paper proposes a method to adapt pretrained vision language models to new tasks in a continual learning manner. 
The reviewers appreciated the contribution of the proposed replay free regularizer and the combination of EMA  and LoRA.
The main concerns of reviewers have been cleared during rebuttal and all reviewers have recommended acceptence.

Reviews: review: paper_summary: paper_summary: The paper presents a novel continual learning (CL) method named ZAF (Zero-shot Antidote to Forgetting) aimed at enhancing the performance of pre-trained vision-language (VL) models in zero-shot prediction tasks. The authors identify zero-shot stability as a key indicator of a model’s ability to retain previously learned information and propose a zero-shot stability regularization technique to maintain this stability. ZAF employs an EMA-LoRA architecture for parameter-efficient learning and applies zero-shot antidote regularization to wild data, decoupling the processes of learning and forgetting.
strength_weakness: strength_weakness: 1.	For the empirical study in Section 3.2 and Figure 1, the evaluations is the same as in [49] except for the datasets used, are there any differences? Additionally, could you provide a more detailed discussion on why “a model’s stability in zero-shot predictions can reflect its anti-forgetting capabilities” (line 152) can be observed in Figure 1?
2.    The proposed method is restrictede to specific pre-trained model (BLIP). Can it be generalzied to more general multi-modal models, such as CLIP?  
3.	For the Loss_ZS, have you performed ablations using other measurements such as cross-entropy, which aligns with the basic Loss_CE optimization of the model, or using KL divergence, etc.?
4.	The idea of using EMA for updating offline LoRA has been proposed in a previous work, LAE [1]. Could you discuss the differences between the two approaches?
5.	The experiment results in Table 1 are lower than those presented in ConStruct-VL [33], and even the proposed method shows lower results compared to [33]. Considering the experiment settings are identical to those in [33], could you explain the reasons behind this discrepancy?
6.	Additionally, could you clarify why, in some cases, the FAA is actually higher than the CAA? Which mechanism in your method allows predictions for previous tasks to improve when learning a new task?

[1] Gao, Q., Zhao, C., Sun, Y., Xi, T., Zhang, G., Ghanem, B., & Zhang, J. (2023). A unified continual learning framework with general parameter-efficient tuning. ICCV 2023.

I will reconsider my rating upon the rebuttal.
questions: questions: please see weakness.
limitations: limitations: The authors discussed some limitations in the supplementary.
paper_summary: paper_summary: This paper tackles continual pretraining / training for vision-language models by introducing both exponentially moving averaged LoRA, and more importantly, replay during training on additional unaligned text and image data. In doing so, the authors show that higher performance on standard continual VL tasks can be achieved. To motivate and support this approach, the authors also conduct a simple zero-shot generalization versus forgetting study, and highlight the connection between retained zero-shot generalization and reduced forgetting, using it as a (theoretically motivated) driver for the regularization / replay on additional data.
strength_weakness: strength_weakness: __[1]__ On the sensibility of zero-shot prediction as indicator for forgetting.

Section 3.2 has effectively no actionability as insights are entirely of qualitative nature. It’s crucial to provide detailed numerical comparisons (L149-151). Moreover, it is quite expected that reduced changes in forward generalization / generalization on yet unseen data come with reduced forgetting, since it likely indicates that the model has forgotten loss / has experienced less catastrophic feature overwriting. For a model that zero-shot generalizes sufficiently well to all tasks before adaptation, retention of this zero-shot performance on remaining tasks naturally results in better performance on tasks encountered during training.

> This has been explored before in e.g. Stojanovski et al. 2022 (https://arxiv.org/abs/2211.03186), which already show that an EMA-style objective helps when deploying pretrained models in a continual scenario. 

In my eyes, the reasoning starts from the wrong premise - it is the choice of a more suitable method that results in higher zero-shot and seen-task / anti-forgetting performance. As a result, it is somewhat meaningless to use these metrics as a regularizer for a method, since these are the exact metrics methods want to optimize for anyway (e.g. also L187 “underscores that the model’s capabilities in zero-shot prediction can reliably indicate its anti-forgetting capabilities”).

> However, I’d love to hear the authors perspective on this. How would the methods zero-shot performance cause the “anti-forgetting” capabilities - as opposed to a confounder (i.e. the choice of method) improving both?

---

__[2]__ Method and Experiments

__2.1__ I would love to get more details on the motivation behind the loss term in Eq. 4.

> What exactly is happening with each loss term, and what is the exact motivation behind these choices?

__2.2__ More importantly however, if I understand correctly, the authors simply introduce additional data during tuning compared to existing approaches. Even if unlabelled, this is effectively semi-supervised training, which is a strict extension over supervised training on a same-sized supervised set. However, the authors do not discuss any semi-supervised learning works (nor do they compare against it).

> Firstly, I might have missed it, but where is the additional data taken from? Secondly, how is the proposed approach not just replay on additional data similar to the original pretraining data - something that is an orthogonal improvement for any replay-free baseline method compared against?

__2.3__ I’m not sure if the authors are claiming EMA training as a contribution (e.g. for EMA-LoRA).

> If so, it would be great if the authors could relate this to existing works such as Stojanovski et al. 2022 linked above.
questions: questions: I am currently advocating for rejection for reasons listed above. In particular, I currently fail to see how improvements don't just stem from conducting semi-supervised training and effectively introducing a replay buffer on data similar to that encountered during original pretraining, which gives an expected, yet unfair advantage to existing methods (and which is not an actual contribution unfortunately).

I'm willing to raise my score, but would need to see this, as well as the smaller issues listed above, addressed.
limitations: limitations: The paper explicitly discusses limitations and societal impact.
paper_summary: paper_summary: The paper introduces a novel continual learning (CL) method called ZAF (Zero-shot Antidote to Forgetting) designed to enhance the retention of previously learned skills in vision-language (VL) models without replaying historical data. The proposed approach leverages zero-shot stability as an indicator of anti-forgetting capabilities, incorporating a stability regularization term and an efficient EMA-LoRA neural architecture.
strength_weakness: strength_weakness: 1. The paper evaluates the performance of ZAF using a relatively small number of subtasks. Conducting experiments with a larger variety of subtasks or across more extensive datasets could provide a more comprehensive validation of the method's effectiveness.

2. Alongside the incremental learning performance on downstream tasks, it would be beneficial to report the model's zero-shot performance across a broad range of domains.
questions: questions: How much wild data this method used in experiments.
limitations: limitations: As a fundamental research in machine learning, the potential negative societal impacts are not immediately apparent at this stage.

Comments: title: Response to Reviewer Comments
Response to Response
Rebuttal by Authors
Response to Reviewer Comments
Re: Rebuttal
comment: Dear Reviewer CbcE,

As the discussion period is set to end in one day, we wanted to kindly remind you that we have posted our latest response addressing your concerns. We have also addressed similar concerns raised by Reviewer DvN8. If you have any further questions or require clarification on any points, please do not hesitate to reach out. We are committed to addressing your queries promptly.

We greatly value your feedback and look forward to your insights.

Best, 

Authors
Thank you for your thoughtful review and for patiently accepting our clarifications. Your feedback has greatly helped us rethink our work and improve our paper. We truly appreciate your time and effort.
I think the authors again for the detailed reply, which has helped clarify my issues regarding the comparison to related works [1,2].

While I am still not convinced w.r.t. the separation from SSL, the newely provided table does provide good experimental support that the proposed zero-shot stability regularization indeed helps to better leverage additional unlabeled data for continual learning.

Given that these two points, alongside my previously raised questions have been consequently mostly addressed, I am raising my score and am leaning towards acceptance.
Thank you for your response. We appreciate the opportunity to address your concerns.

1.**Experiments with CLIP**

- The CLIP model, designed primarily for image-text similarity tasks, is not optimal for reasoning tasks requiring a classification head. In our experiments with three datasets featuring triplets (one image and two texts—one positive and one negative), we **adhered to CLIP's inference by evaluating similarities and assigning labels based on which text is more similar to the image**. For training, **consistent with CLIP's standard methodology**, we treated paired image-text as positive and unpaired ones as negative. However, CLIP's reliance on comparative similarity limits its ability to perform inference on a single image-text pair. Therefore, we chose BLIP and its variants for reasoning tasks, as they demonstrated superior performance.

- To develop LoRA, EMA-LoRA, and ZAF, we integrate low-rank adapters to CLIP's encoders. Specifically, in ZAF, we employed wild data alongside our zero-shot regularization, represented by $\mathcal{L}_{\text{ZS}}(S^t(T\_\text{wild}, I\_\text{wild}), \widehat{S^t}(T\_\text{wild}, I\_\text{wild}))$, where $S^t(\cdot)$ and $\widehat{S^t}(\cdot)$ denote the image-text similarities from the current and EMA-based models, respectively.

- Our ZAF model shows performance gains even when implemented with CLIP. As noted in the previous response, ZAF **clearly outperforms baseline methods when using CLIP**, demonstrating the effectiveness of our zero-shot regularization. Although the improvements are less pronounced than with BLIP, this is expected given CLIP's focus on image-text similarity rather than reasoning tasks. This distinction highlights the adaptability of our method across different pre-trained models.

2.**Performance gains not solely from wild data**

- Foundation: A key innovation of our study is the **empirical and theoretical demonstration** that zero-shot prediction stability when adapting to new tasks, effectively signals anti-forgetting capabilities. This finding underpins our use of wild data and the associated zero-shot stability regularization. Without wild data, the model would simply rely on the basic EMA-LoRA architecture, lacking innovative regularization.

- Methodology: **Simply adding more unlabeled data does not guarantee improved CL performance**. The table below demonstrates that using the same wild data under a self-supervised contrastive consistency loss(SSL), as opposed to zero-shot stability regularization, shows a notable performance decline.

|Method|7TaskVG+VAW|7TaskVG|5TaskVAW
|-|-|-|-
||FAA CAA FFM
|ConStruct-VL|85.97 87.00 6.94|86.96 90.47 7.91|84.36 85.93 5.36
|SSL|80.54 83.20 8.23|77.87 78.85 16.98|82.24 83.14 9.42
|ZAF|89.67 89.30 3.38|91.78 91.74 2.02|88.74 89.03 2.67

- Precedents: The use of additional unlabeled data in CL is an established concept and has been previously explored[1,2]. Typically, these methods selectively use data from external datasets to mimic the distribution of old tasks to combat forgetting, less competitive compared to our method **in both learning and anti-forgetting**. The ZSCL model in Table 1, despite using wild data, underperforms our ZAF method, highlighting the effectiveness of our regularization.

These points articulate our distinct work of using unlabeled data not merely as an additive resource but as a strategic element integral to enhancing CL performance while addressing the limitations of traditional methods.

[1]Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild.

[2]Learning to imagine: Diversify memory for...


3.**ZAF method without loss $L_{ZS}$**

When our ZAF is applied without wild data and $L_{ZS}$, it effectively becomes the EMA-LoRA. As shown in Table 3, even in this reduced form, it surpasses architectures such as LoRA and MoE-Adapters.

4.**reliability to additional data**

- Fig.9 shows that our way of constructing unlabeled data is flexible and robust. Our ZAF outperforms ConStruct-VL, which relies on replaying old data, even when the texts in wild data are ungrammatical(87.23 and 85.97). We adhere to two primary rules for constructing wild data: it covers a broad spectrum of distributions and remains completely distinct from the original pretraining and downstream datasets. This strategy ensures compliance with CL principles without access to old or test data.

- Before submitting our work, we tested ZAF's performance using various compositions of wild data on the 7TaskVG+VAW benchmark. The notation ZAF$^{x}$ represents adding ${x}$% new data whose concept is not from the downstream benchmark to our wild data. The results below demonstrate the robustness of ZAF and its zero-shot loss, showing that performance gains are not dependent on specifically tailored data.

|Method|FAA CAA FFM
|-|-
|ConStruct-VL|85.97 87.00 6.94
|ZAF|89.67 89.30 3.38
|ZAF$^{15}$|89.90 89.21 3.02
|ZAF$^{25}$|88.97 88.35 3.50
|ZAF$^{50}$|88.60 88.03 4.22
|ZAF$^{75}$|89.25 88.67 3.85
Thank you for your thoughtful response. Below we will further address your concerns.

1.**Our EMA-LoRA and existing EMA-style methods**

- **Training**:
  - [1,2] adapt solely to the target distribution by $\min_{\theta_{fast}}L(f_{\theta_{fast}}(x),y)$, where $\theta_{fast}$ is updated through backpropagation. The EMA model $\theta_{slow}$ is updated from $\theta_{fast}$ via interpolation but doesn't partake in loss calculation, primarily serving for inference. For instance, [2] uses $\theta^{on}$ as $\theta_{fast}$ and $\theta^{off}$ as $\theta_{slow}$. 
  - Our approach enhances this by integrating the EMA model into the full training cycle. Here $\mathcal{W}$ and $\widehat{\mathcal{W}}$ correspond to $\theta_{fast}$ and $\theta_{slow}$. We redefine the optimization as $\min_{\mathcal{W}}L(f_{\mathcal{W}}(x),y)+L_{ZS}(f_{\mathcal{W}}(x^{wild}),f_{\widehat{\mathcal{W}}}(x^{wild}))$, **making $\widehat{\mathcal{W}}$ actively contribute to the loss calculations and updates**.

- **Inference**:[1,2] use EMA model $\theta_{slow}$ for inference. In contrast, our model enhances inference flexibility by allowing the use of either the EMA model $\widehat{\mathcal{W}}$ (ZAF) or the current training model $\mathcal{W}$ (ZAF_variant), ensuring strong performance regardless of the model selected.

|BLIPw/CapFilt-L|7TaskVG+VAW|7TaskVG|5TaskVAW
|-|-|-|-
|Method|FAA CAA FFM |
|ConStruct-VL|85.16 87.61 8.75|88.95 90.69 5.22|83.33 85.57 6.28
|ZAF_variant|89.25 88.52 4.80|90.85 89.96 3.39|88.53 89.02 3.89
|ZAF|89.61 89.65 4.18|92.53 92.20 1.72|89.43 90.20 3.02

2.**Relation to SSL**

- A core principle of traditional SSL is that labeled and unlabeled data should come from similar distributions, with deviations potentially leading to performance issues, as clearly noted in [3,4].

- Contrarily, our findings(Fig.9) show that leveraging a diverse range of unlabeled data distributions can be more effective, challenging the conventional SSL assumption that emphasizes similarity between labeled and unlabeled data.

- The classification as SSL depends on **why and how unlabeled data is used**. We use it to bolster anti-forgetting across all CL tasks, not just to enhance performance on current tasks. This is implemented through a novel regularization strategy, diverging from standard SSL methods like self-training.

- Our method is akin to **self-taught learning**[5,6], where the focus is on using unlabeled data to develop a versatile feature extractor, not constrained by class labels or distributions matching those of labeled data.

[3]Safe deep semi-supervised learning for unseen-class unlabeled data

[4]They are Not Completely Useless:Towards...

[5]Self-taught learning:transfer learning from unlabeled data

[6]Robust and Discriminative Self-Taught Learning

3.**Performance gains not solely attributable to unlabeled data**

- **Simply adding more unlabeled data does not guarantee improved CL performance**. The table below demonstrates that using self-supervised contrastive consistency regularization(SSL) instead of zero-shot stability regularization with the same wild data results in a notable performance decline.

|BLIP w/ NLVR|7TaskVG+VAW|7TaskVG|5TaskVAW
|-|-|-|-
|Method|FAA CAA FFM||
|ConStruct-VL|85.97 87.00 6.94|86.96 90.47 7.91|84.36 85.93 5.36
|SSL|80.54 83.20 8.23|77.87 78.85 16.98|82.24 83.14 9.42
|ZAF|89.67 89.30 3.38|91.78 91.74 2.02|88.74 89.03 2.67

- Fig.9 illustrates that **our approach to constructing unlabeled data is flexible and robust**, specifically designed to encompass a broad spectrum of future CL tasks. Importantly, this data is entirely distinct from both the original pretraining and downstream datasets, thereby **upholding CL principles**.

- The concept of **leveraging unlabeled data in CL is not new and has been previously explored**. For instance, [7] discusses the use of data from different datasets to simulate the data distribution of old tasks.

- Table 1 presents a Joint Learning upper bound performance, ConStruct-VL that utilizes **data replay for old tasks**, setting a challenging benchmark, and **ZSCL that employs the same wild data** but with a different methodology.

[7]Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild

4.**Technical novelty**

- A significant innovation is the **empirical and theoretical demonstration** that zero-shot prediction stability, when adapting to new tasks, effectively indicates anti-forgetting capabilities. This is rigorously supported by theoretical proofs.

- We develop **a plug-and-play zero-shot stability regularization**, readily integrable into various frameworks to combat forgetting.

- Our **EMA-LoRA architecture** enhances stabilization efficiency by integrating the EMA mechanism throughout the **training process**, boosting the CL performance.

- Similar to self-taught learning, we **utilize a large volume of easily accessible unlabeled data** but focus distinctly on anti-forgetting, thereby **decoupling** it from new task learning.
I appreciate the authors' response. It addresses part of my concerns but also raises some new concerns (along with checking other reviewers' comments). 
1. Thanks for highlighting the detailed differences between the analysis in ZSCL. Although different specific metrics are used, the overall analysis methodology is still very similar to ZSCL. 
2. I can see that the results of using BLIP are better than CLIP. It is not surprising. I can also understand it is easy to obtain good performance using a strong baseline or pre-trained model, and it is not only about performance. It is mainly for checking whether the proposed techniques are general and robust to obtain good performance in different conditions. 

I am still concerned (and even more concerned) about this related to the baseline model, experimental setting, and the loss functions. 

1) How were the experiments conducted with CLIP? Is the wild reference dataset still used? Is it also used for the experiments with CLIP? 

2) How much performance gain (with BLIP or CLIP) is from the additionally introduced reference dataset? It is also one of the concerns of reviewer DvN8.

3) Can the proposed (rest) techniques work without the additional wild reference dataset, i.e., without loss L_ZS? 

4) I am concerned that the proposed methods (the whole method) and the performances are highly tight with the selected baseline model BLIP and the specifically used unsupervised wild training data. 

Since the continual learning research is mainly about how to maintain performance and avoid forgetting, heavily relying on the specifically trained model and additional data diverges from the original purposes and is slightly misleading, although it can improve performances on the benchmarks. Considering some empirical results are still valuable, I give the score 5. I am still open to reconsidering my score upon the response.
Thank you once again for your valuable feedback and positive evaluation. Your insights have been instrumental in enhancing our work.
I thank the authors for the detailed feedback, clarififying some misunderstandings on my end, and discussing the relation to Stojanovski et al. 2022.

However, I am still not convinced that comparisons made to other methods are fair, and what the actual contributions are that the reader should extract from this paper.

---

For one, e.g. Stojanovski et al. 2022 utilize EMA for CL during training, which is different to what the authors note in the shared response (_"Our approach differentiates from traditional EMA-style methods[1,2] by incorporating EMA-LoRA not only during inference but also throughout training"_). 

---

Moreover, while the proposed method or model may not build on the same intuitions such as standard SSL mehods (_"manifold and low-density separation"_ as per the authors), it does not mean that the overall setup is not conducting semi-supervised learning as training happens on both supervised and unsupervised data (in contrast to other methods compared to). It doesn't matter if certain particular SSL approaches such as self-training aren't utilized - as soon as joint training over supervised and _additional_ unsupervised data occurs, one falls within the SSL regime. This is particularly problematic, as all the other reference methods only utilize the labelled training data. 

Similarly, the authors argue that the results _"challenge a core principle of traditional SSL where similarity between labeled and unlabeled data is preferable"_ - it would be great if the authors could provide references for this claim for completeness.

More importantly however, the authors explicit note in their own rebuttal that their continual learning setting differs from standard CL, in that _"correlation does not extend to dissimilar tasks, such as the reasoning tasks in our studies utilizing BLIP"_. Indeed, the authors evaluate on much more complex and mixed natural image domains, in which cases including data from a diverse set of domains should actually help (as it mimics in nature the data used for pretraining). This is seen in the consistent improvements across methods in Tab. 3 when including additional data. 

---

Put together, in my eyes, the authors get performance gains by adding meaningful (w.r.t. to the evaluated benchmarks) additional unsupervised data, which has limited novelty, and makes e.g. Tab. 1 meaningless. At the same time, exponential moving averages for methods in continual learning have already been explored before (see [1,2] in shared response).
As such, it would be great if the authors could very precisely state what the relevant technical novelty is. I'm still happy to raise my score, however need this aspect to very clearly carved out.
The author has fully answered my question. I maintain a positive evaluation of this paper.