CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models
Authors: Saurav Jha
Dong Gong
Lina Yao
Venue: NeurIPS 2024 poster (2024)
Keywords: Continual Learning
Vision-language models
Finetuning
Primary area: machine_vision
TL;DR: We propose a probabilistic finetuning method for continual learning with pre-trained CLIP model to enable better in-domain knowledge acquisition, generalization, and model calibration.

Abstract: Continual learning (CL) aims to help deep neural networks to learn new knowledge while retaining what has been learned. Owing to their powerful generalizability,  pre-trained vision-language models such as Contrastive Language-Image Pre-training (CLIP)  have lately gained traction as practical CL candidates. However, the domain mismatch between the pre-training and the downstream CL tasks calls for finetuning of the CLIP on the latter. The deterministic nature of the existing finetuning methods makes them overlook the many possible interactions across the modalities and deems them unsafe for high-risk tasks requiring reliable uncertainty estimation. To address these, our work proposes **C**ontinual **L**e**A**rning with **P**robabilistic finetuning (CLAP) - a probabilistic modeling framework over visual-guided text features per task, thus providing more calibrated CL finetuning. Unlike recent data-hungry anti-forgetting CL techniques, CLAP alleviates forgetting by exploiting the rich pre-trained knowledge of CLIP for weight initialization and distribution regularization of task-specific parameters. Cooperating with the diverse range of existing prompting methods, CLAP can surpass the predominant deterministic finetuning approaches for CL with CLIP. We conclude with out-of-the-box applications of superior uncertainty estimation abilities of CLAP including novel data detection and exemplar selection within the existing CL setups. Our code is available at https://github.com/srvCodes/clap4clip.

CLAPACLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models
Saurav Jha', Dong Gong' Lina Yao 1.2
University of New South Wales (UNSW Sydney), CSIRR''s Data61 (saurav.jha, dong gerssggng)munn edu. au; lina. yanddaao@ wwtp:daattai..iii...oooo csiro. au
Abstract
Continual learning (CL. aims help deep neural networks learn new knowledge while retaining what has been learned. Owing to their powerful generalizability. pre- trained vision-language models such as Contrastive Language-Image Pre-training (CLIP) [1] have lately gained traction as practical CL candidates. However, the domain mismatch between the pre-training and the downstream CL tasks often calls for finetuning of the CLIP on the latter. Most existing finetuning methods exhibit deterministic nature. This makes them overlook the many possible interactions across the input modalities and deems them unsafe for high-risk tasks requiring reliable uncertainty estimation. To address these, our work proposes Continual LeArning with Probabilistic finetuning (CLAP) probabilistic modeling frame- work over visual-guided text features per task, thus providing more calibrated CL. finetuning. Unlike recent data-hungry anti-foreetiing CL techniques, CLAP allevi- ates forgetting by exploiting the rich pre-trained knowledge of CLIP for weight initialization and distribution regularization of task-specific parameters. Cooperat- ing with the diverse range of existing prompting methods, CLAP can surpass the predominant deterministic finetuning approaches for CL with CLIP. We conclude with out-of-the-bul applications superior uncertainty estimation abilities CLAP including novel data detection and exemplar selection within the existing CL setups. Our code available at withttt://// /ttt:://gtthlle wtt. homm/srCooees///eeoomocco seaer/oodeee/aapp.......
1 Introduction
Learning in the real world involves dealing with the ever-changing distributions of task streams and their data B 4. Given the constraints on resources and privacy, there is also guarantee for re- re-training a network on all previously seen data [5]. Continual learning (CL) aims to learn from such data/task stream without catastrophic forgetting 66 2 of past data/tasks. A challenging CL setup is the class-incremental learning setting, where new classes emerge with new tasks, and at test time, a model must infer from all seen classes without known task IDs 7 g.
Recent years have seen pre-trained multi-modal foundation models excel on several domains One such example for the vision-language (VL) domain is the CLIP (1 model that comes with strong zero-shot generalizability acquired by learning to match large-scale image-text pairs a contrastive manner [11]. However, adapt well downstream tasks, CLIP must be finetuned on the task-specific data 212 13.. Considering both the need for continually finetuning pre-trained models on streaming tasks and their perks over training from scratch [14. work studies CL with CLIP.
[0]
is that these overlook the uncertainties arising from many possible interactions between the visual and textual cues of downstream tasks. For instance, on the textual side, while good generic hand-crafted prompt for images is "A photo of {class)", there can be instances where further tailored prompts help
An issue with the existing deterministic approaches to finetuning 13 12
*Corresponingg author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In-domain
performance
In-dameein
In-damain performance Prompt- type agnostic Cross-mond mocal cues
In-comain
Promot-type
performanie
performance Prompl-bype agnostic
agnostic
Prompl-lype
Cross-modal
agnostic
cues Distinct task representation
Cross-modal
Cress-mocks cues
cues
1.
3.
4.
M
Twd ensoder
p1
p2
features
Visueld text features
Taek-speoilic adapter madules
Prompts
Image features
Imsaee
Pre-trained CLIP
Figure 1: Concept diagram for probabilistic finetuning of pre-trained CLIP in a CL setup: We identify four such suitable design choices for probabilistic modelling. Choice #1 [17] performs variational modelling by imposing prior on the prompt space which makes it prompt-type dependent while also interfering with the in-domain knowledge learning capability of the prompts a criterion crucial in the deployment of CL models. Choice #2 (see Sec. 3.2.1) instead imposes a prior on the outputs of the text encoder. While this makes it prompt-type agnostic (as the text features can now be derived from arbitrary prompt types), not taking the visual information into account nevertheless leads to the loss of information about cross-modal interactions between the visual and textual cues property essential for preventing modal deviation of finetuned features in CL (see Sec. 3.2.2) Choice #3 (Ours) leverages the best of both worlds by modelling the distribution visual- text features. To further refine the learned distributions of CL tasks, finally introduce lightweight task-specific adapter modules in choice #44 that make the cross-task centroids more distinct while preserving the aforesaid properties (see Sec. 3.2.3
improve the image-to-ters coherence. Similarly, on the visual side, images from the same class can have diverse range backgrounds, poses, orientations, etc. Overlooking the uncertainties in image- text matching can thus cause overfitting on downstream tasks and forgetting of the generalizable knowledge [1]. For CL, where we seek to adapt CLIP on stream of tasks, this can further lead the cross-modal features increasingly deviating from each other to the point of catastrophic forgetting (see Fig. (3b). While existing methods [16 [7] model such uncertainties through probabilistic finetuning. these remain subpar at CL given: (a) their inaptness to leverage existing prompt- based approaches 16]. (b) their excessive trading of in-domain performance for generalization [17]
Lastly, like other autonomous real-world agents, CL models deployed in mission-critical settings (healthcare, transport, etc.) can benefit from uncertainty awareness by calibrating their predictions to reliably assess their confidences [18 []. Hence, to enhance the usage of the pre- pre-trained CLIP for real-world CL tasks, we design a finetuning approach with the following three properties (see Fig. D]: A) probabilistic modeling of cross-modal task cues for better generalization; B) compatibility with prompt-baan based finetuning methods [14 12 19 20 to exploit their finegrained in-domain task knowledge; C) leveraging the rich pre-trained knowledge of CLIP to further counter forgetting.
To this end, we design a principled Variational Inference (VT) framework that learns the functional space of task-specific posterior distributions based on text features that are aligned with their visual counterpart (property #A). To refine these variational task posteriors, we draw our motivation from Bayesian mixture-of-expprt. ensembles (21] (2]. and employ lightweight task-specific adapter modules that model task-specific distributions. Our final prediction is thus a mixture of logits derived from individual task adapter modules. To further counter the forgetting in these modules, we diverge from the recent trend of internet data-hungry CL techniques [23 24]. Instead, we exploit the readily available pre- pre-trained language knowledge of CLIP for weight initialization and task distribution regularization (property # C). Finally, by modelling the distribution of the text feature space, our probabilistic finetuning method boasts a rich modular nature as the features can be derived from arbitrary prompt types (property #B). particular, we show that our framework can inherit the in-domain knowledge of hand-crafted [I] uni-modal [12, multi-modal (20), or conditoond [19] prompts. We backronymize our finetuning approach as CLAP Continual LeArning with Probabilistic finetuning for the pre-trained CLIP model. Our experiments across

---

several settings show that CLAP4CLIP enhances prompt-based finetuning for CLIP, and surpasses the predominant deterministic finetuning methods in terms of in-domain performance. output calibration, and generalization to unseen CL. tasks, all while sharing similar resource overhead. We study some out-of-the-box perks of CLAP's probabilistic nature by leveraging its uncertainty estimation capabilities on proposed post-hoc novel data detection setup and on exemplar selection for CL.
2 Related work
Continual Learning (CL). The existing CL literature is predominated by three categories of methods:
(a) Regularization-basee methods 66 25 26 alleviate forgetting by punishing changes to the param- eters that are important to previous tasks: (b) Architecture-
approaches learn parameters that are specialized for individual tasks either by network expansion [27] or by sub-network composition [2 [9]; (c) Rehearsal- basedd approaches [30 [ rely on storing a fraction of the past task experiences
memory to train with the current task. Each category has its own flaw - methods in (a) struggle to discriminate inter-task classes 31: those in (b) often require task oracle during inference; those in (c) are sensitive to the memory sizes besides being prone to overfitting on the memory samples 32. Hence, practical CL calls for combining these. Our work leverages (a) via function- space
regularization (Sec. 3.4) (b) via task-specific modules (Sec. 3.2.3). and (c) via herding- -bared replay (see App. 1.
Vision-Language Models (VLMs) finetuning. The powerful generalizability of pre-trained VLMs [9.33] like the CLIP [ has enabled their zero-shot applications to a range of downstream tasks, including CL [1]. In practice. their performance on downstream out-of-domain data remains rather weak [4 35 For such cases, finetuning on task-specific data is a natural choice. Instead of
performing extensive full finetuning on all parameters, some parameter-efficien efficienttttttuuuu finetuning (PEFT)
methods learn a lightweight feature adapter module for textual and/or visual paths [13] [6] Another
line of PEFT methods learns soft prompts which are a few continuous tokens serving as inputs to
the frozen visual and/or textual encoder(s) to capture task-specific information [12
37 20 Existing
works on CL with pre-trained CLIP have leveraged either [19 38) or both (39 of these methods.
However, such finetuning methods remain deterministic in nature. This imposes an explicit constraint
on the modeling of the possible ways in which the visual and the textual semantics interact.
To address the aforesaid flaw, one could turn to adapt the existing probabilistic finetuning approaches to capture the cross-modal interactions in CL tasks. For instance, [16] learn the distribution of hand-crafted prompts while [17) propose variational prompt tuning (VPT) conditioned on the input images. Yet these methods are limited in their efficacy. [66] is incompatible with conditional prompt learning [37], which is an extensively studied PEFT field. VPT [17] trades in-domain performance excessively in favor of generalizability - a trait detrimental in the deployment of CL models. Our work aims to bridge these gaps for probabilistic finetuning all while adapting it for CL.
3 Methodology
3.1
Preliminaries
Continual Learning (CL). Clasx-incremental CL 7 aims to learn from sequence of T tasks [(C". ('),
(CT, D²)) Each task E [1, has its training data D' = ((x1, (1), 022) U2...... ...., (XH, yk )}, where and y are the input images and labels, respectively from the set of classes C* = fci, (5, .2,,....,,, C,,,) Following [40 41). we assume any two task-specific sets of classes to be disjoint: C'ncc = 0. A neural network with parameters ő is then trained on task using D' to minimize the cross-entropy loss over C At test time, the model is evaluated on all seen classes C=U (=) C³. where the past task predictions are prone to forgetting. As solution, rehearsal-
methods 42 43 replay past task samples from a memory M during training. Following (30] we
use herding 44
maintain M (see App. I. for details).
CLIP with prompt. CLIP comprises an image encoder fx) acting on an image x E 3X of height N and width W, and a text encoder (((())) acting on a word embedding vector E ((xx)) derived from a text prompt p E (R((()))x)). Here, L is the text length, and e is the text embedding dimension. The encoders's 's outputs are used in the prediction of the class y4 as:
exp ((f(x)T g(t)))/r)
p(ulx)) De=1 exp ((f(x)T, g(t)))/T)*
where 7 is a learnable temperature parameter, (.-)) is the cosine similarity, and the e-th class text feature t. [p. e.] is the result of adding class-specific word embedding e, to the prompt p. The
3

---

Viusl
Saaturii
simiarity
Image Encodes
Sampling
Replay Memorr
gu guided
Attention
dog
(VGA)
Inference
Concatenated logt
Snow Leopard
Pre-Srained CLIP
Module
+
Task distribution encedars
Task logits
Hand--caated prompts
Weight Intistiotioon
Trainable
Functional Regular Regularization
Fro-trainad Larguage-
Knowledgo
Figure 2: CLAP4CLIP overview: the -guuded attention (VGA) inference module uses the text features query (Q), and the visual features keys (K) and values (V) to produce visual-guided text features. The task-specific text features are fed to their respective task distribution encoders (μ², o²). The task distribution samples then fused with the original task features prior to deriving the task logits y². All task logits are concatenated to produce the final prediction y
features (t,)) for all classes are used as the weights of linear classifier. In CL, gt.)) would thus encode the features for classes e E C' seen until task l. Eq. ① forms a contrastive training criterion for the text and visual modalities, whose rich representation allows pre-trained CLIP to be used for zero-shot classification through hard prompt templates, i.e., pd = "A photo of {o class}".
downstream task t, soft prompts use set of learnable vector tokens p = {p> pp= ...,, p²). CoOp [2] shares p with all classes of task. MaPLe (20) learns multi-moda prompts by employing two such token sets P and Py until the J-th layers of the vision and the text encoders of CLIP, respectively. AttriCLIP [9] selects subset of prompts conditioned on the input: {(p'
CLIP finetuning with learnable soft prompts. To improve the CLIP performance
(p')<s/ 15バ5 x Learning p (with frozen context for given task.
CLIP weights) thusly helps encode task wetoooaaityiiistncccoooiiio
CLIP finetuning with adapters. Adapter-based methods like CLIP-Adapter [3] learn lightweight modules over text and/or visual features of the frozen CLIP model. With text adapter A,, the updated text features from Eq. ( can be rewritten (with a slight abuse of notation) as:
g(t;) aAi (gt4)) Bg(t,),
where and I control the strength of the residual connection between the adapted and the pretrained models' features, e.g., 3 = a in 13
3.2 CL with probabilistic finetuning for CLIP
Overview. We develop our CLIP- probabilistic finetuning model using Bayesian VI framework (see Fig. [ Sec. | starts by making the case that, unlike previous VI-based finetuning approaches. the feature embedding space of the text encoder output is superior choice for defining our functional space priors on. While linear adapter layers are often employed for obtaining the mapping from such pre-defined feature-space prior to the function outputs 45 17
we show in Sec. 3.2.2 that CL finetuning with generic adapter leads to the issue of cross-modal deviation. In Sec. 3.2.3] we then propose refining our variational distribution on function space using an ensemble of task task-specific adapters that are built on top of cross-modal aligned text features.
3.2.1 Variational inference with function space prior on text features
We are interested in modeling the stochastic processes that generate the labels y for the inputs x of a CL task t. To this end, we assume a prior distribution Px over the text feature of the c-th class: Px (t)). We can then draw M number of latent variables 2 = {zm ~ Px to represent the c-th class text feature (p) as a linear combination of the text encoder feature (p)) and 2:
(p(p)) (g() 2m M M=1: S.t. 2m ~ Pxx
(3a)
exp (f(()) to(()))
p()[())
E exp ((f()),, tt(())))) (p))dx
(3b)

---

Variation of Acc. & BwT w/
Aog resation anole (degrema)
Test step rotation angle
Backward transfer
SAT
Accumcy
.65
40
Accuracy
50
Continual CLI grasm
Cotp
Colip + Adotte
Colpl I Adaota VGA
40
-35
0.1
Residual strength (a)
0.3
0.5
0.7
0.9
10
10
(a) Effect of a on Accuracy and Backward Transfer (BwT).
(b) Avg. rolation angle 47 per incremental step for image and text features.
Figure 3: Need for Visual-guided Attention (VGA) inference module. Fig. 3a A simple adapter is inadequate at preventing catastrophic forgetting in CL maaked by high BwT scores; Fig. 3b) VGA module encourages cross-modal alignment between the learned text features and the pre-trained visual features marked by a decrease in average angle arccos(t, 1) between them where otherwise the former deviates further with incremental training steps.
where Eq. (bb replaces g(t.) in Eq. ① by (p)). Eq. (3b) thus results into M number of predictions whose distribution gives us the model's 's epistemic uncertainty about the correct prediction. To deal with the intractability of the marginal likelihood, we optimize for the evidence lower bound (ELBO) using a variational posterior go that approximates the prior Px based on the KL-divergence loss xL
logp())>) 2 (ega( p(u|x, =)] DKKL. (1=(2|t.)) -
IIxx)-
By assuming Py to be the (static) standard Gaussian (O, and do to be the (learnable) Gaussian N (t((tt),ottt))) whose mean μ and standard deviation o are parameterized by linear adapter layers, we can ensure that the random variable z remains differentiable by reparameterization trick (45). Accordingly, we refer to the parameters [p; together as probabilistic adapter from here onward.
Imposing the prior over the text feature offers us further advantages over that in the prompt embedding
space as done by VPT [17] (also see App. Fig. for an illustration). First, Eq.
(3b) is prompt-type
agnostic as the text features g(t) could be derived from any existing soft (37 19 20 or hard [14 [3]
prompts Second, by leaving the prompt embedding space intact and injecting stochasticity into the feature space, we can better learn the task-specific knowledge known be encoded by the prompt embeddings [46]. This can help bypass the loss of in-domain performance. In CL., the latter property is crucial for model perform well on all previously seen tasks. Third, as the latent variable is now directly used to infer the logits, it naturally favors generalization by influencing the predictions. On the contrary, the effect of the prompt space prior on the predictions indirect as is mediated by the representations of the entire text encoder layers. This can make the influence of the prior harder to control and can hinder the aforesaid interpretability in the model's predictions.
Efficient continual finetuning with a probabilistic adapter. Finetuming adapters for CL is not straightforward. Namely, we have the overhead of searching for task task-specific residual ratio (see Eq. 22) which is sensitive to the training setup including dataset and prompt-type [3 36] This has partieularly worse implications for a probabilistic adapter like ours, where larger a can inject enough noise to corrupt the pre-trained representations to the point of catastrophic forgetting (see Fig. Bab. For efficient learning of our adapter, we thus seek to retain no additional overhead of hyperparameter search for the residual ratio. Subsequently, we use a = 3 = 1 through our work.
3.2.2 Cross-modal feature deviation in continual finetuning of CLIP
To perform CL with variational modelling in the text feature space, we first take a step back to investigate how CL in general affects the cross-modal deviation [47) between the learned text and the frozen visual features of finetuning methods. To this end, we consider two basic CL models: the CoOp 12] and the CoOp with Alaterr [13]. Then, for the base task (l 1) test samples of CIFAR100, we compute the average of the Rotation Angle Matrix (RAM) 47 using the CL models' frozen visual f(x) and learnable textual (p)) features each incremental test step. Fig. 3bb shows the deviation of the learned textual features from their (frozen) visual counterparts for the CoOp. This implies that the cross-modal retrieval performance of CLIP finetuned with learnable prompts deteriorates with incremental training. Moreover, as a generic adapter (CoOp + Adapter) does not remedy the cross-modal deviation, this sets direct hindrance in employing our probabilistic adapter to learn the variational distribution q6.
è From practitioner's perspective, this enriches the modularity by taking away the overhead of engineering the priors specific to the prompt-type that could, for instance, be spanning multiple layers and/or modalities.
5

---

Variational modeling on visual-guided text features. For variational modeling of text features (p)) that remain aligned with the visual features during continual finetuning, we propose enriching the text features with the visual context through an explicit attention mechanism (see App. A.3 for further justification on this design choice). To this end, we treat the text features as queries Q and the visual features as keys K' and values V. and adopt standard transformer-styled decoder block [48] as a task-shared Visual-guided attention (VGA) module. The VGA module performs text-to-text self-attention followed by text-to-visual cross-attention. To eliminate the influence of the text features from multiple CL tasks, a naive strategy is to perform specialized VGA forward passes with task-specific queries (27]. We seek to replace several such costly VGA passes with a single pass. To do so, we exploit the global nature of our visual context and mask out (set to -co) all inter-task connections in the queries using target mask. This ensures that only the task-specific text features undergo self-attention while the entire query still attends to the visual context:
[t=)== VGA(Q = (t K V (x)),
(5a)
t = 1 + g(t(()))
(5b)
where th is the task-specific visual-guided text feature which is fused with the residual task-specific text encöder feature ((t)) to derive the task embedding t.. We note that in non-CL setup, 49] employ the VGA module using the per-pixel spatial features (obtained before global average-pooling instead of the globally pooled visual features of the ViT [50]. Our choice for the latter favors the
efficiency of our framework for large CL datasets where attending to per-pixel spatial features can incur much higher latency (see App. A.4 a comparison).
3.2.3 Task-specific probabilistic adapters as ensembles for posterior approximation
By exploring diverse modes in function space, ensembles of neural networks can better approximate the variational posterior [51] (2). Motivated by this, we replace our task-shared adapter da with
task-specific adapters ( \ that parameterize the 1-th task-specific posterior N(y², o"") over the
task embeddings t.:
= (N f fmm ~ = ( (t)).
(6)
where zin are the task task-specific MC samples. Task-specifer adapters thus serve as mixture-of-experts
ensemble where each expert is trained on task specificc embeddings E and the logits computed using
each expert is combined to derive the final pre- diction jıst (see Algo D. The experts learn posteriors that are more discriminative across tasks. This is depicted in Fig. X uusing the co- sine distance between the embeddings of the class-specific samples drawn from the poste- riors. With task-specific adapters (right), the cross-task class centroids are more separable.
Effect of task-specific ancoders on inter-class centrold distances
High
low
To prevent interference from current task train- ing data, we freeze the past task encoders dur- ing each incremental training step (t > 1). Moreover, to reduce the forgetting in past task adapters, we follow other parameter-isolation techniques [27 42 52 to finetune on class-
without (Avg. 0.689) with (Avg. 0.804)
Figure 4: Need for task-specific probabilistic adapters: Cosine distance between the centroids of class-specific latent variables produced without (Ieft) and with (right) task-specific adapters on CIFAR100 (10 tasks, 10 classes per task).
balanced dataset of new data and rehearsal data M at the end of each incremental training step (t >> 1). We refer to this as memory consolidation training (see App. A.2). We also provide an overview a test-time forward pass of our framework in App. A.5
Algorithm overview. App. algo. I outlines the pseudo-code of a forward pass of CLAP at t-th th task test step. Here, test image is to be classified into one of the classes { ".....,,"I] Our method executes the computationally heavy VGA layers only once. The task-specific VGA outputs are passed to their respective adapters. By limiting the quadratic complexity of the VGA pass, our method induces minimal time overhead. By only expanding the linear adapters per task, our memory overhead negligible compared to the large backbone of the pre-trained CLIP model (ablation Fig. 5)
3.3 Alleviating forgetting with pre-trained language-aware CLIP knowledge
Like other finetuning methods, our probabilistic adapters are likely to trade the generalizability of text features for downstream task performances 22 37] On the contrary, the pre-trained CLIP text

---

encoder with hand-crafie prompts [5] has strong generalizability because of its rich pre-trained language information. We propose leverage this pre-trained language knowledge help guide the incremental finetuning in CLAP. In the following, we assume (t E RR) [[=1 to be the features corresponding to the L hand-crafted textual prompts for the class y C
3.3.1 Past-task distribution regularization for mitigating forgetting
The functional spaces of the past task distributions are prone forgetting in CL. Though replay helps alleviate the forgetting up to a certain degree, repeated training on the memory samples can lead to overfitting these [32 54). To address this, previous works | 55 exploit functional priors for regularizing the visual space alongside memory replay. Here, we propose to regularize the past task distributions in the textual space by using the hand-crafted prompt-based features (t) to
distill the past task latent samples {=r {=m mm f-1
Namely, the probability of the sample set
z belonging to a class y E C is:
PK 25)
1 M
Σ 1=1
exp ((th) =m)) C* -c=1 exp ((tb' =bm))
The resulting language-awer distillation loss is thus the of cross-entropy between the true label distribution Ye and the predicted probability distribution PKD across all the past-task classes:
T-1 |C*1
LKD ΣΣ
log PkD(cl=')ua
(8)
where LKD serves as data-free (i.e., training samples required) text-to-text distribution regularizer that encourages the latent variable outputs from past-task adapters to stay close to the text features from the hand-crafted prompts. LKD is applied only during the memory consolidation training, that is, when the past-task adapters are trainable. Lastly, as LKD acts on the functional space of past tasks, this sets apart our setting from the non-CL setup of [56) where the language-aware distillation loss regularizes the vector embedding space.
3.3.2 Task-specific adapter initialization considering stability
Stability gap (57) in CL refers the temporary yet substantial forgetting of past task knowledge in the initial phases of updating a network's weights to learn an incremental task. An informed weight initialization can help bridge this gap over random initialization by stabilizing the learning for new ( J= to initialize the weights task components (58). We thus leverage the (-th task text features {th
w w/ R of our t-th task's linear adapter layer. Let S,. E R [C be the mean and the std. dev. of the L text features. We initialize w," and w as:
w d (3/ u sp), w = (8). So).
3.4 Training objective
Approximate ELBO. Building upon Eq. (4) now learn the task-specific adapters g to approx- imate the intractable t [1, task-specific posteriors. The ELBO (see App. F for derivation) is:
E (g(() log pos" (2 x DKL (KL (2 =xxu((((e (+))
logp(y
x:
(10)
Overall objective. Denoting the loss weights by À and our total loss term can be given as f. fce ADxL >>DD where the cross-entropy Lce and the prior-matching lDxL terms on the outputs of all task encoders while the distribution regularization term LKD acts only on the past task encoders. consolidation training stage, X for these is set to 0 during training. is set to 15.
0.001. As the past task encoders are trainable only during the memory
4 Experiments
Datasets. We evaluate our method on CIFAR 100 3 30
ImageNer1 000 41 43 ImageNet-R (59
CUB200 (0), and VTAB (60]. CIFARI00 61 and ImageNet100
62 setups split their respective
original datasets into 10 tasks with 10 classes each. ImageNet-R 63 and CUB200 split 200 classes
7

---

Method
CIPARTOU
ImageNettO) Avg.
ImageNNes I
CUB200
VIAB
Arg.
Last T
Last
Avg.T
Last
Ang
Last
Ang:
Last
Single-iask JOINT
80.28
$1.08
80,92
75.4
89.29
Task spekisp JOINT
82.9
K3.35
83,07
85.72
94.6
iCaRL 100
72.93
57.6
68.62
59.5
66.34
43.71
82.39
75.1
53.36
41.6
L,2 [33
78.92
70,04
77.07
69.33
76.98
68..4
DuilPrompt 150
82.33
20
CODA-P
85.19
76.4
85.93
79,02
$2.05
79.5
54-77
87.5
81.2
PROOF 10
$4.84
76.55
84.89
79. 7
83.50 98
79.35
Continual CLIP 14
78.65
68.25
83,99
74.2
84.43
76.94
67.0
54.8
68.5
00.97
CoOp [
81.17
70.58
79.14
64.9
K4.7
78.66
76.
68.53
87.06
81.25
MaPLe E0
82.74
74.52
79.25
64.06
85.28
64.43
79.31
68.45
82.29
70.76
83.09
78.53
52.12
CliPP.dapper
1
78.75
68.32
84.15
73.96
84.49
78..1
67.41
54.49
68.25
61.02
VPI
73.4
19.33
80..1
61,00
81.06
74.0
09:14
00003
07.2
77.26
Ours w/o V1
$4.36
76.8
$6.11
76.48
85,09
79.83
72.21
61.87
90.74
88.64
Ours
86.13
78.21
87.76
79.16
85.77
79.98
86.93
$1.64
91.37
89.67
CsOp Ours
85.71
77.4
86.8
78.18
85.30
79.52
86.99
81.95
92.51
91.28
MaPLa Ouns
$6.06
78.48
87.67
79.02
80.56
74,24
90.97
AtriCCIP- + Ours
78.06
67.59
87.37
79.3
86.35
80.6
83.71
79.01
74.84
71.12
Table 1: Performance comparison of different methods averaged over three runs. Best scores are in
bold. The second-best scores are in blue. The results for L2P, DualPrompt, and PROOF are taken from 39 See App. Table 12 for statistical significance these results using std. dev. scores.
into 10 tasks with 20 classes each. VTAB has tasks with 10 classes each (64. While CIFARIO 100, ImageNet100, and CUB200 are robust settings for evaluating CL. methods in the face of large forgetting, ImageNet-R and VTAB make challenging settings for CL methods using pre-trained models as these might include test images in their pre-training set (see App. A. for details).
Baselines. We compare CLAP4CLIP against several baselines and state-of-theearr finetuming methods. These include: (a) CLIP-based methods Continual-CLID [4], CoOp [12), CLIP-Adape [3] AttriCLIP [19, MaPLe 20). and PROOF 39), (b) vision-only methods DralPrompt [59] L2P
65
CODA-P 38 (c) the baseline CIL method - iCaRL [30. For a fair comparison, we adhere to the
experimental protocols of PROOF (39 throughout. We adopt ViT-B/16 with the pre-trained weights of OpenAl D as our backbone unless otherwise specified. As the upper bounds on performance, we the CLAPACLIP with single and task-specific encoders, trained on all tasks jointly (IOINT).
Variants. We integrate our method with four prompt-based approaches: Ours uses CLAP with hand- crafted prompt templates. CoOp + Ours with soft prompts [2). MaPLe Ours uses multi-modal soft prompts [20]. and AttriCLIP + Ours uses CLAP4CLIP with instance-conditioned soft prompts [1]] Ours w/o Variational Inference (VI) is the deterministic variant of Ours depicted in App. Fig. We leave the details of the training and the hyperparameters of our models in App. A.2
Performance measures. To quantify CL performances, we report: (a) the final accuracy after the last incremental step (Last) and the average of the accuracies after each step (Avg) 30. and (b) the backward transfer score (BwT) [66] to quantify forgetting. To assess the benefits of probabilistic modelling, we report: (a) the expected calibration error (ECE) [67] that measures the calibration the
model's 's predictions (68), and (b) the (forward) transfer score (69 that quantifies the generalizability
of CL models by measuring the extent of their zero-shot transfer ability after finetuning.
4.1 Results
Accuracy. We report performances in Table on all five datasets. Our method consistently achieves the best results among the methods compared. Notably, on CIFAR 100 and Image our variants using the hand-crafted and multi-modal prompts outperform the others. On the challenging ImageNet-R setup with significant intra-class diversity, our method can better leverage the instance- conditioned prompt knowledge of AttriCLIP 19), which helps it outperform PROOF [39 by 1.46% in terms of average accuracy, On CUB200 and VTAB, sharing the prompt pool among all tasks gives CoOp 12 edge over other baselines. Leveraging CoOp offers the best results on these while surpassing PROOF, which also builds upon CoOp with task-specific soft prompts. We also observe that VPT [17] lags all CL settings. Comparing the performance evolution of our variants against other baselines shows that our variants perform better throughout the incremental steps (App. Fig. {)
Forgetting. Table 13 shows that in general, plugging CLAP4CLIIP with prompt-based finetuning methods helps improve the BwT scores of the latter. It worth noting that on the cross-dataset setting of VTAB (64), our variants the only methods that effectively transfer the knowledge learned from
8

---

Method
ImageNet100
EmageNeri00
CIPARTOO
Duilbrompt
8199
Continal-C11 IP
75.4
Probubilenic
Conolidation
Taak: spesific
Distzbutisn
CsOp
79.3
Brocddn
traimig
Elacndens
Wolpht
failiatizainn
Regulanination
MaPLe 20
84.81
76.2
VGA
AIsCCLIP JP
1V
83.3
78.3
ID
Avg
Last
PRODF 3X
81.26
Ours
83.51
83.83
#1
11.49
CaOp - Cuirs
#2
73.41
MiPLe + Ours
82:97
83.6
493
34.4
74.99
AIICLIP y Ous
84.14
84.56
#4
85.7
77.2
Table 2: Performance comparison on the CDCL setting [19]. All
#6
13
78.21
methods
Table 3: Ablations of the key components of CLAP4CLIP on CIFAR100.
use the ViT-L/14 backbone.
incremental tasks to improve the performance on past tasks (i.e., BwT > 0). This indicates that our probabilistic modeling strategy does not only counter forgetting but can also help bring anti- norrttorg properties onto existing finetuning methods.
Calibration. App. Table compares the ECE scores of our variants and their respective underlying deterministic baselines at the last test step. In general, our variants help enhance (decrease) the ECE scores of the underlying prompt-based methods. This implies that even in the face of forgetting in CL setup, CLAP retains more reliability in assessing the confidence of its predictions.
Generalization. App. Table 14 shows that our method consistently enhances the (forward) transfer scores of the underlying deterministic prompt-based methods. This means that CLAP can better transfer the learned knowledge from seen tasks to help solve future tasks.
Resource-constrained CL. To study the robustness of CLAP towards memory and compute- constrained environments, ablate its performance replay-free [70] and computationally budgeted 23 CL setups, respectively. Taales[T6)ann [7 that for both these setups, leveraging the instance-condittoned and semantically diverse prompts of AttriCLIP provides an edge. Here, our variant leveraging AttriCLIP surpasses the replay-free SOTA, i.e., CODA-P (38 and the budgeted SOTA, ie., AIrrCCII [19]. Further ablating the role of our proposed language-aware distribution regularization and weight initialization components for our AttrCLIP variant shows that the former component remains crucial for avoiding forgetting under resource-constrained settings.
4.1.1 Cross-Datasets Continual Learning (CDCL)
To simulate real-world settings with long sequence of tasks and large distribution shifts, the CDCL setting [19] trains model sequentially ImageNet100 and CIFARI00 (i.e., on 20 tasks), and evaluates it jointly on these. For a fair comparison with [19], we adopt the ViT-L/144 as our CLIP backbone and the train/test batch size 32. All other settings remain the same in Sec. [.. Table reportss the last task accuracy of different methods. While all our variants improve the CDCL performances of their respective baselines, combining ours with AttriCLIP [99] leads to the most gains. This further suggests that our framework can reliably leverage the diverse nature of learned prompts to inherit their -peeciii advantages.
4.2
Ablation Studies
We provide a few ablations of the training pipeline for CLAP4CLIP below and leave more in App. C
Influence of components. We ablate the importance of different components of CLAPACLIP in Table 3 On top of the base CLIP model, we first train a probabilistic encoder. Adding the VGA module and the memory consolidation training stage helps us achieve more stable performances while countering forgetting. We then apply -seciicc encoders which make the centroids of the class-specific latent variable more separable (see Fig, thus improving the last task accuracy by 2.21%. %.. Language-awan weight initialization and regularization help improve the last task accuracies by 0.78% and 0.23%, respectively. Weight initialization further helps tackle the stability gap (see App. C.6 for more ablations on language-awar components).
57 58
Probabilistic vs Deterministic inference. To understand our probabilistic inference modules further, we examine their performance against the deterministic variant ours (Ours w/o VT). Table [ shows that our probabilistic variant consistently outperforms its deterministic counterpart. This emphasizes the advantages of considering uncertainty in finetuning. We further introspect the effects of the number of layers for the VGA and task encoder modules in our framework in App. C.3

---

Time analyses. We compare the inference time per iteration for different methods. As shown in App. Table 19 our variants need more inference time than other finetuning methods for the performance gains. The increased time comes mainly from the VGA and from inferring the M latent variables.
Parameter analyses. The additional parameters in CLAP4CLIP come from the shared VGA module and the task-specific encoders. For a ViT-B/16 backbone of output dimension, d 512 on CI- FARIOO, the VGA module contains 4,204,032 parameters. The mean and the std. dev. layers for 10 tasks have ters each, i.e., 524, 2880 parameters. Hence, the CLAP4CLIP has 9.5 million extra parameters, which negligible compared the pre-trained CLIP with x 150 million parameters. We report the parameter counts Fig.
Decamnate lin milltons
300
MRoor
250-
Codp
200
DuaiPrompt
L2P
Ours
parame-
150
100
Figure 5: Parameter count comparison.
Out-of-the-box utilities of probabilistic finetuning
We study the out-of-the-box utilities of CLAPACLIP's uncertainty quantification (UQ) capabilities. Our motivation for these is not to achieve state-of-ther performance but to highlight the perks of probabilisticc modeling in scenarios where the deterministic CL finetuning methods struggle.
Post-hoc novel data detection (PhNDD). PhNDD uses pre-trained classification model to identify novel data based the output confidence 77 72. For CL., this can help discern the arrival of new tasks, expand the network, etc. To evaluate the PhNDD capabilities of models within CL setup, we design a simple set- ting. Namely, at all but the last test step, we treat the test data from the past and the current tasks as seen while those from all future tasks as novel. We then use FPR95, AUROC (73,, and AUPR [74] scores as our performance metries (see App. D.I) averaged over all but the last incremental test steps. To quantify the output confidence, we rely on the Energy score (75] given its aptness for pre-trained models. Table
Method
AUROC
AUPR
FPR95
Continull-CLPP [4]
74,46
71.11
77.35
Chass win
82.29
78.38
68.83
82.21
79.54
CoOp T Ouus
80.15
66.3
83.73
80.97
62.8
Table 4: PhNDD performances averaged over 3 runs CIFAR100. Best scores for each variant are in bold.
A compares the averaged PhNDD performances. Our probabilistic models enhance the PhNDD capabilities of their underlying prompting frameworks. Moreover, the inferior results of the deterministic (i.e., w/o VI) versions of our models suggest that probabilistic modelling helps a model output predictions that better express what it is not aware of.
Exemplar selection. We employ the entropy (averaged over M predictions) of CLAP's softmax outputs as our exemplar selection criteria (2. Table 20 shows the efficacy entropy- based rehearsal for our method, where other deterministic methods lag due to their inconsistent UQ capabilities. Next, we employ the energy [75] and the variance of the softmax outputs as our selection criterion and contrast these against other criteria proposed in (2]. Fig, 6 shows that variance- based exemplar selection outperforms random, and is only second to iCaRL [30] in terms of Last accuracy. We note that deterministic methods with pointwise predictions cannot use
$5.18
85.73
66.13
8s
81
95.63
Accuracy
11
11.92
7859
mini Bandem Energy Variance
76.21
78
7857
70
Entropy
iCaRL
Exemplar strategy
Figure 6: Analyses of various strate- gies for exemplar selection w/ our method on CIFAR100.
variance for exemplar selection.
Conclusion
In this paper, we propose CLAP4CLIP, a probabilistic finetuning method for learning task-specific distributions over visual-guided textual features. Our model shares the visual-guided text alignment module across all tasks while adding lightweight task specific encoders to learn fine-graud fine-grained task distributions. Besides leading to little memory overhead, this architecture is compatible with several prompt- srappt-tninng--aaee methods thus helping us inherit their respective perks on different CL settings. Our experiments show the superior results of CLAP4CLIP across several datasets and settings. We conclude with two out-of-the-box utilities of our method wherein existing continual learning methods lag: post-hoc novel data detection and uncertainty- limitations, potential future research directions, and the broader impact in App. sec.
exemplar selection. We discuss our
10

---

7 Acknowledgement
This work was partially supported by a Discovery Early Career Researcher Award Fellowship (DE230101591) awarded by the Australian Research Council (ARC) to Dong Gong. We are grateful to Daniel Marezak and M. Jehanzeb Mirza for their insights on the need for continual finetuning of the pre-trained CLIP model.
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 3768. PMLR, 2021.
[2] Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Rie- mannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV), pages 532-547, 2018.
[3] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. International conference on machine learning, pages 3987-3995, PMLR, 2017.
[4] Saurav Jha, Martin Schiemer, Franco Zambonelli, and Juan Ye. Continual learning in sensor- based human activity recognition: An empirical benchmark analysis. Information Sciences, 575:1-21, 2021.
[5] Arslan Chaudhry, Mareus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 1902. 10486, 2019.
[6] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Ttrabsa-BBawwinse et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-322 2017.
[7] Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(5):5513- 5533, 2022.
[8] Saurav Jha, Dong Gong, He Zhao, and Lina Yao. NPCL: Neural processes for uncertainty- aware continual learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[9] Jean- BanttBatistee Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-237366 2022.
[10] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[11] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018.
[12] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-244,, 2022.
[13] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544, 2021.

---

[14] Vishal G. Thengane, Salman A. Khan, Munawar Hayat, and Fahad Shahbaz Khan. Clip model is an efficient continual learner. ArXiv, abs/2210.03114, 2022.
[15] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out dittr---iitiibtiionn ICLR, 2022.
[16] Yuning Lu, Jianzhuang Liu, Yonggang Zhang. Yajing Liu, and Xinmei Tian. Prompt distribu- tion learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2205--215,000 2022.
[17] Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor G Turrisi da Costa, Cees GM Snoek, Georgios Tzimiropoulos, and Brais Martinez. Bayesian prompt learning for image-language model generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15237-15246, 2023.
[18] Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27.
Open Review, 62, 2022.
[19] Runqi Wang. Xiaoyue Duan, Guoliang Kang, Jianzhuang Liu, Shaohui Lin, Songcen Xu, Jinhu Lü, and Baochang Zhang. Attriclip: A non-incremental learner for incremental knowledge learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3654-3663, 2023.
[20] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, 2023.
[21] Yuji likubo, Shunsuke Horii, and Toshiyasu Matsushima. Model selection of bayesian hi- erarchical mixture of experts based on variational inference. In 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), pages 3474-3479, 2019.
[22] Oleksandr Balabanov, Bernhard Mehlig, and Hampus I. Linander. Bayesian posterior approxi- mation with stochastic ensembles. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13701-13711. IEEE, 2023.
[23] Ameya Prabhu, Hasan Abed Al Kader Hammoud, Ser-Nam Lim, Bernard Ghanem, Philip HS Torr, and Adel Bibi. From categories to classifier: Name-only continual learning by exploring the web. arXiv preprint arXiv:2311.11220 2023.
[24] Minhyuk Sco, Diganta Misra, Seongwon Cho, Minjac Lee, and Jonghyun Choi. Just say the name: Online continual learning with category names only via data generation. arXiv preprint arXiv:2403. 1085: 2024.
[25] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139-154, 2018.
[26] Saurav Jha, Shiqi Yang, Masato Ishii, Mengjie Zhao, Christian Simon, Muhammad Jehanzeb Mirza, Dong Gong, Lina Yao, Shusuke Takahashi, and Yuki Mitsufuji. Mining your own secrets: Diffusion classifier scores for continual personalization of text-to-image diffusion models. arXiv preprint 1xXiv:2410.007000 2024.
[27] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Trans- formers for continual learning with dynamic token expansion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9285-9295, 2022.
[28] Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and Laurent Charlin. Continual learning via local module composition. Advances in Neural Information Processing Systems, 34:30298-
30312, 1,, 2021.
[29] Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hossgawa-Johnson, Sung Ju Hwang, and Chang D Yoo. Forget-free continual learning with winning subnetworks. In International Conference on Machine Learning, pages 10734-10750. PMLR, 2022.
12

---

[30] Levvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Speri, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[31] Timothee LESORT and Andrei Stoian. Regularization shortcomings for continual learning. 2021.
[32] Eli Verwimp, Matthias De Lange, and Tinne Tuytelaars. Rehearsal revealed: The limits and merits of revisiting samples in continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9385-93944 2021.
[33] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022.
[34] Mitchell Wortsman, Gabriel Ilhareo, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959-7977 2022.
[35] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh- Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer learning. Neurocomputing, page 126658, 2023.
[36] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In European Conference Computer Vision, pages 493-510. Springer, 2022.
[37] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu, Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[38] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 11909-11919, 2023.
[39] Da- DeiWei Zhou, Yuanhan Zhang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning without forgetting for vision-language models. arXiv preprini arXiv:2305. 19270, 2023.
[40] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pallern analysis and machine intelligence, 40(12):2935-294,, 2017.
[41] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalaneing. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 831-839, 2019.
[42] Francisco M Castro, Manuel Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pages 233-248, 2018.
[43] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 374-382, 2019.
[44] Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 1121-1128, 2009.
[45] Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint axXiv:1312.6114.20 2013.
13

---

[46] Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang. Weijia Wu, et al. Mix-of-show: Decentralized low-rank adap- tation for multi-concept customization of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[47] Zixuan Ni, Longhui Wei, Siliang Tang, Yueting Zhuang, and Qi Tian. Continual vision- language representaion learning with off-diagonal information, 2023.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[49] Longtian Qiu, Renrui Zhang, Ziyu Guo, Ziyao Zeng, Yafeng Li, and Guangnan Zhang. Vt-clip: Enhancing vision-language models with visual- goudee texts. arXiv preprint (rXiv22112.02:99 2021.
[50] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[51] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757, 2019.
[52] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[53] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[54] Dongyue Li and Hongyang Zhang. Improved regularization and robustness for fine-tuning in neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[55] Pingbo Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard Turner, and Mohammad Emtiyaz E Khan. Continual deep learning by functional regularisation of memorable past. In Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4453 4464. Curran Associates, Inc., 2020.
[56] Adrian Bulat and Georgios Tzimiropoulos. Language-aware soft prompting for vision & language foundation models, 2023.
[57] Matthias De Lange, Gido M van de Ven, and Tinne Tuytelaars. Continual evaluation for lifelong learning: Identifying the stability gap. In The Eleventh International Conference Learning Representations, 2023.
[58] Md Yousuf Harun and Christopher Kanan. Overcoming the stability gap in continual learning. arXiv preprint arXiv:2306.01990 2023.
[59] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pages 631-648. Springer, 2022.
[60] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[61] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[62] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
14

---

[63] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli,, Mike Guo, et al. The many faces of robustness: A critical analysis of diitri--iitiitttonn generalization. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8340-8349, 2021.
[64] Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are you need. arXiv preprint
2023.
[65] Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang. Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
[66] David Lopez-Paz. and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
[67] Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, 2015:2901-2907, 2015.
[68] Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34:15682-15694, 2021.
[69] Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Prevent- ing zero-shot transfer degradation in continual learning of vision-language models. In ICCV, October 2023.
[70] Francesco Pelosin, Saurav Jha, Andrea Torsello, Bogdan Raducanu, and Joost van de Weijer. Towards exemplar-free continual learning in vision transformers: An account of attention, func- tional and weight regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, pages 3820-3829, June 2022.
[71] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-o--diittibuttin detection: A survey, urXiv preprint arXiv:21 1334,, 2021.
[72] Jiangpeng He and Fengqing Zhu. Out-of-distribution detection in unsupervised continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3850-3855, 2022.
[73] Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd international conference on Machine learning, pages 233-240, 2006.
[74] Takaya Saito and Mare Rehmsmeier. The precision-recall plot more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3)=001 18432, 2015.
[75] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based -uttr--dist detection. Advances in neural information processing systems, 33:2 24:214664- 2020.
[76] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint 1rXiv:1910.04867, 2019.
[77] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE,
2017. [78] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
15

---

[79] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3498-3505, 2012.
[80] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226,
2019.
[81] M-E Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06), volume 2, pages 1447-1454. IEEE, 2006.
[82] Da- Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep class-incremental learning: A survey. arXiv preprint arXiv:230..03648, 2023.
[83] Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. Interna- tional Conference Machine Learning (ICMI) Workshop, page 9, 2018.
[84] Z. Zheng, M. Ma, K. Wang, Z. Qin, X. Yue, and Y. You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 19068- Los Alamitos, CA, USA, oct 2023. IEEE Computer Society.
[85] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[86] Youngjae Cho, HeeSun Bae, Seungjae Shin, Yeo Dong Youn, Weonyoung Joo, and Il-Chul Moon. Make prompts adaptable: Bayesian modeling for vision-language prompt learning with data-dependent prior. arXiv preprint arXiv:2401.06799 2024.
[87] Tejas Srinivasan, Ting- Yun Chang, Leticia Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. Climb: A continual learning benchmark for vision-and- language tasks. Advances in Neural Information Processing Systems, 35:29440-29453, 2022.
[88] Zifeng Wang, Zheng Zhan, Yifan Gong, Geng Yuan, Wei Niu, Tong Jian, Bin Ren, Stratis Ioannidis, Yanzhi Wang, and Jennifer Dy. Sparel: Sparse continual learning on the edge. Advances in Neural Information Processing Systems, 35:20366-2001 20380, 2022.
[89] Qingsen Yan, Dong Gong, Yuhang Liu, Anton van den Hengel, and Javen Qinfeng Shi. Learning bayesian sparse networks with full experience replay for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 109-118, 2022.
[90] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In The Eleventh International Conference Learning Representations, 2023.
[91] Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the JEEF/CVF International Conference on Computer Vision, pages 15691-15701, 2023.
[92] Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Muzammal Naseer, Luc Van Gool, and Federico Tombari. Learning to prompt with text only supervision for vision-language models. arXiv preprint LrXiv:2401.02418 2024.
[93] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
[94] Tejas Srinivasan, Furong Jia, Mohammad Rostami, and Jesse Thomason. 12i: Initializing adapters with improvised knowledge. arXiv preprint arXiv:2304.02222 2023.
16

---

[95] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sqquenc-to-sequeee pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 7871-7880, 2020.
[96] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.1259,, 2022.
[97] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth word: Personalizing text-to-image generation using textual inversion. arXiv preprint 2rXiv:2208.01618, 2022.
[98] James Seale Smith, Ven-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image to-ta---a diffusion with c-lora. arXiv preprint arXiv:2304.06027 2023.
[99] Vincent Fortuin. Priors in bayesian deep learning: A review. International Statistical Review, 90(3):563-591, 2022.
[100] Andrew Gelman. Prior distributions for variance parameters hierarchical models (comment article by browne and draper). Bayesian Analysis, 1:515-534, 2004.
[101] Bernt Øksendal and Bernt Øksendal. Stochastic differential equations. Springer, 2003.
[102] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo Rezende, SM Es- lami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:/807.0162) 2018.
[103] Donggyun Kim, Seongwoong Cho, Wonkwang Lee, and Seunghoon Hong. Multi-task pro- cesses. In International Conference on Learning Representations, 2022.
[104] Tuan Anh Le, Hyunjik Kim, Marta Garnelo, Dan Rosenbaum, Jonathan Schwarz, and Yee Whye Tch. Empirical evaluation of neural process objectives. In NeurIPS workshop Bayesian Deep Learning, volume 4, 2018
17

---

Appendix
Visual festures
Cosine similanty
Task Task
Task
Encoder
Replay Memory
cat
guided
Text Encoder
Altention
u
dog
(VGA)
1 p
koola
Inference
Concalenated logh
pd
pt
show leopard
Pre-trained CLIP
Textual features
Madule
Task encoders
logits
Hand-crafted promots
Weight Initaaiization
Trainable
Functional Regularization
Frazen
Pre-trained Language-
Aware Knowledge
Figure 7: Illustration of the deterministic variant of Ours (Ours w/o VI in Table I): the task- specific text features are fed to their respective task encoders, consisting of only the mean μ layer each. There is no sampling involved and the task mean outputs are fused directly with the original task features prior to deriving the task logits y". All task logits are concatenated to produce the final prediction lit.
A Experiments and Benchmarks
A.1 Datasets
Dataset
# training instances
# testing instances
# Classes
# Tasks
Link
CIFARI00 100
50,000
10,000
100
10
URL
ImageNet100
130,000
5,000
100
10
URL
ImageNet-R
24,000
6,000
200
CUB200
9,430
2,358
200
10
URL
VTAB
1,796
8,619
50
5
URL
Table 5: Benchmark datasets and their details.
We evaluate our method on five datasets, the details of which are reported in Table 5 Following 39], we shuffle the order of training classes for all but the VTAB dataset with the random seed 1993. While the original VTAB 766) includes 19 evaluation tasks from three categories (natural. specialized, and structured) and their respective sub-domaing we rely on the five datasets cross-domain class- incremental subset proposed SimpleCII. (64. The five datasets (used the same streaming order)
include Resisc45 [77]. DTD (78), Pets [79]. EuroSAT [8], and Flowers [8I] To make the classes emerge from domain domain, we do shuffle the class order for VTAB.
A.1.1 Exemplar selection for memory replay
Following 30 41.
we employ the herding algorithm
44 to choose the exemplars for our main we rely on two typical methods to populate the
experiments. Following the previous works 82 39 memory:
1. Fixed memory budget maintains static memory M with K instances. Upon having seen 1b number of classes after an incremental training stage, the model selects [D%] per class.
& exemplars
18

---

2. Expandable exemplar set dynamically expands the memory M with the arrival of more incremental tasks. After each incremental training stage, the model here stores (3%] ka exemplars, where k. is the number of exemplars per class.
For CIFARI00, ImageNet100, and VTAB, given their lesser number of classes, employ the first policy, and keep a total of 2,000, 1,000, and 1,000 exemplars, respectively. This amounts to the respective sub-totals of 20 and 10 exemplars per class after the last incremental stage. We choose these sizes for a straightforward comparison with the existing works, ie.. PROOF [39 for CIFAR 100 and AttriCLIP [9] for ImageNet100. For VTAB, the chosen memory size reflects the fact that have only 1, 796 training instances in total (see T) For ImageNet-R and CUB200 with 200 classes each, adopt the second policy and store 20 exemplars per class.
A.2 Training and Hyperparameter selection
We train CLAP and its variants using SGD, with batch size of 64, for 5 epochs, including 1 epoch of linear warmup. The initial learning rate (LR) is set to 1e-3 and decays with cosine anncaling. At the end of each incremental task (t > 1), perform memory consolidation training for epochs, with an LR of le-4, on the class-balanced memory dataset. All our experiments were performed on NVIDIA V100 GPUs hosted the Gadi supercomputers of the National Computational Infrastructure (NCI Australia).
Training for memory consolidation. To alleviate the forgetting of past tasks, we finetune on the class-balanced dataset of new data and rehearsal data M at the end of each incremental training step
(t > 1) 41 27). Following other well-established parameter-iss irrameter-isolation CL algorithms (27 42 52),
freeze the past task encoders during the normal training. This helps us avoid knowledge interference from the dominant new task training samples. During the memory consolidation training stage, optimize all the task encoders while freezing the task-shared VGA module parameters.
Hyperparameter tuning. To the end goal of obtaining task-agnostic hyperparameters 83 we tuned our hyperparameters using a validation set comprising 10% of the CIFAR-100 training dataset. Similar to (27]. performing the hyperparameter search only on the CIFAR 100 setup helps us avoid optimizing for the number of tasks while generalizing across our other setups. Table 1 shows the candidate values for the hyperparameter grid search and their best-chosen values. Tables 8 g
and report the last task accuracy scores (Last) corresponding to the hyperparameter search for the number of training epochs, the number of finetuming epochs, the coefficient y and the coefficient X, respectively. Fig. in the main paper reports the accuracy and the runtimes for the different numbers of MC samples M. We will release the full source code upon the acceptance of our paper.
Hyperparameter
Range
Chosen value
Learning rate
Ge-3, 1e-3, 5e-1 7,
le-3
Epochs
Warnup epuchs
0.5, 1,5
Finetuning epochs
1, 3,
2
10, 15, 20,
15
O. 0.0001, 0.001, 0.01,
0.001
5. 10, 15, 20, 30,
Table 6: Hyperparameter tuning: we gridsearch the CIFARTOW TT0 setup with validation set
comprising 10% of the training set. The chosen values are reused across all other setups.
Epochs
3.
Last
77.32
78.21
78.18
Table 7: Accuracy vs. Training epochs
Finctuning ep.
2
Last
77.65
78.21
Table 8: Accuracy vs. Finetuning epochs
19

---

10
15
20
25
Last
78.04
77.94
78.1
78.21
77.96
77.14
Table 9; Accuracy weight "y" for LKD
Last
78.16
78.21
77.99 77.4
Table 10: Accuracy vs. weight "X" for DRI.
A.3 Variational modeling of text feature space vs. image feature space
We opt for the probabilistic modeling of task-specific text feature space rather than the image feature space mainly in light of the practical constraints imposed by the class-incremental learning (CIL) setting. In CIL, test time, we are not given the task labels for images. As such, if we were to use task-specific adapters to model task-specific visual features distribution (rather than task-specific text features distribution), then we must infer which images are to be routed to what adapter. Existing solutions 8 to task id inference would route the text-guided visual features to all available adapters and then infer the correct prediction based the adapter's outputs. Such exhaustive routing mechanism greatly increases the test-time computational burden. Instead, we exploit the multimodal nature of CLIP [I] to model the distribution of visual-guided text features. This helps avoid test-time task inference as now our visual features form a shared context to which all task-specific text features (which distributions over such -iuide- task-specific text features, we compute their cosine similarities with the visual features obtain our predictive logits.
distinguish simply by their labels) can attend. By sampling from the
A.4 Latency comparison for VT-CLIP styled VGA vs Ours
We compare the performance of VT-CLIP-styled VGA with Ours. To align the text features with the image features, the former uses per-pixel spatial features obtained from the ViT prior to global pooling while we use the globally pooled features. Table shows that VT-CLIP styled VGA achieves similar accuracy ours while incurring 6x higher inference time.
Method
Avg.
Last
Inference time (s)
VT-CLIP styled VGA
86.54
77.98
0.94
Ours
86.13
78.21
0.16
Table 11: Performance comparison of VT-CLIP styled VGA with Ours on CIFAR-100.
A.5 Algorithm overview.
B
Results
B.1
Performance evolution
To complement the results in Table 1 Fig. 8 compares the accuracy of different methods at each evaluation step across all datasets. Our major conclusions briefed follows. A) The base task performance of CLAP4CLIP (ours) is consistently higher than other methods including the state-of-the-art PROOF (39 This suggests that our probabilistic finetuning framework is effective for general downstream tasks in a non-incremental setting. B) For the CL settings in Table [ where either of the CLAP4CLIP variants achieve the best performances, their performance curves also consistently retain superior results across all evaluation steps. This validates the effectiveness of our method at tackling forgetting. C) Similar [39], we notice that CLAP4CLIP achieves significant performance improvement over vision-only methods (L2P and DualPrompt). This indicates the merits of considering text and visual cues together for continual learning.
20

---

Algorithm 1: A forward CLAP4CLIP pass at test step &
Input :(t') text features, f(z): image features
Output: (predictions for classes seen till task ()
{t f- -VGA((t') 1i=1-
f(x))
Eq.
(5a)
for i f- 1; += do
t' ='+'t'
// Eq. (5b)
NW σ') q (t:)
// Sec.
3.2.3
8' x 1
// Null prediction set
for m +
1; m S M; m + do
Fm N(u',0))
// Sampling
tm t E + =m
// Fusion
Fn t (f(x)" mm)
// Using Eq. 3bb
G' o' U Um
// Set Union
y 1::
[ 14
// Concatenation
Method
CIFARI00
ImageNet100
ImageNot-R
CUB
VTAB
Comtnual-CLIP
[4
1.416
2.175
1.98
2,087
0.614
+Ours
1.39
2.19
1.86
2.06
0.443
CoOp
1
1.57
2.47
1.95
1.99
0.54
+Ours
1,533
2.074
2.011
MaP1.e M0
1.3
2.052
2.16
1.803
0.49
+Ours
1,36
1,956
1.84
1.62
0.407
AttriCIIP 19
1.781
2.54
2.37
2.419
0.996
+Ours
1.677
2.019
2.388
2.410
0.98
Table 12: Standard deviation (std. dev.) scores comparison for Avg. accuracy scores of Table 1 between our variants and their corresponding baseline prompt-based finetuning methods over three In general, std. dev. scores are comparable or lower than the corresponding baseline methods and are thus statistically significant.
Mothod
CIFAR100
ImageNet100
ImageNet-R
CUB
VTAB
Continual-CLIP
-0.086
-0.041
+Ours
-0.106
-0.117
-0.107
-0.117
0.012
CoDp 12
-0.257
-0.162
+Ours
40.129
-0.139
-0.112
-0.106
0.011
MaPi.e 20
-0.209
-0.352
-0.1
-0.145
0.037
+Ours
-0.105
-0.112
-0.093
-0.102
0.005
AttrCCIIP IP
19
-0.128
-0.152
-0.082
-0.151
-0.099
+Ours
-0.143
-0.1
-0.092
-0.037
0.041
Table 13: Backward Transfer (BwT) scores 1 comparison between our variants and their corre- sponding baseline prompt-base based finetuning methods averaged over three runs. Best scores across each pair is highlighted in bold.
Mathod
CIFARI00
ImageNett100
ImageNet-R
CUB
VTAB
Continual
39.55
+Ours
65.47
53.07
64.05
58.11
66.91
CaOp
64.09
52.6
60.93
62.11
69.38
+Ours
66.2
$5.09
63.44
58.6
74.1
MuPLe R0
68.22
57.04
66.56
61.6
71.51
+Ours
76.17
62.33
70.03
67.8
78.29
AttriCLIPP
[
61.45
50.4
56.41
57.04
61.59
+Ours
61.87
50.56
58.03
$7.95
Table 14: Transfer scores
comparison between our variants and their corresponding baseline prompt-based finetuning methods averaged over three runs. Best scores across each pair is highlighted bold.
84
21

---

95
Accuracy 150
85
80
77
65
60
Durt Cocp
55
CL Aopeer
Curt Codo
tk0p
Outa
100
Classes
Classes
(a) CIFARI00
(b) ImageNet100
95
90
85
Securse (99)
65
55
60
CR
ABNCU
Dua
Das 1000
.00
CLP
ADRCU
Dishorpt
Our
Dopp
35
0002
L2P
Cun CUP
40
123
25
25
50
75
100
25
50
75
100
150
(c) ImageNet-R
(d) CUB200
90
80
Accuracy
70
60
50
40
10
40
50
Number of Classes
(e) VTAB
Figure 8: Performance evolution of different methods. The top-1 accuracy (%) is reported upon learning of each task.
Method
CIFARIO0
ImageNet110
ImageNet.R R
CUB
VTAB
Continul-CLLIP 14
0.206
0.208
+Ours
0.216
0.207
0.201
0.203
0.165
CaOp [P]
0.245
0.3
0.191
0.21
0.191
+Ours
0.224
0.217
0.207
0.204
0.136
MaPLe 20
0.168
0.243
0.149
0.195
0.195
+Ours
0.214
0.208
0.146
0.184
0.159
AttiCLIP 19
0.256
0.256
0.205
0.209
0.191
+Ours
0.304
0.205
0.198
Table 15: Expected Calibration Error (ECE) scores | (computed over 15 bins) comparison between our variants and their corresponding baseline prompt-based finetuning methods averaged over three runs. Best scores across each pair is highlighted in bold.
22

---

B.2 Results for replay-free CL setup
Method
Avg
CIFARTO
Last
Avg
EmageNecT00
Last 7
Avg
ImageNett
Last 7
Avg 7
CUB2000
Last ?
Avg.
VIAB
Last
CODAP AUrCCIPP 15
74.66
63.7
79.3
72.66
76.44
72.19
74,32
70.15
70.59
66.12
71.35
61.4
80.55
73.08
79.57
76.1
73.89
72.36
71.25
66.02
CoOp Ours
74.19
6345
81.07
72.0
81.22
75.8
$2.59
74,98
82,11
80.11
MI/CUI
72.98
ARiCLIP (wio adapter init)
60.35
69.41
70.53
69.82
65,95
65,48
61.35
ABICLIP Ous (wlo distribution sog.)
53.01
52.55
62.81
53.19
58.52
58.22
56.14
Table 16: Performance comparison without memory replay (avg. over runs). For a thorough analysis, the last two rows ablate our proposed pretrained CLIP's language- anti-forgetting components (Sec. 3.3, main paper): distribution regularization and adapter weight initialization. Best results are in bold.
B.3 Results for computationally-bndggedd CL setup
For our computationally-budeeee CL. setup, we follow 223 where each incremental training task, we allocate the number of training iterations equivalent to epoch on the first (base) task of each dataset. Here, our variant utilizing instance-condittoned prompts of AttriCLIP outperforms other compared methods. further ablation shows that our proposed weight distribution regularization technique indeed remains a crucial component at tackling forgetting on the budgeted setup (see the two bottom-most rows Table
CIARROO
ImageNetiOO
Method
Ang
Avg
CODA-P AndCLIP PROOF
55.29
50.3
56.8
56,37
AmrCCI JP+ Ours
61.7
55,89
62.91
60.2
AusiCLIP + Oun (wlo init.)
61.33
54,95
62.14
59.86
AMACLIP Ours rel.
58,95
52.6
60:04
57.93
Table 17: Results for computationally budgeted CL setup [23]: we follow the "Normal" budget setup from (23] where each incremental task is allocated training iterations equivalent of I epoch on the first task of each dataset. Scores reported are averages over three runs. Best results are in bold.
C Ablation studies
78-
01111
C.1 Sensitivity to the number of Monte Carlo (MC) samples.
8.170
0.189
We vary the number of MC samples M from to 50. In Fig. 9 the accuracy is poorer in range [1, 10), grows in range [10, 20). and saturates thereafter. Hence, we set M to 20 for all our experiments.
0168
0.166
10
50
MC (MU
C.2 Efffec of forgetting on individual task adapters.
Figure 9: Accuracy-runtime trade-off with number of MC samples M.
We ablate the task head predictions over the test set of each task on CIFARIO0 (see App. C.2 for more details). Fig. reports the accuracy of test set samples corresponding
the task encoder heads at the end of incremental training on the last task. Here, the first row is to be interpreted as follows: 77% of test samples belonging to test set of the first task (test set ID 1) were correctly allocated to task head 1, 1.4% of test samples belonging to test set of the first task (test set ID 1) were incorrectly allocated to task head 2, and on. Visualizing the task head selection results for the last task evaluation helps us uncover the amount of forgetting among the individual task heads at the end the incremental training.
Fig. 10b compares the evolution of the task head selection accuracy across the incremental test steps. Here, at the first test step, we have only one task head and thus the task head selection accuracy
100%. At the second test step, we have the test samples from two seen tasks well as two available
23

---

task heads. Out of all test samples of task 1, the reported 94.5% were correctly classified into the task head I while the rest 5.5% were incorrectly classified into the task head 2. Similarly, for test samples belonging to task 2, 3.1% were incorrectly classified into the task head 1 while the reported 96.9% were correctly classified into the task head 2, and so on. Hence, by studying the task head selection incremental step, we can investigate the trend of forgetting among the individual task heads.
1 4.3
2.622.6333
4.1
High
2.2.0
2.32
3.1.9
0.4.0.0..8
2.8
1-100
4.2.2
2-96.596
High
55..3 4.034.011
2.1.71.61.0
3. 90.59.66
89:2877 380 792
Accuracy
7. 7.0.8
83
8-0.8 0.88
5.1
7 7814 41447 83 86
9 9.3.
2.1
1.2
1831220
18
777 57 581 694 82
230
10.1.0 0.8.0.8810
1.6
0.9
1.5
2.4
1,4 3,8 252
-Low
78.474 74 63
88
1 2
10
Task head
Task
(a) Last step task head selection accuracies
(b) Per step task head selection accuracies
Figure 10: Task head selection accuracies reported on CIFAR-100 upon: (a) evaluation on the last
step, (b) evaluation each incremental step.
C.3 Effect of inference module architecture.
To further investigate the effects of inference modules performances, vary the number of layers for the VGA module (sec. 3.2.2) and for the task-specific encoders (sec. (.2.3]. Fig. reportsss the results of varying the number of Transformer Decoder layers 48] in the VGA module. As the number of layers grow, the average accuracy (Avg) increases while the last task accuracy (Last) decreases. This indicates that while larger number of layers in the VGA module lead to increase in the initial tasks' performances, these are amenable to larger forgetting on latter incremental steps.
In Fig. 11b we report the performances for varying number of MIP layers in the mean and the standard deviation heads of the task distribution encoders. Unlike the VGA module, here we observe consistent trend of decreasing last and average task accuracy with the increase in the number of layers. This clearly indicates the superiority of using a single-layered task distribution encoder.
78.2
Last Accuracy
78.0
86.0
Accuracy
78.1
86.20
77.5
85.3
78.0
Last
86.15
85.6
77.9
Avg
77.0
Avg
VGA layors
# of task encodar layers
(a)
(b)
Figure 11: Ablation studies on CIFAR100 showing: (a) the variation of accuracy with the number of Transformer decoder layers in the VGA module, (b) the variation of accuracy with the number of linear layers in the task-specific mean and standard deviation encoders.
C.4 Effect of prior type.
To study the role of a more informed prior in our VI framework, we study three choices of priors be used the prior-matchine
term of eq. (10): (a) the static (standard normal) prior, (b) the language-aware prior using the distribution obtained from the task encoders using the hand-crafted prompts' features (th 1/1 (sec (.3 3.3). (c) the data-driven prior using a randomly chosen subset of training minibatch as the context set to condition the prior on (see App. G for more details). App.
Table T8 shows that while (b) and (c) slightly improve over (a) terms accuracies and forgetting,
these come at the cost of poorer model calibration and longer runtime per iteration.

---

Runtime
Prior type
Last 1
Avg 1
BwT 1
ECE +
per iter. +
Static
78.21
86.13
-0.141
0.204
0.169
Data-driven
78.32
86.15
-0.115
0.216
0.172
Language-aware
78.38
86.22
-0.112
0.214
0.17
Table 18: Performances of different priors averaged over 3 runs CIFAR100.
C,5 Inference time for different finetuning methods.
Table 19 investigates the inference time per iteration for different methods. Among the compared prompt-based methods, the inference time for AttriCIII [19] is notably the highest. This is because relies on selecting test instance-conditioned prompt tokens from a pool of prompt tokens. The instance-specific prompts are fed to the text encoder which further outputs equivalent number of instance-specific text features to be used in the derivation of logits through eq. These operations increase the inference time of AttriCLIP beyond our proposed variants of CLAP4CLIP with hand- crafted prompts (Ours), class-conditioned prompts (CoOp + Ours), and multi-modal prompts (MaPLe + Ours) where the latter three outperform AttrICLIP significantly across all our settings.
Method
Inference time (S)
Continual-CLIP
0.017
CoOp 12
0.018
MaPLe 20
0.035
AuriCLIP [99
0.257
CLIP-Adapter
0.019
Ours
0.163
CoOp + Ours
0.182
MaPL. Ours
0.064
AttrICIIIP + Ours
0.299
Table 19: Average inference time for different finetuning methods on CIFARI00.
C.6 Influence of language-aware knowledge components on training dynamics.
Continuing ablations from sec. here visualize the effects of using language-aware pre-trained knowledge, i.e., weight initialization and task distribution regularization on the training dynamics of our model. For thorough analyses, we consider four variants of our model: (a) Ours uses both weight initialization and task distribution regularization, (b) Ours without weight initialization, (c) Ours without task distribution regularization, and (d) Ours without either of the language-aware components.
Does language-aware weight initialization help alleviate stability gap 57 first investigate the evolution of the training loss during the initial training stages of each incremental task. Fig. 14 shows the loss L (Sec. (.4) during the initial 100 training iterations of each task. We observe that our proposed weight initialization technique leads to lower training losses for the scenarios with or without task distribution regularization, i.e., in general, red values green values and blue values < orange values. Following (58), our observations support the hypothesis that larger loss values lead to the stability gap [57 for CL., and that informed weight initialization method can help tackle it by reducing the initial training loss.
To answer this,
To further verify the benefit of our proposed weight initialization strategy for reducing the stability gap, we ablate the accuracy evolution of the first task test samples during the carly training stages of each task. Figures| 12b and 12a contrast these for CIFARI00 100. In general, initialization strategy helps mitigate the drop in accuracy during the initial training phases. On average, the first task accuracy upon the first iteration of training across all tasks remains 78.12 without weight initialization and grows to 79.5 with weight initialization, i.e., a gain of 1.38 percentage points.
proposed weight
How does language-aware knowledge help learning of task distributions in general? To under- stand the effect of language-aware knowledge on task distribution learning, we next investigate the evolution of the means and standard deviations learned by the past and the new task heads throughout
25

---

First 100 training iterations
First 100 training iterations
100
Accuracy
80
Taik Accuracy
60
40-
1.96
5oop
Scop
Step 3
Stap
50ap
Stap 1
Stap
Stop
Step
Step5
80
Accunity
Teck Accuncy
8
60
8
80
9'2
88
500p
Stap
Stop
Stop
(a) With weight initialization
(b) Without weight initialization
Figure 12: Effect of weight initialization on stability gap: Test accuracy with and without weight initializations the first task for the initial 100 iterations of incremental training on all tasks of 11F.R100. The green lines are the means over three different runs, the orange shades denote #1 standard error of the mean. The labels the vertical bars denote the accuracy values for the first iteration of training on each task.
the training iterations. To this end, Fig. 13 and Fig. report the training iterations against the L2 norm of means and standard deviations for the past task heads (at each incremental training step) and the new task heads (at each training step). We observe two consistent trends regarding the evolution of distributions of the past and the task heads. First, the proposed initialization of weights helps stabilize the learning of the means and standard deviations with (red against green) or without (blue against orange) regularizing the task distributions. Second, regularizing the task distributions increases the L2 norms of the learned mean and the standard deviation as these now have to encode more information mimic the distributions of the hand-crafted text features.
26

---

Continual-CLP CoOp
Ours
Ours we
Ours + GoOp (wlc VI
80
FPR95
60
40
20
40
60
80
90
AUROC f
80
70
20
40
80
AUPR
60
40
20
40
60
80
Number of Classes
Figure 13: Performance comparisons for post-hoc novel data detection averaged over 3 runs on CIFAR100: FPR95 (left), AUROC (middle), and AUPR (right). The evaluations are carried over all but the last incremental test step.
D Out-of-the-box utilities of probabilistic finetuning
D.1 Post-hoc novel data detection
Our post-hoc novel data detection (PhNDD) setting aims to evaluate the continual learning methods identifying novel data on the fly. data resource other than that provided by the lataset-specifie CL setting. Starting from the first test step, we treat the test data of the future tasks as novel while those of the seen tasks (including the most recently trained one) seen. Since the last test step of CL dataset has no future tasks, exclude this step for our PhNDD evaluation, i.e., we carry our PhNDD evaluation CL models starting from the first until the penultimate test step.
so, we design an evaluation setup that uses no additional
the Energy scores 75 of the outputs for each test sample as a measure of the model's s confidence score. The samples assigned with a confidence score below the pre-defined confidence threshold are classified as novel. By assuming the seen data as the positive class and the novel data as the negative class, can obtain a series of true positives rate (TPR) and false positive rate (FPR) by varying the confidence thresholds. One of our PhNDD evaluation metrics the FPR95 then measures the FPR when the TPR is 0.95. As such, lower FPR95 score indicates better PhNDD performance. Our other two PhNDD performance metrics include the the area under receiver operating characteristic curve (AUROC (73) calculated based on FPR and TPR, and the precision-recal eurve (AUPR 74) Higher values of AUROC and AUPR indicate better PhNDD performance.
Following other standard practices 75| 722.
27

---

Table 4 reports the PhNDD metrics averaged over all the evaluated steps. Here, in Fig, 13 we show the evolution of these metrics with each evaluation starting from the first test step until the penultimate test step of CIFARI00. We observe that the zero-shot Continual-CI.PP [14] has the poorest PhNDD performances (highest FPR95, and least AUROC and AUPR scores) across all steps given that it has not been finetuned on the downstream CL. tasks. Among the finetuned methods, the CoOp [2] exhibits the poorest performances across all tasks. Among the variants of our method, combining CoOp with ours (CoOp + Ours) achieves the best PhNDD performances across all tasks. Furthermore, the deterministic versions: Ours w/o VI and CoOp + Ours (w/ VI) remain sub-optimal to their respective probabilistic variants, i.e., Ours and CoOp + Ours. The latter results validate the added perks of our probabilistic modeling framework for post-hoe novel data detection.
D.2 Exemplar selection results
Methol Avg
CoOp
76.71
Clip-Adageer
78.78
68.49
Ours wlo VI
84.44
76.55
Table 20: Entropy-based exemplar selection results for different methods on CIFARI00.
E Limitations and further research directions
Few potential directions of research for CLAPACLIP include the design of: (a) parameter-efficient adapters (85] for very large CL settings; (b) better regularization techniques to alleviate forgetting; and (c) more informed [86] yet computationally efficient priors for inference. Similarly, along the direction of alleviating forgetting and mitigating the stability gap [57 58, . it would be interesting to see how class-specific prompts generated by pre-trained Large Language Models (LLMs) can be exploited to obtain task-relevant language-aware CLIP knowledge while preserving the zero-shot transfer ability of the learned prompts (see App. Table 21 for a preliminary investigation). Lastly, we consider applying CLAP4CLIP more sophisticated Vision-Language tasks [87] another possible direction for research. We elaborate further on each of these directions below.
Parameter overhead. For each incoming task, CLAPACLIP initializes new head consisting of d x d parameters where d is the output dimension of the CLIP model's encoder. For a very large number of real-world CL tasks, the number of finetunable parameters for CLAP4CLIP may thus become comparable to or larger than that of the pre-trained CLIP model's R 150 million parameters. For example, using a VIT-B/16 encoder with d 512 brings an overhead of 525, 000 new parameters with each incoming task. After having seen E 300 new tasks, the number of CLAP parameters to be finetuned thus amount to R 158 million, which is larger than the frozen CLIP itself, and thus defeats the purpose of finetuning at the first place. One solid future direction to use CLAPACLIP for very large real-world CL settings could thus be introducing more strict parameter-efficiency measures 88 89 and/or learning probabilistic adapters with low-rank weights [85].
Design choices. Other future directions for improving CLAP4CLIP could include the use of better regularization techniques to further prevent forgetting (see Table 13 for the current forgetting in CLAP), and the search for more informed yet computationally efficient priors (see Table 18 for the computational overhead attached with more informed priors).
LLM-generated class descriptions as language-aware knowledge. In Sec. 3.3 using the text features from hand-crafted prompts as language-award CLIP knowledge to help alleviate forgetting. However, hand-crafted prompts require manual labelling of data which is not always have opted to mining Large Language Models (LLMMs) for efficiently obtaining the class-specifi descriptions. To study the feasibility of alleviating forgetting using such -ggerrrre class descriptions, we leverage the diverse prompts from CuPL [9]] obtained using the GPT-3 [93] model. Our preliminary investigation suggests that the
we proposed
practical. Hence, several recent works 990 91 92
28

---

hand-crafted prompts have an upper hand over GPT-3 based prompts for CLAP4CLIP performance (see 2T). This could be because of the broad range of knowledge encoded in the GPT-generated prompts which at times are irrelevant for the test images.
Prompt type
Runtime per iter. +
Last f
Avg
BwT
ECE
Hand-crafted
78.21
86.13
-0.141
0.204
0.169
GPT-3
77.76
85.7
-0.099
0.219
0.151
Table 21: Performance comparison on CIFARIO0 using hand-crafted vs. LLM-generated prompts for encoding language-aware CLIP knowledge. The results reported are averages over runs. Best results across the metrics are highlighted in bold.
Based on the above finding, suggest leveraging task LLM-generated descriptions as lunguage-aware knowledge to be another promising future research direction. worth noting that number of existing methods that rely on LLM-generated prompts are limited in their transferable knowledge across unseen classes and datasets (90] 9I] (e.g., any new class at test-time would require mining the LLM descriptions in advance). On the contrary, our proposed weight initialization and task distribution regularization strategies provide a natural framework for ALM--eenrater prompts to be used alongside arbitrary learnable prompts (e.g. replacing th in eq. 7 This compliments the idea of LLM-based text-only supervision frameworks [92] that seek to enrich zero-shol transfer of
prompts to new classes by extracting rich contextual information from LLM data 3
Compatibility with Vision-Language datasets The tasks we have covered so far in the paper are based solely on Vision datasets. To further demonstrate that our method compatible with more sophisticated vision-language datasets, we here consider using a toy Visual Question Answering (VQAv2) task from the CLiMB dataset [87] The CLiMB dataset hosts number of tasks/settings to evaluate multi-modal and low-shot transfer abilities of CL algorithms. However, given the intricacies of these tasks (visual question answering, reasoning, etc.). we leave full in in-depth engagement with CLiMB 87 as separate future direction for research
To show the aptness of our method for the dataset's tasks, carry out preliminary experiments on the single-task learning setting (94) of the VQAv2 subset of CLiMB. Following [94). we rely on the BART model (95] for text generation here. Table [2] shows that our method surpasses the Continual-CIa by 9.29 percentage points on the VQAv2 task, thus showing that ours enhances the zero-shot generalization capability of CLIP.
Model
VQAv2 task score
Continual-CLIP
$7.42
Ours
66.71
Table 22: Single-task learning performance on the VQAv2 subset of the CLiMB dataset.
E.1
Broader Impact
Recent years have witnessed the immense popularity of sequential generative models like Stable Diffusion [00 with applications in multimodal content generation as well as scientific research through fast and highly detailed sampling. The CLIP text encoder is widely employed by such generative models for learning personalized concepts conditioned on text prompts [96) 97]. By effective continual finetuning of the text encoder's features, our method can thus aid in customizing such models in a sequential manner using multiple, fine-grained concepts 98 26
*Given that new classes might emerge test time for which we do not have the LLM-gennrated descriptions, it is important that the learned prompts preserve their zero zero-shot generalization ability.
"The CLiMB dataset [87] was introduced as an independent CL benchmark with a number of tasks (Visual Question Seweerin/RaasonnnnnEtaallll
and training settings including low-shot and unimodal learning. Existing works (94 that study CLiMB thus rely solely on it and not additional datasets for evaluations.
29

---

F Derivation of ELBO for the static prior.
We seek to maximize the likelihood p(yl for all observed labels y yl. To derive the predictions, our framework uses the visual- text features t. and the image imputs x (see eq. (I Our evidence thus p(y 1:T xx E!!T)) for which we derive the lower bound (ELBO). In the following, we denote the prior network as p=()") for which the true posterior is pa po |x; (.)) We approximate the true posterior using the variational posterior 4, (=") |x; (2). Our derivation ends up with the reconstruction term po(y"l>", x; f that can be seen as a deterministic function converting a given latent vector 2' and an input image x into an observation y². For our CLIP-based variational framework, this deterministic function is the cosine similarity operation followed by the softmax application (Eq. 3b)).
log poly (y 1:T (
(Log likelitoo of evidence)
log polp"""" 7 (x: I "ols" "T (aRR"))ds "
(:2 f ao 17 ( Jas 1:7 1)
(x. (ute" log volv" AT pe. Jas
(Bring evidence into integral)
R (o(l1:71o. [log pa " (o (xx)))
(Definitio of Expectationn)
(Rewrite miing sum)
Σ (iatt log Po(i Pa(Kf. =x; (x:
(Re-introduce x² by Chain role probubilit))
(aaty llo po x "|x it (a (xx
Multiply by (= [x: E
(= bes 12 (x*) 15)
(a (x: E5)
1og potyf, =1())
x (ottt [Tog 2o(a me; 2
(Splic the expoctation)
(x;
Σ (aatt)) log Poluf, x (Ixs1))
(astof bos (aooa
(Deinition of KL divergence)
(o(="i (x)))))
log Po(e', =f xx:62)
Pagtaf ((
(a( k))
t KL divergence 2 0)
Bagtt ( log pole"la' 1s" ( (2)
(Chain pcobability)
[B Peat Te [.]]] Dea(as(s" (ie (x)))x
( (egg) lospotv'le 1e xx [50] + (aact (x::
Px(
(Split Expectation)
(Defiaition of Kl. divergence)
G Data-driven prior
The choice of prior is pivotal to Bayesian inference workflow like ours 199]. While standard Gaussian prior Px = NO(0, I) adapts well to a range of settings, it is (seemingly) uninformative regarding the nature of a given task [100). With the end goal of deriving more informative priors, thus seek
scek to replace Px with task data-dependent prior distributions p', wherever applicable.
To this end, we first note that the outputs of the VGA module remain invariant not only to the order of the input text features (due to self-attention) but also to the order of the contextual image features (due to cross-attention) The latter invariance implies that the joint task-specific distribution learned by the encoder d (conditioned on the VGA outputs E from eq. 5ap is preserved if we were to permute the elements of the task-specific visual context set. More formally, this observation helps guarantee the (finite) exchangeability and the consistency properties of a stochastic process [IOI].
Motivated by the above, we treat the t-th task image features x" as the target set T and employ a randomly chosen subset of it as our context set Ct to align the t-th task text features and to condition our prior p' on:
t VGA(Q= = Kt
(12)
p² = ( = (p'(t))
where Ee the fused task-specific text feature following eq. (b). The task-specific prior p' thus endows our training framework with a resemblance to the neural process (NP) architectures
30

---

102 103 8) Following NPs, we use the same encoder 9' to parameterize the conditional prior and the variational posterior. This results in the following approximate ELBO (see App. F for the ELBO derivation):
log ply" 1:T xx, (x,C
Σ Busles xx)
( logp(y'lk' x ct)
(13)
DEL (ac((a(()17"))ll00(("
where in practice, the entire set of 1-th images in training minibatch form the target T and a randomly chosen subset of the targets make up the context C² [T04]. Note that unlike NPs, our framework does entirely rely on data-driven priors. Namely, while training on a CL task the past-task encoders are frozen and we have ample t-th task data points to condition the prior on. We thus resort optimizing the ELBO (33) during training. On the other hand, during finetuning, have limited task-specife data points to condition our context on. As such, we empirically found that switching to the static prior yields better results and thus resort to optimizing the ELBO (TO) during finetuning.
G.1 Effect of the context size on data-driven prior.
Table 18 in the main paper compares the results of using data-driven prior against the uniform normal prior and the language-aware prior (see Sec. (.3). where the latter is driven from the pre- trained text encoder using hand-crafted prompts. We observe that data-driven prior leads minor accuracy improvements over the standard normal prior but falls narrowly behind the language-awan prior. Here, study the influence of the batch size of the context selected random to derive our prior from.
Table 23 shows the last task accuracy with varying context sizes and fixed target batch size of 64. We find that a smaller context set size hurts the performance of the model to the extent falling behind the standard normal prior. Given that the context sets are the sampled subsets of the training (target) minibatches, a much smaller context set can lead to the increase in the prior matching loss values. We find that the context set batch size of 40 performs the best, and thus this to ablate the prior- deirndependent performances in the main paper.
Static prior
4
16
32
40
50
Accuracy
76.1
76.99
77.41
78.03
78.32
77.35
78.21
Table 23: Influence of the context set size used to derive the data-driven prior on CIFAR100.
G.2 Derivation of ELBO for the data-driven prior.
Similar to App. we start with the log-likelihood of the evidence which now involves conditioning on an additional context set CI.T" The t-th task context set is used to condition our prior network po(2* (C)). Following the standard practices of other data-driven prior frameworks [102] 8), introduce parameter-sharing between our conditional prior and variational posterior networks. This allows us replace our prior network with the variational posterior network (2'),, where T is
31

---

the target set for task t.
log Po( (x7 c2.2)
Wogg-ieeiioed evidence)
log P0 /xt/".e. gola pxt pxt yatt
( ʃ gata pet" et .e Daz 1)
/ Iosl"" 1x7" ot (V bey" ya
(Bring avidence into integral)
(By definition)
(Rewiite using sum)
log po 1 c*)
(Re intoodzoe a² by probability)
po(="1xt 17, ct)
2011 x ct) pa(="1c")
Chain rule of probabiiit;: T)
pa(o"17t)
(Pagte9iTe) log Po( xb, ( po(s"iTT)
(Equivalen: fraction)
log po(='|e*) 3 (ogoo log 20(="1711
(Split the expectation)
(0(3777))
po(=*ITt)
E (e (Iorno xx.)))))))))))
Dur (ap (o) + Paala " (1pp(((1)77
(By definition KL divergence)
Peoter Tey Tey[ [oopotrrfal xf.c Dalaast (7 "illooc""
divergence 0)
32

---

Ours
Ours w/o: Regularization
Ours w/o: weight init.
Ours w/o: weight init. & Reg.
200
100
100
0
0
50 Step 1
100
50
100
Step
200
150
100
100
50
0
50 Step 3
100
Step
100
100
75
Total loss
50
50
25
50
100
50
100
Step 5
Step
100
100
75
50
25
50 Step 7
100
50 Step
100
100
100
50
50 Step 9
100
50 Step 10
100
Training Iteration
Figure 14: Evolution of the loss value L during the first 100 training iterations of each task on CIFAR100. Training with our proposed weight initialization strategy consistently leads lower training losses thus bridging the stability gap [57] in CL.
33

---

weight
Regularization
weight Reg.
75
100
100
50
L2 norm of past tasks' mean
0
500
1000
0
500
1000
0
500 Step
1000
Step
Step
150
150
150
100
100
50
50
0
500
1000
500
1000
0
50.0
1000
Step
Step
Step
200
150
150
100
100
100
50
500
1000
0
500
1000
0
500 Step 10
10000
Step
Training Iteration
(a) L2 norm mean of past heads
Ours
Ours w/o: weight init.
Ours wio: Regularization
Ours w/o: weight init. & Reg.
80
60
80
60
40
70
L2 norm past tasks' std. dev.
60
0
500
1000
0
500
1000
0
500 Step 4
1000
Step
Step
110
100
80
100
70
90
1000
500 Step 6
1000
500
1000
SSep 5
Step 7
140
120
120
120
100
100
0
500 Step 8
1000
500 Step 9
1000
0
500 [Step 10
1000
Training Iteration
(b) L2 norm of standard deviation of past task heads
Figure 15: Evolution of mean and standard deviation of past task encoders with training iterations.
34

---

Ours
Ours wio: weight init.
Ours w/o: Regularization
Ours wio: weight init. Reg.
60
150
60
60
100
40
40
L2 norm current task's mean
20
50
20
20
500
1000
1000
1000
1000
Step 1
Step
Step 3
Step
Step
60
60
60
60
40
40
40
40
20
20
20
20
0
1000
1000
1000
1000
Step 6
Step
Step 8
Step 9
Step 10
Training Iteration
(a) L2 norm of mean of current task heads
Ours
Ours wio: weight Init.
Ours w/o: Regularization
Ours wio:: weight Init. Reg.
50
50
50
L2 norm current task's std. dev.
45
45
40
30
40
40
35
35
500
1000
1000
1000
1000
Step
Step 2
Step
Step 4
Step 5
55
50
45
45
40
40
40
40
35
35
35
1000
1
$000
1000
1000
1000
Step 6
Step 7
Step 8
Step 9
Step 10
Training Iteration
(b) L2 norm of standard deviation of current task heads
Figure 16: Evolution of mean and standard deviation of task encoders (recorded at the step where they were first introduced) with training iterations.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: To support the claims, we provide in-depth experimental results on four different (with replay memory, cross-datasets, and resource-constrained) CL setups.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We detail the limitations of our work in App. This includes parameter overhead and design choices.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
36

---

Answer: [Yes]
Justification:
We detail the derivation of Evidence Lower Bound for our proposed model in
App. [F]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have made our code public in addition to providing details on hyperparam- eter choices and tuning.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
37

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We use publicly available datasets and have also released our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccipps.c/ppulli//guu /utee//ggudddeeeedSuuuubmiibii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We detail these in App. A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report the standard deviation scores for our main results in Table 12
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
38

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We leave these details in App. A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have followed all the listed NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss these in App.
B.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
39

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: To the best of our knowledge, the pretrained CLIP model does not have any such associated high risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have provided citations wherever necessary.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
40

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not rely on crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not use human subjects for research this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "Cm1W2ULOrm", "reviewer": "Reviewer_giBN", "summary": "This paper proposes Continual Learning with Probabilistic Finetuning (CLAP) for class-incremental learning using CLIP. The key modules of the proposed idea are as follows. First, the authors introduce a CLIP-based probabilistic finetuning model using Bayesian Variational Inference to achieve better generalization during the continual learning process. Second, they propose a visual-guided attention (VGA) model to facilitate cross-modal feature alignment in the continual learning process. Lastly, to alleviate forgetting of pre-trained language-aware CLIP knowledge, they suggest past-task distribution generalization. Additionally, to enable parameter-efficient learning, a probabilistic adapter is used, along with a method for its initialization to enhance stability. Experimental results on various datasets demonstrate that the proposed algorithm achieves superior class-incremental learning performance compared to existing algorithms.", "questions": "See the Weakness section. I don't have significant concerns about the algorithm proposed in this paper, but I couldn't give it a higher score due to some experimental uncertainties and questions about the necessity of the setting. If the authors address these concerns in the rebuttal, I would gladly raise my score.", "limitations": "There is no potential negative societal impact of this paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this paper are as follows:\n\n1. The paper is well-written and easy to understand.\n\n2. The proposed modules for successful class-incremental learning using CLIP are meticulously designed. Through various analyses and ablation studies, the roles of each module are thoroughly demonstrated. Although the algorithm appears somewhat complex compared to existing algorithms, parameter and time analyses show that the actual cost difference is negligible.\n\n3. In class-incremental learning experiments using diverse datasets, the proposed algorithm consistently outperforms existing algorithms across various evaluations.", "weaknesses": "1. I have no major concerns regarding the contribution of the proposed algorithm for class-incremental learning using CLIP. However, I have some questions based on the experimental results.\n\n1-1) Unlike algorithms like L2P and DualPrompt, which utilize exemplar memory, the paper does not use exemplar memory. Are the results in L2P and DualPrompt sections of the paper based on using exemplar memory in a fair comparison? To ensure a fair comparison, results using exemplar memory should be shown.\n\n1-2) The paper only considers prompt-based algorithms like L2P and DualPrompt using Vision Transformer (ViT). However, newer prompt-based algorithms (e.g., CODA-Prompt) that achieve better performance and representation-based algorithms (e.g., Ranpac and EASE) that generally improve performance have been proposed. The authors should consider these algorithms as additional baselines. For more details, please refer to this survey paper [1].\n\n1-3) Among existing baselines, PROOF is currently the state-of-the-art algorithm. While Table 1 shows PROOF's results, Tables 2 and 16 do not. To validate the superiority of the proposed algorithm, results for PROOF should be shown in these tables and other key experiments.\n\n2. Recent research has highlighted discussions on hyperparameter tuning for class-incremental learning algorithms [2] and pointed out issues regarding computational costs[3]. From this perspective, I believe that further discussion and additional analysis on computational costs regarding the proposed algorithm would make the paper more convincing.\n\n3. Lastly, I have personal concerns regarding the necessity of class-incremental learning research using CLIP. As shown in [1], algorithms using only ViT have already achieved excellent performance in class-incremental learning research. Despite using pretrained ViT, one of their major advantages is achieving superior performance without using exemplar memory. Considering this perspective, the proposed research not only uses a pretrained CLIP model but also requires text information and additional exemplar memory usage. Simply comparing numerical experimental results suggests that the proposed algorithm does not achieve overwhelmingly superior performance compared to algorithms using only pretrained ViT. In light of this, I would like to understand: 1) What justifies the necessity of class-incremental learning research using CLIP with exemplar memory? Additionally, 2) I am curious about the potential for the proposed algorithm's ideas to be applied to continual learning in other domains using CLIP or continual pre-training of CLIP itself.\n\n[1] \"Continual learning with pre-trained models: A survey.\" arXiv preprint arXiv:2401.16386 (2024).\n\n[2] \"Hyperparameters in Continual Learning: a Reality Check.\" arXiv preprint arXiv:2403.09066 (2024).\n\n[3] \"Computationally budgeted continual learning: What does matter?.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer giBN, \n\nThank you for your comments and suggestions. We have tried our best to address your concerns below.\n\n1-1) **Fair comparison with L2P and DualPrompt:**\nThe scores we report for L2P and DualPrompt are fair and use a CLIP-based backbone as well as memory replay with the same number of exemplars as our method. As also mentioned in our global rebuttal, both L2P and DualPrompt originally use the ViT-B/16 backbone pre-trained on the ImageNet-21K dataset [1] which leads to higher performance scores on the compared datasets. However, replacing this pretrained ViT backbone with that of OpenAI CLIP’s pretrained ViT backbone leads to a drop in the performance across all datasets and clearly necessitates the need for memory replay to catchup with our proposed method. The aforesaid drop in performance has also been observed by other studies adapting L2P and DualPrompt for continual learning with CLIP [2].\n\n1-2) **CODA-Prompt as baseline:** Based on your comment, we have included CODA-Prompt (reimplemented using the pre-trained OpenAI CLIP’s ViT backbone) into our baselines. Like L2P and DualPrompt, the original CODA-Propmt paper uses ImageNet-21K pre-trained ViT. Replacing this with CLIP’s ViT backbone necessitates the need for memory replay (see Table 1 vs Table 2 in the rebuttal pdf) to catch up with the performance of our method. Hence, our comparison with CODA-P, L2P, and DualPrompt remains fair and consistent.\n\n1-3) Thank you for your concern. We have included the comparison with PROOF on the Cross-Dataset Continual Learning setup in Table 3 of the rebuttal pdf.\n\nAlso, adding to Table 16, the avg. inference time for PROOF remains 0.177s which is slower than the base (Ours) and the MaPLe variant of ours but faster than our other variants utilizing task-conditioned (CoOp) and instance-conditioned (AttriCLIP) prompts. \n\n2) **Computationally-budged CL setup:** Regarding computationally-budgeted CL setup, we would like to state that not all CL training setups require to be computationally-budgeted. In fact, upon finetuning of large pre-trained models like CLIP, it is often the case that their performance post-deployment, both in terms of accuracy/zero-shot transfer ability and inference time, counts the most. Our thorough evaluations make a clear statement about the upper hand of our method in terms of performance based on a number of metrics – accuracy, backward and forward (zero-shot) transfer ability, and calibration. Our inference time is also mainly dependent on the type of prompts used with our method. \n\nIn Table 4 of the rebuttal pdf, we have nevertheless provided an ablation on the “normal” budgeted CL setup of [3] where, on each incremental training task, we allocate the number of training iterations equivalent to 1 epoch on the first (base) task of each dataset. Here, our variant utilizing instance-conditioned prompts of AttriCLIP outperforms other compared methods. A further ablation shows that our proposed weight distribution regularization technique indeed remains a crucial component at tackling forgetting on the budgeted setup (see the two bottom-most rows in Table 4).\n  \n3) **“The proposed algorithm does not achieve overwhelmingly superior performance .. “:** In light of the consistent superior performance scores of our method without using additional exemplar memory replay and on computationally budgeted CL setup, we hope that the reviewer reconsiders their remark.\n\n4) **Justification on exemplar memory:** We would like to state that the use of memory replay in our proposed continual learning method is not redundant, but rather complementary to CLIP's existing strengths. To justify this, in Table 2 of the attached pdf, we show that our method is indeed compatible with setups without memory replay, and performs either on par or better than other compared methods, including ViT-only methods. \n\nMoreover, we would like to highlight two such perspectives to clarify that the usage of a small additional exemplar memory is not to be seen as an overhead in continual learning. **First,** for pre-trained foundation models, a critical performance measure for deploying their  (incrementally) finetuned variants is often their inference time overhead and their downstream/zero-shot transfer abilities. The use of exemplar memory does not affect the inference time overhead of these methods. In fact, comparing Tables 1 and 2 from the rebuttal pdf, it is clear that using replay memory boosts their downstream task performance. Lastly, we clearly state the superior zero-shot (forward) and backward transfer abilities of our proposed method in Sec. 4.1 in the main paper. **Second,** if we were to look at exemplar memory from a prior regularization perspective, then it becomes immediately clear how predominant the usage of these is across different domains, e.g. the adaptation of Vision-Language Models [4], latent diffusion models [5], and LLMs [6].\n\nReferences:\n\n[1] Dosovitskiy, Alexey *et al.* “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.” ICLR 2021.\n\n[2] Zhou, Da-Wei *et al.* “Learning without Forgetting for Vision-Language Models.”\n\n[3] Prabhu, Ameya *et al.* “From Categories to Classifier: Name-Only Continual Learning by Exploring the Web.” ArXiv abs/2311.11293 (2023).\n\n[4] Smith, James Seale *et al.* “Adaptive Memory Replay for Continual Learning.” CVPR 2024 Workshops.\n\n[5] Kumari, Nupur *et al.* “Multi-Concept Customization of Text-to-Image Diffusion.” CVPR 2022.\n\n[6] Wang, Yifan *et al.* “InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions.” ACL 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HznVX8JFG5", "reviewer": "Reviewer_bibK", "summary": "The paper emphasizes on the existing limitations of deterministic approaches in fine-tuning and highlights the need for probabilistic fine-tuning approach. Following this, it proposes a probabilistic parameter efficient fine-tuning method for continually learning vision language models like CLIP.", "questions": "1. Could you address the points I raised in the weakness section?\n\n2. The paper demonstrates a ~2% performance improvement with the addition of the memory consolidation component. However, I'm interested in seeing how the proposed method performs when the replay memory size is set to zero.", "limitations": "The authors have discussed the limitations of their work in the paper.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach seems novel.", "weaknesses": "1. Justification of probabilistic modelling of the text feature and not the image feature space is not clear.\n2. The approach is highly inefficient in terms of inference time.\n3. Recent approaches like ConvPrompt[a], CODA-Prompt[b], HiDe-Prompt[c], and SLCA[d] not compared.\n\n\n[a]Roy, Anurag, et al. \"Convolutional Prompting meets Language Models for Continual Learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024\n\n[b]Smith, James Seale, et al. \"Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[c]Wang, Liyuan, et al. \"Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[d]Zhang, Gengwei, et al. \"Slca: Slow learner with classifier alignment for continual learning on a pre-trained model.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer bibK,\n\nThank you for your comments and suggestions. In what follows, we have tried our best to address your concerns.\n\n- **Why do we do probabilistic modeling of text feature space?** We opt for probabilistic modeling of task-specific text feature space rather than image feature space mainly in light of the practical constraints imposed by the class-incremental learning (CIL) setting. In CIL, at test time, we are not given the task labels for images. As such, if we were to use task-specific adapters to model task-specific visual features distribution (rather than task-specific text features distribution as we do now), then we must know which images should be routed to what adapter – something not plausible at test time. A naive getaway would be to route the text-guided visual features to all available adapters and then infer the correct prediction based on the adapter outputs. Such an exhaustive routing mechanism would greatly increase our computational burden at test time. Modeling the distribution of visual-guided text features helps us overcome this because now our visual features serve as a shared context to which all task-specific text features (which we can distinguish simply by their labels) can attend. By sampling from the distributions over such visual-guided task-specific text features, we can compute their cosine similarities with the visual features to obtain our predictive logits.\n\n- **Comparison with zero replay memory size:**  In Table 2 of the rebuttal pdf, we have provided a comparison of our proposed method with the SOTA models for continual learning without replay with ViT (i.e., CODA-Prompt) and with CLIP (i.e., AttriCLIP). Here, leveraging the instance-conditioned and semantically diverse prompts of AttriCLIP provides an edge. Our variant leveraging AttriCLIP further improves its performance surpassing the SOTA. In the bottom two rows of the table, we ablate the role of our proposed language-aware distribution regularization and weight init. components and find that the former is crucial in avoiding forgetting in this setting.\n\n\n- **Inference time overhead:** The inference time of our method is highly dependent on the number of MC samples as well as the prompt type being used. For instance, in Table 16 Appendix C.5, we show that using multi-modal prompts (MaPLe) with Ours leads to an average inference time of 0.064s which is comparable to and even lower than other existing methods like AttriCLIP (0.257s). Also, in our rebuttal to Reviewer **giBN**, we state that the inference time of several variants of our method remains lower than the compared SOTA (PROOF) -- 0.163s (Ours) vs 0.177s (PROOF).\n\nWe nonetheless agree that MC sampling for probabilistic modeling endows our method with higher inference time overhead compared to other deterministic methods (something we have thoroughly ablated in Fig. 8, Appendix C.1). However, such a caveat is known to be general for probabilistic models, and we believe that rejecting our method based on this (while downweighing the wide range of performance advantages) would be unfair. Put together, our proposed method outperforms several existing SOTA with better accuracy, backward and forward (zero-shot) transfer ability, and calibration, and also performs better on settings without memory replay and with restricted computational budget, as shown in Table 2 and Table 4 in the rebuttal pdf, respectively.\n\n- **Comparison with every single recent work:** We would like to state that continual learning with pre-trained foundation models is a rapidly evolving field, as is the evolution of new pre-trained foundation models itself. Given this fast-moving environment, it is challenging to provide an exhaustive comparison with every single recent work. While we strive for comprehensive analysis, we hope the reviewer understands that an all-encompassing comparison may not be practically achievable within the scope of this work. That said, we have thoroughly covered the **most relevant** SOTA models for vision-language models (PROOF and AttriCLIP) as well as for vision-only models (DualPrompt and CODA-Prompt) across a range of datasets and settings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yhfXt9R04u", "reviewer": "Reviewer_KrxH", "summary": "This paper introduces CLAP4CLIP, a method designed to enhance continual learning (CL) using CLIP, a pre-trained vision-language model. The method leverages probabilistic fine-tuning with task-specific adapters to mitigate the issue of catastrophic forgetting commonly faced in CL. By incorporating visual-guided attention (VGA) modules, the model aims to align text features with frozen visual features during incremental training. The proposed method is evaluated on several datasets, including CIFAR100, ImageNet100, ImageNet-R, CUB200, and VTAB, and compared against multiple baselines and state-of-the-art fine-tuning methods.", "questions": "1. Could you provide more insights into why the replay strategy is necessary and how it complements the use of CLIP in your method?\n2. How is the VGA module handled during incremental tasks? Are its parameters updated, and if so, what strategy is used to ensure consistency and stability across tasks?\n3. What are the computational requirements of your method compared to the baselines, especially concerning the additional components introduced? How do you balance the trade-off between performance improvements and computational efficiency?", "limitations": "*", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a novel approach by integrating probabilistic fine-tuning and visual-guided attention into CLIP for continual learning. The use of task-specific adapters and the Bayesian variational inference framework adds a unique angle to the existing methods.\n2. The methodology is well-structured, and the experiments are thorough, comparing the proposed method against a wide range of baselines and state-of-the-art approaches. \n3. Addressing catastrophic forgetting in CL is a significant challenge, and the proposed method offers a promising solution. The integration with CLIP, known for its zero-shot learning capabilities, highlights the potential impact on real-world applications.", "weaknesses": "1. Given CLIP’s strong generalization abilities and its effectiveness in zero-shot learning, the reliance on replay strategies might seem redundant and potentially underutilizes CLIP’s full capabilities. \n2. The paper does not explicitly detail whether the VGA module is updated during incremental tasks. This is a critical aspect, as the update strategy could significantly impact the model's performance and stability.\n3. The method involves multiple components, such as probabilistic fine-tuning, VGA modules, and task-specific adapters, which may introduce significant computational overhead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer KrxH,\n\nThank you for your comments and suggestions. In what follows, we have tried our best to address your concerns.\n\n- **Our method stands out without replay, replay further boosts its performance:** Thank you for your comment. We would first like to mention that our additional rebuttal experiments without memory replay (Table 2 in the rebuttal pdf) clearly show that our method outperforms the compared SOTA even without replay. \n\nThe use of memory replay in our proposed continual learning method is thus not redundant, but rather complementary to CLIP's existing strengths. CLIP's power primarily comes from its strong alignment between visual and textual modalities. As we finetune the model to incremental tasks, memory replay helps maintain this crucial alignment for previously learned tasks. Without it, the model might drift, compromising its image-text retrieval performance and in turn, its zero-shot capabilities on previously seen concepts while learning new ones. More importantly, like other CL setups, our use of replay is confined to the training process. Once the model is fine-tuned, there's no additional computational overhead during inference brought about by memory replay. \n\nIn addition, from the perspective of prior regularization, memory replay has broadly been used for adapting/finetuning a range of foundation models, e.g. the adaptation of vision and language models [1], latent diffusion models [2], and large language models [3]. These practical scenarios often demand that the deployed foundation models maintain their zero-shot transfer ability as well as their inference-time efficiency. Replay thus helps boost the former without affecting the latter. \n\nLastly, in Section 5 of the main paper, we discuss that our proposed method is **agnostic** of the strategy used to select exemplars for memory replay. To support this, in Table 18, Appendix D.2, we show that our method works well even with an entropy-based exemplar selection strategy – a setup where existing deterministic methods lag due to their poor predictive confidences.  \n\n- **Working of VGA module:** Thank you for your comment and apologies for the working of the VGA module being unclear from the main paper – we mention it explicitly in Appendix A.2 “Training for memory consolidation”.  The VGA module is indeed shared across different tasks and its parameter is updated normally during the training phase when our training data comprises the current task data as well as the replay memory data. This is followed by the training phase for memory consolidation where we finetune our model on the class-balanced dataset of new data and rehearsal data. Following other well-established parameter-isolation CL algorithms [4-5], here we freeze the task-shared VGA parameters to avoid interference with the knowledge acquired during the normal training phase (given that during the consolidation phase, our training data is vastly reduced and comprises only the class-balanced data maintained in the small replay buffer). \n\n- **Computational requirements of our method:** We provide a detailed comparison of the parameter and time analyses for different methods in Fig. 4 and Appendix Table 16, respectively. A major computational overhead for our proposed probabilistic method is the number of Monte Carlo samples. In App C.1, we thoroughly report the accuracy-runtime tradeoff for our proposed method with the varying numbers of MC samples. Namely, the accuracy remains poorer in the range [1,10], grows in the range [10, 20], and in general, saturates thereafter. On the other hand, the runtime grows roughly by 103% as we increase the number of MC samples from 1 to 50. \n\nFrom the perspective of probabilistic finetuning, we also analyze the performance vs efficiency trade-off for the choice of prior type in Appendix C.4. As shown in Table 15, compared to the static standard normal prior, the data-driven and language-aware priors offer us minor performance gains in terms of last and avg. accuracy and backward transfer scores. However, these performance gains are neutralized by the higher cost of runtime per inference iteration. As a result, we stick to using the standard normal prior throughout our work. \n\nReferences: \n\n[1] Smith, James Seale *et al.* “Adaptive Memory Replay for Continual Learning.” CVPR 2024 Workshops.\n\n[2] Kumari, Nupur *et al.* “Multi-Concept Customization of Text-to-Image Diffusion.” CVPR 2022.\n\n[3] Wang, Yifan *et al.* “InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions.” ACL 2024.\n\n[4] Castro, Francisco Manuel *et al.* “End-to-End Incremental Learning.” ECCV 2018.\n\n[5] Douillard, Arthur *et al.* “DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion.” CVPR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P4yrsLF08V", "reviewer": "Reviewer_ey1B", "summary": "The authors present a novel method for CL called CLAP (Continual LeArning with Probabilistic finetuning) applied to the CLIP model. The technique employs probabilistic modeling to refine task-specific modules aligned with visual-guided text features, improving model adaptation to new tasks while mitigating the forgetting of old ones. It also benefits from the knowledge of pre-trained CLIP for this purpose. This method also offers compatibility with prompt-based finetuning methods as well.  The approach demonstrates superiority over traditional deterministic finetuning methods through enhanced performance and better uncertainty estimation in diverse experimental setups.", "questions": "Please address the mentioned weaknesses.", "limitations": "The limitations of this work are discussed and presented in Appendix E.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. **Originality and Significance:** The paper introduces a unique approach to CL by integrating probabilistic finetuning with the CLIP model. The proposed pipeline is innovative and significant, as it addresses the critical issue of catastrophic forgetting in CLIP when trained on streams of tasks using well-justified solutions.\n   \n2. **Quality and Clarity:** The paper is well-written, with clear explanations of the methodology and its advantages. \n\n3. **Technical Soundness:** The experimental setup is comprehensive, covering several datasets and comparative baselines. The results convincingly demonstrate the effectiveness of CLAP4CLIP in improving in-domain performance and generalization to new tasks. Especially achieving positive back-ward transfer in VTAB (Table 12) was a very interesting finding.", "weaknesses": "1. **Presentation Issues:** Figure 1 could be larger to improve readability and clarity. The paper would benefit from better visualization to help convey the complex mechanisms of the proposed method more effectively.\n\n2. **Reference Order and Citations:** The order of the references seems incorrect, which could potentially confuse readers. Additionally, Reference [36] should be discussed in the related works section as well to better contextualize the contributions of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ey1B,\n\nThank you for your comments. \n\n- We assure you that we will enlarge Figure 1 in the next version of the paper to improve its readability and clarity.\n\n- We will correct the reference order accordingly and discuss the relevant reference in the related works. We apologize for any confusion this may have created.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their comments and constructive suggestions on our manuscript. Here, we provide a pdf containing results for the experiments asked in the reviews. We also highlight the three major points raised in the reviews and how our rebuttal has addressed these:

- **Comparison with CODA-Prompt:** We have included CODA-P into our baselines and have compared it with our method across all datasets with and without replay in Table 1 and 2 of the rebuttal pdf, respectively. Note that similar to L2P and DualPrompt, CODA-P paper uses ImageNet-21K pre-trained ViT-B/16 model as the baseline. Given the similarity of ImageNet-21K images with the compared datasets, this leads to quite high performances.  For a **fair** comparison, we thus reimplement CODA-P using pretrained OpenAI CLIP’s ViT backbone (similar to how we compare L2P and DualPrompt in our paper). The latter reimplementation then demands memory replay to catch up with the performance of our proposed method, thus making our comparisons fair. Lastly, as shown in Table 1, our method surpasses CODA-Prompt across all datasets.
- **Our method works without replay, replay further boosts its performance:** In Table 2 of the rebuttal pdf, we have provided a comparison of our proposed method with the SOTA models for replay-free continual learning using ViT-based (i.e., CODA-Prompt) and CLIP-based (i.e., AttriCLIP) backbones. Here, leveraging the instance-conditioned and semantically diverse prompts of AttriCLIP provides an edge. Our variant leveraging AttriCLIP further improves its performance surpassing the SOTA. In the bottom two rows of the table, we ablate the role of our proposed language-aware distribution regularization and weight init. components and find that the former is crucial in avoiding forgetting in this setting.
- **Our method is agnostic of the exemplar selection strategy used for replay:** In Section 5 in the main paper, we discuss that our proposed method is agnostic of the strategy used to select exemplars for memory replay. To support this, in Table 18, Appendix D.2, we show that our method works well even with entropy-based exemplar selection strategy – a setup where deterministic methods generally lag due to their poor predictive confidences [1]. 

References:

[1] Chaudhry, Arslan *et al.* “Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence.” ECCV 2018.

Meta-review: Initially, the paper receives three negative scores and one positive score; after the rebuttal, one reviewer increases the score to borderline accept. I would like to thanks the authors for providing the detailed responses to the reviewers’ questions and addressed many concerns from the reviewers. The AC read through the manuscript, all reviews, and the rebuttal. Some reviewers did not actively participate in the discussion, although the AC tried several times to initiate one. The AC carefully read the authors' response and think it addresses the concerns from the reviewers well and would like to recognize the originality of the paper. The AC would like to accept this paper. The AC also strongly encourage the authors to polish the writing and incorporating the suggestions of the reviewers in the final version.
Decision: Accept (poster)