RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Related work
Continual Learning (CL). The existing CL literature is predominated by three categories of methods: (a) Regularization-based methods [ 33 , 2 , 44 ] alleviate forgetting by punishing changes to the parameters that are important to previous tasks; (b) Architecture-based approaches learn parameters that are specialized for individual tasks either by network expansion [ 14 , 27 ] or by sub-network composition [ 57 , 28 ] ; (c) Rehearsal-based approaches [ 65 , 7 ] rely on storing a fraction of the past task experiences in a memory to train with the current task. Nevertheless, each category has its own flaw – methods in (a) struggle to discriminate inter-task classes [ 41 ] ; those in (b) often require task oracle during inference and can induce dramatic memory overhead for a larger number of tasks; those in (c) are sensitive to the memory sizes besides being prone to overfitting on the memory samples [ 72 ] . Hence, practical CL calls for combining these. Our work leverages (a) via function-space regularization (Sec. 3.7 ), (b) via task-specific modules (Sec. 3.5 ), and (c) via herding-based replay [ 65 ] (Sec. 4 ).

Vision-Language Models (VLMs) finetuning. The powerful generalizability of pre-trained VLMs [ 1 , 84 ] like the CLIP [ 63 ] has enabled their zero-shot applications to a range of downstream tasks, including CL [ 70 ] . In practice, their performance on downstream out-of-distribution data remains rather weak [ 79 , 60 ] . For such cases, finetuning on task-specific data is a natural choice [ 79 ] .
Instead of performing full finetuning on all parameters, some parameter-efficient finetuning methods learn a lightweight feature adapter module for textual and/or visual paths [ 17 , 87 ] . Another line of parameter-efficient finetuning methods learns soft prompts which are a few continuous tokens serving as inputs to the frozen visual and/or textual encoder(s) to capture task-specific information [ 94 , 93 ] . Existing works on CL with pre-trained CLIP have leveraged either [ 74 , 67 ] or both [ 92 ] of these methods. However, such finetuning methods remain deterministic in nature. This imposes an explicit constraint on the modeling of the possible ways in which the visual and the textual semantics interact.

To address the aforesaid flaw, one could turn to adapting the existing probabilistic finetuning approaches to capture the cross-modal interactions in CL tasks. For instance, [ 47 ] learn the distribution of hand-crafted prompts while [ 12 ] leverage VI to model the distributions of soft prompts conditioned on the input images. Yet these methods are limited in their efficacies. While [ 47 ] is incompatible with conditional prompt learning [ 93 ] , [ 12 ] lags in terms of in-domain performance and computational efficiency. Hence, our work aims to bridge these gaps for probabilistic finetuning all while adapting it for CL.
Preliminaries
Continual Learning (CL). Class-incremental CL [ 48 ] aims to learn from a sequence of T tasks [(C^{1},D^{1}),(C^{2},D^{2}),...,(C^{T},D^{T})] . Each task t\in[1,T] has its training data D^{t}=\{(\mathbf{x}_{1},y_{1}),(\mathbf{x}_{2},y_{2}),...,(\mathbf{x}_{k^{t}},y_{k^{t}})\} , where \mathbf{x} and y are the input images and labels, respectively from the set of classes C^{t}=\{c_{1}^{t},c_{2}^{t},...,c^{t}_{n^{t}}\} . Following [ 44 , 24 , 48 ] , we assume any two task-specific sets of classes to be disjoint: C^{i}\cap C^{j}=\emptyset . A neural network with parameters \phi is then trained on task t using D^{t} to minimize the cross-entropy loss over C^{t} . At test time, the model is evaluated on all seen classes \bigcup_{i=1}^{t}C^{i} , where the past task predictions are prone to forgetting. As a solution, rehearsal-based methods [ 5 , 80 ] replay past task samples from a memory \mathcal{M} during training. We use herding [ 78 ] to maintain \mathcal{M} .
CLIP with prompt design
CLIP comprises an image encoder f(\textbf{x}) acting on an image \mathbf{x}\in\mathbb{R}^{3\times H\times W} and a text encoder g(\textbf{t}) acting on a word embedding vector \mathbf{t}\in\mathbb{R}^{(L\times e)} derived from a text prompt \mathbf{p}\in\mathbb{R}^{((L-1)\times e)} . Here, H and W are the height and the width of the image, respectively, L is the text length, and e is the text embedding dimension. Both the encoders output a d- dimensional encoding, which are used in the prediction of the class y_{i} of the image as:

p(y_{i}|\mathbf{x})=\frac{\exp\big{(}\langle f(\mathbf{x})^{T},g(\textbf{t}_{i})\rangle/\tau\big{)}}{\sum_{c=1}^{|C^{t}|}\exp\big{(}\langle f(\mathbf{x})^{T},g(\textbf{t}_{c})\rangle/\tau\big{)}},\vspace{-0.1cm} (1)

where \tau is a learnable temperature parameter, \langle\cdot,\cdot\rangle is the cosine similarity, and \mathbf{t}_{c}=[\mathbf{p},\mathbf{e}_{c}] is the result of adding a class-specific word embedding \mathbf{e}_{c} to the prompt \mathbf{p} .
The textual encoding g(\mathbf{t}_{c}) ’s for all classes are used as the weights of a linear classifier. In CL, g(\mathbf{t}_{c}) encodes the class names seen until task t .
Eq. ( 1 ) thus forms contrastive training criteria for the text and visual modalities, whose rich representation allows pre-trained CLIP to be used for zero-shot classification through hard prompt templates, i.e., \mathbf{p}^{c}= “ A photo of a {c th class} ”.
CLIP finetuning with soft prompts
To improve the CLIP performance on a downstream task t , prompt learning employs soft prompts as a set of learnable vector tokens \mathbf{p}=\{\mathbf{p}^{1},\mathbf{p}^{2},...,\mathbf{p}^{L}\} . CoOp [ 94 ] shares \mathbf{p} with all classes and averages them to use in t . MaPLe [ 29 ] learns multi-modal prompts by employing two such token sets \mathbf{p}_{v} and \mathbf{p}_{t} until the J -th layers of the vision and the text encoders of CLIP, respectively. AttriCLIP [ 74 ] selects input-conditioned tokens \{\{\mathbf{p}^{j}\}_{1\leq j\leq L}|\mathbf{x}_{k}\} . Learning \mathbf{p} (with frozen CLIP weights) thusly helps encode class/modality/instance-conditioned context for a given task.
CLIP finetuning with adapters
Adapter-based methods, e.g . , CLIP-Adapter [ 17 ] , learn a lightweight adapter module over the text and/or visual features of the frozen CLIP model. With a text adapter A_{t} , the updated text encoder features from Eq. ( 1 ) can now be rewritten (with a slight abuse of notation) as:

g(\mathbf{t}_{i})=\alpha A_{t}(g(\mathbf{t}_{i}))+\beta g(\mathbf{t}_{i}),\vspace{-0.1cm} (2)

where \alpha and \beta control the strength of the residual connection between the adapted and the pretrained features, e.g . , \beta=1-\alpha in [ 17 ] .
Unlike soft prompts, however, the gradients of the adapter do not back-propagate through all the text encoder layers. This gives them a computational edge in training.
Probabilistic modeling and inference in CLAP 4 CLIP
In Sec. 3.3 - 3.7 , we develop our CLIP-based probabilistic finetuning model with a variational inference (VI) framework, as shown in Fig. 1 . We start by introducing a basic prediction-aware latent variable model for CLAP 4 CLIP in this section.
Cross-modal feature alignment
To investigate the source of catastrophic forgetting in finetuning methods, we study how CL effects the cross-modal deviation [ 53 ] between the text and the image features of CLIP.
Here, we consider two basic CL models: the CoOp [ 94 ] and the CoOp with a CLIP-Adapter [ 17 ] .
Then, for the base task ( t=1 ) test samples of CIFAR100, we compute the average of the Rotation Angle Matrix (RAM) [ 53 ] using the CL models’ frozen visual f(\mathbf{x}) and learnable textual g(\mathbf{t}_{c}(\mathbf{p})) features at each incremental test step.
Fig. 2(b) shows the deviation of the learned textual features from their (frozen) visual counterparts for the CoOp. This implies that the cross-modal retrieval performance of CLIP finetuned with learnable prompt deteriorates with incremental training. Moreover, a simple adapter (CoOp + Adapter) does not remedy the cross-modal deviation.
Modelling with visual-guided text features
To alleviate the cross-modal deviation due to incremental training, we consider explicit enriching of communication among the encoders’ features. To this end, we adopt a standard transformer-styled decoder block [ 71 ] as a visual-guided attention (VGA) module that refines the task-specific text features based on the visual features as context. While in a non-CL setup, the VGA module has been shown to guide the text features to attend to the informative per-pixel spatial features (obtained before global average-pooling in a ViT [ 13 ] ), here we use the globally pooled visual features of the ViT for guiding the text features. This setting reflects our primary motivation for aligning the text features which we can later employ for variational modelling. Further, using globally pooled visual features favors the computational efficiency of our framework for large CL datasets where attending to per-pixel spatial features can incur much higher latency (App. 8.3 ).

Leveraging the task-specific text features g(\mathbf{t}_{c}^{t}) as queries Q and the visual global context f(\mathbf{x}) as keys K and values V , the VGA performs context-context self-attention followed by context-query cross-attention. To eliminate the influence of the queries from other tasks, a naive strategy is to perform specialized VGA forward passes [ 14 ] with task-specific query. Instead, we replace several such costly passes with a single pass. To do so, we exploit the global nature of our context and mask out (set to -\infty ) all inter-task connections in the query using a target mask. This ensures that only the task-specific text features in the query undergo self-attention while the entire query cross-attends to the visual context:

\begin{split}O&=\{O^{k}\}_{k=1}^{t}\ &=\{\hat{\mathbf{t}}_{c}^{k}\}_{k=1}^{t}\ &=VGA\big{(}Q=\{\mathbf{t}_{c}^{k}\}_{k=1}^{t},K=V=f(\mathbf{x})\big{)},\end{split} (5)

where O is the set of task-specific visual-guided text features \hat{\mathbf{t}}_{c}^{t} . We fuse the original task-specific text features \mathbf{t}_{c}^{t} to derive the task embeddings \tilde{\mathbf{t}}_{c}^{t} :

\tilde{\mathbf{t}}_{c}^{t}=\hat{\mathbf{t}}_{c}^{t}+\mathbf{t}_{c}^{t}.\vspace{-0.3cm} (6)
Task-specific distribution encoders
Instead of feeding the concatenated text encoding \{\tilde{\mathbf{t}}_{c}^{1},\tilde{\mathbf{t}}_{c}^{2},...,\tilde{\mathbf{t}}_{c}^{t}\} to one shared adapter q_{\phi} , we maintain task-specific encoders \{q_{\phi}^{1},q_{\phi}^{2},...,q_{\phi}^{t}\} ; each comprising a probabilistic adapter to parameterize the t- th task-specific posterior distribution \mathcal{N}(\mu^{t},\sigma^{t}) over its task embeddings \tilde{\mathbf{t}}_{c}^{t} :

\{z^{t}_{m}\}_{m=1}^{M}\sim q_{\phi}^{t}(z|\tilde{\mathbf{t}}_{c}^{t})=\mathcal{N}\big{(}\mu^{t}(\tilde{\mathbf{t}}_{c}^{t}),\sigma^{t}(\tilde{\mathbf{t}}_{c}^{t})\big{)}, (7)

where \{z^{t}_{i}\}_{m=1}^{M} are the M task-specific MC samples. Task-specific encoders help us sample latent variables that are more discriminative across tasks. This is depicted in Fig. 3 using the cosine distance between the embeddings of class-specific samples. With task-specific encoders (right), the cross-task class centroids are further apart than those with one global encoder (left).

During each incremental training step ( t>1) , we keep the past task encoders frozen. Lastly, to reduce the forgetting of past task encoders, we follow other parameter-isolation techniques [ 14 , 5 , 82 ] to finetune on a class-balanced dataset of new data and rehearsal data \mathcal{M} at the end of each incremental training step ( t>1 ). We refer to this as memory consolidation training (see App. 8.2 ).
Alleviating forgetting with language-aware CLIP knowledge
Finetuning methods are prone to trading generalization for downstream task performances [ 94 , 93 ] . On the contrary, a pre-trained CLIP with hand-crafted prompts has strong generalizability. To alleviate forgetting ( i.e . , losing generalization) in CL, we propose to leverage the rich language information captured by the text encoder with hand-crafted prompts. In the following, we assume \{\mathbf{t}_{y}^{h,l}\in\mathbb{R}^{d}\}_{l=1}^{L} to be the features corresponding to the L hand-crafted textual prompts for the class y\in C^{t} .
Weight initialization
An informed
weight initialization can help bridge the stability gap [ 37 ] in CL by stabilizing the learning for new task components over random initialization [ 20 ] . We thus leverage the t- th task text features \{\mathbf{t}_{y}^{h,l}\}_{l=1}^{L} to initialize the weights \mathbf{w}_{t}^{\mu},\mathbf{w}_{t}^{\sigma}\in\mathbb{R}^{d\times d} of our t- th task mean and std. dev. modules. Let \mathbf{s}_{\mu},\mathbf{s}_{\sigma}\in\mathbb{R}^{|C^{t}|\times d} be the mean and the std. dev. of the L text features. We initialize \mathbf{w}_{t}^{\mu} and \mathbf{w}_{t}^{\sigma} as:

\mathbf{w}_{t}^{\mu}=\frac{1}{d}\langle\mathbf{s}_{\mu},\mathbf{s}_{\mu}^{T}\rangle,\mathbf{w}_{t}^{\sigma}=\frac{1}{d}\langle\mathbf{s}_{\sigma},\mathbf{s}_{\sigma}^{T}\rangle
Past-task distribution regularization
The functional spaces of the past task distributions are prone to forgetting in CL. Though replay helps alleviate the forgetting up to a certain degree, repeated training on the memory samples can lead to overfitting on these [ 72 , 43 ] .
To address this, previous works [ 27 , 58 ] exploit functional priors for regularizing the visual space alongside memory replay. Here, we propose regularizing the past task distributions in the textual space by using the features \{\mathbf{t}_{y}^{h,l}\}_{l=1}^{L} to distill the past task samples \big{\{}\{z^{i}_{m}\}_{m=1}^{M}\big{\}}_{i=1}^{t-1} . Namely, the probability of z^{t}_{m} belonging to class y\in C^{t} is:

P_{KD}(y|z^{t}_{m})=\frac{1}{M}\sum_{m=1}^{M}\frac{1}{L}\sum_{l=1}^{L}\frac{\exp\big{(}\langle\mathbf{t}_{y}^{h,l},z^{t}_{m}\rangle\big{)}}{\sum_{c=1}^{|C^{t}|}\exp\big{(}\langle\mathbf{t}_{c}^{h,l},z^{t}_{m}\rangle\big{)}}

The resulting language-aware distillation loss using P_{\text{KD}} is thus given as:

\mathcal{L}_{KD}=-\sum_{t=1}^{T-1}\sum_{c=1}^{|C^{t}|}\log P_{KD}(c|z^{t})y_{c},\vspace{-0.08in} (10)

where \mathcal{L}_{\text{KD}} is a data-free ( i.e. , no training samples required) text-to-text distribution regularizer that encourages the latent variables to stay close to the text features from hand-crafted prompts.
Lastly, as \mathcal{L}_{\text{KD}} acts on the functional space of past tasks, this sets apart our setting from the non-CL setup of Bulat et al . [ 4 ] where the language-aware distillation loss regularizes the vector space.
Approximate ELBO
Unlike Eq. ( 4 ), the task-specific encoders q_{\phi}^{t} now serve to approximate the intractable task-specific functional posteriors. The resulting ELBO (see App. 10 for derivation) with the static prior p_{\chi}=\mathcal{N}(\mathbf{0},\mathbf{I}) is:

\small\begin{split}\log p(y^{1:T}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})\geq\sum_{t=1}^{T}&\bigg{[}\mathbb{E}_{q_{\phi}^{t}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})}\Big{[}\log p_{\theta}(y^{t}|z^{t},\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})\Big{]}\ &\;\;\;\;-\mathbb{D}_{KL}\big{(}q_{\phi}^{t}(z^{t}|\mathbf{x};\tilde{\mathbf{t}}_{c}^{t})\|p_{\chi}(z^{t})\big{)}\bigg{]}.\end{split} (11)
Integrated objective
Denoting the loss weights by \lambda and \gamma , our total loss consists of the following three terms:

\mathcal{L}=\mathcal{L}_{CE}-\lambda\mathbb{D}_{KL}+\gamma\mathcal{L}_{KD},\vspace{-0.1in} (12)

where the cross-entropy \mathcal{L}_{\text{CE}} and the prior-matching \mathbb{D}_{\text{KL}} terms act on the outputs of all task encoders while the distribution regularization term \mathcal{L}_{\text{KD}} acts only on the past task encoders during finetuning. \lambda is set to 0.001. As the past task encoders are trainable only during the memory consolidation training stage, \lambda for these is set to 0 during training. \gamma is set to 15.
Algorithm overview
Algo. 1 outlines the pseudo-code of a forward pass of CLAP 4 CLIP at the t- th task test step. Here, a test image is to be classified among one of the seen classes \{1,...,|C^{t}|\} . Our method executes the computationally heavy VGA layers only once. The task-specific VGA outputs are passed through their respective task encoders. By limiting the quadratic complexity of the VGA module pass, our method induces minimal time overhead. Moreover, by only expanding the task encoder per task, our memory overhead is negligible compared to the large backbone of the pre-trained CLIP model (see Fig. 5(b) ).
Results
We report performances in Table 1 on all five datasets. Our method consistently achieves the best results among all the methods compared. Notably, on CIFAR100 and ImageNet100, our variants using the hand-crafted and multi-modal prompts outperform the others. On the challenging ImageNet-R setup with significant intra-class diversity, our method can better leverage the instance-conditioned prompt knowledge of AttriCLIP [ 74 ] , which helps it outperform PROOF [ 92 ] by 1.46\% in terms of average accuracy. On CUB200 and VTAB, sharing the prompt pool among all tasks gives CoOp [ 94 ] an edge over other baselines. Leveraging CoOp offers us the best results on these, while surpassing PROOF, which also builds upon CoOp with task-specific soft prompts. Lastly, Fig. 4 shows that our variants perform consistently better than their baseline prompting methods throughout the incremental steps.
Cross-Datasets Continual Learning (CDCL)
To simulate real-world settings with long sequence of tasks and large distribution shifts, the CDCL setting [ 74 ] trains a model sequentially on ImageNet100 and CIFAR100 ( i.e. , on 20 tasks), and evaluates it jointly on these. For a fair comparison with [ 74 ] , we adopt the ViT-L/14 as our CLIP backbone and set the train/test batch size to 32. All other settings remain the same as in Sec. 4.1 . Table 2 reports the last task accuracy of different methods. While all our variants improve the CDCL performances of their respective baselines, combining ours with AttriCLIP [ 74 ] leads to the most gains. This further suggests that our framework can reliably leverage the diverse nature of learned prompts to inherit their setting-specific advantages.
Influence of components.
We ablate the importance of different components of CLAP 4 CLIP in Table 3 . On top of the base CLIP model, we first train a probabilistic encoder. Adding the VGA module and the memory consolidation training stage helps us achieve much stable performances while countering forgetting. We then apply task-specific encoders which makes the centroids of the class-specific latent variable more separable (see Fig. 3 ) thus improving the last task accuracy by 2.21%. Language-aware weight initialization and regularization help improve the last task accuracies by 0.78% and 0.23%, respectively. Weight initialization further helps us tackle the stability gap [ 37 , 20 ] (see App. 8.5.4 for more ablations on language-aware components).
Effect of forgetting on task heads
We ablate the task head predictions over the test set of each task on CIFAR100 (see App. 8.5.1 for more details). Fig. 9(a) shows that the past task heads are relatively less accurate at identifying their respective test instances. However, their amount of forgetting do not necessarily correspond to the task order. This is further evident in Fig. 9(b) where at each step, the past task heads lag unevenly in performance. The uneven performances could be because of the role of task semantics behind forgetting [ 64 , 40 ] .
Probabilistic vs Deterministic inference
To understand our probabilistic inference modules further, we examine their performance against the deterministic variant of ours (Ours w/o VI). Table 1 shows that our probabilistic inference module consistently outperforms its deterministic counterpart in terms of Avg and Last accuracy. This emphasizes the advantages of considering uncertainty in finetuning. We further introspect the effects of the number of layers for the VGA and task encoder modules in our framework in App. 8.5.2 .
Sensitivity to the number of Monte Carlo (MC) samples
We vary the number of MC samples M from 1 to 50. In Fig. 5(a) , the accuracy is poorer in range [1,10] , grows in range [10,20] , and saturates thereafter. Hence, we set M to 20 for all our experiments.
More informed priors
To study the role of a more informed prior in our VI framework, we study three choices of priors to be used in the prior-matching term of eq. ( 11 ): (a) the static (standard normal) prior, (b) the language-aware prior using the distribution obtained from the task encoders using the hand-crafted prompts’ features \{\mathbf{t}_{y}^{h,l}\}_{l=1}^{L} (sec 3.6 ), (c) the data-driven prior using a randomly chosen subset of a training minibatch as the context set to condition the prior on (see App. 11 for more details). Table 4 shows that while (b) and (c) slightly improve over (a) in terms of accuracies and forgetting (BwT), these come at the cost of poorer model calibration and longer runtime per iteration.
Parameter analyses
The additional parameters in CLAP 4 CLIP come from the shared VGA module and the task-specific encoders. For a ViT-B/16 backbone of output dimension d=512 on CIFAR100, the VGA module contains 4,204,032 parameters. The mean and the std. dev. layers for 10 tasks have d\times d parameters each, i.e., 524,2880 parameters. Hence, the CLAP 4 CLIP has 9.5 million extra parameters, which is negligible compared to the pre-trained CLIP with \approx 150 million parameters. We report the parameter counts in fig. 5(b) .
Time analyses
We investigate the inference time per iteration for different methods. As shown in Table 18 , our variants need more inference time than other finetuning methods for the performance gains. The increased time comes mainly from the VGA and from inferring the M latent variables.
Added utilities of probabilistic finetuning
We study the out-of-the-box utilities of CLAP 4 CLIP’s uncertainty quantification (UQ) capabilities. Our motivation for these is not to achieve state-of-the-art performance but to highlight the perks of probabilistic modeling in scenarios where the deterministic CL finetuning methods struggle.
Post-hoc novel data detection (PhNDD)
PhNDD uses a pre-trained classification model to identify novel data based on the output confidence [ 83 , 21 ] . For CL, this can help discern the arrival of new tasks, expand the network, etc. To evaluate the PhNDD capabilities of models within a CL setup, we design a simple setting. Namely, at all but the last test step, we treat the test data from the past and the current tasks as seen while those from all future tasks as novel . We then use FPR95, AUROC [ 11 ] , and AUPR [ 66 ] scores as our performance metrics (see App. 8.6 for more details). We average these metrics over all but last incremental test steps.

To quantify the output confidence, we rely on the Energy score [ 45 ] given its aptness for pre-trained models. Table 5 compares the averaged PhNDD performances. Our probabilistic models enhance the PhNDD capabilities of their underlying prompting frameworks. Lastly, the inferior results of the deterministic ( i.e. , w/o VI) versions of our models imply that considering uncertainty-awareness for PhNDD helps us maintain richer and more diverse features [ 88 ] .
Exemplar selection
To study uncertainty-based exemplar selection, we first employ entropy of the output softmax distributions as our selection criteria [ 6 ] . Table 6 shows the efficacy of entropy-based rehearsal for our method, where other deterministic methods lag behind due to their inconsistent UQ capabilities. Next, we employ the energy [ 45 ] and the variance of softmax distributions of the M predictions as our selection criterion, and contrast these against other criteria proposed in [ 6 ] . Figure 6 shows that variance-based exemplar selection outperforms random, and is only second to iCaRL [ 65 ] in terms of Last accuracy. Lastly, we note that deterministic finetuning methods are inherently limited at leveraging variance for exemplar selection.