RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Conclusion
In this paper, we propose CLAP 4 CLIP, a probabilistic finetuning method for learning task-specific distributions over visual-guided textual features. Our model shares the visual-guided text alignment module across all tasks while adding lightweight task-specific encoders to learn fine grained task distributions. Besides leading to little memory overhead, this architecture is compatible with several prompt-tuning based methods thus helping us inherit their respective perks on different CL settings. Our experiments show the superior results of CLAP 4 CLIP across several datasets and settings. We conclude with two out-of-the-box utilities of our method wherein existing continual learning methods lag: post-hoc novel data detection and uncertainty-based exemplar selection.
Future research directions
Few potential directions of research for CLAP 4 CLIP include the design of: (a) parameter-efficient adapters [ 25 ] for very large CL settings; (b) better regularization techniques to alleviate forgetting; and (c) more informed [ 9 ] yet computationally efficient priors for inference. Similarly, along the direction of alleviating forgetting and mitigating the stability gap [ 37 , 20 ] , it would be interesting to see how class-specific prompts generated by pre-trained Large Language Models (LLMs) can be exploited to obtain task-relevant language-aware CLIP knowledge while preserving the zero-shot transfer ability of the learned prompts (see App. table 19 for a preliminary investigation). Lastly, we consider applying CLAP 4 CLIP to more sophisticated Vision-Language tasks [ 68 ] as another possible direction for research (see App. table 20 for a preliminary investigation). We elaborate further on each of these directions in App. 9 .
Datasets
We evaluate our method on five datasets, the details of which are reported in Table 7 . Following Zhou et al . [ 92 ] , we shuffle the order of training classes for all but the VTAB dataset with the random seed 1993.
While the original VTAB [ 86 ] includes 19 evaluation tasks from three categories ( natural , specialized , and structured ) and their respective sub-domains, we rely on the five datasets cross-domain class-incremental subset proposed in SimpleCIL [ 91 ] . The five datasets (used in the same streaming order) include Resisc45 [ 8 ] , DTD [ 10 ] , Pets [ 59 ] , EuroSAT [ 22 ] , and Flowers [ 54 ] . To make the classes emerge from domain to domain, we do not shuffle the class order for VTAB.
Exemplar selection
Following [ 65 , 24 ] , we employ the herding algorithm [ 78 ] to choose the exemplars for our main experiments. Following the previous works [ 90 , 92 ] , we rely on two typical methods to populate the memory:

Fixed memory budget maintains a static memory \mathcal{M} with K instances. Upon having seen |\mathcal{Y}_{b}| number of classes after an incremental training stage, the model selects \frac{K}{|\mathcal{Y}_{b}|} exemplars per class.

Expandable exemplar set dynamically expands the memory \mathcal{M} with the arrival of more incremental tasks. After each incremental training stage, the model here stores |\mathcal{Y}_{b}|\times k_{c} exemplars, where k_{c} is the number of exemplars per class.

For CIFAR100, ImageNet100 and VTAB, given their lesser number of classes, we employ the first policy, and keep a total of 2,000, 1,000, and 1,000 exemplars, respectively. This amounts to the respective sub-totals of 20 and 10 exemplars per class after the last incremental stage. We choose these sizes for a straightforward comparison with the existing works, i.e., PROOF [ 92 ] for CIFAR100 and AttriCLIP [ 74 ] for ImageNet100. For VTAB, the chosen memory size reflects the fact that we have only 1,796 training instances in total (see Table 7 ). For ImageNet-R and CUB200 with 200 classes each, we adopt the second policy and store 20 exemplars per class.
Hyperparameter selection and tuning
To the end goal of obtaining task-agnostic hyperparameters [ 15 ] , we tuned our hyperparameters using a validation set comprising 10\% of the CIFAR-100 training dataset. Similar to [ 14 ] , performing the hyperparameter search only on the CIFAR100 setup helps us avoid optimizing for the number of tasks while generalizing across all our other setups. Table 8 shows the candidate values for the hyperparameter grid search and their best chosen values. Tables 9 , 10 , 11 , and 12 report the last task accuracy scores (Last) corresponding to the hyperparameter search for the number of training epochs, the number of finetuning epochs, the coefficient \gamma and the coefficient \lambda , respectively. Fig. 5(a) in the main paper reports the accuracy and the runtimes for the different number of MC samples M . We will release the full source code upon the acceptance of our paper. Training for memory consolidation. To alleviate the forgetting of past tasks, we finetune on the class-balanced dataset of new data and rehearsal data \mathcal{M} at the end of each incremental training step ( t>1 ) [ 24 , 14 ] . Following other well-established parameter-isolation CL algorithms [ 14 , 5 , 82 ] , we freeze the past task encoders during the normal training. This helps us avoid knowledge interference from the dominant new task training samples. During the memory consolidation training stage, we optimize all the task encoders while freezing the task-shared VGA module parameters.
Latency comparison for VT-CLIP styled VGA vs Ours
We compare the performance of VT-CLIP styled VGA with Ours. To align the text features with the image features, the former uses per-pixel spatial features obtained from the ViT prior to global pooling while we use the globally pooled features. Table 13 shows that VT-CLIP styled VGA achieves similar accuracy as ours while incurring \approx 6\times higher inference time.
Performance evolution
To complement the results in Table 1 , Fig. 8 compares the accuracy of different methods at each evaluation step across all datasets. Our major conclusions are briefed as follows. A) The base task performance of CLAP 4 CLIP (ours) is consistently higher than other methods including the state-of-the-art PROOF [ 92 ] . This suggests that our probabilistic finetuning framework is effective for general downstream tasks in a non-incremental setting. B) For the CL settings in Table 1 where either of the CLAP 4 CLIP variants achieve the best performances, their performance curves also consistently retain superior results across all evaluation steps. This validates the effectiveness of our method at tackling forgetting. C) Similar to Zhou et al . [ 92 ] , we notice that CLAP 4 CLIP achieves a significant performance improvement over vision-only methods (L2P and DualPrompt). This indicates the merits of considering text and visual cues together for continual learning.
Task head selection interpretation and results.
To elaborate on Sec. 4.3 , Fig. 9(a) reports the accuracy of test set samples corresponding to the task encoder heads at the end of incremental training on the last task. Here, the first row is to be interpreted as follows: 77\% of test samples belonging to test set of the first task (test set ID 1) were correctly allocated to task head 1, 1.4\% of test samples belonging to test set of the first task (test set ID 1) were incorrectly allocated to task head 2, and so on. Visualizing the task head selection results for the last task evaluation helps us uncover the amount of forgetting among the individual task heads at the end of the incremental training.

Fig. 9(b) compares the evolution of the task head selection accuracy across the incremental test steps. Here, at the first test step, we have only one task head and thus the task head selection accuracy is 100\% . At the second test step, we have the test samples from two seen tasks as well as two available task heads. Out of all test samples of task 1, the reported 94.5\% were correctly classified into the task head 1 while the rest 5.5\% were incorrectly classified into the task head 2. Similarly, for test samples belonging to task 2, 3.1\% were incorrectly classified into the task head 1 while the reported 96.9\% were correctly classified into the task head 2, and so on. Hence, by studying the task head selection per incremental step, we can investigate the trend of forgetting among the individual task heads.
Effect of inference module types.
To further investigate the effects of inference modules on performances, we vary the number of layers for the VGA module (sec. 3.4 ) and for the task-specific encoders (sec. 3.5 ). Fig. 10(a) reports the results of varying the number of Transformer Decoder layers [ 71 ] in the VGA module. As the number of layers grow, the average accuracy (Avg) increases while the last task accuracy (Last) decreases. This indicates that while a larger number of layers in the VGA module lead to an increase in the initial tasks’ performances, these are amenable to larger forgetting on latter incremental steps.

In Fig. 10(b) , we report the performances for varying number of MLP layers in the mean and the standard deviation heads of the task distribution encoders. Unlike the VGA module, here we observe a consistent trend of decreasing last and average task accuracy with the increase in the number of layers. This clearly indicates the superiority of using a single-layered task distribution encoder.
Inference time for different finetuning methods.
Table 18 investigates the inference time per iteration for different methods. Among the compared prompt-based methods, the inference time for AttriCLIP [ 74 ] is notably the highest. This is because it relies on selecting test instance-conditioned prompt tokens from a pool of prompt tokens. The instance-specific prompts are fed to the text encoder which further outputs an equivalent number of instance-specific text features to be used in the derivation of logits through eq. 1 . These operations increase the inference time of AttriCLIP beyond our proposed variants of CLAP 4 CLIP with hand-crafted prompts (Ours), class-conditioned prompts (CoOp + Ours), and multi-modal prompts (MaPLe + Ours) where the latter three outperform AttriCLIP significantly across all our settings.
Influence of language-aware knowledge components on training dynamics.
Continuing our ablations from sec. 4.3 , here we visualize the effects of using language-aware pre-trained knowledge, i.e. , weight initialization and task distribution regularization on the training dynamics of our model. For thorough analyses, we consider four variants of our model: (a) Ours uses both weight initialization and task distribution regularization, (b) Ours without weight initialization, (c) Ours without task distribution regularization, and (d) Ours without either of the language-aware components.

Does language-aware weight initialization help alleviate stability gap [ 37 ] ? To answer this, we first investigate the evolution of the training loss during the initial training stages of each incremental task. Figure 11 shows the loss \mathcal{L} (eq. ( 12 )) during the initial 100 training iterations of each task. We observe that our proposed weight initialization technique leads to lower training losses for the scenarios with or without task distribution regularization, i.e. , in general, red values < green values and blue values < orange values . Following Harun et al . [ 20 ] , our observations support the hypothesis that larger loss values lead to the stability gap [ 37 ] for CL, and that an informed weight initialization method can help tackle it by reducing the initial training loss.

To further verify the benefit of our proposed weight initialization strategy for reducing the stability gap, we ablate the accuracy evolution of the first task test samples during the early training stages of each task. Figures 12 and 13 contrast these for CIFAR100. In general, our proposed weight initialization strategy helps mitigate the drop in accuracy during the initial training phases. On average, the first task accuracy upon the first iteration of training across all tasks remains 78.12 without weight initialization and grows to 79.5 with weight initialization, i.e . a gain of 1.38 percentage points.

How does language-aware knowledge help learning of task distributions in general? To understand the effect of language-aware knowledge on task distribution learning, we next investigate the evolution of the means and standard deviations learned by the past and the new task heads throughout the training iterations. To this end, Fig. 14 and Fig. 15 report the training iterations against the L2 norm of means and standard deviations for the past task heads (at each incremental training step) and the new task heads (at each training step). We observe two consistent trends regarding the evolution of distributions of the past and the new task heads. First, the proposed initialization of weights helps stabilize the learning of the means and standard deviations with ( red against green ) or without ( blue against orange )
regularizing the task distributions. Second, regularizing the task distributions increases the L2 norms of the learned mean and the standard deviation as these now have to encode more information to mimic the distributions of the hand-crafted text features.
Post-hoc novel data detection
Our post-hoc novel data detection (PhNDD) setting aims to evaluate the continual learning methods at identifying novel data on the fly. To do so, we design an evaluation setup that uses no additional data resource other than that provided by the dataset-specific CL setting. Starting from the first test step, we treat the test data of the future tasks as novel while those of the seen tasks (including the most recently trained one) as seen . Since the last test step of a CL dataset has no future tasks, we exclude this step for our PhNDD evaluation, i.e., we carry our PhNDD evaluation of CL models starting from the first until the penultimate test step.

Following other standard practices [ 45 , 21 ] , we use the Energy scores [ 45 ] of the outputs for each test sample as a measure of the model’s confidence score. The samples assigned with a confidence score below the pre-defined confidence threshold are classified as novel. By assuming the seen data as the positive class and the novel data as the negative class, we can obtain a series of true positives rate (TPR) and false positive rate (FPR) by varying the confidence thresholds. One of our PhNDD evaluation metrics – the FPR95 then measures the FPR when the TPR is 0.95. As such, a lower FPR95 score indicates better PhNDD performance. Our other two PhNDD performance metrics include the the area under receiver operating characteristic curve (AUROC [ 11 ] ) calculated based on FPR and TPR, and the precision-recall curve (AUPR [ 66 ] ). Higher values of AUROC and AUPR indicate better PhNDD performance.

In Table 5 in the main paper, we report the PhNDD metrics averaged over all the evaluated steps. Here, in Fig. 16 , we show the evolution of these metrics with each evaluation starting from the first test step until the penultimate test step of CIFAR100.
We observe that the zero-shot Continual-CLIP [ 70 ] has the poorest PhNDD performances (highest FPR95, and least AUROC and AUPR scores) across all steps given that it has not been finetuned on the downstream CL tasks. Among the finetuned methods, the CoOp [ 94 ] exhibits the poorest performances across all tasks. Among the variants of our method, combining CoOp with ours (CoOp + Ours) achieves the best PhNDD performances across all tasks. Furthermore, the deterministic versions: Ours w/o VI and CoOp + Ours (w/o VI) remain sub-optimal to their respective probabilistic variants, i.e., Ours and CoOp + Ours. The latter results validate the added perks of our probabilistic modeling framework for post-hoc novel data detection.
Data-driven prior
The choice of prior is pivotal to a Bayesian inference workflow like ours [ 16 ] . While a standard Gaussian prior p_{\chi}=\mathcal{N}(0,I) adapts well to a range of settings, it is (seemingly) uninformative regarding the nature of a given task [ 19 ] .
With the end goal of deriving more informative priors, we thus seek to replace p_{\chi} with task data-dependent prior distributions p^{t} , wherever applicable.

To this end, we first note that the outputs of the VGA module remain invariant not only to the order of the input text features (due to self-attention) but also to the order of the contextual image features (due to cross-attention). The latter invariance implies that the joint task-specific distribution learned by the encoder q^{t}_{\phi} (conditioned on the VGA outputs \hat{\mathbf{t}}_{c}^{t} from eq. 5 ) is preserved if we were to permute the elements of the task-specific visual context set. More formally, this observation helps guarantee the (finite) exchangeability and the consistency properties of a stochastic process [ 55 ] .

Motivated by the above, we treat the t- th task image features \mathbf{x}^{t} as the target set \mathcal{T}^{t} and employ a randomly chosen subset of it as our context set \mathcal{C}^{t} to align the t- th task text features and to condition our prior p^{t} on:

\begin{split}\hat{\mathbf{t}}_{c}^{t}&=VGA\big{(}Q=\mathbf{t}_{c}^{t},K=V=\mathcal{C}^{t}\big{)},\ p^{t}&=q_{\phi}^{t}(\tilde{\mathbf{t}}^{t}_{c})=\big{(}\mu^{t}(\tilde{\mathbf{t}}_{c}^{t}),\sigma^{t}(\tilde{\mathbf{t}}_{c}^{t})\big{)}\end{split} (14)

where \tilde{\mathbf{t}}^{t} is the fused task-specific text feature following eq. ( 6 ). The task-specific prior p^{t} thus endows our training framework with a resemblance to the neural process (NP) architectures [ 18 , 31 , 27 ] .
Following NPs, we use the same encoder q_{\phi}^{t} to parameterize the conditional prior and the variational posterior. This results in the following approximate ELBO (see App. 10 for the ELBO derivation):

\begin{split}\log\;&p(\mathbf{y}^{1:T}|\mathbf{x},\mathcal{C}^{1:T})\geq\ &\sum_{t=1}^{T}\bigg{[}\mathbb{E}_{q_{\phi}(z^{t}|\mathbf{x})}\Big{[}\log p(\mathbf{y}^{t}|z^{t},\mathbf{x}_{\mathcal{T}}^{t},\mathcal{C}^{t})\Big{]}\ &-\mathbb{D}_{KL}\big{(}q_{\phi}(z^{t}|\mathcal{T}^{t})\|q_{\phi}(z^{t}|\mathcal{C}^{t})\big{)}\bigg{]}\end{split} (15)

where in practice, the entire set of t- th images in a training minibatch form the target set \mathcal{T}^{t} and a randomly chosen subset of the targets make up the context \mathcal{C}^{t} [ 38 ] . Note that unlike NPs, our framework does not entirely rely on data-driven priors. Namely, while training on a CL task t , the past-task encoders are frozen and we have ample t- th task data points to condition the prior on. We thus resort to optimizing the ELBO ( 15 ) during training. On the other hand, during finetuning, we have limited task-specific data points to condition our context on. As such, we empirically found that switching to the static prior yields better results and thus resort to optimizing the ELBO ( 11 ) during finetuning.
Effect of the context size on data-driven prior.
Table 4 in the main paper compares the results of using a data-driven prior against the uniform normal prior and the language-aware prior (see Sec. 3.6 ), where the latter is driven from the pre-trained text encoder using hand-crafted prompts. We observe that data-driven prior leads to minor accuracy improvements over the standard normal prior but falls narrowly behind the language-aware prior. Here, we study the influence of the batch size of the context set selected at random to derive our prior from.

Table 21 shows the last task accuracy with varying context sizes and a fixed target batch size of 64. We find that a smaller context set size hurts the performance of the model to the extent of falling behind the standard normal prior. Given that the context sets are the sampled subsets of the training (target) minibatches, a much smaller context set can lead to the increase in the prior matching loss values. We find that the context set batch size of 40 performs the best, and thus use this to ablate the prior-dependent performances in the main paper.
Figure 1 : CLAP \mathbf{4} CLIP overview: the visual-guided attention (VGA) inference module uses the text features as query (Q), and the visual features as keys (K) and values (V) to produce visual-guided text features. The task-specific text features are fed to their respective task distribution encoders (\mu^{t},\sigma^{t}) . The task distribution samples are then fused with the original task features prior to deriving the task logits y^{t} . All task logits are concatenated to produce the final prediction y^{1:t} .; CLAP \mathbf{4} CLIP overview: the visual-guided attention (VGA) inference module uses the text features as query (Q), and the visual features as keys (K) and values (V) to produce visual-guided text features. The task-specific text features are fed to their respective task distribution encoders (\mu^{t},\sigma^{t}) . The task distribution samples are then fused with the original task features prior to deriving the task logits y^{t} . All task logits are concatenated to produce the final prediction y^{1:t} .
(a) Effect of \alpha on Accuracy and Backward Transfer (BWT).; (b) Avg. rotation angle [ 53 ] per step between image and text features.; Need for Visual-guided Attention (VGA) inference module. Fig. 2(a) : A simple adapter is inadequate at preventing catastrophic forgetting in CL – marked by high BWT scores; Fig. 2(b) : VGA module encourages cross-modal alignment among text and image features – marked by a decrease in average angle \texttt{arccos}\langle t,1\rangle between them – where otherwise the former deviates further with incremental training steps.
Figure 3 : Need for task-specific distribution encoders: Cosine distance between the centroids of class-specific latent variables produced without ( left ) and with ( right ) the use of task-specific encoders on CIFAR100 (10 tasks, 10 classes per task). Centroids of class-specific latent variables are more separable on the right.; Need for task-specific distribution encoders: Cosine distance between the centroids of class-specific latent variables produced without ( left ) and with ( right ) the use of task-specific encoders on CIFAR100 (10 tasks, 10 classes per task). Centroids of class-specific latent variables are more separable on the right.
(a) CIFAR100; (b) ImageNet100; Performance evolution of different methods. The top-1 accuracy (%) is reported upon learning of each task (see App. fig. 8 for results on all datasets).
(a) Accuracy-runtime tradeoff; (b) Parameter count comparison; Ablations on CIFAR100 showing: (a) performance trade-off with the number of MC samples M , (b) the number of trainable parameters in different finetuning methods.
Figure 6 : Analyses of various strategies for exemplar selection w/ our method on CIFAR100.; Analyses of various strategies for exemplar selection w/ our method on CIFAR100.
Figure 7 : Illustration of the deterministic variant of Ours (Ours w/o VI in Table 1 ): the task-specific text features are fed to their respective task encoders, consisting of only the mean \mu layer each. There is no sampling involved and the task mean outputs are fused directly with the original task features prior to deriving the task logits \mathbf{y}^{t} . All task logits are concatenated to produce the final prediction \mathbf{y}^{1:t} .; Illustration of the deterministic variant of Ours (Ours w/o VI in Table 1 ): the task-specific text features are fed to their respective task encoders, consisting of only the mean \mu layer each. There is no sampling involved and the task mean outputs are fused directly with the original task features prior to deriving the task logits \mathbf{y}^{t} . All task logits are concatenated to produce the final prediction \mathbf{y}^{1:t} .
(a) CIFAR100; (b) ImageNet100; (c) ImageNet-R; (d) CUB200; (e) VTAB; Performance evolution of different methods. The top-1 accuracy (%) is reported upon learning of each task.
(a) Last step task head selection accuracies; (b) Per step task head selection accuracies; Task head selection accuracies reported on CIFAR-100 upon: (a) evaluation on the last step, (b) evaluation on each incremental step.