Introduction
Learning in the real world involves dealing with the ever-changing distributions of task streams and their data [ 6 , 85 , 49 , 26 ] .
Given the constraints on resources and privacy, there is also no guarantee on re-training a network on all previously seen data [ 7 ] . Continual learning (CL) aims to learn from such data/task stream without catastrophic forgetting [ 33 , 6 ] of past data/tasks. A challenging CL setup is class-incremental learning, where new classes emerge with new tasks, and at test time, a model must infer from all seen classes without known task IDs [ 48 ] .

Recent years have seen a surge of pre-trained multi-modal foundation models achieving state-of-the-art performances on several domains [ 63 , 1 , 84 ] .
One such example for the vision-language (VL) domain is the
Contrastive Language-Image Pre-training (CLIP) [ 63 ] model. CLIP comes with strong zero-shot generalization ability acquired by learning to match large-scale image-text pairs with contrastive loss [ 56 ] . However, to adapt well to downstream tasks, CLIP must be finetuned on the task-specific data [ 94 , 17 ] . Considering both the need for continually finetuning pre-trained models to streaming tasks and their perks over training from scratch [ 70 ] , our work investigates CL with CLIP.

An issue with the existing deterministic approaches [ 17 , 94 ] is that these overlook the uncertainties arising from many possible interactions between visual and textual cues. For instance, on the textual side, while a good default hand-crafted prompt for a number of tasks is “ A photo of a {class} ”, there can be tasks where a more specific prompt can help improve the image-to-text coherence [ 63 ] .
On the visual side, images from the same class have diverse range of backgrounds, poses, orientations, etc.
Overlooking the uncertainties in image-text matching may cause overfitting to the downstream task data and forgetting of the generalizable knowledge [ 36 ] . This has particularly worse implications for CL where we seek to adapt CLIP on a stream of tasks. While some methods model the uncertainties through probabilistic finetuning [ 47 , 12 ] , these remain subpar at CL given: (a) their inaptness to existing prompt-based approaches [ 47 ] , (b) their trading of in-domain performance for generalization [ 12 ] .

Uncertainty-awareness can further be crucial for CL models deployed in mission-critical settings (healthcare, transport, etc.) as it can help calibrate predictions to reliably assess the models’ predictive confidences [ 39 , 27 ] . Hence, to enhance the usage of pre-trained CLIP models for real-world CL tasks, we design a probabilistic finetuning approach with three properties : A) uncertainty-aware modeling of cross-modal task cues for better generalization; B) compatibility with existing prompt-based finetuning methods [ 70 , 94 , 74 , 29 ] to exploit their task-specific knowledge; C) leveraging the rich pre-trained knowledge of CLIP to further counter forgetting without extra training data.

To this end, we design a Variational Inference (VI) framework to model the distribution of visual-guided text features (rather than modeling the distribution of either modality alone), thus validating property \# A . This helps our method implicitly address the biases in the image-text alignment [ 62 , 95 ] (see Fig. 2(b) ). Namely, we design a task-shared visual-guided attention module to encourage the alignment, and then perform specialized forward passes through lightweight task-specific inference modules to learn stochastic factors for CL tasks. To further counter the forgetting in these modules, we propose using the pre-trained language information of CLIP for weight initialization and task distribution regularization (property \# C ).
Lastly, the plug-and-play nature of our probabilistic fine-tuning method makes it
compatible with other model adaption/fine-tuning methods (such as [ 74 ] ) and diverse types of prompts that are hand-crafted [ 63 ] , uni-modal [ 94 ] , multi-modal [ 29 ] , or input-specific [ 74 ] (property \# B ).
We backronymize our finetuning approach as CLAP – C ontinual L e A rning with P robabilistic finetuning – for the pre-trained CLIP model.

Our experiments across several settings show that CLAP 4 CLIP enhances prompt-based finetuning for CLIP and surpasses the predominant deterministic finetuning methods for CL all while sharing a similar resource overhead. To further explore the out-of-the-box perks of CLAP’s probabilistic nature, we study its superior uncertainty quantification capabilities on a proposed post-hoc novel data detection setup and on exemplar selection for CL.