## 1 Introduction  
In recent years, deep learning models have excelled in processing large datasets in a one-shot manner [1, 2]. However, extending the knowledge of pretrained models to new datasets often necessitates retraining on these entire datasets [3], incurring significant economic and temporal costs. This challenge is particularly acute for open-source multimodal models like Contrastive Language-Image Pretraining (CLIP) [4], which are pretrained on extensive and complex image-text pairing datasets and demonstrate robust zero-shot capabilities across various tasks [5]. The prohibitive cost of retraining the entire CLIP model renders it impractical for real-world applications. As illustrated in Figure 1, various continual learning methods have been developed to train models on sequential tasks without degrading performance on previous tasks [3; 6; 7]. Replay-based methods, in particular, have proven effective [8; 9; 10; 11; 12; 13]. These methods involve sampling instances from a stored data pool of previous tasks and combining them with samples from new tasks, thus preserving earlier task performance. Nevertheless, identifying suitable replay strategies for models like CLIP is challenging due to the vast and diverse nature of their original datasets. Concurrently, research has explored continual learning strategies based on CLIP, including knowledge distillation methods [14; 15]. However, these approaches might necessitate adjustments to the pretrained parameters of CLIP for downstream tasks, potentially compromising its zero-shot capabilities. Therefore, it is essential to explore innovative methodologies for continual learning in multimodal models like CLIP that maintain zero-shot performance, reduce training costs, and enhance training efficiency. Prompt-based continual learning methods are emerging as promising solutions. These methods keep the parameters of pretrained feature extractors intact while introducing a small set of learnable key-prompt parameters tailored to new task sequences [2; 16; 17; 18; 19]. However, existing prompt-based methods primarily utilize the knowledge from a single modality within pretrained models [20; 21]. For multimodal models like CLIP, effectively harnessing pretrained knowledge from both image and text modalities is crucial. Inspired by this, we propose IT-prompt, a novel prompt-based method that guides key-prompt learning from both image and text perspectives, thus enabling comprehensive knowledge expansion within the pretrained CLIP model. Building on IT-Prompt, we introduce a C-NCM classifier designed to reduce dependence on Softmax classifiers, which require the continuous storage and retraining of old task samples. The C-NCM classifier significantly cuts training resource consumption and enhances training efficiency by emulating human-like adaptive learning mechanisms. In summary, the contributions of this paper can be summarized as follows:  
* We introduce IT-Prompt, a novel prompt-based continual learning method for pre-trained CLIP models, which leverages prompts for both image and text modalities to harness the correlation between visual and textual modalities, effectively extending the pretrained CLIP models' knowledge. IT-Prompt preserves the zero-shot capabilities of the pretrained CLIP models and mitigates the catastrophic forgetting problem inherent in previous CLIP-based continual learning methods. Figure 1: For class incremental learning, the process is divided into two phases: a training phase and a testing phase. The model is trained solely on the current task’s category-specific data during training and evaluated comprehensively across all previous and current task categories during testing. * Building on IT-Prompt, we further introduce a C-NCM classifier, which eliminates the need for Softmax classifiers to store and retrain old task samples, significantly improving training efficiency and reducing resource consumption.