## 3 Proposed Method
### Task-Specific Prompt Learning
#### 3.3.3 Optimization of the Prompt  
Recent advancements in vision-language models demonstrate that language typically provides information complementary to visual data [4; 20]. In this context, we employ a pretrained text encoder to extract features pertinent to image categories, which guide the optimization of prompts. Due to the extensive textual data involved in the pretraining phase, the extracted category text features exhibit a degree of discretization. As a result, prompts influenced by these text features are somewhat discrete in nature. This discretization helps  
Figure 3: Illustration of prompt-based learning. Whenever a new task is learned, a prompt corresponding to the task is added to the prompt pool. The prompt of the current task is combined with the MSA module in the ViT architecture using Prefix Tuning.  
minimize knowledge interference across different tasks, promotes task-specific prompt learning, and aids in mitigating the phenomenon of catastrophic forgetting.  
Upon receiving the key associated with the input image \(x\), the model selects a task-specific prompt \(P_{Task}\). This prompt is then concatenated with several MSA modules within the image encoder of CLIP, yielding the image's feature representation \(F_{I}\) guided by the \(P_{Task}\). \(F_{I}\) encapsulates rich task-specific information. Simultaneously, based on the label name of \(x\), a textual prompt \(P_{text}\) is constructed, such as \"A photo of a dog\". \(P_{text}\) is fed into the text encoder of CLIP to obtain the textual feature representation \(F_{T}\) of the image. Given the discrete nature of textual data, we aim to compute a constrastive-based loss [47] between the prompt-guided \(F_{I}\) and \(F_{T}\), denoted as \(L_{P}\), as elaborated in Equation (3).  
\[L_{P}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(F_{I}\cdot F_{Ti}/\tau)}{\sum_{j =1}^{M}\exp(F_{I}\cdot F_{Tj}/\tau)} \tag{3}\]  
where \(N\) is the batch size, \(M\) is the number of categories so far and \(\tau\) is a temperature parameter controlling the scale of similarity scores.  
In addition, for prompt updates, a prompt ensemble (PE) strategy is utilized to efficiently transfer knowledge from previously learned prompts (which encode task-specific knowledge) to the new task's prompt. The new task's prompt is optimized as follows:  
\[P_{t}=\alpha\sum_{i=1}^{t-1}P_{i}+(1-\alpha)P_{t}, \tag{4}\]  
where \(P_{i}\) represents the learned prompts of the previous tasks, \(P_{t}\) is the prompt for the new task, and \(\alpha\) is a weighting factor that balances the contributions of the previous prompts and the new task's prompt.  
To ensure that the new task's prompt does not overlap with the previously learned prompts, a contrastive learning loss is employed during the optimization of the new task's prompt. The contrastive loss is defined as  
\[\text{sim}_{ij}=\frac{M_{i}\cdot M_{j}}{\|M_{i}\|\|M_{j}\|}, \tag{5}\]  
where \(M=[F_{I};\mu_{t}]\) is the concatenated matrix of the prototype of previous classes and the feature of current sample. The similarity values are normalized by a temperature factor (e.g., 0.8). Finally, the contrastive loss is computed as  
\[\mathcal{L}_{\text{C}}=-\frac{1}{|M|}\sum_{i=1}^{|M|}\log\frac{\exp(\text{sim }_{ii}/\tau)}{\sum_{j=1}^{|M|}\exp(\text{sim}_{ij}/\tau)}, \tag{6}\]  
where \(\tau\) is the temperature factor. This loss encourages the new task's prompt to remain distinct from the previously learned prompts, thereby preventing overlap and improving task-specific prompt optimization.  
In this way, the overall optimization objective is defined as Equation (7).  
\[L=L_{\text{C}}+\lambda L_{P} \tag{7}\]  
where \(\lambda\) denotes balance factors.