Calibrating Prompt from History for Continual Vision-Language Retrieval and Grounding

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 19:37:37

Abstract: In the field of machine learning, continual learning is a crucial concept that allows models to adapt to non-stationary data distributions. However, most of the existing works focus on uni-modal settings and ignore the multi-modal data. In this paper, to enable neural networks better understand diverse modalities in real-world scenario, we investigate continual learning for two typical vision-language applications, i.e. retrieval and grounding. Instead of conventional exemplar-based methods, we leverage the pre-trained transformer model (e.g. CLIP/GLIP) and the prompt technique to tackle this problem. Under this scheme, we identify two critical limitations in existing methods: (1) Unfamiliarity across tasks, which prevents task-specific prompts from achieving forward propagation; and (2) Heterogeneity between modalities, which makes it difficult to guarantee a consistent optimization direction for prompts of different modalities. To overcome these constraints, we design Historical Prompt Calibration that includes two objectives to calibrate prompts. First, the intra-modal relevance estimation helps encode sufficient task-specific information for prompts, with the help a relevance estimator developed for recognizing task relevance. Second, the inter-modal consistency alignment enhances the agreement of the two modality-specific prompts in the current task by contrasting them with the prompts from previous tasks. We evaluate the superiority of our strategy over state-of-the arts methods by four vision-language applications, including two retrieval tasks (i.e. image- and video-text retrieval) and two grounding tasks (i.e. referring expression comprehension and segmentation).

Main paper: 
 related works: 
 Continual Learning. Existing approaches to continual learning can be categorized into three main groups: (1) Replay-based methods [6, 7, 14, 20, 33, [41] [42] [43] 46 ] store a subset of data from previous tasks for future rehearsal via experience replay, representation consolidation or constrained optimization. The data can be either stored directly or synthesized by generative models.(2) Regularization-based methods [1, 2, 22, 55] enforce constraints on parameter changes to mitigate interference with prior tasks.(3) Architecture-based methods [17, 23, 53] learn separate sets of parameters dedicated to individual tasks. Despite their promises, replay-based methods suffer from diminishing performance with smaller buffers and raise data privacy concerns [5] , the regularized methods struggle to achieve satisfactory performance in challenging settings [41, 51] and the architecture-based methods [19] require substantial amount of additional parameters. Recently, the promptbased methods are investigated [12, 50] , which leverage learnable prompt parameters upon pre-trained models to encode knowledge more succinctly and hence avoid memory consumption. Following this paradigm, we establish our base framework upon S-Prompt [48] and further explore continual learning for vision-language retrieval and grounding.Visual-Language Transformer. With the remarkable progress in language tasks [4, 39, 47] , the transformer architecture is also being rapidly transferred to the field of computer vision [8, 10] . Recently, pretraining visual-language transformer [16, 25, 35, 37, 38, 45, 58] has yielded substantial improvements across various downstream tasks, e.g. visual question answering, image captioning [58] , and referring image comprehension [25] . The large-scale models demonstrate extremely powerful capabilities of multi-modal pretraining. Among them, the single-stream architecture [16, 24, 37, 44] employs a single transformer to jointly model a pair of text and image, while the two-stream architecture [25, 38, 45] applies two transformers to separately learn the representations of the text and the image, respectively. In this work, we primarily leverage the two-stream architecture transformers as our modal encoders.Prompt Technique. The prompt technique is initially applied in the NLP (Natural Language Processing) domain and subsequently adapted to vision and vision-language models. The fundamental concept behind prompt technique is to learn a function that modifies input texts or images, enabling language or image models to acquire additional task-related information [30] . A variety of prompting methods have emerged, including notable contributions such as [3, 12, 15, 18, 27, 50, 56, 57] . For example, [27] employs prefix-tuning to prompt the pre-trained language model. VPT [18] introduces a small number of learnable parameters into the input space of Vision Transformers (ViT) [11] and Swin Transformers (Swin) [32] , outperforming full fine-tuning in many cases while also reducing storage costs. CoOp [56] incorporates prompts into the input of the vision encoder for the vision-language model. Recent work [48] [49] [50] exploit the prompt technique for continual learning in uni-modal applications. S-Prompts [48] introduces learnable prompts into the input space of both the visual and text encoder within a visionlanguage model. However, these methods fail to realize the taskwise and modality-wise connection for prompts, thereby falling 
historical prompt calibration 3.1 problem formulation: 
 Our research focus on the problem of continual learning for visionlanguage retrieval and grounding. In this problem, the model is required to learn knowledge from a series of tasks (i.e. input data), ultimately evolving into a universal expert capable of handling all tasks. Formally, the model learns tasks 𝑇 = {𝑇 1 ,𝑇 2 , ...,𝑇 𝑘 } in a sequential manner, where 𝐾 denotes the total number of tasks. Each task 𝑇 𝑘 is associated with data D 𝑘 from different or even highly heterogeneous contents. At task 𝑇 𝑘 , the model receives theincoming data D 𝑘 = 𝑞 𝑘 𝑖 , 𝑣 𝑘 𝑖 , 𝑎 𝑘 𝑖 𝑁 𝑘 𝑖=1where 𝑞 𝑘 𝑖 , 𝑣 𝑘 𝑖 is the 𝑖-th input language modality and visual modality (e.g. image/video) from task 𝑘 respectively, and 𝑎 𝑘 𝑖 is the corresponding label, (e.g. 𝑎 𝑘 𝑖 ∈ {0, 1} for image-text retrieval and 𝑎 𝑘 𝑖 = (𝑥, 𝑦, ℎ, 𝑤) for referring expression comprehension), and 𝑁 𝑘 is the total number of samples at task 𝑇 𝑘 . In order to prioritize data privacy and minimize memory consumption, we adhere to the exemplar-free setting, which prohibits the utilization of previously seen data during both training and inference stages.Conventional continual learning methods have long been classified into task-, class-, and domain-incremental settings according to various task transition environments. However, the domain of continual vision-language retrieval and grounding introduces a heightened level of complexity, defying straightforward categorization. In our research, we delineate the notion of continuous tasks based on distinct semantic contents, such as "human", "animal" and "food". Besides, we follow the class-and domain-incremental settings, where the identity of the task remains unknown during test time, posing a more common and formidable challenge.In this section, we first build a simple but effective framework with existing prompt-based approaches [48, 50] to serve as our baseline for continuous vision-language retrieval and grounding.In the base framework, we adopt the pre-trained multi-modal transformer model, such as the CLIP [38] or GLIP [25] , as the feature extractor. These pre-trained transformer can be simplified to consist of a language encoder 𝜙 𝑞 (•) and a vision encoder 𝜙 𝑣 (•), while the rest components such as interaction module are omitted here for clarity. Throughout the training process, we keep these models frozen to preserve their learned representations. To tailor these models for downstream vision-language applications, we introduce a trainable downstream head 𝐻 (•) on top of the pre-trained transformer. The head is designed to be relatively lightweight to fully leverage the potential of large pre-trained model. Concretely, for image-text retrieval, the head comprises a single MLP layer for each encoder output, as the feature similarity can be directly calculated. For video-text retrieval, the head further incorporates a mean-pooling layer on the vision side to aggregate the clip-level features. For referring expression comprehension and segmentation, we follow [26] to build the head as a simple multi-modal transformer model with a one-layer encoder, two-layer decoders, and an MLP layer to obtain center coordinates or several fully-connected convolutional layers to derive segmentation masks.Prompt Design. Following previous works [48] , for each task 𝑇 𝑘 , we use an independent set of continuous learnable parameters 𝑃 𝑘 ∈ R 𝐿×𝐷 as a part of inputs to the pre-trained encoder, where 𝐿 ∈ R and 𝐷 ∈ R indicate the prompt's length and embedding dimension respectively. To incorporate the prompt, we extend the embedding of any single modality input from task 𝑇 𝑘 as x = [𝑥, 𝑃 𝑘 ], where 𝑥 ∈ {𝑞, 𝑣 } denotes the original input tokens of the vision or language modality. This extended embedding is then fed into the transformer blocks. When trained on a new task 𝑇 𝑘+1 , a new independent set of prompts 𝑃 𝑘+1 is added. Hence, learning all the tasks sequentially results in a task-wise prompt pool 𝑃 = {𝑃 1 , 𝑃 2 , ..., 𝑃 𝐾 }, where 𝑃 𝑘 ∈ R 𝐿×𝐷 is a single set of prompts of task 𝑇 𝑘 , and 𝐾 is the total number of tasks. Since we have separate encoders for vision and language, we also build two modality-specific prompt pools 𝑃 𝑣 = {𝑃 𝑣,𝑘 } 𝐾 𝑘=1 and 𝑃 𝑞 = {𝑃 𝑞,𝑘 } 𝐾 𝑘=1 . Downstream Training. Given the extended embedding q and v, we computed encoded features by q = 𝜙 𝑞 ( q) and ṽ = 𝜙 𝑣 ( v). Then we input these features into the downstream head and obtain the problem-specific output 𝑜, where 𝑜 = 𝐻 ( q, ṽ). At task 𝑇 𝑘 , we develop the problem-specific loss L base = L problem (𝑜 𝑘 , 𝑎 𝑘 ), where L problem corresponds to the contrastive loss for two retrieval tasks [38] , the L1 loss for referring expression comprehension and the dice loss for referring expression segmentation [25] . During training, only the parameters of the prompt and the downstream head are updated while the remaining parameter are kept frozen.Inference. During inference, since we learn independent prompt groups for each task, we need to identify which prompt group should be leveraged for the input test data. Here we follow S-Prompts [48] to apply K-Means to store the data centriods of each task during training, and use K-NN to search for the nearest centroid of the given test feature, so as to identify its corresponding prompt group during inference. This simple strategy can work generally well in our experiments.As shown in Figure 2 , we introduce intra-modal relevance estimation (IRE) for both vision and language encoders, where we leverage the task relevance within each modality to optimize prompts.The objective of IRE is to learn a binary classifier to judge whether the input prompt is relevant or irrelevant to the data of current task 𝑇 𝑘 . Inspired by BERT [9] , we insert a special token [REL] to obtain the extended embedding asx = [𝑥, 𝑃 𝑘 , [REL]],where 𝑥 ∈ {𝑞, 𝑣 }. Then we input x to the transformer encoder and obtain the encoded features x ∈ { q, ṽ }. By adding a linear layer on the output corresponding to the token [REL], we calculate a scalar representing the logits 𝑅. To estimate the relevance, we regard the prompt 𝑃 𝑘 that corresponds to the current 𝑇 𝑘 as the positive instances (i.e. relevant pairs) and construct negative instances (i.e. irrelevant pairs) by randomly substituting 𝑃 𝑘 with one of the historical prompts 𝑃 𝑖 from past tasks 𝑇 𝑖 , where 0 ≤ 𝑖 < 𝑘. The full intra-modal relevance estimation is performed using a binary-cross entropy loss:EQUATIONwhere 𝑅 + and 𝑅 -are the logits of relevant and irrelevant pairs respectively. As we employ two distinct encoders for the vision and language modalities, we perform relevance estimation separately for each modality.For the initial task 𝑇 1 , as no previous tasks exist, we only apply the loss L base . As we progress to the second task 𝑇 2 , we introduce the IRE loss L IRE and update both the special token [REL] and the linear layer. For subsequent tasks 𝑇 𝑖 where 𝑖 > 2, we fix the special token [REL] and update the linear layer with a momentum way. This step is key to learn effective prompt representations that contain much task-specific information, since the relevance estimation also suffers the catastrophic forgetting problem and continuous updating the token or linear layer will lead to severe forgetting of previous relevance knowledge. Instead, we fix the token to preserve the initial relevance information and slowly update the linear layer to avoid rapid forgetting. In this process, the prompt parameters of the current task can be gradually distinctive from the previous ones and therefore be more task-specific.We design inter-modal consistency alignment (ICA) to uniformly guide the optimization of prompts across modalities, as illustrated in Figure 2 .The objective of ICA is to perform contrastive learning over the encoded cross-modal features generated from current and historical prompts. Given the input data 𝑣 𝑘 , 𝑞 𝑘 and the prompt 𝑃 𝑣,𝑘 , 𝑃 𝑖,𝑘 that are associated with the current task 𝑇 𝑘 , we can calculate the encoded features ṽ𝑘 , q𝑘 and utilize the problem-specific loss L base to learn cross-modal alignment. However, relying solely on the data from the current task during training can limit the potential for achieving optimal alignment between the features ṽ and q, as presented in 1.Hence, we adopt the contrastive scheme by integrating the historical prompts to construct negative pairs, which can regularize the learning for current prompts. Concretely, given ṽ𝑘 and q𝑘 as the positive pair, we can generate 2 × (𝑘 -1) negative instances { ṽ𝑖,-} 𝑘 -1 𝑖=1 and { q𝑖,-} 𝑘 -1 𝑖=1 by replacing the prompt 𝑃 𝑣,𝑘 and 𝑃 𝑞,𝑘 with the previous prompts 𝑃 𝑣,𝑖 and 𝑃 𝑞,𝑖 where 𝑖 < 𝑘. Then we develop a contrastive alignment loss to pull each modal feature ṽ𝑘 and q𝑘 close to each other while push them apart from all other negative instances { q𝑖,-} 𝑘 -1 𝑖=1 and { ṽ𝑖,-} 𝑘 -1 𝑖=1 . On the language side, the loss can be given by:EQUATION)where 𝜏 is a temperature parameter [13] set to 0.1. The loss L vis ICA on the vision side can be obtain similarly. The full loss L ICA = L lang ICA + L vis ICA . The training process of the inter-modal consistency alignment follows a similar pattern to intra-modal relevance estimation, which starts from the second task as it requires the preparation of historical tasks. By utilizing the same modal features with prompts of different tasks, our constructed contrastive samples can more effectively enhance the consistency of cross-modal features, and ensure that this enhancement comes from the prompt corresponding to the task, thereby calibrating the cohesive optimization direction of the prompt.We combine the problem-specific loss and two proposed calibration loss to train the model, i.e.,EQUATIONwhere 𝜆 1 , 𝜆 2 and 𝜆 3 are set to 1.0, 0.1, 0.1 to balance the three losses. As our above discussion, we introduce two calibration losses exclusively after the initial task. It is important to note that our HPC paradigm is solely incorporated into the training process, ensuring that it has no impact on the inference speed or runtime efficiency. 
experiments: 
 In this section, we elaborate the experiment setting, performance evaluation of the proposed method, ablation study, in-depth analysis and hyper-parameter analysis. Additional dataset details and implementation details are provided in the supplementary materials.Task and Dataset. We perform experiments on four mainstream vision-language tasks, including two retrieval tasks and two grounding tasks:• Image-text Retrieval: We select MS-COCO [29] for evaluation and divide it into 12 tasks based on content categories, e.g. person, vehicle, animal, etc. • Video-text Retrieval: We select MSR-VTT [52] for evaluation and divide into 10 tasks based on content categories, e.g. news, movie, sports, etc. • Referring Expression Comprehension: We select RefCOCO [54] for evaluation and divide into 12 tasks based on content categories as MS-COCO. • Referring Expression Segmentation: Similar to the comprehension task, we select RefCOCO [54] for evaluation and divide into the same 12 tasks. The former two tasks focus on global-level alignment while the latter two are more difficult with local-level grounding. To generate separate tasks for continual learning for the image-text retrieval, we use the "category" annotation in original dataset to define 12 tasks, as shown in Table 1 . To generate separate tasks for continual learning for the video-text retrieval, we use the "category" annotation in original dataset to define 10 tasks, as shown in Table 2 . We evaluate the referring expression comprehension and segmentation on this dataset via its box-level and mask-level annotations respectively. Similar to MS-COCO, we adopt the same division split in Table 1 . Metric. Under the settings where the task boundaries are unknown and each task has an associated test set, we follow previous works [33, 50] to adopt two widely-used metrics, i.e., average accuracy and forgetting. Specifically, for two retrieval tasks, we report the accuracy as R@K (Recall at K) that calculates the percentage of test samples for which the correct result is found in the top-K retrieved points to the query sample. For two grounding tasks, we compute the accuracy as mIoU (Mean Intersection-over-Union) by the averaging over the IoU of each testing sample where IoU is the intersection area divided by the total union area. To better reflect the computational cost of methods, we also report the number of parameters that need to be tuned.Implementation Details. For input data, all the training images or video frames are resized to 240 x 240 and 320 x 320 for retrieval and grounding tasks, respectively. For model selection, we follow the prompt-based pipeline to select two powerful and widely-used pre-trained transformer models as encoders: CLIP(ViT-B/16) [38] for retrieval tasks and GLIP-T [25] for grounding tasks. For prompt design, we adopt the deep prompt [18] to insert prompt for all encoder layers. In CLIP, the prompt length is set to 16 and 12 for language and vision encoders, respectively. In GLIP, the prompt length is set to 12 and 20 for language and vision encoders, respectively. To train our model, we use the Adam optimizer with the cosine scheduler.We adopt Adam optimizer with a momentum of 0.9 and a cosine annealing scheduler [34] . For image-text retrieval, we train the model for 5 epochs for each task. The batch size is 128. The learning rate is set to 0.05 and 0.01 for prompt tokens and the head, respectively. For video-text retrieval, we train the model for 10 epochs for each task. The batch size is 24. The learning rate is set to 0.01 and 0.0001 for prompt tokens and the head, respectively. The head structure is the same as [36] . For referring expression comprehension and segmentation, we train the model for 20 and 10 epochs for each task, respectively. The batch size is 16. The learning rate is set to 0.001 and 0.0001 for prompt tokens and the head, respectively. The head structure is the same as [26] .Methods. We compare the proposed HPC strategy with the following state-of-the-art methods for continual learning, which can be divided into three groups: (1) Replay-based method. The replayhead is the naive sequential fine-tuning approach with the pretrained model frozen. The replay-all instead fine-tunes pre-trained model weights as well. (2) Regularization-based methods including EWC [22] , LwF [28] . (3) Prompt-based methods including Dy-Tox [12] , L2P [50] , S-Prompts [48] , DualPrompt [49] , MaPLe [21] and DCP [31] . Specifically, L2P and S-Prompts extract features using the uni-encoder, whereas DualPrompt, MaPLe and DCP take into account the interactions between modalities during feature extraction. As these methods are designed for uni-modal application, we extend them by separately applying their strategy on two modal encoders. To compare fairly, we use the same pre-trained model (i.e., CLIP/GLIP) for all compared methods as well as ours. The full implementations details of baselines are shown in the appendix.Baseline Implementation. For all baselines, we adopt the same base network as our HPC that consists of pre-trained CLIP or GLIP.(1) For replay-based methods, we simply maintain a extra memory space to store the training samples from the previous tasks, where the memory size is set to 50 or 100 samples per task. (2) For regularization-based methods EWC [22] and LwF [28] , we adopt the same hyper-parameter setting as reported in their work. (3) For prompt-based methods: In DyTox [12] , we use 3 Self-Attention Blocks and 1 Task-Attention Block for retrieval, and 1 Self-Attention Blocks and 1 Task-Attention Block for grounding, where all have 8 attention heads. To transfer the task-specific classifier in original classification problem, we also design task-specific structure by keeping separate MLP layers for different tasks in retrieval and separate MLP layer/fully-convolutional layers for different tasks in referring expression comprehension/segmentation. In L2P [50] , we set the prompt pool size to 20/10, and the number of key selection Table 3 : Performance Evaluation on Image-Text Retrieval Task. Bold: best results of exemplar-free methods, Underline: second best results of exemplar-free methods. Upper-bound: supervised finetuning on the i.i.d. data of all tasks, which is usually regarded as the upper bound performance [50] .Buffer size Text Retrieval Image Retrieval Param (↓) R@1 (↑) R@5 (↑) Forget(↓) R@1 (↑) R@5 (↑) Forget(↓) of exemplar-free methods. Upper-bound: supervised finetuning on the i.i.d. data of all tasks, which is usually regarded as the upper bound performance [50] .Video Retrieval Param (↓) R@1 (↑) R@5 (↑) Forget(↓) R@1 (↑) R@5 (↑) Forget(↓) to 5/5 for vision and language modality. For DualPrompt [49] , the sharable prompt length is set to 16 and 12 for language and vision encoders in CLIP, respectively, and is set to 6 and 10 for language and vision encoders in GLIP. For S-Prompts [48] , MaPLe [21] , and DCP [31] we adopt the same parameter setting as our HPC, i.e. the prompt length is set to 16 and 12 for language and vision encoders in CLIP, respectively; the prompt length is set to 12 and 20 for language and vision encoders in GLIP, respectively.Results of Retrieval Tasks. The results of two retrieval tasks are shown in Table 3 and 4 . It demonstrates that our proposed HPC method significantly outperforms the other exemplar-free methods, e.g. S-Prompts, DualPrompt. HPC slightly outperforms MaPLe and DCP with fewer additional parameters. We also find the proposed HPC methods' forgetting degrees are much less than those of the others. An interesting point is that the replay-based methods achieve limited performance. We argue the reason is that the limited data in each task restrict the retrieval learning since Table 5 : Performance Evaluation on Two Vision-Language Grounding Tasks. Bold: best results of exemplar-free methods, Underline: second best results of exemplar-free methods. Upper-bound: supervised finetuning of HPC on the i.i.d. data of all tasks, which is usually regarded as the upper bound performance [50] .Buffer it requires large contrastive samples, thereby making the model biased. This can explain the fact that freezing the pre-trained model and only tune the head can bring better performance. Instead, our method only fine-tune the prompt parameter and avoid wasting the pre-trained model's knowledge. The inferior performance of other prompt-based methods might be due to the less transfer learning capability of their dependent prompt learning on heterogeneous tasks and modalities.Results of Grounding Tasks. The results of two challenging grounding tasks are summarized in Table 5 . We can find HPC still surpass most exemplar-free methods. Different from the retrieval, we observe the replay-based method is effective and tuning all model can bring further gains. While these methods can reach closer to our method, it requests a large memory overhead that increases linearly with the class number for the storage of classwise exemplars and it also requires a large amount of parameters for tuning. Besides, we notice our method brings more remarkable gains in segmentation than comprehension. We argue the reason is that segmentation is more difficult and designed with complex downstream head, which indirectly poses a greater obstacle for continual learning. This fact again validates the superiority of our method.We conduct ablation studies of Calibrate Prompt and show the results in Table 6 . First, we develop two overall ablated methods HPC (w/o. IRE) and HPC (w/o. ICA) by discarding the intra-modal relevance estimation and inter-modal consistency alignment respectively. The results demonstrate that the two calibration tasks can both help improve the model performance in continuous tasks, showing our method can effectively calibrate prompt for the pretrained transformer.Besides, we further explore detailed design by generating two ablation methods HPC (IRE w/o. estimator) and HPC (ICA w/o. contrastive). The HPC (IRE w/o. estimator) leverages the contrastive loss to enlarge the gap between prompt groups of different tasks instead of our estimator on the transformer model. The HPC (ICA w/o. contrastive) directly enhances the cross-modal alignment without contrasting to historical prompts. The results indicate that our design is effective, since the estimator can leverage the discriminative power of pre-trained model and the contrastive loss can better achieve cross-modal alignment by utilizing historical information.Sensitivity to Downstream Head. We validate the robustness of our method by replacing the lightweight task-specific head with more complex one. We consider the challenging grounding tasks and follow [26] to build a 12 layer multi-modal transformer architecture and introduce cross-modal interaction mechanism. From the results shown in Table 7 , we can clearly observe that all methods suffer performance degradation since the complex neural network of downstream head lead to more severe forgetting. However, our method still outperforms all other methods, validating it can promote the pre-trained encoder to obtain more generalizable features for even complex downstream structure and alleviate the latent forgetting of it.Performance with Task Number. To study the correlation between the effect of our method and the number of tasks, we visualize the accuracy curves with the incremental tasks in Figure 3 . It can be seen that the gap between our HPC method and other methods continues to grow with the input of the task. This fact is because that our intra-modal relevance estimation and inter-modal consistency alignment can leverage the historical information of previous tasks to improve the learning of current and upcoming tasks. Compared with S-Prompts that builds independent prompt pools, our method can explicitly utilize the knowledge of each past task, thus bringing a more direct improvement. Effectiveness of the temperature parameter. We explore the effect of the temperature parameter 𝜏, which is a crucial hyperparameter in loss L ICA of inter-model consistency alignment. We set the value of 𝜏 to [0.01, 0.1, 0.2, 0.5, 1.0] and display the results in Figure 4 . We observe that the performance of our method decreases when the temperature is too small or too large. The optimal temperature value is around 0.1. This indicates a proper temperature value can promote feature learning, which is consistent with previous studies [13] . Training Stability. Further, we draw the three loss curve that corresponds to downstream task, IRE and ICA respectively during training. It is visually apparent in Figure 5 that the two losses L IRE and L ICA converge at a faster rate than the primary task loss L base , reaching its order of magnitude within one percent. Hence, our proposed strategy will not impede the training of the primary task, but instead enables prompt calibration in the early training stage, leading to more effective prompt representations for continual learning.  
conclusions: 
 In this work, we investigate the continual learning for visionlanguage retrieval and grounding tasks. Inspired by the recent success of pre-trained multi-modal transformer and the prompt technique, we adopt the prompt-based pipeline to solve this problem. To solve two critical limitations, i.e. the unfamiliarity across tasks and the heterogeneity between modalities, we propose Historical Prompt Calibration. First, we design a intra-modal relevance estimation task to help prompt encode more task-specific information by distinguishing the previous prompts of historical tasks, which is achieved by a relevance estimator on top of the pre-trained encoder. Second, we develop a inter-modal consistency alignment training to enhance cross-modal alignment by contrasting one modal features with another modal features, where contrastive samples are generated by prompts of current and previous tasks. In two vision-language retrieval tasks and two vision-language grounding tasks, HPC shows superior performance over the state-of-the-art methods. 
: 
 been proposed to tackle this problem. Regularization-based methods [1, 2, 22, 55] regularize key parameters during continual learning, which is simple but limited in some challenging applications. Replay-based and architecture-based methods are more effective, which save a small set of examples in a memory [7, 33, 41, 43, 46] and learn separated parameters for each task [17, 23, 53] respectively. While promising, these two methods are still limited due to the requirement for extra memory space and large amount of additional parameters. Recently, prompt-based methods [12, 40, 49, 50] has gained popularity, which adapt the pre-trained models to continuous tasks by learning task-specific prompt tokens for each individual task. This method has been demonstrated to be highly effective with minimal parameter tuning, leading to increased interest and research in this field.Despite the remarkable progress made in addressing the continual learning problem, the majority of existing works have been limited to uni-modal tasks such as image classification [50] and fake detection [48] . This is a significant drawback in modern society where multi-modal data is ubiquitous, spanning a range of modalities such as vision, language, sound, and more. Therefore, it is imperative to expand the current research in continual learning to support the diverse modalities.In this paper, we strive to advance deep learning in real-world scenarios by tackling the multi-modal challenge in continual learning. We focus on two major and common modalities, i.e. vision and language, and study the continual learning in two representative applications: retrieval and grounding. In light of the efficiency of prompt technique as well as the recent success of multi-modal pre-training models, we can seamlessly follow the prompt-based pipeline to develop our research.However, existing prompt-based methods have limitations for continual learning and incorporating another modality poses more difficulty. As illustrated in Figure 1 , we observe two inevitable challenges: (1) Unfamiliarity across tasks. Before diving into the multi-modal property, we note previous studies [48, 50] merely focus on avoiding catastrophic forgetting while ignore the forward transfer. They typically learn independent prompts for each task, which fails to leverage the knowledge from past tasks for efficient learning of new tasks. (2) Heterogeneity between modalities. The two-stream architecture [25, 38, 44] is a typical vision-language transformer model that applies two separate transformers to encode each modality. Directly learning modality-specific prompt for each encoder may not ensure the consistent optimization for these prompts.Based on the aforementioned discussions, we propose Historical Prompt Calibration, a novel strategy to explicitly calibrate taskwise and modality-wise prompt relation by learning from history knowledge. First, to alleviate the task unfamiliarity, we design intramodal relevance estimation that encourages prompts to be more closely related to the current task by leveraging prompts of past tasks. This technique involves developing a relevance estimator on the pre-trained model to discern whether the input prompt corresponds to the current task, where the positive samples are the learnable prompts of the current task and the negative samples are sampled from frozen prompts of the previous tasks. This objective help improves the quality of the prompt, because it explicitly forces each task-specific prompt group to be distinctive from others and hence enable them preserve much task-specific information. Second, to bridge the gap between modalities, we present inter-modal consistency alignment to enhance consistency between two modal prompts within the same task. This approach employs contrastive learning to ensure the agreement between two modal features that are encoded under prompt guidance of the current task can surpass all other combinations, where either modal feature is encoded by historical prompts. By similarly utlizing historical prompts in a cross-modal manner, we can enhance the cross-modal alignment to a greater extent.To validate the effectiveness of our proposed approach, we conduct the extensive experiments on four important vision-language applications, including image-text retrieval, video-text retrieval, referring expression comprehension and segmentation. Our comprehensive evaluation reveals that our HPC approach can deliver state-of-the-art results, underscoring its impressive power. 
 
 Appendix:  


Decision: All reviewers found the proposed method to be novel and effective.  The authors are encouraged to improve the final version by following reviewer recommendations.

Reviews: review: paper_summary: paper_summary: The paper investigates the use of continual learning in multi-modal settings, focusing on vision-language applications such as retrieval and grounding. The authors identify two critical limitations in existing methods: unfamiliarity across tasks and heterogeneity between modalities. To address these issues, they propose Historical Prompt Calibration, which includes intra-modal relevance estimation and inter-modal consistency alignment. The proposed approach is evaluated on four vision-language applications, demonstrating its superiority over state-of-the-art methods.
strength_weakness: strength_weakness: - The use of pre-trained transformer models (CLIP/GLIP) and prompt techniques to address the identified limitations is well-grounded in theory.

- The proposed objectives (intra-modal relevance estimation and inter-modal consistency alignment) are logically sound and effectively address the identified issues.

- The findings have significant implications for improving vision-language models, making them more adaptable to real-world, non-stationary data distributions.
limitations: limitations: - Although the paper claims superiority over state-of-the-art methods, a more detailed comparative analysis with a diverse set of baseline models would strengthen the findings.

- The approach might require task-specific adjustments for optimal performance, which could limit its applicability in a wider range of scenarios without further tuning. This suggests that while the method is effective for the evaluated tasks, it may not be readily applicable to other tasks without significant modifications. 

- The reliance on pre-trained transformer models like CLIP/GLIP may require significant computational resources, which could be a barrier for some applications. Additionally, the environmental impact of extensive computational requirements should be considered.

- The paper fails to cite many important prior works.

1. Oh, Changdae, et al. "Towards calibrated robust fine-tuning of vision-language models." NeurIPS Workshop on Distribution Shifts. 2023.

2. Cheng, Zhi-Qi, et al. "Gsrformer: Grounded situation recognition transformer with alternate semantic attention refinement." Proceedings of the 30th ACM International Conference on Multimedia. 2022.

3. Fang, Kaipeng, et al. "ProS: Prompting-to-simulate Generalized knowledge for Universal Cross-Domain Retrieval." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
paper_summary: paper_summary: This article presents an approach, Historical Prompt Calibration (HPC). This approach is used to address the problems of task unfamiliarity and inter-modal heterogeneity that arise in multimodal continuous learning. For task unfamiliarity, HPC utilizes cues from past tasks to improve the relevance between the current task and the cues. For the inter-modal heterogeneity problem, this paper proposes to use contrast learning to achieve inter-modal consistency alignment, which in turn enhances the consistency between cues from two modalities under the same task. Experiments on four visual-verbal tasks demonstrate the effectiveness and superiority of HPC.
strength_weakness: strength_weakness: 1. The paper designs a reasonable experimental protocol for this paper and implements a Baseline for all the current better-performing continuous learning scenarios to achieve a fair comparison in the context of cross-modal continuous learning.
2. The experiments in 4.4 on downstream headers, temperature coefficients, and training stability are persuasive.
limitations: limitations: 1. In Figure 5, IRE and ICA achieve the calibration of cues early in the training, is it possible to explore the performance performance and knowledge forgetting of HPC doing continuous learning in four tasks in the few-shot case?
2. Could you provide some visualized grounding results?
3. The experiments under the text-image retrieval, and text-video retrieval tasks in this paper are performed by splitting the COCO dataset and using a series of subsets of the COCO dataset as data streams. In fact, the data stream can also be constructed with many different datasets. The differences in image features between multiple different datasets may be greater than using a subset of the same dataset, and the challenge of catastrophic forgetting during continuous learning may be more difficult. Some recent work on continuous learning across modalities [1] has used multiple different datasets as data streams to conduct experiments, could this paper also use this approach to construct data streams to conduct experiments?

Reference
[1] Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. arXiv preprint arXiv:2303.06628, 2023.
paper_summary: paper_summary: This paper explores continual learning for vision-language applications, focusing on retrieval and grounding tasks, and proposes a novel strategy called Historical Prompt Calibration to address challenges related to task unfamiliarity and modality heterogeneity by leveraging pre-trained transformer models and prompt techniques.
strength_weakness: strength_weakness: The paper introduces the concept of Historical Prompt Calibration (HPC), which is a novel approach to addressing the challenges of continual learning in vision-language tasks.
limitations: limitations: W1: In the experiments conducted in this paper, the methods compared are mostly those of few-shot VLM adaptation, and the comparison with methods for Continual Learning for retrieval and localization is relatively outdated. The authors should compare with newer methods, i.e., [1][2]. However, in addition to the suggestions mentioned above, the authors could also reference relevant work from the past two years.

W2: Formatting Issue. The authors should ensure that the text in the figures and tables matches that in the main text, for example, in Figure 5.


[1] Continual Vision-Language Retrieval via Dynamic Knowledge Rectification. AAAI, 2024.
[2] Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters. CVPR, 2024.
paper_summary: paper_summary: This paper introduces a novel continual learning paradigm tailored for two vision-language applications: retrieval and grounding. Specifically, the authors propose a method termed Historical Prompt Calibration, which calibrates prompts in two key ways. Firstly, it involves intra-modal relevance estimation that encodes task-specific information. Secondly, it features inter-modal consistency alignment to enhance the agreement between modality-specific prompts. Extensive experiments across four vision-language tasks—namely, image-text retrieval, video-text retrieval, referring expression comprehension, and segmentation—demonstrate the effectiveness of the proposed method. This approach not only addresses the complexities inherent in cross-modal interactions but also shows promising improvements in performance across diverse tasks.
strength_weakness: strength_weakness: 1. $\textbf{Innovative Approach}$: This manuscript introduces Historical Prompt Calibration (HPC), a novel strategy for continual learning in vision-language tasks. This method addresses critical limitations in existing prompt-based methods by introducing intra-modal relevance estimation and inter-modal consistency alignment, which enhance task-specific information encoding and cross-modal alignment, respectively.
2. $\textbf{Logical Writing}$: The manuscript is well-structured, beginning with a clear introduction that outlines the limitations of existing methods and logically progresses to present the proposed HPC approach. Each section flows coherently, making it easy for the reader to follow the development of ideas and understand the contributions of the work.
limitations: limitations: 1. $\textbf{Lacking of Innovation}$: While the Historical Prompt Calibration (HPC) strategy introduces intra-modal relevance estimation and inter-modal consistency alignment, these techniques may be seen as incremental improvements rather than groundbreaking innovations. The core ideas are extensions of existing prompt calibration and continual learning methods, which are quite incremental.
2. $\textbf{Lacking of Discussions}: While the manuscript emphasizes the effectiveness of Historical Prompt Calibration (HPC), it lacks a detailed discussion on the computational overheads introduced by the intra-modal relevance estimation and inter-modal consistency alignment processes. Understanding the trade-offs between improved performance and computational complexity is crucial for practical applications.