Low-rank Prompt Interaction for Continual Vision-Language Retrieval

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 10:33:59

Abstract: Research on continual learning in multi-modal tasks has been receiving increasing attention. However, most existing work overlooks the explicit cross-modal and cross-task interaction. In this paper, we innovatively propose the Low-rank Prompt Interaction (LPI) to address this general problem of multi-modal understanding, which considers both cross-modal interaction and cross-task interaction. Specifically, as for the former, we employ multi-modal correlation modules for corresponding Transformer layers. Considering that the training parameters scale to the number of layers and tasks, we propose Low-rank Interaction-augmented Decomposition to avoid memory explosion, while enhancing the cross-modal association through sharing and separating common-specific low-rank factors. In addition, due to the multi-modal semantic differences carried by the low-rank initialization, we adopt hierarchical low-rank contrastive learning to ensure training robustness. As for the latter, we initially employ visual analysis and identify that different tasks have clear distinctions in terms of proximity. Therefore, we introduce explicit task contrastive constraints in the prompt learning process based on task semantic distance. Experiments on two retrieval tasks show performance improvements with the introduction of a minimal number of parameters, demonstrating the effectiveness of our method.

Main paper: 
 related work 2.1 continual learning: 
 Within the context of continual learning, an intelligent system is expected to acquire a sequence of knowledge. Unlike traditional models that are designed to learn from a single data distribution, these models are required to adapt to dynamically changing data distributions. A significant challenge arises if the model overfits on a particular task, leading to catastrophic forgetting, a significant decline in performance on previously learned tasks. To address this problem, common methods include: (1) rehearsal-based methods [5, [26] [27] [28] [29] that preserve representative or pseudo-samples to avoid catastrophic forgetting; (2) regularization-based methods [1, 13, 40 ]that constrain parameter updates; (3) architecture-based methods [14, 37] that allocate distinct parameters for each task to learn independently. Recent works [34] [35] [36] employ prompt-based learning techniques to train unique prompts for each task. These approaches have only been applied to simple uni-modal tasks and do not take into account the interactions between modalities or tasks. Consequently, they have not fully leveraged the potential of the pre-trained model's performance in more complex, multi-modal application scenarios.The Transformer architecture model has achieved tremendous success in natural language processing [3, 25, 33] . Subsequently, works [4, 8] have demonstrated their effectiveness in the field of computer vision as well, effectively establishing connections between multiple modalities. Due to its powerful representational capabilities and flexible structure, the Transformer-based architecture has become the ideal choice for linking multi-modal data. Following this, numerous effective pre-trained vision-language models like CLIP [24] , GLIP [16] , ALIGN [10] and Florence [39] have emerged. These models are trained on extensive datasets, learning rich joint visual-language representations, and exhibit commendable performance on zeroshot and few-shot learning. By employing fine-tuning techniques, these pre-trained models can be applied to specific downstream tasks, such as image-text retrieval, visual question answering, and more, finding widespread application across both the academic and industrial sectors.Prompt learning was initially applied in the field of natural language processing [15, 17] , enhancing the performance of pre-trained models through artificially designed or learnable prompts. Works [2, 11] introduces prompt learning into computer vision by adding prompts to visual input. Works like CoOp [42] and CoCoOp [41] apply it to Vision-Language (V-L) models. Furthermore, works like L2P [36] , DualPrompt [35] and S-Prompts [34] adopt prompt learning in the continual learning setting. To further boost model performance, MaPle [12] and DCP [19] integrate prompts into both the visual and language encoders, taking into account the cross-modal interaction of prompts. In this work, we make full use of prompt tuning to maximize the performance of pre-trained vision-language models in the continual learning setting. 
preliminary 3.1 problem formulation: 
 We concentrate on tackling the problem of multi-modal continual learning for vision-language retrieval in the class-incremental learning(Class-IL) scenario [30] . Specifically, we conduct experiments on text-image retrieval and referring expression comprehension tasks. For the image-text retrieval task, we need to retrieve images based on captions while retrieving captions based on images. For the referring expression comprehension task, given a HPA utilizes contrastive learning to align prompts across different layers. CPF achieves cross-layer information fusion between modalities. As for the cross-modal interaction, we introduce the cross-task prompt alignment (CPA) module which aligns prompts with task semantic distance. For each task, our method learns unique visual and textual prompts and interact modules.We store the prompts in the visual and textual prompt pool with task-specific keys which are used for predicting task identity during the inference period. As the task identity is unknown during the inference stage, we employ a method similar to that used in S-Prompts [34] to predict the task identity.caption and an image, we need to locate the position of the content described by the caption within the image. Our model is designed to progressively acquire knowledge across a sequence of tasks, with the ultimate goal of achieving comprehensive expertise in all tasks. Formally, the model sequentially acquires knowledge on a series of tasks, denoted as 𝑇 = {𝑇 1 ,𝑇 2 , ...,𝑇 𝐾 }, where 𝐾 represents the overall count of tasks seen. For each task𝑇 𝑘 , we denote D 𝑘 = {𝑣 𝑘 𝑖 , 𝑞 𝑘 𝑖 , 𝑎 𝑘 𝑖 } 𝑁 𝑘 𝑖=1as the available data with 𝑁 𝑘 samples, where 𝑣 𝑘 𝑖 , 𝑞 𝑘 𝑖 is the 𝑖-th input vision and language representation respectively and 𝑎 𝑘 𝑖 is the corresponding label. During training period, given visual input 𝑣 𝑖 and textual input 𝑞 𝑖 , we can compute the task-specific output 𝑜 𝑖 = 𝑀𝑂𝐷𝐸𝐿(𝑣 𝑖 , 𝑞 𝑖 ), L 𝑏𝑎𝑠𝑒 = L 𝑝𝑟𝑜𝑏𝑙𝑒𝑚 (𝑜 𝑖 , 𝑎 𝑖 ) where L 𝑝𝑟𝑜𝑏𝑙𝑒𝑚 is the task-specific loss. For the image-text retrieval task, L 𝑝𝑟𝑜𝑏𝑙𝑒𝑚 is the cross-entropy loss between logits and the truth label. For the referring expression comprehension, L 𝑝𝑟𝑜𝑏𝑙𝑒𝑚 is the same as that in GLIP. During the testing phase, we evaluate the model's performance on both the current and all previously learned tasks.Our proposed method is based on the pre-trained vision-language models, CLIP for the image-text retrieval task and GLIP for the referring expression comprehension task. Both CLIP and GLIP can be extracted as a vision encoder, a language encoder, and a downstream head. We omit the downstream head here for clarity. We only make improvements at the feature extraction stage, while maintaining consistency in structure and parameters for the rest of the model. Both the vision encoder and the language encoder have𝑁 𝐿 transformer layers, denoted as {V 𝑖 (•)} 𝑁 𝐿 -1 𝑖=0 for vision encoder and {L 𝑖 (•)} 𝑁 𝐿 -1𝑖=0 for language encoder. The feature embedding dimension of the vision encoder is 𝑑 𝑣 , while the feature embedding dimension of the language encoder is 𝑑 𝑙 . Visual input 𝑣 𝑘 𝑖 is divided into patches and projected into patch embeddings as the vision encoder input. Textual input 𝑞 𝑘 𝑖 is tokenized and projected into word embeddings as the language encoder input.Deep prompting attaches learnable visual and textual prompts to the input of the first 𝐷 transformer layers. Deep Prompting. Deep vision prompting and deep language prompting adopt the same strategy. To learn the modality prompts, we introduce learnable tokens {𝑃 𝑚 𝑖 ∈ R 𝐿 𝑚 ×𝑑 𝑚 } 𝐷 𝑖=1 where m represents v(ision) or l(anguage) modality, 𝐿 𝑚 is the prompt length and 𝑑 𝑚 is the prompt dimension. The input token can be denoted as [𝐸 𝑚 0 ,𝑊 𝑚 0 ]. 𝐸 𝑚 0 and 𝑃 𝑚 0 have the same token length and embedding dimension. For the first D transformer layer, the forward process can be formulated as:EQUATIONFor the rest transformer layers, the forward process can be formulated as:EQUATIONwhere M represents V (vision encoder) or L (language encoder). 
lpi: low-rank prompt interaction: 
 Our proposed method, Low-rank Prompt Interaction is illustrated in Fig. 2 . Concretely, LPI consists of two parts, the cross-modal and cross-task interaction. As for the cross-modal interaction, we introduce three modules, low-rank interaction decomposition in Sec. The Low-rank Interaction Decomposition decomposes the highdimensional prompts into multiple low-rank vectors, which is shown in Fig. 2 (a) . To establish the interaction between modalities, we have the visual prompts and textual prompts share the same lowrank vector, which not only facilitates the cross-modal interaction but also compresses the number of parameters introduced.For a three-dimensional vector P ∈ R 𝑖 × 𝑗 ×𝑘 , we can decompose it into three low-rank two-dimensional vectors, 𝐷 1 ∈ R 𝑖 ×𝑟 , 𝐷 2 ∈ R 𝑗 ×𝑟 and 𝐷 3 ∈ R 𝑘 ×𝑟 . P can be formulated as:EQUATIONwhere @ means the dot product and [i] means the index 𝑖 of the specific dimension.The visual prompts 𝑃𝑉 ∈ R 𝐷 ×𝐿 𝑣 ×𝑑 𝑣 in the vision encoder are decomposed into 𝐷 1 ,𝐷 𝑣 2 and 𝐷 𝑣 3 and textual prompts 𝑃𝐿 ∈ R 𝐷 ×𝐿 𝑙 ×𝑑 𝑙 in the language encoder are decomposed into three low-rank factors 𝐷 1 , 𝐷 𝑙 2 and 𝐷 𝑙 3 . The visual prompts and textual prompts share the same low-rank factor 𝐷 1 , thus establishing the interaction between visual and textual modality.As low-rank interaction decomposition fails to consider the hierarchical relationships among prompts from different transformer layers, we design Hierarchical Prompt Alignment (HPA) to mitigate the multi-modal semantic discrepancy between different layers.As shown in Fig. 2 (b), the objective of HPA is to perform contrastive learning over the learnable visual and textual prompts between different transformer layers. We consider visual prompts and text prompts that are on the same layer as positive samples, and those that are not on the same layer as negative samples, utilizing the cross-entropy loss function to calculate the semantic disparity of cross-modal prompts. For 𝑃𝑉 = {𝑃 𝑣 1 , 𝑃 𝑣 2 , • • • , 𝑃 𝑣 𝐷 } ∈ R 𝐷 ×𝐿 𝑣 ×𝑑 𝑣 and 𝑃𝐿 = 𝑃 𝑙 1 , 𝑃 𝑙 2 , • • • , 𝑃 𝑙 𝐷 ∈ R 𝐷 ×𝐿 𝑙 ×𝑑 𝑙 ,We set 𝐿 𝑣 = 𝐿 𝑙 . At this point P 𝑉 and P𝐿 have the same shape, thus we can compute their score matrix 𝑀𝑆 and cross-modal loss L 𝑚𝑜𝑑𝑎𝑙 ,𝑀𝑆 = ( P 𝑉 /𝜏 𝑚𝑜𝑑𝑎𝑙 )( P𝐿 𝑇 /𝜏 𝑚𝑜𝑑𝑎𝑙 ), L 𝑚𝑜𝑑𝑎𝑙 = - 1 𝐷 𝐷 ∑︁ 𝑖=1 𝐷 ∑︁ 𝑗=1 𝑦 𝑖 𝑗 log 𝑀𝑆 𝑖 𝑗 , 𝑦 𝑖 𝑗 = 0 i != j, 1 i = j.(EQUATIONwhere 𝜏 𝑚𝑜𝑑𝑎𝑙 represents the temperature for the hierarchical prompt alignment module and 𝑦 is the corresponding label.The LPI module establishes cross-modal associations, while the HPA accomplishes hierarchical alignment. Furthermore, we introduce the Cross-modal Prompt Fusion (CPF) to achieve cross-layer information fusion, which involves considering the outputs of previous layers from different modalities during the forward propagation process of the encoder. We conduct the cross-modal prompt fusion across the remaining 𝐷 -1 layers except for the first layer. The specific interaction process is shown in Fig. 3 . Before feeding the 𝐼 𝑣 𝑖 and 𝐼 𝑙 𝑖 into the encoders, we facilitate the cross-modal interaction and employ a fully connected layer to perform cross-modal mapping, thereby projecting different modalities onto the same dimensional space. To take previous prompt information into account, we combine the interactive information with the output from the preceding layer using momentum summation,EQUATIONwhere 𝑉 𝐼𝑆 is the language to vision projection layer, 𝑇 𝐸𝑋𝑇 is the vision to language projection layer, 𝜆 𝑣 is the visual momentum and 𝜆 𝑙 is the textual momentum. Then 𝐼 𝑖 will be replaced with Ĩ 𝑖 as input to the transformer layer.To avoid the introduction of an excessive number of parameters by the linear layer, we use the same decomposition strategy in Sec. 4.1 to decompose 𝑝 𝑡 into three low-rank factors. As for the parameters in the interaction module, assuming the input dimension is 𝑑 𝑖𝑛 and the output dimension is 𝑑 𝑜𝑢𝑡 . The weight can be denoted as 𝑝 𝑤 ∈ R (𝐷 -1) ×𝑑 𝑖𝑛 ×𝑑 𝑜𝑢𝑡 and the bias can be denoted as 𝑝 𝑏 ∈ R (𝐷 -1) ×𝑑 𝑜𝑢𝑡 . Thus the total learnable parameters can be denoted as 𝑝 𝑡 ∈ R (𝐷 -1) × (𝑑 𝑖𝑛 +1) ×𝑑 𝑜𝑢𝑡 . Thus we can decompose 𝑝 𝑡 into 𝐷 1 ∈ R (𝐷 -1) ×𝑟 , 𝐷 2 ∈ R (𝑑 𝑖𝑛 +1) ×𝑟 and 𝐷 3 ∈ R 𝑑 𝑜𝑢𝑡 ×𝑟 .When training on task 𝑇 𝑘 , we utilize the word embedding to convert the task name seen, denoted as 𝑁 1 , 𝑁 2 , •, 𝑁 𝑘 , to vectors as the task semantic embeddings. Using the visualization technique, we observe a clear distance relationship between tasks, which is shown in Fig. 4 .Figure 4 : Task Distance Visualization using t-SNE [32] .Subsequently, we calculate the cosine similarity for each task's semantic embeddings. Pairs of tasks with cosine similarity greater than and equal to a predefined threshold are considered positive tasks, while those below the threshold are treated as negative tasks, we calculate the label 𝑧 for cross-entropy loss as follows:EQUATIONWe denote 𝑃𝑉 𝑖 , 𝑃𝐿 𝑖 as the prompts for task 𝑇 𝑖 . Prompts in positive tasks are considered positive of each other, and prompts in negative tasks are considered negative of each other. Thereby, we can calculate visual score 𝑇 𝑆 𝑣 and textual score 𝑇 𝑆 𝑙 among different tasks, For visual prompts and textual prompts in task 𝑇 𝑘 , We first merge the second and third dimensions, stretching each prompt into a one-dimensional vector. We then compute the scores between visual prompts and textual prompts across different tasks,EQUATIONUsing the visual and textual scores, we can calculate the visual and textual losses 𝐿 𝑚 . Considering the numerical imbalance between positive and negative samples, we employ the NT-BXent (Normalized Temperature-scaled Binary cross-entropy) [7] loss function for computation, which is based on NT-Xent (Normalized Temperature-Scaled Cross-Entropy) [6] . Cross-task prompt loss between task 𝑇 𝑖 and 𝑇 𝑗 can be presented as follows:EQUATION) where 𝜏 𝑡𝑎𝑠𝑘 represents the temperature for the cross-task interaction, 𝜎 stands for the Sigmoid function, Thus we can compute the loss 𝐿 𝑚 𝑖 for each task 𝑇 𝑖 ,EQUATIONwhere 𝑁 𝑝𝑜𝑠 and 𝑁 𝑛𝑒𝑔 are positive and negative sample number of task 𝑇 𝑖 respectively. The final cross-task loss L 𝑡𝑎𝑠𝑘 can be formulated as following:EQUATION)We combine the problem-specific loss, the cross-modal loss and cross-task loss to train the model, i.e.,EQUATIONwhere 𝜆 1 ,𝜆 2 and 𝜆 3 are set to 0.8,0.1,0.1 to balance the three losses.Additional training and inference details are provided in the supplementary material. 
experiments: 
 In this section, we perform experiments on two vision-language retrieval tasks under the same class-incremental learning scenario [31] , where we need to solve each task seen so far with task identity unknown. We compare our proposed LPI with state-of-the-art methods and conduct ablation studies. LPI-M can extract uni-modal features separately, and saving the uni-modal features of the retrieved objects in advance can enhance retrieval speed. LPI-P, on the other hand, carries out the cross-modal interaction, demonstrating higher performance.We conduct experiments on two visual-language retrieval tasks, namely image-text retrieval and referring expression comprehension. For the image-text retrieval task, we select the MS-COCO [18] dataset, while for the referring expression comprehension task, we choose the RefCOCO [38] , Re-fCOCO+ [38] , and RefCOCOg [22] datasets. We divide the whole Image-to-Text Retrieval Text-to-Image Retrieval R@1(↑) R@5(↑) R@10(↑) Forgetting(↓) R@1(↑) R@5(↑) R@10(↑) Forgetting(↓) CLIP [24] 67 dataset into 12 categories according to the super category of the images, which are appliance, sports, outdoor, electronic, accessory, indoor, kitchen, furniture, vehicle, food, animal, and person. Further details can be found in the supplementary materials.We conduct experiments under the classincrement setting, utilizing Average Accuracy and Forgetting as metrics following previous work [20, 21] . Specifically, we compute the Recall at K, denoted as 𝑅@𝐾, which means the recall rate of samples containing the target within the top K results ranked by the predicted probability of retrieval. The forgetting rate, represented as 𝐹 @𝐾, is derived by subtracting the maximum value of 𝑅@𝐾 from previous tasks from the value of 𝑅@𝐾 obtained from the most recent task test. 𝐹𝑜𝑟𝑔𝑒𝑡𝑡𝑖𝑛𝑔 is the average of 𝐹 @1, 𝐹 @5, and 𝐹 @10.We set the prompt length 𝐿 𝑣 = 𝐿 𝑙 = 16, 𝜏 𝑚𝑜𝑑𝑎𝑙 = 0.01, 𝜏 𝑡𝑎𝑠𝑘 = 0.01, and 𝜆 𝑣 = 𝜆 𝑙 = 0.9. The prompt and interaction rank is set to 4. For the image-text retrieval task, all the training images are resized to 224 × 224. We select CLIP(ViT-B/16) as the backbone. We set 𝑑 𝑣 = 768, 𝑑 𝑙 = 512, 𝑒𝑝𝑜𝑐ℎ = 5, learning rate 𝑙𝑟 = 0.01. For the referring expression comprehension task, all the training images are resized to 448 × 448. We select GLIP-T(A) as the backbone. We set 𝑑 𝑣 = 96, 𝑑 𝑙 = 768, 𝑒𝑝𝑜𝑐ℎ = 10, learning rate 𝑙𝑟 = 0.05, 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 = 0.4. To train Our model, we use Adam optimizer with the cosine annealing scheduler. Further experimental details can be found in the supplementary material.5.2.1 Baselines. We compare our proposed method with pre-trained CLIP [24] , GLIP [16] , and prompt-based methods including L2P [36] , S-Prompts [34] , MaPLe [12] , and DCP [19] . L2P and S-Prompts employ the uni-encoder for feature extraction. MaPLe and DCP account for the cross-modal interaction. Given that some of these methodologies are not originally tailored for a continual learning framework, we adapt them by incorporating their prompt tuning strategies.Image-text Retrieval. The results of the image-text retrieval are shown in Tab. 1. Compared to CLIP, LPI-M shows improvements across all metrics, yet it has a higher 𝐹𝑜𝑟𝑔𝑒𝑡𝑡𝑖𝑛𝑔. In contrast to L2P, LPI-M demonstrates enhancements in every benchmark. LPI-M marginally surpasses S-Prompts in 𝑅@1, 𝑅@5, and 𝑅@10, while maintaining a similar 𝐹𝑜𝑟𝑔𝑒𝑡𝑡𝑖𝑛𝑔.Tab. 2 presents the results of the referring expression comprehension. Our method shows significant improvements in 𝑅@1, 𝑅@5, and 𝑅@10, while its performance on the 𝐹𝑜𝑟𝑔𝑒𝑡𝑡𝑖𝑛𝑔 is average. We observe that L2P has the lowest 𝐹𝑜𝑟𝑔𝑒𝑡𝑡𝑖𝑛𝑔 in most tasks but limited improvement in 𝑅𝑒𝑐𝑎𝑙𝑙. In some tasks, its performance is even lower than the base model GLIP. This is attributed to training prompts for each task, where an incorrect prediction of task identity can mislead the inference. S-Prompts achieves a more balanced improvement in Recall compared to the base model. MaPLe and DCP show significant improvements in 𝑅@5 and 𝑅@10, though their 𝑅@1 is lower than GLIP on some test datasets. DCP, LPI-M, and LPI-P exhibit a high 𝐹𝑜𝑟𝑔𝑒𝑡𝑡𝑖𝑛𝑔 of up to 10% on testA of refcoco and refcoco+. A deeper analysis reveals that testA datasets have very few test samples in most tasks, leading to a significant drop in recall if even one sample is forgotten.LPI-M and LPI-P surpass other comparative approaches on most metrics for the referring expression comprehension task. Specifically, LPI-M is well-suited for scenarios with a small number of training samples per task, while LPI-P is appropriate when there is an abundance of training samples for each task, as this allows for better learning of cross-modal and cross-task interactions, resulting in improved performance.We conduct ablation experiments on RefCOCO for the referring expression comprehension and calculate the metrics on the val dataset. We adopt the approach known as Decomposition Prompting (DP), which utilizes the low-rank interaction decomposition module. The method without prompt compression is referred to as Common Prompting (CP). Tab. 3 presents the metrics associated with the incorporation of different modules. TP indicates the type of prompting, DP or CP. The ablation experiment on prompt and interaction depth, as well as qualitative analysis, are provided in the supplementary material.We conduct an ablation study to explore the effectiveness of different modules, Hierarchical Prompt Alignment (HPA), Cross-modal Prompt Fusion (CPF), and Cross-task Prompt Alignment (CPA). The results are shown in the Tab. 3. It can be observed that HPA, CPF, and CPA all contribute to improvements in 𝑅@1, 𝑅@5, and 𝑅@10. Specifically, CPF allows the encoder to extract features more effectively by leveraging information from other modalities, which significantly enhances the 𝑅@1, 𝑅@5, and 𝑅@10. CPA reduces the distance between prompts of similar tasks, meaning that even if the task identity predicted during the inference stage is for a similar task, the prompts of these similar tasks still provide a certain level of guidance. Therefore, this increases the model's fault tolerance. The HPA achieves hierarchical alignment of encoders across different modalities, effectively constraining prompt updates and reducing the 𝐹𝑜𝑟𝑔𝑒𝑡𝑡𝑖𝑛𝑔.We further conduct experiments on Low-rank Interaction Decomposition and find that not compressing the prompt results in very poor performance on 𝑅@1. Although there is a slight improvement in the 𝑅@5 and 𝑅@10 metrics, it leads to a higher 𝐹𝑜𝑟𝑔𝑒𝑡𝑡𝑖𝑛𝑔. Utilizing the CP strategy on LPI-P even underperforms the method that only uses DP. The analysis suggests that the prompts require more resources to train and random initialization cannot align well between modalities during the training process. 5 (a) and (b), indicate that 𝑅@𝐾 initially increases and then decreases with the increase of the prompt rank, peaking at a rank of 4. Conversely, 𝐹𝑜𝑟𝑔𝑒𝑡𝑡𝑖𝑛𝑔 follows the exact opposite pattern. We set the prompt rank to 4 and experiment with interaction rank at 1, 2, 4, 8, and 16, obtaining comparable results, as shown in Fig. 5 (c) and (d). Overall, as the interaction rank increases, 𝑅@𝐾 initially rises and then falls, with a significant decrease in the 𝑅@1 only observed when the interaction rank is set to 8. As for the 𝐹𝑜𝑟𝑔𝑒𝑡𝑡𝑖𝑛𝑔, the value initially increases and then decreases, similar to that of prompt rank.(a) Recall@K (b) Forgetting (c) Recall@K (d) Forgetting From the figure, it is evident that for the Recall metrics, LPI-P performs the best in most tasks, followed by LPI-M, then DP, with GLIP showing the lowest performance. In terms of the 𝑅@1 metric, the GLIP outperforms other methods in the appliance, person, and furniture tasks. For the 𝑅@5 and 𝑅@10, the use of the prompt technique results in improvements across all tasks. Regarding the 𝐹𝑜𝑟𝑔𝑒𝑡𝑡𝑖𝑛𝑔, DP, LPI-M, and LPI-P exhibit the high 𝐹𝑜𝑟𝑔𝑒𝑡𝑡𝑖𝑛𝑔 in sports, furniture, kitchen, accessory, and electronic tasks.R@1 R@5 R@10 Forgetting  
conclusion: 
 In this paper, we propose a novel Low-rank Prompt Interaction method that considers both cross-modal and cross-task interactions.We further conduct experiments in the class-incremental setting on two vision-language tasks, image-text retrieval and referring expression comprehension. The results compared with state-of-theart methods demonstrate the effectiveness of our method. 
: 
 Traditional solutions to address catastrophic forgetting can be categorized into three types. The replay-based approach involves storing samples from previous tasks in a memory bank for training on future tasks. However, this can lead to data leakage and security issues, making it unacceptable in certain scenarios. The regularization-based approach seeks to prevent model forgetting by regularizing parameter updates, but it can only mitigate forgetting to a certain extent. The architecture-based approach allows for the independent learning of parameters for each task, which introduces many additional parameters.Recently, prompt-based learning has made significant progress in the field of natural language processing (NLP). Inspired by this technology, some works have incorporated prompt techniques within pre-trained multi-modal models. Pre-trained vision-language models, such as CLIP (Contrastive Language-Image Pre-Training) [24] and GLIP (Grounded Language-Image Pre-training) [16] are trained on web-scale data, exhibit strong generalization capabilities. When we need to employ the model for downstream tasks, full model fine-tuning consumes a significant amount of computational and storage resources. By freezing the pre-trained model and training only a small number of prompt tokens, the model can achieve satisfactory results. We summarize the application of prompt learning in continual learning into three paradigms, as illustrated in Fig. 1 . Paradigm (a) uses the uni-encoder to extract features, Paradigm (b) considers the cross-modal interaction, and Paradigm (c) considers both cross-modal and cross-task interactions.Works [35, 36] have been developed for continual learning scenarios, however, numerous aspects could be improved. Such approaches are confined to single-modality tasks, such as image classification, and they have not taken into account the interactions between different modalities and tasks, thus failing to fully leverage the performance of pre-trained models. Works [12, 19] have incorporated the cross-modal interaction within single-task scenarios, demonstrating the effectiveness of the interaction between different modalities. However, the introduction of interaction modules results in a substantial increase in the number of parameters, which becomes unacceptable as the number of tasks grows. How can we reduce the introduction of parameters while still incorporating interaction modules? In large language models, LoRA (Low-Rank Adaptation) [9] compress updates to network parameters. Inspired by this approach, we can employ a similar strategy to compress all learnable parameters within our model.Taking into consideration the issues discussed above, we adhere to the third paradigm shown in Fig. 1 (c ) to propose a novel Lowrank Prompt Interaction (LPI) method which accounts for both the cross-modal and cross-task interactions. For the cross-modal interaction, we integrate a low-rank interaction decomposition module, utilizing a shared low-rank factor to construct high-dimensional visual and textual prompts, which not only establishes cross-modal prompt associations but also reduces the number of introduced parameters. Since the low-rank interaction decomposition does not account for the hierarchical relationships among prompts, we introduce hierarchical prompt contrastive learning to achieve the hierarchical alignment of cross-modal prompts. Furthermore, we employ a cross-modal prompt fusion module to achieve the integration of cross-modal information within visual and textual prompts from different layers. For the cross-task interaction, we compute semantic distances across various tasks, classifying tasks into positive or negative samples based on the closeness of these distances in the semantic space. By applying contrastive learning, positive samples are drawn closer while negative samples are pushed further apart.We conducted experiments on two retrieval tasks: image-text retrieval and referring expression comprehension. The results demonstrate that our method outperforms other approaches in the classincremental learning setting [30] , thus proving the efficacy of our method. In summary, our contributions are as follows:• We propose Low-rank Prompt Interaction, a novel method that considers the cross-modal and cross-task interactions. The cross-modal interaction facilitates interaction between modalities, achieves hierarchical prompt alignment, and reduces introduced parameters while the cross-task interaction constrains prompt updates based on task semantic distance. • In the class incremental setting, we conduct experiments on two complex vision-and-language tasks: image-text retrieval and referring expression comprehension. Our experimental results surpass the state-of-the-art approaches, demonstrating the efficacy of our method. 
 
 Appendix:  


Decision: The paper introduces Low-rank Prompt Interaction (LPI) for multi-modal understanding, which integrates cross-modal and cross-task interactions via multi-modal correlation modules in Transformer layers. It employs Low-rank Interaction-augmented Decomposition to balance memory efficiency and cross-modal connectivity with minimal parameter increase. Enhanced by hierarchical low-rank contrastive learning and task-specific contrasts, LPI shows improved performance on two retrieval tasks, demonstrating its effectiveness.

Reviews: review: paper_summary: paper_summary: The paper introduces Low-rank Prompt Interaction (LPI) for multi-modal understanding, integrating cross-modal and cross-task interactions. It incorporates multi-modal correlation modules in Transformer layers and utilizes Low-rank Interaction-augmented Decomposition to manage memory and enhance cross-modal ties without adding excessive parameters. Hierarchical low-rank contrastive learning improves robustness, while task-specific contrasts are used to tailor learning processes based on task differences. The method demonstrates improved performance on two retrieval tasks, confirming its effectiveness with minimal parameter increase.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow.
2. The visualizations are pretty.
3. Good experimental results on several scenarios.
limitations: limitations: 1. It is suggested that future revisions include a deeper exploration of the motivations behind specific design decisions, enhancing the reader's understanding of the reasons for these choices beyond just a description of the implemented actions.
2. I see the overall loss is a combination of several losses, how do authors balance the contribution of each loss?
paper_summary: paper_summary: This paper introduces a LPI method for continual vision-language retrieval tasks. The proposed LPI enables effective cross-modal and cross-task interaction with a low-rank interaction decomposition module to address the parameter explosion issue in multimodal understanding. Additionally, the hierarchical low-rank contrastive learning strategy enhances training robustness, ensuring prompt alignment across different layers.
strength_weakness: strength_weakness: 1.The proposed LPI effectively addresses the key issue in current multimodal understanding by combining cross-modal and cross-task interactions.
2.The low-rank interaction decomposition module overcomes the parameter explosion problem caused by the increasing number of tasks.
3. The extensive experiments on image-text retrieval and referring expression comprehension demonstrate the efficacy of LPI.
limitations: limitations: 1. The paper lacks a detailed analysis of computational complexity and scalability. While the method optimizes parameter quantity, it does not discuss computational complexity in depth, which may hinder its implementation in larger and more complex tasks.
2. The necessary comparative methods in Table.1 are lacking:
[1] Multilateral Semantic Relations Modeling for Image Text Retrieval, CVPR '23
[2] HGAN: Hierarchical Graph Alignment Network for Image-Text Retrieval, TMM 2023
[3] MKVSE: Multimodal Knowledge Enhanced Visual-semantic Embedding for Image-text Retrieval, TOMM 2023
[4] External Knowledge Dynamic Modeling for Image-text Retrieval, ACMMM '23
3. The source code is not available.
paper_summary: paper_summary: This paper considers both cross-modal and cross-task interactions during continual learning and  proposes a novel method, LPI, which extends the prompt learning for multi-modal vision-langugae retrieval in the continual learning setting.
Overall, the paper is well written and easy to follow. Extensive experiments are conducted to show the effectiveness of the proposed method.
strength_weakness: strength_weakness: This paper considers both cross-modal and cross-task interactions during continual learning and  proposes a novel method, LPI, which extends the prompt learning for multi-modal vision-langugae retrieval in the continual learning setting.
Overall, the paper is well written and easy to follow. Extensive experiments are conducted to show the effectiveness of the proposed method.
limitations: limitations: 1. The second paragraph of the introduction, the references should be cited when the authors talk about the related work.
2. The resolution of Fig.2 needs improvement. And for Task i-1, it's better to add lines between the model and CPA module, just like the Task i.
3. In Line 412, it missed the braces.
4. For the cross-task interaction, how do you choose the predefined threshold of the task? 
And what is the meaning of "Pairs of tasks with cosine similarity greater than and equal to a predefined threshold are considered positive tasks, while those below the threshold are treated as negative tasks,". 
From my understanding, during training, the task identity is already known, why the threshold is still needed?
5. For the ablation study in Table 3, it's interesting that why CP is much lower than DP. CP without compression should contain more information.