Bilateral Adaptive Cross-Modal Fusion Prompt Learning for CLIP

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 09:12:40

Abstract: In the realm of CLIP adaptation through prompt learning, it is important to emphasize the pivotal role that the proper alignment of visual and textual representations plays when adapting the CLIP to downstream tasks. We propose that the proper alignment for downstream tasks is determined by the $\textbf{flexibility}$ of the interaction between cross-modal information, which compensates for the absence of contrastive loss during the adaptation process. However, the current prompt learning methods, such as isolated modifications to the visual or language branches of CLIP or the employment of uni-directional cross-modal fusion, are not sufficient to explore the full potential of the mutual interaction between visual and textual modalities. To overcome this limitation, we propose a new paradigm for the CLIP prompt learning community, named $\textbf{B}$i$\textbf{l}$ateral Adaptive Cr$\textbf{o}$ss-Modal Fusi$\textbf{o}$n Pro$\textbf{m}$pt Learning~($\textit{Bloom}$)， which includes two enhancements. First, we propose using projection functions for bi-directional modality transformation and fusion functions to encourage the mutual interaction between corresponding layers within both the image and text encoders. Second, we propose an adaptive manner that automatically searches the optimal combination of cross-modal information at each layer. These two improvements ensure a more efficient and flexible integration of the two modalities, thereby achieving proper alignment for specific downstream tasks. We put our method to the test in terms of base-to-novel, cross-dataset, and cross-domain evaluations on 15 image classification datasets. The results demonstrate a significant performance enhancement achieved by $\textit{Bloom}$.

Main paper: 
 introduction: 
 The fundamental vision-language pre-trained models, particularly the groundbreaking CLIP model [20] , have demonstrated extraordinary generalization capabilities across a wide array of downstream computer vision tasks. Examples of these tasks include but are not limited to, image classification [10, 12, 29, 30] , semantic segmentation [6, 15, 21, 31] , object detection [4, 14, 28] , and action recognition [11, 17, 25, 26] . The primary objective of CLIP is to [20] and the prompt learning method CoOp [30] fail as they focus on irrelevant regions, which results from the misalignment between visual and textual representations. Bloom effectively addresses it leveraging sufficient flexibility of cross-modal interaction.effectively align language and vision representations into a unified feature space, utilizing an extensive web-scale dataset comprising approximately 400 million image-text description pairs. In contrast to the conventional approach of fine-tuning the entire model, the prevailing research methodology primarily leverages prompt learning, which is inspired by advancements in the field of natural language processing (NLP), to adapt the capabilities of CLIP to downstream tasks.Current prompt learning methodologies can be broadly classified into two distinct categories. The first category encompasses uni-modal prompt learning techniques [10, 29, 30] , which exclusively adjusts either the language or vision branch of the CLIP model, as shown in Fig. 2 (a) . The second category is multimodal prompt learning (MaPLe) [12] , which simultaneously adjusts both language and vision branches, as shown in Fig. 2 (b) . MaPLe incorporates a learnable textual prompt in conjunction with a layer-wise coupling function, such as a single layer MLP, to facilitate languageto-vision (𝐿 → 𝑉 ) transformation and generate a visual prompt.Despite achieving a certain level of efficiency, the prompt learning adaptation process consistently exhibits a distinct objective function compared to the pre-training process. For example, singlelabel image classification tasks employ CrossEntropy loss for training, whereas contrastive loss is absent. This difference in optimization objectives inevitably disrupts the original alignment between language and visual representations. In Fig. 1 , we analyze the failure L i-1 ❄ 🔥 Image Encoder V i-1 V i ❄ L i Text Encoder L i-1 ❄ Image Encoder V i-1 V i ❄ L i 🔥 Image Encoder V i-1 V i ❄ 🔥 Text Encoder L i-1 ❄ 🔥 L i T→V Proj V i-1 V i Text Encoder L i-1 L i 🔥 🔥Image Encoder cases of CLIP and prompt learning CoOp by examining the gradient class attention maps for the ground-truth category (i.e., cat, TV, and zebra, respectively). The misalignment of the two modalities causes both CLIP and CoOp to concentrate on irrelevant regions. As a result, an efficient prompt learning method should strive to establish proper alignment for a specific downstream task. In this paper, we believe that the proper alignment is determined by the flexibility of cross-modal information interaction. However, upon reviewing existing prompt learning methods, we find that uni-modal prompt learning inadvertently overlooks the multi-modal aligned structure inherent to the CLIP model. As a result, solely adjusting the vision or language branch proves inadequate for the effective adaptation of CLIP. Although MaPLe has recognized the importance of cross-modal interaction, the uni-directional prompt transformation remains insufficient. Consequently, a key question arises: How can we make prompt learning as flexible as possible?To that end, we propose a new paradigm for prompt learning community, named Bilateral Adaptive Cross-Modal Fusion Prompt Learning (Bloom) with two enhancements to address these shortcomings. First, we ensure a bi-directional interaction between the two modalities. Specifically, as shown in Fig. 2 (c), we simultaneously initialize individual vision and language prompts for the corresponding branches at each layer. Subsequently, we employ two bottleneck-style projection functions to achieve language-tovision (𝐿 → 𝑉 ) and vision-to-language (𝑉 → 𝐿) prompt transformations. Second, we use a fusion function with an adaptive strategy to attain cross-modal fusion in an end-to-end manner, which allows Bloom to automatically search an optimal convex combination of bilateral cross-modal information at each layer, thereby ensuring proper alignment.We conduct a comprehensive experimental evaluation on 15 image classification datasets to validate the generalization capabilities of the Bloom in terms of base-to-novel generalization, cross-dataset, and cross-domain generalization. The results provide evidence that Bloom effectively enhances the prompt learning of the CLIP model by improving the generalization performance for unseen classes, as well as in varying data distribution and domain scenarios.Our contributions can be summarized in three aspects:• We propose a new Bilateral Adaptive Cross-Modal Fusion Prompt Learning (Bloom) paradigm for prompt learning that explores flexible cross-modal interactions to attain appropriate alignment for specific downstream tasks. • We propose an adaptive cross-modal fusion function to ensure our proposed Bloom to automatically search the optimal combination of cross-modal information, which further enhances the flexibility of prompt learning.Bloom significantly advances the current state of multimodal prompt learning, achieving new state-of-the-art results across 15 image classification datasets.  
related work: 
 "a photo of [class]" Layer V 1 Layer V 2 Layer V K [c 0 ,E 0 ] … … … … Text Encoder Word Embedding Layer L 1 Layer L 2 Layer L K W 0 W 1 W K-1 ❄ ❄ ❄ ❄ ❄ ❄ 𝑃 !→# $ 𝑃 # $ 𝑃 ! $ 𝑃 #→! $ Projv→l Projl→v 𝑀 # $ 𝑀 ! $ F F 𝑃 !→# % 𝑃 # % 𝑃 ! % 𝑃 #→! % Projv→l Projl→v 𝑀 # % 𝑀 ! % F F [c 1 ,E 1 ] [c k-1 ,E k-1 ] 𝑀 # & Softmax α 1-α 𝐺 # $ 𝐺 # 'Fusion Function main streams. First, uni-modal prompts focus on learning either textual or visual prompts for CLIP. For instance, CoOp [30] aims to learn prompts for the language branch of CLIP, while CoCoOp [29] incorporates instance-dependent bias as a condition to guide the learning of prompts, thereby enhancing the generalization ability to unseen classes. Additionally, [26] introduces motion-aware prompts to adapt CLIP for action recognition tasks. VPT [10] integrates a small number of learnable parameters, which are then prepended to the input sequence of each Transformer layer. Second, multimodal prompt [12] involves learning prompts for both language and vision branches to improve the alignment between visual and textual representations. 
revisiting clip: 
 Our method is built upon the Transformer-based CLIP [20] , which integrates both a text encoder and an image encoder with the primary objective of generating textual and visual representations, respectively. To effectively understand our method, it is important to revisit the structure and functionality of the CLIP model.The image encoder, denoted as V, is composed of 𝐾 transformer layers. Given an RGB image as input, we initially partition the image into 𝑁 non-overlapping patches, subsequently projecting these image patches into image embeddings𝐸 0 = [𝑒 0 0 , 𝑒 1 0 , • • • , 𝑒 𝑁 0 ] ∈ R 𝑁 ×𝑑 𝑣 .From the perspective of the 𝑖 𝑡ℎ layer within V, the image embeddings 𝐸 𝑖 -1 serve as input to V 𝑖 , accompanied by a learnable class token 𝑐 𝑖 -1 . This class token is instrumental in capturing global contextual information. This calculation can be represented as:EQUATIONwhere [•, •] denotes concatenate operation. Lastly, the class token 𝑐 𝐾 is projected by ImageProj(•) to obtain the final image representation:EQUATIONThe text encoder, denoted as L, features a layer count that is identical to that of the image encoder, V. The textual description corresponding to the input image undergoes an initial projection to text embeddings𝑊 0 = [𝑤 0 0 , 𝑤 1 0 , • • • , 𝑤 𝐻 0 ] ∈ R 𝐻 ×𝑑 𝑙via the Tokenizing process. In this context, 𝐻 is typically set to 77. With respect to the 𝑖 𝑡ℎ layer within L, the text embeddings 𝑊 𝑖 -1 are utilized as input to L 𝑖 , namely that:EQUATIONAt last, the final text representation 𝑧 is obtained by projecting the text embeddings corresponding to the last token of the 𝐾 𝑡ℎ transformer layer via TextProj(•):EQUATION)The CLIP model is remarkably well-suited for zero-shot prediction tasks. Given an image and a set of corresponding 𝐶 classes, we first extract the image representation 𝑥 by leveraging the image encoder. Subsequently, we construct 𝐶 descriptions based on a handcrafted prompt, such as "a photo of a  
bilateral adaptive cross-modal fusion prompt learning: 
 Prior research [12] has determined that uni-modal adaptation [10, 29, 30] , which involves exclusively learning prompts for either the vision or language branch, results in a sub-optimal solution. This is due to the degradation of CLIP's flexibility in aligning textual and visual representations. Consequently, the primary principle for an optimal prompt learning methodology is to establish a proper alignment for specific downstream tasks. Although MaPLe [12] proposes the first multimodal prompt learning method to investigate crossmodal interaction via a uni-directional coupling function, such as a single layer of MLP, it still remains insufficient in terms of the flexibility. In this section, we first bridge the image and text encoders at each layer through a bi-directional transformation which consists of two light-weight projectors. Then we introduce the adaptive cross-modal fusion function in details.We initially propose a comprehensive framework designed to facilitate bi-directional fusion across multimodal prompts, as illustrated in Fig. 3 . Specifically, at each Transformer layer, a visual prompt 𝑃 𝑣 and a textual prompt 𝑃 𝑙 are concurrently assigned to their respective branches. Both prompts comprise a series of learnable vectors, with the vector length 𝐿 being consistent across 𝑃 𝑣 and 𝑃 𝑙 . We utilize distinct methods to initialize the visual and the textual prompts. In particular, textual prompts are initialized using handcrafted prompts, such as "a photo of a [CLS].", while visual prompts are established through a Gaussian distribution N (0, 0.2). Formally, this can be expressed as:EQUATIONwhere 𝑑 𝑣 and 𝑑 𝑙 denote the dimensions of visual and textual representations, respectively. Projection Functions. To facilitate the transformations between vision and language branches, a pair of projection functions is employed, symbolized as Proj 𝑣→𝑙 (•) and Proj 𝑙→𝑣 (•). The projection functions serve a dual purpose: they facilitate cross-modal transformations and simultaneously align the dimensions of the prompts, thereby enabling the implementation of a cross-modal fusion strategy during the fusion stage.A straightforward approach would be to employ a single MLP, analogous to the coupling function of MaPLe. However, the considerable number of trainable parameters adversely impacts efficiency. As a result, we design a bottleneck-style projection function. For instance, in the case of Proj 𝑣→𝑙 (•), the initial reduction layer 𝑆 𝑟 𝑣→𝑙 compresses the dimension of 𝑃 𝑣 from 𝑑 𝑣 to 𝑑 𝑚 , where 𝑑 𝑚 is significantly smaller than 𝑑 𝑣 . This reduction layer is followed by a non-linear ReLU and an expansion layer 𝑆 𝑒 𝑣→𝑙 , which increases the dimension from 𝑑 𝑚 to 𝑑 𝑙 . At this point, the 𝑉 → 𝐿 transformation is complete and vice versa. As a result, the trainable parameters of Proj 𝑣→𝑙 (•) or Proj 𝑙→𝑣 (•) is merely 𝑑 𝑚 × (𝑑 𝑣 + 𝑑 𝑙 ), which is much smaller than the one of MaPLe, i.e, 𝑑 𝑣 × 𝑑 𝑙 . The computation of projection functions can be represented as:EQUATIONFinally, these functions generate two transformed prompts, called 𝑃 𝑣→𝑙 and 𝑃 𝑙→𝑣 , namely that:EQUATIONConsequently, we propose employing a fusion function, denoted as F (•, •), to generate a synthesis of cross-modal prompts, ultimately achieving mutual interaction between the two modalities. This process can be mathematically represented as follows:EQUATIONHere, 𝑀 𝑣 and 𝑀 𝑙 symbolize the fused prompts, which encapsulate information derived from both visual and language branches, effectively bridging the gap between the two modalities and enhancing the overall performance.A prevalent approach for cross-modal fusion involves the direct addition of bilateral cross-modal information. However, this method remains inflexible, as it presumes uniform interaction information sharing across every layer between the two modalities. Our adaptive cross-modal fusion strategy is predicated on a hypothesis that the adaptation process necessitates varying degrees of additional information from other modalities at different layers. Consequently, we propose an adaptive strategy designed to enable prompt learning to autonomously search for an optimal combination of cross-modal information, thereby offering maximum flexibility.As shown in Fig. 3 , we introduce a learnable gate vector, G 𝑣 ∈ R 2 , for the vision branch, and another learnable gate vector, G 𝑙 ∈ R 2 , for the language branch within each layer. These gate vectors are initialized by uniform distribution, while produce binomial distributions during training, generating convex combinations of multimodal prompts and allowing for flexibility in the search for optimal solutions. This gating function, termed F g , can be represented as:EQUATION)The mixture of cross-modal prompts is subsequently appended to the corresponding image or text tokens. Specifically, at the 𝑖 𝑡ℎ layer V 𝑖 within the image encoder, 𝑀 𝑖 𝑣 is first concatenated with the image embeddings 𝐸 𝑖 -1 and the class token 𝑐 𝑖 -1 from the previous layer. This concatenated output is then fed into V 𝑖 to facilitate 𝐿 → 𝑉 interaction, namely: where [•, •, •] denotes concatenate operation. Conversely, at the 𝑖 𝑡ℎ layer L 𝑖 within the text encoder, 𝑀 𝑖 𝑙 is first concatenated with the text embeddings 𝑊 𝑖 -1 obtained from the (𝑖 -1) 𝑡ℎ layer. This combined output is then fed into L 𝑖 to enable 𝑉 → 𝐿 interaction:EQUATIONEQUATION)As a result, both 𝑃 𝑣 and 𝑃 𝑙 are updated by the language and vision branches simultaneously, thereby enhancing the overall flexibility of tuning process. In general, we apply Bloom to all 12 Transformer layers, resulting in a prompt depth of 12. We also investigate various configurations to evaluate the impact of prompt depth and placement position in experiments. 
experiments 5.1 experimental protocol and datasets: 
 Building upon the foundation laid by previous research [12, 29, 30] , we undertake a comprehensive evaluation of our method's generalization capabilities from three distinct perspectives. i) Base-to-Novel Generalization: Our evaluation of base-to-novel generalization encompasses 11 diverse image classification datasets. These include two generic-objects datasets, namely ImageNet [3] and Caltech101 [5] ; five fine-grained datasets, specifically Oxford-Pets [19] , StanfordCars [13] , Flowers102 [18] , Food101 [1] , and FGVCAircraft [16] ; a scene recognition dataset, SUN397 [27] ; an action recognition dataset, UCF101 [23] ; a texture dataset, DTD [2] ; and a satellite image dataset, EuroSAT [7] . Each dataset's categories are bifurcated into "base" and "novel" classes. The "base" classes serve as the training and evaluation ground for base performance, while the "novel" classes are leveraged to assess the generalization to unseen classes in a zero-shot setting. We present the Top-1 accuracy for both the "base" and "novel" classes, along with their Harmonic Mean (HM). ii) Cross-dataset Generalization: To ascertain the robustness and adaptability of our method across varied data distributions, we employ a zero-shot learning approach. This involves the direct transfer of the optimal model, initially trained on ImageNet, to the aforementioned 10 classification datasets. This rigorous testing process provides a comprehensive understanding of our method's effectiveness across various data landscapes. iii) Cross-domain Generalization: Lastly, we scrutinize cross-domain generalization capabilities of Bloom. This is achieved by transferring the optimal model trained on ImageNet in a few-shot setting, to its four variants, namely, ImageNetV2 [22] , ImageNetSketch [24] , ImageNet-A [9] , and ImageNet-R [8] . This final step of evaluation further solidifies our understanding of the method's adaptability and effectiveness across different domains. We only present the 𝐿 → 𝑉 fusion process, while the 𝑉 → 𝐿 fusion procedure follows a similar approach.EQUATION# % 𝐺 # & 𝑀 # $ 𝑀 # $ 𝑃 !→# $ 𝑃 # $ 𝑃 # $ 𝑃 !→# $We train all models using a single Tesla V100 GPU for 5 epochs, employing a batch size of 4 and a learning rate of 0.0035 via the SGD optimizer. As the foundational model, we select the pre-trained ViT-B/16 CLIP model, in which the dimensions of the visual and language representations, 𝑑 𝑣 and 𝑑 𝑙 , are set to 768 and 512, respectively. We report the Top-1 accuracy for both base and novel classes, as well as their harmonic mean (HM), averaged over three runs with random seeds ∈ {1, 2, 3}. The language prompt 𝑃 𝑙 is initialized using a static prompt, such as "a photo of a [CLS].". It is worth noting that the model trained on all 1,000 classes of Ima-geNet, which is utilized for evaluating cross-dataset generalization, undergoes training for only 2 epochs with a learning rate of 0.0026.In this study, we investigate the base-to-novel generalization capabilities of Bloom across 11 image classification datasets. We employ a basic setting in which the prompt depth is set to 12, signifying the application of Bloom to every layer. The prompt length 𝐿 is configured to 2, and the training is conducted in a 16-shot setting, which entails randomly selecting 16 samples for each category. Additionally, the middle dimension 𝑑 𝑚 within the projection functions is set to 64. The results are presented in Tab. 1. We benchmark the performance of our approach against state-of-the-art prompt learning methods, which include CoOp [30] , CoCoOp [29] , VPT [10] , MaPLe [12] , and the zero-shot CLIP model utilizing a hand-crafted prompt. Specifically, with respect to the harmonic mean (HM) of the accuracy for both "base" and "novel" classes, Bloom surpasses the state-of-the-art methods on all of 11 datasets, with significant improvements in terms of "base" accuracy (+0.96%), "novel" generalization (+1.73%), and their HM (+1.13%), averaged across the 11 datasets. Although Bloom's performance is sub-optimal in some instances, such as a -0.80% decrease in "Base" accuracy on the FGVCAircraft dataset, it demonstrates significant improvements in "Novel" classes and attains the best Harmonic Mean (HM) compared to state-of-the-art methods. This highlights the method's ability to adapt and excel in various scenarios, despite occasional shortcomings.We investigate the cross-dataset generalization capabilities of Bloom by utilizing ImageNet as the source dataset and employing the remaining 10 datasets as target datasets. Following the recommendations from prior research [12] , we adopt a shallow and slim prompt configuration, setting the prompt depth to 2 and 𝐿 = 1. The results, as depicted in Tab. 2, reveal that our Bloom surpasses the stateof-the-art methods on 8 out of 10 datasets in terms of zero-shot accuracy. On average, Bloom achieves an improvement of +0.57%, signifying enhanced generalization performance in comparison to existing state-of-the-art approaches.Analogous to the cross-dataset generalization evaluation, we directly transfer the source model trained on ImageNet to four crossdomain datasets. As illustrated in Tab. 3, the Bloom outperforms state-of-the-art methods on all of the tested datasets. This result underscores the superior generalization capabilities of Bloom in addressing domain shifts, highlighting its effectiveness in adapting to diverse and challenging scenarios.We carry out an ablation study, delving into the effects of crucial designs and hyper-parameters on the ImageNet dataset within the context of a base-to-novel training setting.The Formulation of Fusion Function. To validate the crucial role of the adaptive cross-modal fusion function, we evaluate 3 additional fusion functions that exhibit lower flexibility compared to Bloom: i) Addition (F a ), which entails the addition of corresponding elements from the two prompts. ii) Concatenate (F c ), which involves concatenating the two prompts. iii) Routing (F r ), which is built upon F g (Eq. 10). We propose a routing function that replaces G 𝑣 and G 𝑙 with two binarized gate vectors via a differentiable Gumbelsoftmax function. Consequently, the routing strategy permits only uni-modal prompts across each gate, while preserving the potential for cross-modal interaction from a macroscopic perspective. An illustration of the variations is provided in Fig. 4 . We employ the same basic setting across all variations of the fusion function, with the results depicted in Fig. 5 (a). The superior performance of Bloom is evident, thus corroborating the notion that a more flexible cross-modal interaction leads to a more proper alignment. Middle Dimension. The middle dimension 𝑑 𝑚 within projection functions plays a pivotal role in achieving a better trade-off between enhanced performance and minimal additional learnable parameters. In the basic setting, we set 𝑑 𝑚 to 16, 32, 64, 128, 256, respectively, while maintaining other hyper-parameters constant.As illustrated in Fig. 5 (b), the HM on ImageNet increases as 𝑑 𝑚 becomes larger with diminishing marginal effects. When setting 𝑑 𝑚 to 256, the HM surpasses 80%, but the learnable parameters double, contradicting the desired trade-off objective.Where to insert Bloom? In this study, we examine the impact of varying the insertion position of Bloom. We establish 5 distinct intervals, namely [1, 9), [2, 10), [3, 11) , [4, 12) , [1, 12) prompt is inserted into (𝑏 -𝑎) consecutive layers, spanning from the 𝑎 𝑡ℎ layer to the (𝑏 -1) 𝑡ℎ layer. The other hyper-parameters in basic setting are constant. The results are depicted in Fig. 5 (c).Our findings indicate that as the insertion position shifts further toward the latter layers, there is a gradual decline in performance. Consequently, we deduce the first principle for the usage of Bloom that the earlier the mutual interaction, the better the alignment. This implies that the Bloom plays a more crucial role during the initial stages of CLIP. Incorporating cross-modal fusion at an earlier point in the model leads to improved alignment. Prompt Depth. In this study, we investigate the influence of prompt depth by adjusting it to {1, 2, 5, 8, 12}, while remain the other hyper-parameters in basic setting constant. The results are presented in Fig. 5 (d ). Our analysis reveals a positive correlation between the prompt depth and the HM, indicating an improvement in performance as the depth increases. Consequently, we deduce the second principle for the application of Bloom that the deeper the mutual interaction, the better the alignment. In particular, a more effective Bloom implementation, which integrates the two modalities across multiple layers, promotes enhanced cross-modal alignment. This, in turn, results in an optimized balance between performance on "base" classes and generalization capabilities on "novel" classes. Prompt Length. In our subsequent analysis, we examine the impact of the prompt length 𝐿 in Eq. 6 by adjusting it to {1, 2, 4, 6, 8}. The other hyper-parameters are identical to basic setting. As shown in Fig. 5 (e), our findings indicate that, in the majority of cases, an increase in prompt length results in a decline in the HM. This suggests that employing a larger prompt length may not be optimal. We postulate that the Bloom with an extended length exacerbates the challenges associated with cross-modal alignment due to the absence of adequate prior information. results are illustrated in Fig. 5 (f). Our analysis reveals a favorable effect of data scaling, as evidenced by the increase in the HM when more few-shot samples are incorporated. However, it is worth noting that the benefits derived from augmenting the training dataset exhibit diminishing marginal returns. As the number of training samples increases, the incremental gains in performance become progressively smaller.The Effect of Cross-modal Fusion Direction. We conduct a comparative analysis of the performance of two distinct uni-directional cross-modal fusion strategies, designated as V→L and L→V, and Bloom, denoted as V↔L. The comparison, as illustrated in Tab. 4, reveals that Bloom outperforms both uni-directional strategies. The superiority of Bloom can be attributed to its ability to leverage the mutual information across two modalities, thereby facilitating a more comprehensive and proper alignment. In contrast, unidirectional strategies may not fully capitalize on the potential benefits of integrating information from both modalities, resulting in sub-optimal performance. The Trade-off between Performance and Parameters. In Tab. 5, we compare the HM on ImageNet and additional trainable parameters of Bloom with state-of-the-art approaches, including CoOp [30] , CoCoOp [29] , VPT [10] , and MaPLe [12] . The results demonstrate that Bloom achieves a superior trade-off, requiring only 1.8% of additional CLIP parameters while attaining a 79.68% HM.The Search Decision of Gate Vectors. We visualize the gate vectors (after softmax) within the fusion functions of both the vision and language branches in Fig. 6 . This intuitively showcases Bloom's varying requirements for mutual information from the two different modalities across distinct layers. Moreover, this observation corroborates our hypothesis, suggesting that providing prompt learning with greater autonomy positively impacts its performance. 
conclusions: 
 In this paper, we introduce a new paradigm, termed Bloom, for prompt learning community. Bloom exhibits more flexibility for cross-modal interaction via a bilateral cross-modal fusion framework and an adaptive fusion function ensuring Bloom to search optimal combination of interaction information, ultimately resulting in new state-of-the-art performance in terms of base-to-novel generalization, cross-dataset, and cross-domain generalization.  
 
 Appendix: & 𝑃 !→# & 


Decision: This paper introduces a cross-modal fusion prompt learning method for CLIP. The original reviews are two weak accept, one borderline accept, and one borderline reject. The main concerns center around the novelty of the proposed method. The rebuttal successfully addressed some of the concerns. The authors are encouraged to incorporate all the comments in the final version.

Reviews: review: paper_summary: paper_summary: This article proposes a simple  Bilateral Adaptive Cross-Modal Fusion method. It implements prompt tuning of CLIP by implanting learnable vectors and projection modules between the two encoders of CLIP, and achieves good results.
strength_weakness: strength_weakness: 1. This paper is clearly written and easy to understand.
2. The method is simple and effective, and its sufficient generalization ability has been verified on a variety of tasks.
3. A large number of experiments fully explored the effectiveness and versatility of this method.
limitations: limitations: 1. Although it does not use complex distillation or ensemble operations, the performance of this article does not exceed prompt tuning methods such as PromptSRC. Can you try to add your method on PromptSRC?
2. This article seems to be relatively uninnovative. It is just a simple replacement of the fusion module on methods such as MaPLe.
3. The H values in Table 4 look different from those in the other charts, where H usually reaches 79.68, and only 74.08 in Table 4. It is expected that you can align the experimental settings.
4. Although this is not necessary, it is recommended to add time and space metrics such as FLOP and Memory to enrich Bloom's efficiency comparison with other methods.
5. I would like to know why Bloom has fewer parameters than MaPLe? Bloom implanted Prompt at more layers and learned bidirectional mapping, which appears to have more parameters.
paper_summary: paper_summary: This paper proposes Bilateral Adaptive Cross-Modal Fusion Prompt Learning (Bloom), a new paradigm for the CLIP prompt learning community. Bloom enhances cross-modal interaction through a bilateral fusion framework and an adaptive fusion function, ensuring optimal interaction combinations. It features two main enhancements: projection functions for bi-directional modality transformation and fusion functions for layer interaction within image and text encoders, and an adaptive method to find the best combination of cross-modal information at each layer. Extensive experiments on 15 image classification datasets demonstrate its effectiveness.
strength_weakness: strength_weakness: 1. This paper is well-written.
2. The organization of the paper is logical.
limitations: limitations: 1. From my point of view, the proposed bilateral adaptive cross-modal fusion prompt learning is more like a variant of the MaPLe and similar to its architecture, this may weaken the novelty of the proposed method. The authors are expected to further explain and refine their motivation and contribution.
2. It is better to provide the image embeddings of the proposed method via t-SNE or UMAP and compare it with previous methods such as MaPLE and Co-CoOp.
paper_summary: paper_summary: The paper presents Bilateral Adaptive Cross-Modal Fusion Prompt Learning (Bloom), a novel approach for improving the adaptation of the CLIP model to downstream tasks. Bloom enhances cross-modal alignment by introducing bi-directional fusion of visual and textual information across both the image and text encoders. It employs adaptive fusion functions that automatically determine the optimal integration of cross-modal data at each layer, leading to a more comprehensive alignment. Experimentally, Bloom surpasses existing state-of-the-art methods in three critical areas: base-to-novel generalization, cross-dataset, and cross-domain evaluations.  The work validates the importance of early and deep cross-modal interactions for better alignment and generalization, and visualizes gate vectors to showcase Bloom's dynamic adjustment of cross-modal information requirements. Overall, Bloom represents a significant step forward in prompt learning for CLIP, offering a more flexible and efficient way to leverage cross-modal representations in diverse applications.
strength_weakness: strength_weakness: 1)	Importance of key insights: the paper proposes an innovative Bloom approach that addresses the lack of adaptability of existing CLIP models through bi-directional adaptive cross-modal fusion;.
2)	Experimental or theoretical validation: through exhaustive experimental design, including cross-dataset, cross-domain and base-to-new generalisation evaluation, as well as meticulous ablation studies, the effectiveness of Bloom's approach is empirically demonstrated, proving that more flexible cross-modal interactions enhance the model's generalisation capability;.
3)	Writing quality: the article is well-organised and logical, detailing the principles of the method, experimental design and analysis of the results, as well as being rich in graphs and charts.
4)	Data contribution: comprehensive evaluation data on 15 image classification datasets are provided, demonstrating the significant advantages of Bloom over existing top methods.
limitations: limitations: 1)	The paper discusses less about the theoretical basis of the proposed two-way fusion function and adaptive strategy；
2)	Lack of sufficient implementation details and hyperparameter settings in the paper；
paper_summary: paper_summary: This paper introduces a new paradigm for prompt learning, termed Bloom. Bloom exhibits more flexibility for cross-modal interaction via a bilateral cross-modal fusion framework and an adaptive fusion function ensuring Bloom to search optimal combination of interaction information, ultimately resulting in new SOTA performance in terms of base-to-novel generalization, cross-dataset, and cross-domain generalization.
strength_weakness: strength_weakness: 1. The idea is clear, simple, yet effective.
2. The writing is clear and logical.
3. The experiments are solid and sufficient, demonstrating the generalization ability of the proposed method.
limitations: limitations: 1. The training GPU memory usage should be discussed in the discussion of trade-off.
2. Fig.6 is hard to understand.
3. Though on different tasks, the idea of bi-directional projection of prompt fusion was proposed in PMF [1], which reduces the novelty of this paper.

[1] Yaowei Li, Ruijie Quan, Linchao Zhu, and Yi Yang. 2023. Efficient Multimodal Fusion via Interactive Prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2604–2613.