Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-22 03:50:28

Abstract: Cross-lingual cross-modal retrieval aims to retrieve visually relevant content based on non-English queries, without relying on human-labeled cross-modal data pairs during training. One popular approach involves utilizing machine translation (MT) to create pseudo-parallel data pairs, establishing correspondence between visual and non-English textual data. However, aligning their representations poses challenges due to the significant semantic gap between vision and text, as well as the lower quality of non-English representations caused by pre-trained encoders and data noise. To overcome these challenges, we propose LECCR, a novel solution that incorporates the multi-modal large language model (MLLM) to improve the alignment between visual and non-English representations. Specifically, we first employ MLLM to generate detailed visual content descriptions and aggregate them into multi-view semantic slots that encapsulate different semantics. Then, we take these semantic slots as internal features and leverage them to interact with the visual features.  By doing so, we enhance the semantic information within the visual features, narrowing the semantic gap between modalities and generating local visual semantics for subsequent multi-level matching. Additionally, to further enhance the alignment between visual and non-English features, we introduce softened matching under English guidance. This approach provides more comprehensive and reliable inter-modal correspondences between visual and non-English features. Extensive experiments on two cross-lingual image-text retrieval benchmarks, Multi30K and MSCOCO, as well as two cross-lingual video-text retrieval benchmarks, VATEX and MSR-VTT-CN, demonstrate the effectiveness of our proposed method.

Main paper: 
 introduction: 
 Cross-lingual cross-modal retrieval (CCR) aims to develop models that can retrieve relevant visual context based on non-English queries, without relying on human-labeled non-English cross-modal data pairs. Compared to traditional cross-modal retrieval, CCR goes beyond the limitations of English and can transfer to other languages. Currently, most research [23, 34-36, 47, 50] in this field resorts MT to generate pseudo-parallel data pairs, as depicted in Figure 1 (a). These studies have achieved remarkable performance by establishing direct correspondences between visual and non-English data. However, the representation quality of non-English captions tends to exhibit inferior performance compared to English data, which can be attributed to the limitations of pre-trained text encoders when handling non-English languages and the quality of translations. Consequently, aligning visual features with non-English features poses a significant challenge. To overcome this challenge, some methods [23, 27, 47, 50] adopt a single-stream structure (as shown in Figure 2 (a)), utilizing cross-modal fusion modules to capture fine-grained interactions between the modalities. While these methods have demonstrated significant performance improvements, they suffer from increased computational cost and inference time. This is because all possible query-candidate pairs need to be Comparison of CCR methods. Among them, "𝐹 𝑣 ", "𝐹 𝑐 ", "𝐹 𝑠 ", and "𝐹 𝑡 " represent the visual encoder, description encoder, English encoder, and non-English encoder, respectively. Our method follows the two-stream structure and incorporates MLLM to enhance the semantics within visual features, helping bridge the gap between modalities. Additionally, to improve the alignment between visual and non-English features, we propose employing English representations as guidance. This can provide more comprehensive and reliable inter-modal correspondence for visual and non-English features.processed by the fusion modules to calculate similarity scores. In contrast, alternative approaches [34] [35] [36] employ a more efficient dual-stream structure (as shown in Figure 2 (b)), where similarity is calculated using the dot product of their global features. They improve the cross-lingual cross-modal alignment by introducing noise-robust designs. However, these methods extract visual and linguistic features independently without any information fusion, leading to a persistent semantic gap between modalities. Additionally, the image (or video) often conveys richer content than text, as the saying goes, "a picture is worth a thousand words. " The captions in existing datasets are typically brief and may only capture partial visual information (see Figure 1 (a)). Therefore, relying solely on global features extracted independently may not fully achieve semantic alignment between visual and non-English features.Considering the exceptional capabilities demonstrated by MLLMs such as GPT4 [1] and videochat2 [19] in multimodal understanding, we argue that we can generate complementary contextual descriptions using MLLM to enhance the semantics within visual features (as depicted in Figure 1 (b) and Figure 2 (c) ). This approach enables us to bridge the semantic gap between modalities more effectively. Similarly, other works [25, 38, 40, 41, 44, 45] have also incorporated MLLMs to enhance existing vision-language models. However, these works simply utilize the [CLS] token or all descriptions to interact with visual features, potentially resulting in incomplete context information or high computational costs. Therefore, it is crucial to further investigate how to effectively utilize these visual descriptions to enhance visual representations, thereby improving the alignment between visual and non-English features.To address the aforementioned challenges, we propose LECCR (Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval), a novel two-stream solution. Specifically, given the descriptions generated by MLLM, we first aggregate these descriptions into the multi-view semantic slots, and incorporate the regularization loss to force these semantic slots to focus on the different semantics present in descriptions (e.g., different objects in the image). Subsequently, we introduce a multi-view visual-semantic interaction module to interact these semantic slots with the visual features. This module not only enhances the semantics within visual features but also generates local visual context semantics, enabling multi-level cross-lingual cross-modal matching and effectively narrowing the semantic gap between modalities. Finally, we use English features as guidance to establish a more comprehensive correspondence between visual and non-English features.The proposed method, LECCR, is evaluated and compared with previous work on two text-image retrieval benchmarks, Multi30K and MSCOCO, as well as two text-video retrieval benchmarks, VATEX and MSR-VTT-CN. Our method consistently outperforms previous approaches in the majority of evaluation settings, highlighting its effectiveness in CCR task. Our contributions can be summarized as follows: (1) We propose a new two-stream CCR solution, LECCR, which incorporates MLLM to improve the alignment between visual and non-English features. (2) To bridge the semantic gap between modalities, we utilize detailed visual descriptions generated by MLLM and aggregate them into multi-view semantic slots to enhance visual features. We then introduce multilevel matching and softened matching under English guidance to improve the alignment between visual and non-English features. (3) We conduct extensive experiments on four image-text and videotext cross-modal retrieval benchmarks across different languages, demonstrating the effectiveness and potential of our method. 
related work 2.1 cross-lingual cross-modal retrieval (ccr): 
 CCR has been attracting growing interest among researchers. Compared with the traditional cross-modal retrieval [3, 7, 11, 14, 16-18, 20, 24, 37, 49] , this approach offers a highly efficient and costeffective solution for non-English-based retrieval, reducing the reliance on human-labeled data. Existing methods can be grouped into two categories in terms of the model architecture. The first approach [23, 27, 47, 50] adopts a single-stream structure that incorporates cross-lingual cross-modal fusion modules to model both image regions and multilingual text word representations in a unified semantic space, capturing the fine-grained relationship between them. For instance, Ni et al. [27] use a code-switch strategy and masked modeling loss to model the interaction between vision and multiple languages. However, this approach includes an extra crossmodal fusion module, which may lead to computation overhead and slower inference speeds. Therefore, it may not be practical for large-scale CCR tasks in real-world settings.The second approach [13, 15, [34] [35] [36] involves a two-stream structure, where each stream is dedicated to modeling either the vision or language inputs. For instance, Jain et al. [15] use scalable dual encoder models trained with contrastive losses to learn encoders for both language and images, combining image-text matching and text-text matching tasks. This approach maps the global features of different modalities into a common semantic space to learn crossmodal alignment. However, despite its efficiency, this approach still encounters challenges due to the absence of explicit cross-modal interaction. Additionally, existing dataset captions are typically brief and may only capture partial visual information. The information imbalance between two modalities further exacerbating the difficulty in aligning their features. In this paper, we adopt the twostream structure and incorporate the MLLM to provide additional contextual semantics for visual features. This can help narrow the semantic gap between modalities and further improve alignment between visual and non-English features.With exceptional language understanding capabilities, LLMs [2, 30, 31, 31] have exhibited remarkable capabilities in a variety of tasks, such as image captioning, and visual question answering. Recently, a new line of work has emerged [25, 28, 38, 40, 41, 44, 45] that incorporates them to enhance the vision-language models (VLMs). For example, in the classification task, menon et al., [25] and pratt et al., [28] leverage the LLMs to generate the textual descriptors for each category, comparing images to these descriptors rather than estimating the similarity of images directly with category names. Similarly, Yang et al., [41] use GPT-3 along with text descriptions of images for the Visual Question Answering (VQA) task. In this work, we extend the descriptive LLMs to generate visual context descriptions in a CCR task. Another relevant work in cross-modal retrieval, Wu et al. [38] , employs LLMs to generate auxiliary captions to enhance text-video matching. However, their approach primarily utilizes auxiliary captions for data augmentation and simply interacts with visual features using [CLS] embeddings. In contrast, our objective is to leverage rich descriptions to offer semantic context for visual features. Moreover, we introduce multi-view semantic slots to comprehensively represent the description content, thereby providing contextual semantic information for visual features. 
methods: 
 Figure 3 shows an overview of our approach. In what follows, we will briefly describe the CCR definitions and our Baseline method (in Section 3.1). Then, we present our method LECCR, including multiview semantic slots generation (in Section 3.2), multi-view visualsemantic interaction (in Section 3.3), multi-level matching (in Section 3.4), and softened matching under English guidance (in Section 3.5), respectively.The objective of the CCR task is to retrieve the relevant visual content (i.e., image or video ) using the non-English query (see Figure 1 (c)), while solely relying on annotated paired visual-English sample pairs during training. Following previous studies [34, 36, 47] , we employ the MT to generate the translated captions based on the English captions. This enables us to construct a dataset consisting of triplet sample pairs D = (𝑉 , 𝑆,𝑇 ), where 𝑉 , 𝑆, and 𝑇 represent the visual items (e.g., images or videos), English captions, and non-English captions, respectively. Then, we take them as the input, and use the vision encoder F 𝑣 , English encoder F 𝑠 , and non-English encoder F 𝑡 to extract the corresponding sequential representationsZ 𝑣 ∈ R 𝑁 𝑣 ×𝑑 𝑣 , Z 𝑠 ∈ R 𝑁 𝑠 ×𝑑 𝑠 , and Z 𝑡 ∈ R 𝑁 𝑡 ×𝑑 𝑡 , where 𝑁 𝑥 ∈ {𝑣,𝑠,𝑡 }denotes the length of each sequence, and 𝑑 𝑥 ∈ {𝑣,𝑠,𝑡 } denotes the channel dimension. Finally, we project them into a multi-lingual multi-modal common space. This process can be represented as:EQUATIONEQUATIONwhere 𝜙 (•) denotes the linear projection function used to project the features into a common space, Z 𝑐𝑙𝑠 𝑥 is the [CLS] features, and h 𝑥 ∈ R 𝑑 denotes the corresponding latent features in the common space, where 𝑥 ∈ {𝑣, 𝑠, 𝑡 }.Following this, we introduce the contrastive loss to pull the paired samples closer to each other and push the non-paired samples away from each other. It can be defined as:EQUATIONAmong them, the contrastive loss L 𝑐𝑡𝑟𝑎 can be formulated as:EQUATIONwhere 𝐵 represents the mini-batch size, 𝜏 represents the temperature coefficient, and𝑆 𝑔 (a 𝑖 , b 𝑗 ) = a 𝑖 𝑇 b 𝑗 | |a 𝑖 | | • | |b 𝑗 | |represents the similarity function to calculate the similarity between the 𝑖-th feature vector a and the 𝑗-th feature vector b. The final objective can be calculated as:L 𝑏𝑎𝑠𝑒 = L 𝑣𝑠 + L 𝑡𝑠 + L 𝑣𝑡 .During the inference, we calculate the similarities 𝑆 𝑔 (h 𝑣 , h 𝑡 ) between the visual features h 𝑣 and the non-English features h 𝑡 to perform retrieval. Note that we take the above process as our Baseline method.In this section, we aim to aggregate visual descriptions generated by MLLM into the multi-view semantic slots, enabling us to capture diverse semantics within them. This is substantially different from existing methods that only utilize the [CLS] token or all representations, which may not fully leverage the contextual information in the descriptions. Specifically, given the images (or videos), we feed them into the MLLM with the prompt, such as "### Please describe the contents of this image in detail." to generate the rich visual descriptions 𝐶 ( as illustrated in Figure 4 ). It is worth noting that our visual descriptions are generated in English, and no additional translations are introduced, avoiding any extra loss in quality. Then, we extract description embeddings Z 𝑐 ∈ R 𝑁 𝑐 ×𝑑 𝑐 using the description encoder F 𝑐 and employ the 𝑁 𝑞 learnable queries that are randomly initialized to aggregate them. This aggregation process enables us to generate multi-view semantic slots Visual-Semantic InteractionLearned Queries Q 𝑚𝑎𝑥 q k v v q Interaction Non-English caption Generation 𝐹 $ Interaction 𝑍 ! " 𝑍 ! # 𝑀 𝑀 Image Description 𝐹 &The image shows a plate of food on a 𝐹 - MLLM + ℎ ! ℎ # ℎ " ℎ # ℎ # ℎ " + ℎ ! + ℎ !Figure 3 : Overview of the proposed LECCR framework. We utilize the multi-modal large language model (MLLM) to generate detailed visual descriptions, which are then employed as internal features to enhance the visual representations. Additionally, we introduce multi-level matching and softened matching under English guidance to improve the alignment between visual and non-English representations.The video depicts a news report on a restaurant event, with the news anchor discussing the event. Delicious food is displayed on the table. The chef is seen cutting the food and the news anchor is seen eating the food. M ∈ R 𝑁 𝑞 ×𝑑 , where each view encapsulates distinct semantics of input descriptions. Mathematically, this can be represented as:EQUATIONM = 𝐿𝑁 (𝜙 𝑞 ( Q)) + Q (7)where 𝑀𝐻𝐶𝐴(, , ) represents the multi-head cross-attention module, 𝜙 𝑐 (•) and 𝜙 𝑞 (•) represent the linear projection functions, and 𝐿𝑁 (•) represents the layer norm.Additionally, to prevent the generated multi-view semantic slots from overly focusing on the same semantic features, we introduce a multi-view regularization loss to encourage diversity among the slots. This objective can be formulated as:EQUATIONwhere𝑝 𝑖 𝑗 = 𝑒𝑥𝑝 (m 𝑖 𝑗 𝑇 m 𝑖 𝑗 ) 𝑁𝑞 𝑘=1 𝑒𝑥𝑝 (m 𝑖 𝑗 𝑇 m 𝑖𝑘 )represents the similarity distribution between views, and m 𝑖 𝑗 denotes the 𝑗-view semantic slot of the 𝑖 -𝑡ℎ sample.We propose a vision-semantic interaction module that uses the above multi-view semantic slots as the internal representation to bridge the semantic gap between modalities. This module serves two primary purposes: (1) semantic slots to vision (C2V), which provides additional contextual semantic information to enhance the semantics within visual features; (2) vision to semantic slots (V2C), enabling the multi-view semantic slots to capture their corresponding visual information and generate local contextual visual semantics. Specifically, we take the visual representations Z 𝑣 and the multi-view semantic slots 𝑀 as inputs to the dual attention block. This module enables us to generate two outputs: semanticenhanced visual features Ẑ𝑣 and local contextual visual semantics M. Here, we present two alternative options for the interaction as below:(1) Dual cross-attention: we utilize two cross-attention blocks, where the visual features Z 𝑣 and multi-view semantic slots M are used as queries, respectively. These blocks then aggregate information from the features of each other. It can be formally defined as:EQUATIONEQUATION)where 𝜙 𝑧 (•) and 𝜙 𝑚 (•) represent the linear projection functions.(2) Co-attention: we concatenate the visual features Z 𝑣 and semantic slots M together and process them through a self-attention block:EQUATIONEQUATION)where 𝑀𝐻𝑆𝐴(, , ) represents the multi-head self-attention, and [; ] represents the concatenate operation along the length dimension of the sequence.After the vision-semantic interaction, the multi-view semantic slots aggregate the corresponding local contextual visual features, which can be considered as the local features. Next, we introduce multilevel matching, including caption-slots matching (local level) and caption-vision matching (global level), to facilitate cross-lingual cross-modal alignment.Caption-slots matching. Considering that semantic slots from each view may be associated with different aspects of the visual content, there could be significant semantic disparities between the different views. Hence, we align the caption with the slot whose semantics are most closely related, instead of imposing strict alignment constraints on all view semantic slots. To achieve this, the similarity scores between the caption and the semantic slots are computed as follows:EQUATIONwhere S 𝑙 (h 𝑖 𝑥 , m𝑗 ) represents the similarity score between the 𝑖-th caption and the 𝑗-th multi-view semantic slots. Then, the captionslots matching objective L 𝑐 can be formulated as:EQUATIONEQUATIONEQUATIONCaption-vision matching. Similar to Equation 3, in this case, we utilize the semantic-enhanced visual features Ẑ𝑣 to align with the caption features, represented as:EQUATIONEQUATIONwhere ĥ𝑣 = 𝜙 𝑣 ( Ẑ𝑐𝑙𝑠 𝑣 ) represents the global visual features.The ground-truth labels used in Equation 3 are hard one-hot labels, which assume no correlation between unpaired samples. This approach assigns equal weights to all negative samples, disregarding the potentially valuable inter-modal relationships. This makes aligning visual and non-English features more challenging. To address this issue, we propose utilizing English features as guidance for non-English ones. Our goal is to use the visual-English similarity as a softened target to guide the alignment between visual and non-English features. This softened target helps establish comprehensive relationships between modalities. To capture the relationships between modalities more effectively, we calculate vision-English similarity at multiple levels, including both local and global levels.We then integrate these similarities to generate softened targets that direct the non-English features. This process can be represented mathematically as follows:EQUATION)EQUATIONwhere 𝑌 ∈ R 𝐵×𝐵 represents the softened targets, and 𝛼 represents the weight parameter. Following, we use the KL-Divergence to supervise the visual-non-English correspondence:EQUATIONEQUATIONFinally, the matching between visual and non-English features in Equation 17 can be modified as:EQUATION)where 𝜆 represents the weight parameter.Training. The final training objective can be formulated as:EQUATION)where 𝜇 1 and 𝜇 2 represent the loss weights. In the experiments, we set them to 0.1 and 0.01, respectively.Inference. After training the model, given a sentence query in non-English, we sort candidate videos/images in descending order based on their similarity scores with the query. Specifically, we first compute the similarity between visual features ĥ𝑣 and query features ℎ 𝑡 , as well as between semantic slots M and query features ℎ 𝑡 . Then, we combine the two scores to obtain the final similarity score. This can be formulated as:EQUATION)where 𝛽 denotes the weight parameter, which we set to 0.8 in our experiments. 
experiment 4.1 experimental settings: 
 Datasets. We conduct experiments on two public multilingual image-text retrieval datasets (Multi30K [10] ) and MSCOCO [6] ), as well as two video-text retrieval datasets, VATEX [32] and MSR-VTT-CN [34] . Notably, we only use the annotated vision-English data pair in the training process, while using the non-English query (human labeled) to evaluate. Following previous work [34] , the MT we adopt the Google translator.• Multi30K [10] : This dataset consists of 31,000 images and is a multi-lingual version of Flickr30K [43] . It involves four languages, i.e., English(en), German(de), French(fr), and Czech(cs). For the dataset partition, we split the data into train/dev/test sets in a 29000/1000/1000, similar to [43] .• MSCOCO [6] : This dataset consists of 123,287 images, and each image has 5 captions. We translate the training set from English into Chinese(zh) and Japanese(ja) by resorting to MT, and using the test sets from the [21] and [42] , respectively. We follow the data split as in [50] .• VATEX [32] : VATEX is an extensive video-text retrieval dataset that provides bilingual captions for over 41,250 videos. Each video is paired with 10 English sentences and 10 Chinese sentences that describe its contents in detail. Similar to the approach taken in [5, 9] , we train our models on 25,991 video clips, while we reserve 1,500 clips each for validation and testing. • MSR-VTT-CN [34] : MSR-VTT-CN is the multilingual version of the MSR-VTT, which covers English and Chinese. We follow the partition of [46] containing 9,000 and 1,000 for the training and testing, respectively.Evaluation metrics. For video-text retrieval, we follow the previous works [22, 33] , and use rank-based metrics, namely 𝑅@𝐾 (𝐾 = 1, 5, 10), and the sum of all Recalls (SumR) to evaluate the performance. 𝑅@𝐾 is the fraction of queries that correctly retrieve desired items in the top 𝐾 of ranking list. Higher 𝑅@𝐾 and mAP mean better performance. For image-text retrieval, we only report the SumR.Implementation details Following [34] , we use the CLIP [29] to extract the image representations, and use mBERT [8] to extract the text representations. For video features, on MSR-VTT-CN, we use the frame-level features ResNet-152 [12] and concatenate framelevel features ResNeXt-101 [39] [26] to obtain a combined 4,096dimensional feature. On VATEX, we adopt the I3D [4] video feature which is officially provided. For all experiments, we train the modal for 40 epochs with a cosine decay scheduler and an initial learning rate of 1 × 10 -5 . We use the videochat2 to generate the visual descriptions for images and videos. Besides, all text encoders are parameters shared.We conducted a comprehensive evaluation of our LECCR method compared to state-of-the-art approaches on two widely used imagetext datasets, Multi30K and MSCOCO. It is worth noting that M 3 P, UC 2 , UMVLP, CCLM, MURAL, and MLA were pre-trained on largescale vision-language datasets, while NRCCR, DCOT, CL2CM, and Table 2 : Cross-lingual video-text retrieval results on VATEX (en2zh). *: Model pre-trained on a large-scale dataset Multi-HowTo100M [13] .Method T2V V2T SumR R@1 R@5 R@10 R@1 R@5 R@10 MMP w/o pre-train [13] our method do not require additional pre-training data. In Table 1 , we can observe that our LECCR achieves better performance than all two-stream methods. Specifically, compared to the baseline method CL2CM, our approach demonstrates improvements of 1.4%, 1.8%, 1.9%, 2.6%, and 3.4% on all languages in terms of sumR. Unlike the single-stream models that incorporate cross-modal fusion modules to capture detailed interactions between image regions and text words, our LECCR incorporates LLM to provide additional semantic context information for visual features. This can narrow the gap between modalities and improve the alignment between visual and non-English representations. Note that our method is xx faster than CCLM in inference times. Moreover, when equipped with a stronger backbone pre-trained on large-scale datasets, our LECCR † achieves comparable performance to the single-stream method CCLM. Method T2V V2T SumR R@1 R@5 R@10 R@1 R@5 R@10 NRCCR [34] The experimental results on the VATEX and MSR-VTT-CN datasets are reported in Table 2 and Table 3 . We can see that our method LECCR outperforms all baseline methods using a two-stream structure. This further demonstrates the effectiveness of our approach in incorporating LLM to provide additional contextual semantic information. Additionally, even when compared to the MMP trained on large-scale multi-lingual multi-modal datasets, our method still achieves superior results.In this section, we provided detailed ablation studies on Multi30K to verify the effectiveness of each part of our design.Analysis of each component. To validate the effectiveness of the components within our model, we conduct ablation experiments on them. As shown in Table 4 , we observed consistent improvements in performance as the components were incrementally added. Specifically, the introduction of multi-view semantic slots and their corresponding interaction resulted in performance increases of 2.8%, 1.6%, and 1.2%, respectively. This clearly demonstrates that the complementary context information provided by MLLM helps bridge the gap between modalities. Furthermore, when we incorporated multi-level matching and English guidance, we observed a significant performance gain. Analysis of multi-view visual-semantic interaction. In Table 5, we investigate various interaction modules mentioned in Section 3.3. Both modules employ a transformer structure and utilize the multi-head self-attention mechanism to fuse visual and semantic information. The results indicate that the dual cross-attention module achieves superior performance, and we use it in the following experiments. Furthermore, we also conduct the experiment to analyze the interaction direction. The results reveal that dual interaction leads to significant performance improvements. This is because dual interaction not only enhances the semantic information within visual features, but also generates local visual semantics that can be utilized for multi-level matching. In summary, the results demonstrate that the multi-view visual-interaction module can lead to better visual representations and improved alignment between visual and non-English features.Analysis of soft matching under English guidance. In Table 6, we conduct the experiment to investigate the effectiveness of soft matching and analyze the impact of the source of the softened targets. Notably, we observed a substantial performance drop when soft matching was removed. This suggests that the rich relations between samples are important to improve the alignment between visual and non-English features. Moreover, the semantic slots after interaction integrate local visual contextual semantics, enabling the capture of more detailed local visual information and helping to provide the better guidance. The results demonstrate that the combination of 𝑆 𝑚 with 𝑆 𝑔 further improves the performance.Analysis of the description representation. We conducted a comparison between our proposed multi-view semantic slots and other methods for extracting description representations, namely CLS, Mean, and All. As shown in Table 7 , our method consistently outperforms the others. Using the [CLS] token and mean pooling methods both extract global information from descriptions, but this may lead to a loss of rich contextual semantics. Consequently, it becomes challenging to provide comprehensive semantic information for visual features. On the other hand, directly employing all description representations may result in information redundancy and make it difficult to extract useful and diverse key information.Table 7 : Ablation study of the extraction of MLLM-generated descriptions. "CLS" denotes using the [CLS] token of description representations; "mean" denotes using the mean pooling applied to the description representations; "All" denotes using the all description representations; and "Multi-view" denotes our proposed multi-view semantic slots. In contrast, our proposed multi-view semantic slots effectively aggregate diverse semantics within descriptions. This enables us to capture rich contextual semantics of visual representations in the subsequent interaction module.Analysis of the description encoding. In Table 8 , we perform an ablation study to investigate different approaches to encoding the description representation. The results indicate that utilizing a frozen text encoder to extract the description representation leads to inferior results. In contrast, we find that the best performance is achieved when using shared parameters with the caption encoder. We hypothesize that this is due to the significant distributional differences between description and caption features when using a frozen text encoder. Therefore, we adopt the shared text encoder for description representation encoding, and it does not incur any additional computational costs.Analysis of the number of semantic slots. In Figure 5 , we investigate the impact of the number of semantic slots on performance. As shown, performance is best when 4 views are used. WhenAttention Map more views are added, performance slightly drops. We hypothesize that this is because too many views may lead to information redundancy.In Figure 6 , attention maps from the last attention layer in the multi-view visual-semantic interaction module (V2C) are visualized. These attentions are designed to capture visual information corresponding to the semantic slots. Through this interaction module, we obtain diverse local visual semantics that specifically focus on different objects or contexts. These semantics are then utilized in subsequent multi-level alignment objectives to improve the alignment between visual and non-English features. As shown, each slot focuses on distinct objects in the images, showcasing the diverse semantics captured by the multi-view semantic slots. For instance, in the third line, the semantic slots highlight the "cat", "oranges", and "cake" respectively in the image. The visualization results provide further evidence of the effectiveness of the interaction module in capturing and integrating related visual information. 
summary and conclusions: 
 In this paper, we present LECCR, a novel two-stream solution for the CCR task. Our approach aims to bridge the gap between modalities by incorporating the multi-modal large language model (MLLM) to generate detailed visual descriptions. These descriptions serve to provide additional contextual semantics for the visual representations. Additionally, considering the lower quality of non-English representations, we utilize English representations as guidance to improve the alignment between visual and non-English features by providing comprehensive inter-modal relationships. Extensive experiments demonstrate that LECCR significantly improves the alignment quality between vision and non-English features on various benchmarks.  
image image: 
 Description: The image shows a plate of food on a table with two glasses of beer. The plate has a sandwich on it, and there are vegetables and a tomato on the sandwich. The table is set up in a restaurant, and there is a napkin on the table as well.  
 
 Appendix:  


Decision: **Conclusion: Accept as a Poster Paper**

After a thorough evaluation of the four independent reviews, I recommend accepting this sumission as a Poster Paper for ACM MM 2024. The paper proposes a novel LECCR framework for cross-lingual cross-modal retrieval, which effectively addresses the modality gap between non-English and visual features using a multimodal large language model (MLLM). This framework's ability to enhance semantic information and improve alignment between modalities across different languages makes it a valuable contribution to the field.

**Strengths:**

1. **Innovative Approach**: The paper introduces a dual-stream solution leveraging MLLM to generate detailed visual descriptions, enhancing the semantic information within visual features. This innovative approach contributes significantly to multimedia/multimodal processing.
   
2. **Comprehensive Experiments**: Extensive experiments across multiple benchmarks, including image-text and video-text retrieval tasks, validate the method's effectiveness, universality, and robustness. The comparisons with existing methods and ablation studies further confirm the method's superiority.

3. **Detailed Analysis and Visualizations**: The paper includes thorough evaluations and visualizations, such as attention maps, which illustrate how the proposed method focuses on local semantics within images. This adds depth to the understanding and validation of the model.

**Weaknesses:**

1. **Complexity and Reproducibility**: The model training process involves seven different objective functions, making it complex and potentially difficult to reproduce. There is also a lack of comprehensive ablation experiments to clarify the contribution of each loss function.
   
2. **Evaluation Consistency**: The method's performance on certain benchmarks, such as Multi30K, is inconsistent compared to others like MSCOCO. This raises questions about the robustness and generalizability of the approach across different datasets.

3. **Dependency on MLLM Quality**: The effectiveness of the proposed method depends on the quality of MLLM-generated descriptions, which may not always be accurate. This could affect the universality and reliability of the results.

In summary, despite the identified weaknesses, the strengths of the paper, particularly its innovative approach and comprehensive evaluation, justify its acceptance as a Poster Paper. The proposed LECCR framework offers a promising solution to cross-lingual cross-modal retrieval, making a notable contribution to the multimedia research community. Further refinement and additional evaluations could enhance the paper's impact and reproducibility, but its current state is sufficiently strong for presentation at ACM MM 2024.

Reviews: review: paper_summary: paper_summary: The strengths of the paper include understanding alignment among non-English and visual features by identifying a modality gap in cross-lingual cross-modal retrieval scenarios, proposing the LECCR framework for generating visual descriptions, conducting baseline evaluations using a set of pre-trained large multimodal models and showing how inter-modal relationships among non-English and visual features shape alignment via softened matching.
strength_weakness: strength_weakness: - The idea of multi-view visual-semantic interaction is promising for characterizing the alignment among non-English and visual features. And, so the cross-lingual cross-modal alignment via multi-level matching to consolidate both local and global features.
- Careful utilization of the single-stream and two-stream methods, and so the softened matching is an interesting way to guide the alignment among visual and non-English features.
limitations: limitations: - In Figure 1(b), the text feature in EN for guidance is a highly abstract caption, which reflects that the visual element slots attend to capture distinctive semantics, which wholly carries for retrieving related. Also, given the KL-divergence formulation in Eq. 21 and Eq. 22, it is plausible for one language to cater supervision but it doesn’t satisfy the invariance measure if more than one non-English language is used. The basic assumption should lie in the fact that it approximates the semantic distribution of each language which is not the case here. For instance, the textual captions are less semantic (such as the one in Figure 1(b) for EN/non-EN description which misses the information about a glass of beer).
- In the formulated similarity function in Eq. (8), authors should justify whether their multi-view regularization loss supports multiple instance learning [1] which is utilized in a cross-modal retrieval case [2]. Though, [1] lacks sparse supervision which is resolved in [3] by matching probability as a similarity function, however, Eq. (8) doesn’t reflect denser supervision scenarios. Authors should justify on the selection of their objective function and its aspects with the existing literature as aforementioned.

Comments:
- In L692, a missing information on inference time in this statement “our method is xx faster than CCLM in inference times”.


[1] Dietterich, T. G., Lathrop, R. H., & Lozano-Pérez, T. (1997). Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence, 89(1-2), 31-71.

[2] Song, Y., & Soleymani, M. (2019). Polysemous visual-semantic embedding for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 1979-1988).

[3] Chun, S., Oh, S. J., De Rezende, R. S., Kalantidis, Y., & Larlus, D. (2021). Probabilistic embeddings for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8415-8424).
paper_summary: paper_summary: This paper proposes LECCR, a novel solution for the CCR task. It aims to bridge the gap between modalities by incorporating the multi-modal large language model (MLLM) to generate detailed visual descriptions. Additionally, considering the lower quality of non-English representations, this paper utilizes English representations as guidance to improve the alignment between visual and non-English features by providing comprehensive inter-modal relationships.
strength_weakness: strength_weakness: 1. Employing MLLM to improve the alignment between visual and non-English features is interesting.
2. These paper conducts extensive experiments on four image-text and video-text cross-modal retrieval benchmarks across different languages, demonstrating the effectiveness of method.
limitations: limitations: 1. Model training is complex, and seven different objective functions were used during the training process. This could make the paper difficult to reproduce. Additionally, there is a lack of comprehensive ablation experiments and analysis, making it unclear what each loss function contributes to the model. 

2. Utilizing large models to solve visual language tasks is a popular practice. For example [38] employs LLMs to generate auxiliary captions to enhance Text-Video Retrieval. The authors should compare and analyze their method against existing retrieval methods based on large models. 

3. In the English Guidance module, the similarities between the source text and the vision are used at both local (semantic slots) and global (visual features) levels to guide the similarity of the target language text with the global vision (visual features). This setup is confusing. Why not constrain the similarity of the target text with the local vision (the semantic slots) as well? Furthermore, a more natural approach would be to use the global alignment of the source text with the visual features to constrain the global alignment of the target text with the visual features, and the local alignment of the source text with the semantic slots to constrain the local alignment of the target text with the semantic slots. 

4.In Table 1, compared to CCLM [47], why the proposed model performs much worse on Multi30K but performs well on MSCOCO? 

5.Is using MLLM to generate descriptions for all images offline or online?  This takes extra time. Compared with other dual-stream methods, how is the training speed and inference speed of this model? 

6.How are the hyperparameters $\mu_1$ and $\mu_2$ determined?
paper_summary: paper_summary: The paper introduces a new method for cross-lingual cross-modal retrieval called LECCR, which enhances the alignment of visual and non-English textual representations using a Multimodal Large Language Model (MLLM). Without relying on manually labeled cross-modal data pairs, LECCR generates detailed visual content descriptions using the MLLM and aggregates them into multi-view semantic slots. These slots interact with visual features to enhance semantic information and narrow the semantic gap between modalities. Additionally, LECCR introduces soft matching under the guidance of English, establishing more comprehensive and reliable cross-modal correspondences. Experimental results demonstrate the effectiveness of LECCR on cross-lingual image-text and video-text retrieval benchmarks, showcasing its potential to improve cross-lingual cross-modal retrieval tasks.
strength_weakness: strength_weakness: - The paper introduces a dual-stream solution for cross-lingual cross-modal retrieval that utilizes a Multimodal Large Language Model (MLLM) to generate detailed visual content descriptions, thereby enhancing the semantic information within visual features, which is sufficiently novel.

-  Established models are employed for the multi-view visual-semantic interaction module, and the use of MLLM improves cross-modal alignment. The proposed concept of multi-view semantic slots adds depth to the model.
- The method has a broad range of applications, suitable for multilingual content retrieval, cross-cultural content analysis, and more.
- Comprehensive experiments are conducted on multiple benchmarks, including image-text and video-text retrieval tasks, to validate the method's effectiveness, universality, and robustness. Comparisons with existing methods and ablation studies further confirm the superiority of the method and the rationality of its design choices.
limitations: limitations: The proposed method may depend on the quality of MLLM generated descriptions, which may not always be accurate and affect the universality of the results.
paper_summary: paper_summary: This paper presents a novel Cross-lingual cross-modal retrieval model that leverages a multimodal large language model that generates detailed visual descriptions to use them as internal representations and  enhance visual representations and also utilizes English features to guide alignment between visual and the non-English modalities. This allows to tackle the gap between vision and text and especially the lower quality of non-English representations. Their extentive experiments demonstrate that their proposed method LECCR (LLM Enhanced Cross-lingual Cross-modal Retreival) method significantly improves the quality of the alignment between the visual features and the non-english textual features on a wide variety of various benchmarks that the authors evaluated their method on.
strength_weakness: strength_weakness: The idea is clever, the architecture and the methods are well described.
Results are consistent over multiple datasets and multiple languages and the ablation studies are provided to verify the effectiveness of the various components.
Personally I think that these kind of studies should always be emphasised given that methods on other languages always lack behind methods presented for the English language. Additionally, Cross-lingual models that can exploit the performance on the english language are always a nice addition to methods that are trained on datasets solely in another language. This is especially true for languages where the datasets typically lag behind those for the most popular languages.
I think that the evaluations are thorough and that the various graphs and even the attention map visualisation are nice addition and show how nicely each semantif slot nicely distinctly focus on ocal semantics within the image.
limitations: limitations: I personally don't see any significant limitations to this work. It appears to be sufficiently novel and of good quality for ICMR 2024 and it's definitely a good fit for such a conference where these methods that are Multimodal or crossmodal are at core of the interest.
To me the evaluation also appears sufficient but I might not be aware of the latests datasets in mulitmodal retrieval.