Title: Efficiently Maintaining the Multilingual Capacity of MCLIP in Downstream Cross-Modal Retrieval Tasks
Venue: NeurIPS 2025
Accepted: yes
Keywords: multilingual CLIP

multilingual learning

cross-lingual transfer
Abstract: While existing research on Multilingual CLIP (MCLIP) has prioritized model architecture design, our work uncovers a critical challenge in practical adaptation: fine-tuning MCLIP through a single source language risks diminishing its multilingual capabilities in downstream tasks due to cross-linguistic disparities. To bridge this gap, we systematically investigate the role of ‌token similarity‌ in cross-lingual transferability for image-text retrieval, establishing it as a key factor governing fine-tuning efficacy. Building on this insight, we propose two novel strategies to enhance efficiency while preserving multilinguality: 1) ‌TaPCL dynamically optimizes training by prioritizing linguistically distant language pairs during corpus sampling, reducing redundant computation, and 2) CiPCL enriches the source corpus with multilingual key terms, enabling targeted knowledge transfer without reliance on exhaustive parallel data. By strategically balancing token similarity and domain-critical information, our methods significantly lower computational costs and mitigate over-dependence on parallel corpora. Experimental evaluations across diverse datasets validate the effectiveness and scalability of our framework, demonstrating robust multilingual retention across languages. This work provides a principled pathway for adapting MCLIP to real-world scenarios, where computational efficiency and cross-lingual robustness are paramount. Our codes are available at https://github.com/tiggers23/TaPCL-CiPCL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes TaPCL and CiPCL strategies to address the problem of multilingual capability degradation during fine-tuning of Multilingual CLIP (MCLIP). TaPCL reduces redundant computation by reducing the sampling probability of languages with high token similarity, and CiPCL achieves precise knowledge transfer by translating key terms in the source language and inserting prompts. The experiments were verified on the MSCOCO₃₆ and XM3600 datasets, providing a new path for MCLIP to balance computational efficiency and multilingual robustness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nDynamically optimize sampling based on token similarity to avoid redundant training of highly similar languages, reducing the number of iterations by 30%-50%.\n\nOnly key terms are translated and inserted into the source corpus, without the need for full parallel data, reducing the amount of data while maintaining performance.\n\n**Weaknesses:**\n\n1. After the data set is selected and processed, the amount of data is small, and the evaluation of the model's true capabilities may not be very accurate. There is also a risk of overfitting.\n\n2. Token similarity and key information extraction strategies may have certain limitations. \n\nFor example, token overlap cannot distinguish the semantic differences of polysemous words, such as the English word \"bank\" and the German word \"Bank\" (bench) are similar in form but unrelated in meaning. TaPCL may misjudge them as high-similarity language pairs, resulting in a lower sampling probability.\n\nCiPCL only extracts the top-1 key terms and ignores the semantic associations of multiple term combinations. For example, \"artificial intelligence\" needs to be translated into both \"artificial\" and \"intelligence\" to fully convey the semantics.\n\n3. Are all batch sizes set to the same? Because for retrieval tasks, the batch size also has a significant impact on the convergence speed and effect.\n\n4. Are the initial losses of different experimental settings in Figures 4 and 5 consistent? It seems inconsistent. Why?\n\n5. This paper only verifies the effectiveness of the experiment through cross-modal retrieval, and the generalization of multi-tasks is questionable, especially with such a small amount of training data.\n\nQUESTIONS:\n1. After the data set is selected and processed, the amount of data is small, and the evaluation of the model's true capabilities may not be very accurate. There is also a risk of overfitting.\n\n2. Token similarity and key information extraction strategies may have certain limitations. \n\nFor example, token overlap cannot distinguish the semantic differences of polysemous words, such as the English word \"bank\" and the German word \"Bank\" (bench) are similar in form but unrelated in meaning. TaPCL may misjudge them as high-similarity language pairs, resulting in a lower sampling probability.\n\nCiPCL only extracts the top-1 key terms and ignores the semantic associations of multiple term combinations. For example, \"artificial intelligence\" needs to be translated into both \"artificial\" and \"intelligence\" to fully convey the semantics.\n\n3. Are all batch sizes set to the same? Because for retrieval tasks, the batch size also has a significant impact on the convergence speed and effect.\n\n4. Are the initial losses of different experimental settings in Figures 4 and 5 consistent? It seems inconsistent. Why?\n\n5. This paper only verifies the effectiveness of the experiment through cross-modal retrieval, and the generalization of multi-tasks is questionable, especially with such a small amount of training data.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the issue that MCLIP requires training data for each language when fine-tuning for downstream tasks. From the perspective of optimizing data, the authors propose two methods: TaPCL, which performs sampling based on language similarity, and CiPCL, which retains key knowledge in the data. Experiments conducted on two cross-model retrieval benchmarks demonstrate the effectiveness of the proposed methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n1. Reasonable Motivation and Preliminary Evidence: The proposed methods are based on sound motivation and are supported by initial experimental results, which provide some validation for their effectiveness.\n\n2. Clear Introduction: The introduction is well-structured and clearly identifies the gaps in existing work, as well as the positioning of this study. This makes the contribution of the paper easy to understand.\n\n## Weakness\n1. Limited Practical Applicability: The real-world applicability of the proposed methods might be constrained. This limitation could be inherent in this type of approach. Specifically, under what circumstances would it be necessary to fine-tune the model for multiple languages? Wouldn't it be simpler to fine-tune a version of the model for each language separately and provide these versions to the relevant users? The authors are encouraged to propose a practical scenario where their methods would be advantageous.\n\n2. Experimental Scope and Task Variety: The experimental setup is limited to two cross-modal retrieval benchmarks. The authors should explain why the experiments were conducted exclusively in this context and whether it would be appropriate to extend the evaluation to other task types.\n\n3. Performance Concerns: The performance improvement brought by the proposed methods is not significant and, in some cases, slightly underperforms compared to PCL-base. While the time savings of approximately ten minutes are noted, such a minor difference in real-world scenarios is unlikely to be impactful. The authors are encouraged to provide further clarification regarding these aspects.\n\nQUESTIONS:\n1. Practical Applicability: Under what specific real-world scenarios would it be necessary to fine-tune a model for multiple languages using the proposed methods? Would it not be simpler to fine-tune separate versions of the model for each language and provide these versions to relevant users? Please elaborate on potential use cases or advantages of your approach in comparison to this alternative.\n\n2. Experimental Scope: Why were the experiments limited to only two cross-modal retrieval benchmarks? Is there a specific reason for focusing on these tasks, and do you believe the proposed methods are applicable to other types of downstream tasks? If so, could you provide additional evidence or reasoning to support this?\n\n3. Performance and Impact: The proposed methods yield only marginal performance improvements and, in some cases, slightly underperform compared to PCL-base. Additionally, the time savings of approximately ten minutes may not be impactful in real-world scenarios. Could you provide further explanation or justification regarding these results, and clarify the significance of the observed improvements in practical settings?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n# Method summary\n\nThe paper focuses on a problem in downstream adaptation of Multilingual CLIP (MCLIP) models - maintaining performance in other languages that was not present in the downstream fine-tuning, which is usually the case without directly using parallel corpora in other languages. The authors first analyse token overlap degree (Otk) between a source and target lanaguage (IOU over the token vocab), syntactic, phonological and geographic similarlity between the MCLIP model before and after fine-tuning on the source language. The analysis shows that only token overlap degree correlates well to drop in performance. The authors then present TaPCL and CiPCL: the idea behind TaPCL is uniform sampling from target languages can be improved since the analysis shows high overlap indicates less adaptation effort and hence propose a weighted sampling based on the token overlap degree. Furthermore, in CiPCL, the authors propose focusing on top-k critical word key in the source lanaguages and only translating those into target languages and similar weighted sampling strategy to TaPCL is used for source and target languages.\n\n# Experiment setup\n\nThe authors evaluate on MSCOCO and XM3600. \n\n# Results summary\nTable one shows ~1% absolute improvement over the closest baseline on MSCOCO and XM3600 while having a faster runtime and fewer iterations than PCL-base. Ablations suggest all the similarlity measures roughly perform the same and Otk performs the best in CiPCL.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n1. Clearly written paper with well-described problem statement, methodology, and contributions.\n2. The analysis and motivations are clear in Section 3\n\n# Weaknesses\n\n1.  The authors do not provide any information on seeds etc. With the improvements being really close to the baseline, I wonder how much can we really infer from a single run. \n2. The ablations indicate Otk roughly performs the same - if the correlations were really that bad, the performance changes should also have been reflected the same, however, only runtime / iterations are effected. I'm not sure I see an explanation or hypothesis as to why this happens.\n\nQUESTIONS:\nNo additional questions to the authors, just the ones listed in the weaknesses.\n\nLIMITATIONS:\nYes, the authors have addressed limitations but no mention of the broader impact statement which is clearly a part of the author guidelines.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the computational inefficiency of maintaining multilingual capabilities when fine-tuning MCLIP models for downstream tasks. The authors find that token similarity is the key factor affecting cross-lingual transferability in multimodal retrieval. They propose two strategies: TaPCL, which adjusts sampling probabilities based on token similarity, and CiPCL, which augments source text with translated key terms instead of full parallel sentences. Experiments on MSCOCO36 and XM3600 show both methods achieve comparable performance to baselines while reducing training time by 30-50%.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: The paper provides a systematic empirical analysis to identify token similarity as the primary factor affecting cross-lingual transfer in multimodal settings, which is a novel contribution to the field. The insights about token similarity could inform future work on multilingual multimodal models. Furthermore, I also think that the proposed methods are intuitive and motivated, with clear mathematical formulations for both TaPCL and CiPCL strategies. That said, I am not super familiar with multimodal models literature to know how novel these two strategies are. The wide language coverage also adds to the strengths of the paper.\n\n**Weaknesses**: A critical weakness is that token similarity is inherently confounded with pretrained representation quality - languages with high token overlap typically receive better multilingual pretraining due to shared subwords and data availability. This makes it unclear whether the observed transferability stems from token similarity itself or simply reflects which languages already have stronger representations in the base model. In addition, the authors say that in Section 5 they only consider \"10 languages which can be well perceived by the pre-trained MCLIP model\" which complicates this fundamental confound. To disentangle this, it'd be encouraging to see results on languages that have low digital resources but have a lot of token overlap with English. \n\nFurthermore, in my opinion, the paper doesn't adequately discuss failure cases or provide guidance on when to use TaPCL versus CiPCL. Scalability concerns can arise from the need to compute pairwise token similarities and the additional translation model requirement for CiPCL. The key term selection in CiPCL (Equation 8) seems simplistic - using only attention weights might miss important contextual information (such as knowledge retrieval pairs [1]). However, I don't think this is a fundamental weakness.\n\n\n[1] Geva, Mor, et al. \"Transformer feed-forward layers are key-value memories.\" EMNLP 2021.\n\nQUESTIONS:\nDo you have results on languages that have high token overlap but usually are poor in digital resources according to this taxonomy [1]?\n\n[1] Joshi, Pratik, et al. \"The state and fate of linguistic diversity and inclusion in the NLP world.\" ACL 2020.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}