Efficiently Maintaining the Multilingual Capacity of MCLIP in Downstream Cross-Modal Retrieval Tasks
Authors: Fengmao Lv
Jitong Lei
Guosheng Lin
Desheng ZHENG
Jianyang Zhang
Tianrui Li
Venue: NeurIPS 2025 poster ()
Keywords: multilingual CLIP
multilingual learning
cross-lingual transfer
Primary area: applications

Abstract: While existing research on Multilingual CLIP (MCLIP) has prioritized model architecture design, our work uncovers a critical challenge in practical adaptation: fine-tuning MCLIP through a single source language risks diminishing its multilingual capabilities in downstream tasks due to cross-linguistic disparities. To bridge this gap, we systematically investigate the role of ‌token similarity‌ in cross-lingual transferability for image-text retrieval, establishing it as a key factor governing fine-tuning efficacy. Building on this insight, we propose two novel strategies to enhance efficiency while preserving multilinguality: 1) ‌TaPCL dynamically optimizes training by prioritizing linguistically distant language pairs during corpus sampling, reducing redundant computation, and 2) CiPCL enriches the source corpus with multilingual key terms, enabling targeted knowledge transfer without reliance on exhaustive parallel data. By strategically balancing token similarity and domain-critical information, our methods significantly lower computational costs and mitigate over-dependence on parallel corpora. Experimental evaluations across diverse datasets validate the effectiveness and scalability of our framework, demonstrating robust multilingual retention across languages. This work provides a principled pathway for adapting MCLIP to real-world scenarios, where computational efficiency and cross-lingual robustness are paramount. Our codes are available at https://github.com/tiggers23/TaPCL-CiPCL.

Efficiently Maintaining the Multilingual Capacity of MCLIP in Downstream Cross-Modal Retrieval Tasks
Fengmao Lv Jitong Lei', Guosheng Lin', Desheng Zheng Jianyang Tianrii Li
'Southwest Jiaotong University, *Nanyang Technological University
"Southwest Petroleum University, "Kash Institute of Electronics and Information Industry, UUniversity of Electronic Science and Technology of China
fengmaolv@126.comm 20232018180my swjtu, edu. en, gslinentu. edu. sg, deeng__d__senn@063.. com, jyzhang3120gmai com, trli@swjtu. edu. cn
Abstract
While existing research on Multilingual CLIP (MCLIP) has prioritized model architecture design, our work uncovers critical challenge in practical adaptation: fine-tuning MCLIP through single source language risks diminishing its multilin- gual capabilities in downstream tasks due to cross-linguistic disparities. To bridge this gap, we systematically investigate the role of token similarity in cross-lingual transferability for image-text retrieval, establishing it key factor governing fine-tuning efficacy. Building on this insight, we propose two novel strategies to enhance efficiency while preserving multilinguality: 1) TaPCL dynamically opti- mizes training by prioritizing linguistically distant language pairs during corpus sampling, reducing redundant computation, and 2) CiPCL enriches the source corpus with multilingual key terms, enabling targeted knowledge transfer without reliance exhaustive parallel data. By strategically balancing token similarity information, our methods significantly lower computational costs and mitigate over-dependence on parallel corpora. Experimental evaluations across diverse datasets validate the effectiveness and scalability of our framework, demonstrating robust multilingual retention across languages. This work provides a principled pathway for adapting MCLIP to real-world scenarios, where com- putational efficiency and cross-lingual robustness are paramount. Our codes are available at /tttps://githann com/ Camm/iiiggorr33/TPCL-CCCC
and
1 Introduction
Cross-modal retrieval aims to retrieve the corresponding data across different modalities using a query from one modality. With the emergence of large numbers of images on social media platforms such as Twitter and Facebook, accurately retrieving multimedia contents has become a significant challenge, making research on cross-modal retrieval a hot topic. recent years, visual-language pre-training models like CLIP [ have gained significant attentions. Models such as CLIP effectively calibrate the visual and language modality representations during the pretraining phase, making them well-suited for tasks such image-text cross-modal retrieval. Early research on CLIP primarily focuses on English. Considering that there are more than 7,000 languages worldwide, recent studies have extended the multilingual processing capability of the CLIP model to build Multilingual CLIP (MCLIP) models, which can support image-text retrieval applications in different languages.
However, current studies on MCLIP primarily concentrate on the construction of multilingual pre-training models, e.g., building multilingual pre-trained CLIP models through joint learning of multiple languages (2 6 7 g 9j, or expanding the multilingual capability of the CLIP model
corresponning author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Comoocal hewwlokge of Enghes
Cenerca koowiedoo ofFireet French
New domein- specific English
New domaits specilic French knowledge
knowledge
standing forest
haar
Jears
kaga equal French
French description of the key teminology
New demaiie- French knowledge
domaio specitll Koowlalge in Enghish
Figure 1: A person who has learned new domain-specific knowledge in English can comprehend that knowledge in French by simply learning the French description for the key terminology about the corresponding knowledge.
incrementally based on the continual learning framework 10] In this work, we further focus on how to maintain the multilingual capability of the MCLIP model during its adaptation for downstream applications. practical applications, MCLIP models are usually fine-tuned via one language (i.e., source language) to enhance its performance in relevant tasks. Due to inherent differences between languages, the performance improvements for other languages (i.e., target languages) are typically inferior compared to the improvement in the source language, leading the multilingual capability of the MCLIP model on downstream tasks. Therefore, this work will focus on multilingual generalizable fine-tuning strategies for MCLIP, enabling each language to perform well in the corresponding downstream tasks.
degradation of
Similarly to existing cross-lingual learning studies [1 12 13 common approach is to translate the source language corpus into target languages, and use the resulting parallel corpus of target languages alongside the source language corpus to fine-tune MCLIP models. Considering that the MCLIP model usually involves a large number of target languages, directly incorporating parallel corpora for fine-tuning will lead to massive increase of training data for each downstream task-specific adaptation of the MCLIP model. As result, this will cause substantial computational overhead, significantly compromising fine-tuning efficiency, and hindering the flexible deployment of MCLIP models for downstream applications. Therefore, this work will focus on how to implement efficient fine-tuning of MCLIP models in downstream tasks, i.e., achieving fine-tuning with reduced computational time while preventing performance degradation compared to directly utilizing the parallel corpora.
In general, visual knowledge can be transferred across similar languages. For instance, when learning visual knowledge domain through English, person who is proficient in multiple languages can easily understand the corresponding visual knowledge via other linguistically similar languages (e.g., French Italian), without relearning via French or Italian. Inspired by this observation, this work will attempt to investigate what factor can influence the knowledge transferability across languages and how leverage the knowledge transferability across languages to reduce excessive reliance on parallel corpora, thereby lowering the computational cost of fine-tuning MCLIP models for downstream tasks, and achieving a more efficient fine-tuning
Existing works 14 16 18 19 primarily focus on studying cross-lingual transferability in unimodal language models such mBERT (20) and BLRRRRaERTTa (21], exploring different similarity factors (e.g., token similarity, syntactic similarity, and phonological similarity) for cross- lingual transfer in tasks such as document classification [22), natural language inference [11] and pfrt-of-speech tagging. This work conducts the pioneer attempt to explore influential similarity factors for gross-lingual transfcrability in multimodal scenarios. To this end, according to some preliminary experiments on the correlation between performance improvement in target languages and different similarity factors (see Figure 2. find that token similarity between languages significantly affects the transfer performance, i.e., target languages with higher token similarity to the source language exhibit more substantial performance improvements after fine-tuning the MCLIP model with the source language.
With the above findings, this work proposes two efficient fine-tuning strategies to enhance the computational efficiency in fine-tuning MCLIP models with parallel corpora. Firstly, we propose the Transferability-wwer aware Parallel Corpora Learning (TaPCL) mechanism by incorporating the influence

---

of token similarity factors into the downstream adaptation of MCLIP models. For target languages that exhibit high token similarity to the source language, knowledge transfer from the source the target language is more effective, so that those target languages demonstrate notable performance improvements after fine-tuning on the source language, which can be equivalent to fine-tuning on those languages. Consequently, directly utilizing all parallel corpora in target languages during fine-tuning may introduce training redundancy. Under this consideration, our approach employs language-specific sampling probabilities for different target languages during fine-tuning, rather than directly considering all target language parallel corpora. By lowering the sampling probability for languages with high token similarity to the source language, this strategy mitigates the over- werruutiizzation of parallel corpora and reduces redundant computational costs caused by redundant parallel corpora in target languages, preliminary experiments demonstrate that TaPCL improves fine-tuning efficiency.
Based on the preliminary success of TaPCI,, we propose the Critical-information Parallel Corpora Learning (CiPCL. to further reduce the reliance on parallel corpora. In general, this method is inspired by the learning mechanism of humans for foreign language knowledge acquisition. Specifically, as shown in Figure [ when person (who is proficient in both English and French) has learned new Nomain-speccife knowledge in English, he can also comprehend and articulate that knowledge in French by simply learning the French description for the key terminology about the corresponding knowledge, without the need to relearn the entire knowledge through French. Similarly, during pre-training MCLIP models have already acquired syntactic structures across languages, while the primary objective of downstream task learning is to gain domain domain-specific knowledge. Therefore, by treating the pre-trained MCLIP model as a person who has multilingual capability, the CiPCL approach mainly focus on applying the learning principle of humans to MCLIP models. To this end, CiPCL first recognizes key information in downstream image-text retrieval tasks, and incorporates their descriptions in target languages as critical prompts in the source language corpus, e.g. "A kagu (cagou in French) standing on log in forest". By fine-tuning MCLIP models using the source language corpus augmented with target language prompts for critical information, CiPCL does not need specifically construct complete parallel sentences for target languages, significantly enhancing computational efficiency during fine-tuning Similarly to our first strategy, we also employ language-specffe sampling probabilities add the target language prompts for critical information. Extensive experiments further demonstrate the effectiveness of CiPCL.
To sum up, our contributions of maintaining the multilingual capability of MCLIP models in downstream tasks. To this end, we also conduct initial investigation explore which similarity factors significantly impact the cross- lingual transferability in visual-language pretraining models. (2) Based on preliminary experiments and the learning mechanism of human, we propose two efficient fine-tuning strategies for MCLIP models to reduce computational time and alleviate reliance on parallel corpora while maintaining the multilingual capability of MCLIP models. (3) Extensive experiments are conducted demonstrate both the effectiveness and scalability of our proposed methods.
be summarized as follows: (1) We present initial exploration
2 Related Work
2.1 Multilingual CLIP Models
Models like CLIP 1 calibrate visual and language modal representations in a shared feature space, enabling cross-modal retrieval by computing the similarity between image and text representations. The initial works primarily focus on English, while subsequent works also attempt to construct CLIP models capable of handling multiple languages. Specifically, Reimers et al. 2 and Carlsson et al. propose to construct multilingual CLIP by distilling knowledge from the text encoder of the original CLIP to a multilingual text encoder via parallel corpora. Chen et al. (6) propose to further distill knowledge from the vision encoder of the original CLIP to improve the image-text alignment capability of multilingual CLIP. The above works construct multilingual CLIP primarily by extending the multilingual capability of the original CLIP. OpenCLIP (5) also proposes to directly construct multilingual CLIP via pre-training from multilingual image-text data. Recently, Yang et al. [10] further propose the continual learning paradigm which enables the continuous integration of new language processing mechanisms into multilingual CLIP models. However, existing works on multilingual CLIP mainly focus on the construction of multilingual CL.P. This work will further consider how to maintain the multilingual capability of MCLIP models during its adaptation for downstream applications.
3

---

2.2 Cross-lingual Transfer
Fine-tuning multilingual models typically requires considering cross- Progsllingual transfer mechanisms between different languages. Existing works primarily focus studying cross-lingual transferability for unimodal language models. Specifically, Wu al. [14 analyze the relationship between token similarity and cross-lingual transferability across multiple tasks, including document classification, natural language inference, named entity recognition, part-of-speech tagging. and dependency parsing. Ahuja et al. [IS] develop multitask performance prediction framework to analyze the importance of influential factors, thereby enabling more accurate analysis by leveraging data of other tasks when dealing with tasks involving limited language quantities. de Vries et al. [16] consider different multiple languages as the source languages and perform analyzes for POS tasks. Limisiewicz et
al. [17] consider multiple methods and investigate the effects of tokenizer quality and token similarity on cross- messs-lingual transferabilitt. To further promote cross- pooss-lingual transfer when fine-tuning unimodal language models, David Schmidt al. (18] propose to improve the performance of multilingual models on downstream tasks by integrating two independent fine-tuning phases into a unified stage, thus providing multifaceted supervision for the fine-tuning process. To maintain robust cross-lingual sentence representations with minimal alteration to the model's output representations during fine- tuning, Tu al. [23 leverage a prompt-based structure for model adaptation. Yu et al. [24] propose leveraging mutual supervision between language models during fine-tuning to enhance cross-lingual transferability. With the recent advances in machine translation 25 26 (e.g., language translation enhanced by LLM), cross- lonss-l transfer can be enhanced by directly leveraging parallel corpora translated from the source language during fine-tuning [ [13] However, existing works mainly focus on single-modal models/tasks. For MCLIP models typically involve numerous target languages, which requires to construct large-scale parallel corpora and introducing substantial training time. In this work, we present an initial exploration of cross-lingual transfer for image-text retrieval tasks in MCLIP models and reduce the high computational time caused by the large-scale parallel corpora.
3 Analysis
In this section, we explore influential similarity factors for cross-lingual transferability in multi-moda retrieval scenes, including the identification of potential similarity factors, their measurement methods, and the correlation between these factors and transferability.
3.1 Similarity Factors
We adopt token vocabulary overlap to measure token similarity between languages. The token overlap degree Or, between the source language S and the target language T3 is defined as:
V n VT Or, VUVV
(I)
where V denotes the source language token vocabulary, and 4 represents the target language token vocabulary. Furthermore, we calculate syntactic similarity (Saym), phonological similarity (Spho), and geographic similarity (Sgeo) between the source language and the target language using the linguistic features of the URIEL project (27), following Lauscher et al. (28)
3.2 Analysis Methodology
We fine-tune the MCLIP model using only the image-text dataset of the source language D. The model is then directly evaluated target language image-text datasets to measure lingsalingual retrieval performance. We calculate the performance change AT; for each language before and after fine-tuning and analyze the impact of similarity factors on cross-lingual transferability through visualization techniques and Pearson correlation. To avoid the influence of other factors such as the different language capabilities of pre-training models in different languages [15 16), we only consider languages which can be well perceived by the pre-trained MCLIP model, resulting in 10 languages"
'The 10 languages are Greek, Spanish, French, Italian, Polish, Portuguese, Swedish, Ukrainian, Chinese- Simplitied.

---

Token Sivilerity ity
Syntactio Sie Sieilerity
Foreunce shanet
Perforaanc
P--0 and
Geographic Similariit
Phonol og lcal Simi lar
a70. 12618
uno 66156
Por for seoee phorae
Perforgmeo
0.
Figure 2: The relationship between the four types of similarity and cross-lingual transferability,
Pearson correlation coefficient and p-value is shown the box.
3.3 Analysis Results
As shown in Figure 2] observe that token similarity significantly impacts cross-lingual transfer- ability, and the p-value the Pearson correlation test is much lower than 0.05, indicating a strong positive correlation between token similarity and cross-lingual transferability. In contrast, syntactic similarity (Sxyn), phonological similarity (Spho), and geographic similarity (Sgea) do not exhibit significant influence cross-lingual transferability. This demonstrates that token similarity is critical similarity factor that influences cross-lingual transfer performance.
4 Methods
In this section, present proposed efficient multilingual cross-modal retrieval fine-tuning strategies, including the Transferabilit-awwar Parallel Corpora Learning (TaPCL) and the Critical- information Parallel Corpora Learning (CiPCL) strategies, the overview of strategies is shown in Figure 3
4.1 Problem Formulation
We formulate multilingual cross-modal retrieval as follows. Given a source language S and M target languages { we aim to adapt a multilingual language model to downstream tasks using: manually annotated source language image-text pairs D {(v,, ()) where v E V is image and of¹ c text deseription and machine-translated target language corpora CT² [T 1=-1 generated from C W The target language training is defined as:
DT
UUD? D..
(2)
1=1
where each D ((v,, of & contains target language parallel corpora and image pairs.
4.2 Transferabilit-awware Parallel Corpora Learning
uniformly sample all target language training data. Given the target language training set D each sample (v% TT) )) is sampled in mini- batch with equal probability:
Conventional parallel corpora learning approaches 11] 12 13]
B
Pruse (1s, (Ts sTt)
N
5

---

An insicaall
dengied
ball
bust
colle
Salacied Secele
Eour
woggt.
Sampler
(40
Kry "boa"
Comprnn infemmaii soog!
Pronpel'
Eacoder
As resicaictyy dodggne bost hasce French) ee back
Common propeetiio
201
Mililogll visker baggage medel
scatarcly dolgad
(a) T&PCL
(b) CIPCL
Figure 3: Illustration of our methods: (a) The overview of TaPCL. (b) The overview of CiPCL.
where B is the mini-batch and B is the batch size. This uniform treatment assumes equitable utilization of parallel corpora across languages.
However, as discussed in Section 3.1 languages with high token overlap require less adaptation effort. To improve fine-tuning efficiency, we propose to reduce the repeated participation of redundant target language samples that are similar to the source language in training by decreasing their sampling probabilities. Specifically, based on the token similarity factor Or, calculated by Eq, [ the Transferability-awee Weighted Sampling (taws) function Paws is formulated as follows:
B N
exp(-Onx
Pinwes Punes((Vi, ofth)) B)
(4)
Σ expl exp((-0,/7))
where is temperature hyperparameter controlling the sharpness of the probability distribution.
By replacing Plase with Puwes target language data DT), with fewer overlaps with the source language will be assigned a higher sampling probability, while those with large overlaps with the source language will be assigned a lower sampling probability. In this way, our approach encourages the model to focus more adapting to low-overlap languages while reducing computational overhead as- sociated with high-overlap languages, thereby accelerating downstream task adaptation and lowering the overuse of parallel corpora.
The overall loss combines source and target language objectives through adaptive weighting:
L Ls + aLr,
where Ls implements image-text alignment in source language and Lr implements weighted sam- pling fine-tuning in target language. Both terms use the CLIP contrastive loss:
Ls
Σ
ExCR (NCA(11,))))
(6)
[DS
CT = E
LNCE(VE, 87
where LNCE denotes the probability of samples that are dissimilar from the source language and reducing the sampling probability of similar samples, our optimization objective pays more attention to difficult samples, thereby improving the training efficiency.
Estimation loss 29 Based on increasing the sampling
4.3 Critical-information Parallel Corpora Learning
To further improve fine-tuning efficiency, we discard the full parallel corpus the target language and instead use only source-language samples with key prompts in the target language for fine-tumin sonettunngg

---

thereby avoiding redundant training on semantically similar samples. Specifically, for a source language text S², instead of constructing complete parallel sentences in target languages, we translate only the critical top-k word key* identified in s into target languages key"*. The critical word selection is based on attention weights between the [cls] token and other tokens in the last transformer layer:
11
key argmatr.ea.tl
Σ oh (t., M(as)):
where H denotes the number of attention heads, and 0% represents the attention weight between token & and the (cls) token in head h. If the selected key* is a subword unit, we merge it with adjacent tokens until form a complete word. We construct key information prompts using the template:
Prompt = "Key" Th in Tk";
where T& specifies the target language (e.g., Chinese or French). As shown in Figure 3 the prompt is inserted into the text in the source language, yielding the prompted text 8 ((P.Ts), formally,
(P.T.) = (f1...., fpos (Key"t in T&), Epos +1, +1,... t., t.
(9)
The training data is sampled through hybrid strategy:
DP {(vi, 8:));
(10)
where
with prob c% with prob (1 - c%) Pmws) (TR)
8 (P.T))
(11)
To maintain source-language transferability, we randomly retain e% of source-language texts without prompts. The training objective becomes:
LNCE(US; 8s). (D,,,))DPP
Σ
(12)
DDP
By fine-tuning without using complete parallel sentences in target languages, eliminate the depen- dency on parallel corpora in target languages. This strategy significantly improves computational efficiency while maintaining multilingual generalization. We use the parameter-cfficient learning approach Adapter [30] to further improve fine-tuning efficiency.
5
Experiments
5.1 Experimental Settings
Datasets and Baseline. We evaluate our proposed methods on MSCOCOg6 10 and XM3600 B1 both which are cross-modal retrieval benchmarks involving 36 parallel corpora. The pre-trained MCLIP models usually have imbalanced abilities across different languages. As shown in the original ability of the pre-trained multilingual language model will affect the performance of the model in the corresponding language after fine-tuning. As discussed in Section 3 only consider 10 languages which can be well perceived by the pre-trained OpenCLIP model in our experiments (experiments considering all the target langauges are also reported in Table and Figure 7 of the appendix). For MSCOCO36 and XM3600, we respectively randomly select 5000 and 2880 samples for training and English the source language. For baseline comparisons. we consider: (1) methods that only use the source language to fine-tune, (2) existing cross-lingual transfer approaches (prompt-tuning [23], VME (32]) that do not leverage parallel corpora, and (3) the recent parallel corpora learning method PCL-base across target languages.
16.16]
12 13
which applies uniform sampling
Evaluation Metries. Our experiments consider both retrieval performance and training efficiency.
To this end, we use rank-based metrics for performance evaluation: Recall@ (R@K) for K 1, 5, 10 and R@Auy Aug. Specifically, R@ K measures the proportion of samples where the correct
7

---

Table 1: Performance comparison of multilingual image-text retrieval and number of iterations on MSCOCO3s and XM3600.
Image-to- Textt
Text-to-Imape -maage
Method
@ Avg
Iteration
Runtime
R@]
R@5
R@ 10
R@1 1
R@S
R@ 10
MSCOCOM
Source-only
50.90
80.73
90.70
49.30
79.19
89.16
73.33
Prompt
23
52.71
81.72
90.54
49.73
78.31
88.68
73.61
VME 32
50.75
80.16
80.91
51.24
79.92
89.83
73.62
PCI.-base 1
53.02
82.75
91.68
51.71
81.48
91.05
75.28
7039
21m02s
TaPCL (Ours)
54.17
82.94
91.66
51.61
81 81.23
90.72
75.39
4929
14m31s
CiPCL (Ours)
54.17
83.17
91.64
50.68
80.87
90.38
75.15
3519
09m55s
Source-only
75.03
93.25
96.66
71.29
91.10
94.87
87.03
XM336000
Prompt 23
75.17
93.42
96.81
71.46
91.25
95.03
87.19
VME 32
74.00
93.24
96.38
71.97
91.96
95.78
87.22
PCL-base 1
76.21
94.53
97.26
72.86
92.49
96.12
88.25
8100
24m42s
TaPCL (Ours)
76.32
94,54
97.29
72.93
92.46
96.08
88.27
5680
16m49s
CiPCL (Ours)
76.08
94.47
97.26
72.72
92.29
96.11
88.16
4050
12m05s
Loss
Loss
PCL-kae
TxPCL
CINCE
PCL-basee
TxPCL
CINCE
0.45
0.3
0.3
0.5
24
24
30 36
24
4c
3c
(8)
Figure 4: The loss convergence curve on MSCOCO and XM3600.
image/text is retrieved within the top-K K ranked candidates, while R Aug is the average of text-to- image and image-to-ters retrieval R&K, K. Higher values indicate better performance. We report the average metrics across all languages:
ROK- = M Σ (RGK)) 1=1
(13)
where M is the number of languages, and ROKi is the R@K the 1-th language. For training efficiency, we consider the training iteration number (with each method utilizing the same batch size) and computation cost until training convergence.
Implementation Details. We adopt the OpenCLIP
10--3-XXILM-Rohereee
model 5 as our
pre-trained multi-lingual CLIP model. For CiPCL, we use M2M 0000 1.2B (66 translate, and we translate the critical top-1 word (we also report the detailed results with the top-2 and top-3 words in (n444)) into target languages. the detailed results under different trainable parameter numbers in A.2 A For the adapter, up layer output dimension to 256 and the down layer output dimension to 768, and set the dropout probability of the ReLU activation function to 0. The weights of the upper layer initialized with a uniform distribution, while the remaining parameters of the adapter are initialized as zero. conducted Linux with NVIDIA 3090 GPUs, using AdamW optimizer with cosine learning rate scheduler, and the initial learning rate 1e-4. We set the batch size to 128 and 0.2, B = 0.8. The model trained for 10 epochs with PCL-base MSCOCO,s. and for 20 epochs on XM3600.
insert the adapter into the last layer in the model (we also report set the
The experiments
5.2 Main Results
To validate the multilingual image-text retrieval capability of our fine-tuning strategies, conduct experiments on MSCOCO3s and XM3600. As shown in Table our fine-tuning strategies achieve comparable R@Kae performance to PCL-base, demonstrating equivalent retrieval capabilities.

---

Loss
Loss
TaPCL-SyE
TaPCL 200
TaPCL ph0
TaPCL
CIPCL-oya
CiPCL
CIPCL
0.
0.4
03
03
154
Figure 5: The loss convergence curve of ablation study,
In particular, our methods require fewer training iterations than the PCL-base. Specifically, for MSCOCOgs. TaPCL outperforms PCI.-base by 0. % with reduced training iterations, and CiPCL shows acceptable performance degradation of 0.13% compared PCL-base, offset by 50% fewer training iterations, And for XM3600, TaPCL outperforms PCL-base by 0.01 %, and CiPCL shows less performance degradation of 0.09%. These results validate that our strategies maintain downstream task performance while significantly improving training efficiency. We also report the detailed results on each target language in Table
To verify the training efficiency of our fine-tuning strategies, we plot loss convergence curves of the three methods MSCOCO%6 and XM3600. As shown in Table and Figure 4 compared to PCL-bbsse requiring over 7,000 iterations for loss convergence on MSCOCOss and 8100 iterations on XM3600, TaPCL reaches loss convergence with only 70% of PCL-base iterations and runtime") CiPCL achieves a significantly lower loss than PCL-base with only 50% its iterations and runtime. This shows that both proposed strategies achieve stable loss convergence with fewer training iterations and runtime, demonstrating their superior capability to adapt to downstream tasks with reduced computational costs.
5.3 Ablation Study
Ablation Study of TaPCL. For TaPCL, we
Table 2: Ablation study of TaPCL on MSCOCO
explore whether the improved fine-tuning effi- ciency originates from using token similarity the primary criterion for adjusting sampling probabilities. We fine-tune the model by sepa- rately adopting syntactic similarity, geographic similarity, and phonological similarity as the al- ternative primary criterion for adjusting sampling probabilities. shown in Table and Figure (ja), we observe that the fine-tuning convergence speeds using other similarity factors as the primary criterion are significantly slower than TaPCL, which shows that the efficiency improvement of TaPCL. is specifically attributed to using token similarity the primary criterion.
Sayn
Spho
Sye
Or.
R@
Iteration
Runtime
75.28
7039
21m20s
75.12
6329
19m12s
75.26
6350
19m26s
75.01
6343
19m14s
75.39
4929
14m31s
Ablation Study of CiPCL. For CiPCL, to verify the effectiveness of critical information prompts, we randomly select words from sam- ples of the source language and translate them into target languages as prompts. The model is then fine-tuned using samples augmented with these random prompts. As shown in Table and Figure(5]bb, when non-critical information used prompts, the retrieval performance decreases sig- nificantly, indicating that randomly selected prompts fail to facilitate the learning of visual knowledge for target languages. Similarly TaPCL, we test CiPCL using syntactic similarity as the primary criterion for adjusting sampling probabilities. The result shows a significant decline in retrieval performance, further validating the effectiveness of using token similarity as the primary criterion.
Table 3: Ablation study of CiPCL on MSCOCO
S agm On
keyr,
RØ Avg
Iteration
Runtime
75.28
7039
21m20s
74.15
3560
10m07s
74.43
3594
10m25s
75.15
3519
09m55s
'The initial loss difference is caused by the sampling strategies differ across methods; thus, the first mini- batch contains different samples, leading to slight difference in initial losses. By comparing the staye-sagee slopes of the loss curves across methods, we assess the convergence speed.

---

RCD
A person with a shopping cart on city street
A woman standing on a tennis court holding raequet
A girl with running mascara and bottle of Bacardi
Figure Sample cases for critical-information selection
5.4 Low-resource Languages Analysis
Based on the analysis Section we further investigate the factors that influence cross-lingual transferability low-resource languages. Following the categorization proposed by Pratik Joshi [33], we select several low-resource languages and evaluate the Source-only method on them, and calculate the Pearson correlation between token similarity and performance variation, shown in Table 4 We observe that token similarity still has a significant impact on performance gains-language with higher token overlap tend to achieve larger improvements. In addition, token similarity remains strongly and positively correlated with performance variation.
Table 4: Correlation between token similarity and performance variation in low-resource languages.
Danish
Norwegian
Romanian
Quechua
Swahili
Indonesian
Thai
Hebrew
Ukrainian
Token similarity
0.19
0.17
0.16
0.16
0.14
0.14
0.03
0.01
0.02
R Avg variation
1.89
1.77
1.08
0.99
1.05
1.02
0.33
0.23
0.68
Correlation
0.84, P= 0.004
5.5
Critical-information Modeling
Figure shows sample cases for citical-informatioon selection. We can see that the selected critical information (c.g., the shopping cart) carries the key visual knowledge about the corresponding image. supporting our claim for our proposed CiPCL method.
6 Conclusion and Limitations
This work focuses on MCLIP models and analyzes the similarity factors that influence tross-lingual transfer in image-text retrieval tasks. Based on the analysis, we propose two efficient parallel corpus learning strategies to alleviate the high computational overhead caused by directly incorporating all parallel corpora. Experiments on two multilingual image-text retrieval datasets demonstrate that the proposed methods enhance the efficiency of parallel corpus-based fine-tuning for downstream tasks.
TaPCL and CiPCL demonstrate high efficiency when fine-tuning for cross-modal retrieval tasks, suggesting their potential scalability. In our future work, we will not only focus on coarse-grained multilingual retrieval tasks but will further investigate the application of TaPCL and CiPCL in finer-grained vision- languae tasks, such as multilingual VQA.
Acknowledgments and Disclosure of Funding
This work was supported by the Natural Science Foundation Sichuan (No. 2025YFH20124), the Sichuan Science and Technology Program (No. 2024NSFTTD0036), the Frontier Cross Innovation Team Project of Southwest Jiaotong University (YH1 12432297), and the Engineering Research Center of Sustainable Urban Intelligent Transportation, Ministry of Education.
10

---

References
[1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and L. Sutskever, "Learning transferable visual models from natural language supervision." " in ICML, ser. Proceedings of Machine Learning Research, vol.
139, 2021, pp. 8748-8763.
[2] N. Reimers and I. Gurevych, "Making monolingual sentence embeddings multilingual using knowledge distillation," in EMNLP. Association for Computational Linguistics, 2020, pp. 4512-4525.
[3] G. Chen, Hou, Y. Chen, W. Dai, Shang, X. Jiang, Q. Liu, J. Pan, and W. Wang, "melip: Mul- tilingual CLIP via cross- loogsll transfer," in ACL. Association for Computational Linguistics, 2023, pp. 13 12:028111 13 043.
[4] Carlsson, Eisen, F. Rekathati, and M. Sahlgren, "Cross-lingual and multilingual CLIP," in LREC. European Language Resources Association, 2022, pp. 6848-6854.
[5] G. Ilharco. M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave. V. Shankar,
H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L.. Schmidt, "Openclip," Jul. 2021.
[6] Z. Chen, G. Liu, B. Zhang, Q- Yang, and L. Wu, "Altelip: Altering the language encoder in CLIP for extended language capabilities," in Findings of ACL. Association for Computational Linguistics, 2023, pp. 8666-8682,
[7] P. Huang. M. Patrick, J. Hu, G. Neubig, F. Metze, and A. Hauptmann, "Multilingual multi- modal pre-training for zero-shot cross-lingual transfer of vision- language models," in NAACL. Association for Computational Linguistics, 2021, pp. 2443-2459.
[8] Y. Wang, J. Dong, T. Liang, M. Zhang, R. Cai, and X. Wang. "Cross-lingual cross-modal retrieval with nobse-robust learning," in MM. ACM, 2022, pp. 422-433.
[9] Y. Wang, F. Wang, J. Dong, and H. Luo, "CL2CM: improving cross-lingual cross-modal retrieval via cross-lingual knowledge transfer," in AAAI. AAAI Press, 2024, pp. 5651-5659.
[10] B. Yang. Y. Dai, X. Cheng, Y. Li, A. Raza, and Y. Zou, "Embracing language inclusivity and diversity in CLIP through continual language learning," in AAAI. AAAI Press, 2024, pp. 1458-6466.
[11] A. Conneau, R. Rinotr, G. Lample, A. Williams, R. Bowman, H. Schwenk, and Stoy- anov, "XNLI: evaluating cross-lingual sentence representations," in EMNLP. Association for Computational Linguistics, 2018, pp. 2475-2485.
[12] J. Hu, S. Ruder, A. Siddhant, G. Neubig, O. Firat, and M. Johnson, "XTREME: massively multilingual multi-task benchmark for evaluating cross-lingual generalisation." in ICML, ser. Proceedings of Machine Learning Research, vol. 119. PMLR, 2020, pp. 4411 1-411.
[13) M. Artetxe, V. Goswami, S. Bhosale, A. Fan, and L. Zettlemoyer, "Revisiting machine transla- tion for cross-lingual classification," in EMNLP. Association for Computational Linguistics, 2023, pp. 6489-6499.
[14] S. Wu and M. Dredze, "Beto, bentz, becas: The surprising coos-linggall effectiveness of BERT." in EMNIP. Association for Computational Linguistics, 2019, pp. 833-844.
[15] K. Ahuja, S. Kumar, S. Dandapat, and M. Choudhury. "Multi task learning for zero shot performance prediction of multilingual models," in ACL. Association for Computational Linguistics, 2022, pp. 5454-5467.
[16] W. de Vries, M. Wieling, and M. Nissim, "Make the best of cross- cross-lingual transfer: Evidence from POS tagging with over 100 languages," in ACL. Association for Computational Linguisties, 2022, pp. 7676-7685.
[17] T. Limisiewicz, J. Balhar, and D. Marecek, "Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages," in Findings of ACL. Association for Computational Linguistics, 2023, pp. 5661-5681.
[18] F. D. Schmidt, I. Vulic, and G. Glavas, "Don't 't stop fine-tuning: On training regimes for few- shot cross-lingual transfer with multilingual language models," " EMNLP. Association for Computational Linguistics, 2022, pp. 10 10725-10742

---

[19] V. Patil, P. P. Talukdar, and S. Sarawagi. "Overlap-basee vocabulary generation improves cross- lingual transfer among related languages." " in ACL. Association for Computational Linguistics. 2022, pp. 219-233.
[20] J. Devlin, M. Chang, K. Lee, and K. Toutanova, "BERT: pre-training of deep bidirectional trans- formers for language understanding." in NAACI. Association for Computational Linguistics, 2019, pp. 4171-4186.
[21] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, "Unsupervised cross-lingual representation learning at scale,"
in ACL. Association for Computational Linguistics, 2020, pp.
[22] H. Schwenk and X. Li, "A corpus for multilingual document elassification in eight languages." in LREC. European Language Resources Association, 2018
[23] L. Tu, C. Xiong, and Y. Zhou, "Prompt-tuning can be much better than fine-tuning on cross- lingual understanding with multilingual language models," in Findings of EMNLP. Association for Computational Linguisties, 2022, pp. 5478-5485.
[24] T. Yu and S. R. Joty, "Effective fine-tuning methods for cross-lingual adaptation,"" in EMNLP. Association for Computational Linguistics, 2021, pp.
[25] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng. D. Grangier, and M. Auli, "fairseq: A fast, extensible toolkit for sequence modeling," in NAACL. Association for Computational Linguistics, 2019, pp. 48-53.
[26] A. Fan. S. Bhosale, H. Schwenk, Z. Ma. A. El-Kishky, S. Gooyl,, M. Baines, O. Celebi, G. Wenzek, V. Chaudhary, N. Goyal, T. Birch, V. Liptchinsky, S. Edunov, M. Auli, and A. Joulin, "Beyond english-centric multilingual machine translation," JMLL,, vol. 22, pp. 107:1-107:48 2021.
[27] Littell, D. R. Mortensen, K. Lin, K. Kairis, C. Turner, and L. S. Levin, "URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors," in EACL. Association for Computational Linguistics. 2017, pp. 8-14.
[28] A. Lauscher, V. Ravishankar, I. Vulic, and G. Glavas, "From zero to hero: On the limitations of zero-shot language transfer with multilingual transformers," in EMNLP. Association for Computational Linguisties, 2020, pp.
[29] A. v. d. Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding." arXiv preprint arXiv: 18XXvv:800.000 1807.037453 2018.
[30] Gao, Geng, Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao, "Clip-adapter: Better vision-language models with feature adapters," International Journal of Computer Vision. vol. 132. no. 2. pp. 581-595, 2024.
[31] A. V. Thapliyal, J. Pont-Tuset, X. Chen, and R. Soricut, "Crossmodal-3600. A massively multilingual multimodal evaluation dataset," in EMNLP. Association for Computational Linguisties, 2022, pp. 715-729.
[32] K. Ding, W. Liu, Y. Fang, W. Mao. Z. Zhao, T. Zhu, H. Liu, R. Tian, and Y. Chen, "A simple and effective method to improve zero-shot cross- pooss-inguul transfer learning," in COLING. International Committee Computational Linguistics, 2022, pp. 4372-4380.
[33] P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury, "The state and fate of linguistic diversity and inclusion in the NL. world," in ACI. Association for Computational Linguistics, 2020, pp. 6282-6293.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please check the last sentence of the abstract and the last paragraph of Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please check the second paragraph of Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
This work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please check Sec. 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification; Upon acceptance, we will include the links to our data and code in the camera-ready version of this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.cc/pil wwwppbblccgguiicuceccccccoe.oopp /eegguddssseCeebbbinninn htt.SSbbbiiiiooooiiiii.oooi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please check the 'Implementation Details' section in Sec. 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
This work does not run experiments multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please check the 'Implementation Details' section in Sec. 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This work complies with the NeurlPS Code of Ethics in all aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work does not present any dataset or benchmark. This work provides strategies enhance the efficiency of parallel corpus- based fine-tuning for downstream tasks.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in this work cited in the References.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This work does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This work does not involve LLMs as any important, original, non-standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
19

---

A
Additional Results
In this section, we report findings from the following experiments: (1) multilingual fine-tuning with the expanded language set, (2) multilingual fine-tuning under scalable trainable parameters, (3)
unimodal multilingual model adaptation, and (4) critical information quantity effects CiPCL. All
experiments are conducted on MSCOCO36-
Loss
PCL-base
CIPCI
08
07
06
05
0.4
03
Figure 7: The loss convergence curve MSCOCOy6 in 29 languages.
A.1 Results with All the Target Languages Considered
To further validate the effectiveness of our method when applied to a wider range of lan- guages, expand the number of languages and exclude languages not involved the pre- training phase of the MCLIP model, resulting in 29 F As shown in Table 5 and Figure 7 our method maintains strong retrieval performance with fewer training iterations even when scaling to more languages. The conver- gence speeds of TaPCL and CiPCL are not slowed by the increased number of languages, remaining faster than the PCL-base. This demonstrates that the effectiveness of our method not constrained by language quantity and consistently enhances fine-tuning efficiency when handling multilingual scenarios.
Table 5: Performance of multilingual image-text retrieval and number of iterations on MSCOCOg6 in 29 languages.
Method
R@ Avg
Iteration
Runtime
PCL-base TaPCL
72.73
10940
36m02s
73.13
7660
25m1 1s
CiPCI
72.51
5470
17m45s
A.2 Scalability under Different Trainable Parameter Numbers
To verify the effectiveness of our method under varying quantities of learnable parameters, configure adapters in different transformer layers. We set i = 8, 9, and 10, and insert adapters into the layer where / followed by model fine-tuning. As shown in 6a and Figure & our method consistently improves line-tuning efficiency across different adapter configurationss demonstrating that our fine-tuning strategy remains effective regardless of the number of learnable parameters.
Table Performance multilingual image-text retrieval and number of iterations on MSCOCOg6 with different amounts of adapters.
Layer >=
Layer >=
Layer >
Method
R@ Avg
Iteration
Runtime
R@ Avg
Iteration
Runtime
R@@ Avg
Iteration
Runtime
PCL-buse
74.31
7039
22m25s
74.53
7039
23m27s
74.70
7039
24m31s
TaPCL
74.51
4929
15m22s
74.72
4929
16m06s
74.94
4929
17m01s
CiPCL
74.39
3519
11m05s
74.52
3519
11m42s
74.46
3519
12m08s
'The 29 languages are Arabic, Czech, Danish, German, Greek, Spanish, Persian, Finnish, French, Croatian, Hungarian, Indonesian, Italian, Hebrew, Japanese, Korean, Dutch, Norwegian, Polish. Portuguese, Romanian, Russian, Swedish, Thai, Turkish, Ukrainian, Vietnamese, Siinese-Simplitidd.
20

---

Lass
Loss
Luss
CUCL-ayen
KCI-tnnolayeri
CUNLL-ayari
PCL-hme-layes
02
557
5xp
24
3t
3+
14
28
(b)
Figure 8: The loss convergence curve on MSCOCO36 36 with different amounts of adapters.
Loss
PCI--ase
TaPCI.
CIPCI
15k
25k
Figure 9: The loss convergence on XNLI.
A.3 Scalability to Unimodal Multilingual Model
To validate the portability of TaPCL and CiPCL, apply them unimodal fine-tuning see- nario. We fine-tune the multilingual pre-trained model
Table 7: Performance of natural language inference and number of iterations on XNLI.
Method
Accuracy
Iteration
Runtime
[21] on the XNLI dataset [1I], which covers 15 languages, with English the source language and target lan- guage training sets derived from English transla- tions. Following Wu al. [14, we adopted full fine-tuning while freezing the parameters of the first 6 layers. As shown in Table 7 and Figure 9 our methods significantly improve fine-tuning efficiency when applied to the unimodal multilingual model, demonstrating their effectiveness even when applied to single-modal scenarios.
PCL--bse
59.97
21874
1h17m
TaPCL
60.46
15314
0h52m
CiPCL
59.84
10939
0h38m
Loss
CIPCL-1p33
CIPCL-op22
CIPCI
300
1.5c
Figure 10: The loss convergence with different quantities of critical information prompts.
A.4 Effeets of Critical-informaatie Quantity
This work further investigates the impact of the number of critical information on CiPCL. Specifically, we select tokens with top-k attention weights relative to the [CL.]] token as key words, where k=2,3. Subsequently, we construct key information prompts using the selected keywords. As presented in Table 8 and Figure 10 observe that neither retrieval performance fine-tuning efficiency
21

---

improved further despite an increase in the number of key information prompts. We speculate that when excessive key information prompts are introduced, these prompts may function as noise relative to the original English samples. This noise effect potentially creates a counterbalance where the positive influence of key information prompting offset by the diminishing effectiveness of cross-lingual transfer.
Table 8: Performance of multilingual image-text retrieval and number of iterations on MSCOCO36 with different quantities of critical information prompts.
Method
@@ Avg
Iteration
Runtime
CiPCL-top-3
75.05
3519
10m46s
CiPCL-top-2
75.01
3519
10m22s
CiPCL-top-1
75.15
3519
09m55s
A.5 Results on Each Target Language
Table 9 reports the detailed results on each target language. We can see that our proposed approaches can obtain competitive performance on each target lanagueg compared to the PCL-base model which directly utilizes the parallel corpus.
Table Performance multilingual image-text retrieval and number of iterations on MSCOCO.s 36 with different quantities of critical information prompts.
Method
English
Greek
Spanish
French
Italian
Polish
Portuguese
Swedish
Ukrainian
Chinese
Source-only 78.87
71.73
73.33
74.30
73.88
70.88
72.23
PCL-hase
78.63
74.45
76.02
75.92
76.05
74.57
75.78
74.72
72.83
73.85
TaPCL
78.70
74.63
76.47
75.37
76.40
75.17
75.93
73.52
73.12
74.58
CiPCL
79.43
74.13
75.98
75.73
76.23
74.33
75.43
74.00
72.33
73.90
22


Reviews:
[{"review_id": "Xjg379qhKD", "reviewer": "Reviewer_vHM2", "summary": "This paper addresses the computational inefficiency of maintaining multilingual capabilities when fine-tuning MCLIP models for downstream tasks. The authors find that token similarity is the key factor affecting cross-lingual transferability in multimodal retrieval. They propose two strategies: TaPCL, which adjusts sampling probabilities based on token similarity, and CiPCL, which augments source text with translated key terms instead of full parallel sentences. Experiments on MSCOCO36 and XM3600 show both methods achieve comparable performance to baselines while reducing training time by 30-50%.", "questions": "Do you have results on languages that have high token overlap but usually are poor in digital resources according to this taxonomy [1]?\n\n[1] Joshi, Pratik, et al. \"The state and fate of linguistic diversity and inclusion in the NLP world.\" ACL 2020.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**: The paper provides a systematic empirical analysis to identify token similarity as the primary factor affecting cross-lingual transfer in multimodal settings, which is a novel contribution to the field. The insights about token similarity could inform future work on multilingual multimodal models. Furthermore, I also think that the proposed methods are intuitive and motivated, with clear mathematical formulations for both TaPCL and CiPCL strategies. That said, I am not super familiar with multimodal models literature to know how novel these two strategies are. The wide language coverage also adds to the strengths of the paper.\n\n**Weaknesses**: A critical weakness is that token similarity is inherently confounded with pretrained representation quality - languages with high token overlap typically receive better multilingual pretraining due to shared subwords and data availability. This makes it unclear whether the observed transferability stems from token similarity itself or simply reflects which languages already have stronger representations in the base model. In addition, the authors say that in Section 5 they only consider \"10 languages which can be well perceived by the pre-trained MCLIP model\" which complicates this fundamental confound. To disentangle this, it'd be encouraging to see results on languages that have low digital resources but have a lot of token overlap with English. \n\nFurthermore, in my opinion, the paper doesn't adequately discuss failure cases or provide guidance on when to use TaPCL versus CiPCL. Scalability concerns can arise from the need to compute pairwise token similarities and the additional translation model requirement for CiPCL. The key term selection in CiPCL (Equation 8) seems simplistic - using only attention weights might miss important contextual information (such as knowledge retrieval pairs [1]). However, I don't think this is a fundamental weakness.\n\n\n[1] Geva, Mor, et al. \"Transformer feed-forward layers are key-value memories.\" EMNLP 2021.", "was_revised": true, "final_justification": "I jumped up my score to 5 because my main concern (about seeing results on the low-resource languages with a lot of token overlap with the English) is addressed.", "rebuttal": "**Q1: Are the improvements from token similarity, or from uneven base‑model language quality?**\n\nA1: Thank you for the question. In this work, we selected 9 target languages with comparable zero‑shot performance to more clearly  isolate the effect of token similarity on transferability. This design effectively removes the confounding influence of base‑model pretraining quality on transfer. The zero‑shot results for these 9 target languages are reported in Table 1, which shows that the base model has similar zero-shot capability on those target languages. Moreover, our conclusions also hold for low‑resource languages: following your suggestion, we evaluated the Source-only method (fine‑tuned only on the source language) on several low‑resource languages from [1], with results in Table 2. We observe that token overlap still has a significant effect on performance gains—languages with higher token overlap achieve larger improvements. We will make this more explicit in the final version.\n\n[1] Pratik Joshi, Sebastin Santy, et al. \"The state and fate of linguistic diversity and inclusion in the NLP world.\" ACL 2020.\n\n**Table 1: Zeroshot performance on 9 target languages**\n|Language|Greek|Spanish|French|Italian|Polish|Portuguese|Swedish|Ukrainian|Chinese|\n|:------:|:----:|:------:|:------:|:-----:|:------:|:--------:|:-------:|:--------:|:-------:|\n|Zeroshot|71.50|72.10|71.99|72.59|70.03|72.06|70.12|70.20|71.91|\n||||||||||\n\n**Table 2: Performance of low digital resource languages**\n|Language|Performance change|Zeroshot|After fine-tuning|Token overlap %|\n|:---|:---:|:---:|:---:|:---:|\n|Danish|1.89|65.88|67.77|18.92|\n|Norwegian|1.77|63.98|65.75|17.17|\n|Romanian|1.08|55.05|56.13|16.03|\n|Quechua|0.99|28.23|29.22|15.87|\n|Swahili|1.05|13.38|14.43|14.39|\n|Indonesian|1.02|66.55|67.57|13.92|\n|Thai|0.33|55.93|56.26|2.67|\n|Hebrew|0.23|66.08|66.31|1.45|\n|Ukrainian|0.68|70.20|70.88|1.85|\n|**R=0.84 P=0.004**|||||\n||||||\n\n**Q2: When to use TaPCL vs. CiPCL?**\n\nA2: Thank you for the question. TaPCL and CiPCL can be chosen as needed. Empirically, TaPCL achieves slightly higher accuracy than CiPCL (+0.24% on MSCOCO, +0.11% on XM3600), but requires more fine‑tuning time (+46% on MSCOCO, +39% on XM3600). We therefore recommend selecting between them by trading off accuracy and training time for the target use case. We will make this guidance explicit in the final version.\n\n**Q3: Scalability and dependence on computing token similarities and translation model.**\n\nA3: Thank you for the question. As target-language training samples are hard to collect, the target-language training data in MSCOCO and XM3600 are obtained by translating source‑language datasets. Under this setup, the use of a translation model aligns with the existing pipeline. As for token similarity computation, since it only uses input-layer token embeddings and is performed once as a pre-training step, computing token similarities for the entire dataset takes only a few seconds and thus does not affect scalability. Therefore, these two designs do not limit the scalability of our approach. We will discuss this more clearly in the final version.\n\n**Q4: Key‑term selection in Eq. 8 may be simplistic.**\n\nA4: Thank you for the question. Since the essence of retrieval is to match images and texts by computing the similarity between their cls token features, considering the attention weights from other tokens to the cls token alone can effectively reflect the importance of different tokens during keyword selection. When additionally considering the memory stored in the Key layer of the feed-forward network (FFN) for identifying input patterns [1], the performance improvement is limited. Specifically, according to the guidance in [1] for knowledge retrieval, the key most relevant to the cls token can be regarded as the one that best identifies its pattern. We compute the relevance between other tokens and this key, and use this value together with the attention weights to guide keyword selection. However, the experimental performance did not improve as expected and instead showed a slight decline, as shown in Table 3. We will provide a more detailed discussion of this in the final version.\n\n[1] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. \"Transformer feed-forward layers are key-value memories.\" EMNLP 2021.\n\n**Table 3: Performance of CiPCL with additional factor**\n|Method|R@Avg|Iteration|Runtime|\n|:---|:---:|:---:|:---:|\n|CiPCL with Key memory|75.00|3521|09m56s|\n|CiPCL|75.15|3519|09m55s|\n||||"}, {"review_id": "9bfvTnWp7m", "reviewer": "Reviewer_Unow", "summary": "# Method summary\n\nThe paper focuses on a problem in downstream adaptation of Multilingual CLIP (MCLIP) models - maintaining performance in other languages that was not present in the downstream fine-tuning, which is usually the case without directly using parallel corpora in other languages. The authors first analyse token overlap degree (Otk) between a source and target lanaguage (IOU over the token vocab), syntactic, phonological and geographic similarlity between the MCLIP model before and after fine-tuning on the source language. The analysis shows that only token overlap degree correlates well to drop in performance. The authors then present TaPCL and CiPCL: the idea behind TaPCL is uniform sampling from target languages can be improved since the analysis shows high overlap indicates less adaptation effort and hence propose a weighted sampling based on the token overlap degree. Furthermore, in CiPCL, the authors propose focusing on top-k critical word key in the source lanaguages and only translating those into target languages and similar weighted sampling strategy to TaPCL is used for source and target languages.\n\n# Experiment setup\n\nThe authors evaluate on MSCOCO and XM3600. \n\n# Results summary\nTable one shows ~1% absolute improvement over the closest baseline on MSCOCO and XM3600 while having a faster runtime and fewer iterations than PCL-base. Ablations suggest all the similarlity measures roughly perform the same and Otk performs the best in CiPCL.", "questions": "No additional questions to the authors, just the ones listed in the weaknesses.", "limitations": "Yes, the authors have addressed limitations but no mention of the broader impact statement which is clearly a part of the author guidelines.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\n1. Clearly written paper with well-described problem statement, methodology, and contributions.\n2. The analysis and motivations are clear in Section 3\n\n# Weaknesses\n\n1.  The authors do not provide any information on seeds etc. With the improvements being really close to the baseline, I wonder how much can we really infer from a single run. \n2. The ablations indicate Otk roughly performs the same - if the correlations were really that bad, the performance changes should also have been reflected the same, however, only runtime / iterations are effected. I'm not sure I see an explanation or hypothesis as to why this happens.", "was_revised": true, "final_justification": "> Thank you for the clarification and updated results on the seeds, the relatively lower std deviation boosts my confidence in inferring the efficiency gains while preserving / slightly improving the performance. Bumping score to 4.", "rebuttal": "**Q1: Reproducibility & statistical reliability.**\n\nA1: Thank you for this question. There may be a misunderstanding on our main focus. As stated in the Introduction, this work primarily targets the fine‑tuning efficiency of multilingual CLIP models rather than maximizing performance. As stated in the Introduction, directly incorporating parallel corpora for fine-tuning will cause substantial computational overhead, hindering the flexible deployment of MCLIP models for downstream applications; thus, it is necessary to improve fine-tuning efficiency. PCL‑base can be regarded as an upper bound on multilingual performance as it uses the full parallel corpus; therefore, our goal is to improve training efficiency while achieving comparable performance with PCL‑base. Regarding random seeds, all experiments in the paper used a fixed, randomly chosen seed (42). To more reliably validate our results, we additionally ran experiments with three seeds (i.e., 43, 44, and 45); the results are reported in Table 1 and Table2. The outcomes are consistent across different seeds, showing performance comparable to PCL‑base along with >30% improvements in training efficiency, which confirms the robustness of our method and the reliability of our conclusions. We will discuss this more clearly in the final version.\n\n**Table 1: Performace on MSCOCO$_{36}$ with different seeds**\n|Method|R@Avg|Iteration|Runtime|Average of R@avg|SD|\n|:---|:---:|:---:|:---:|:---:|:---:|\n|PCL-base seed1|75.30|7036|21m02s|||\n|PCL-base seed2|75.15|7039|21m07s|75.27|0.11|\n|PCL-base seed3|75.36|7035|21m04s|||\n|TaPCL seed1|75.42|4940|14m51s|||\n|TaPCL seed2|75.50|4957|15m06s|75.45|0.04|\n|TaPCL seed3|75.44|4946|14m58s|||\n|CiPCL seed1|75.16|3523|09m59s|||\n|CiPCL seed2|75.10|3518|09m54s|75.16|0.06|\n|CiPCL seed3|75.21|3522|10m01s|||\n|||||||\n\n**Table 2: Performace on XM3600 with different seeds**\n|Method|R@Avg|Iteration|Runtime|Average of R@avg|SD|\n|:---|:---:|:---:|:---:|:---:|:---:|\n|PCL-base seed1|88.22|8100|24m41s|||\n|PCL-base seed2|88.26|8100|24m42s|88.24|0.02|\n|PCL-base seed3|88.23|8110|24m54s|||\n|TaPCL seed1|88.24|5688|16m55s|||\n|TaPCL seed2|88.29|5680|16m50s|88.27|0.03|\n|TaPCL seed3|88.28|5678|16m46s|||\n|CiPCL seed1|88.03|4058|12m14s|||\n|CiPCL seed2|88.10|4044|11m53s|88.06|0.04|\n|CiPCL seed3|88.06|4042|11m52s|||\n|||||||\n\n**Q2: Why weak Otk correlations slow convergence but leave final accuracy essentially unchanged?**\n\nA2: Thank you for this question. As stated in our rebuttal to the above question, our objective is to improve fine‑tuning efficiency, rather than maximize accuracy. When parallel corpora are used, different sampling strategies will, given sufficient iterations, converge towards the same upper‑bound accuracy (comparable to PCL‑base); the difference is how quickly they reach it. Sampling based on other similarity measures leverages source‑language transferability less effectively and thus has lower training efficiency than Otk. Moreover, as shown in Table 3, when training is stopped after the same iteration number, the variants using other sampling methods underperform Otk. We will discuss this in more detail in the final version.\n\n**Table 3: Ablation study of TaPCL under the same number of iterations**\n|Mehtod|R@Avg|Iteration|Runtime|\n|:---|:---:|:---:|:---:|\n|TaPCL-syn|74.21|4929|14m39s|\n|TaPCL-pho|74.42|4929|14m50s|\n|TaPCL-geo|74.17|4929|14m35s|\n|TaPCL-token|75.39|4929|14m31s|\n|||||\n\n**Q3: No mention of the broader impact statement, which is clearly a part of the author guidelines.**\n\nA3: Thank you for this question. As required by the author guidelines, we explained in the checklist why this work does not include a broader impact statement. This work does not present any dataset or benchmark. It proposes strategies to enhance the efficiency of parallel corpus-based fine-tuning for downstream tasks. Therefore, there is no potential societal harm associated with this work. We will more clearly clarify this in the final version."}, {"review_id": "lyxuY182fL", "reviewer": "Reviewer_aAer", "summary": "This paper addresses the issue that MCLIP requires training data for each language when fine-tuning for downstream tasks. From the perspective of optimizing data, the authors propose two methods: TaPCL, which performs sampling based on language similarity, and CiPCL, which retains key knowledge in the data. Experiments conducted on two cross-model retrieval benchmarks demonstrate the effectiveness of the proposed methods.", "questions": "1. Practical Applicability: Under what specific real-world scenarios would it be necessary to fine-tune a model for multiple languages using the proposed methods? Would it not be simpler to fine-tune separate versions of the model for each language and provide these versions to relevant users? Please elaborate on potential use cases or advantages of your approach in comparison to this alternative.\n\n2. Experimental Scope: Why were the experiments limited to only two cross-modal retrieval benchmarks? Is there a specific reason for focusing on these tasks, and do you believe the proposed methods are applicable to other types of downstream tasks? If so, could you provide additional evidence or reasoning to support this?\n\n3. Performance and Impact: The proposed methods yield only marginal performance improvements and, in some cases, slightly underperform compared to PCL-base. Additionally, the time savings of approximately ten minutes may not be impactful in real-world scenarios. Could you provide further explanation or justification regarding these results, and clarify the significance of the observed improvements in practical settings?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "## Strengths \n1. Reasonable Motivation and Preliminary Evidence: The proposed methods are based on sound motivation and are supported by initial experimental results, which provide some validation for their effectiveness.\n\n2. Clear Introduction: The introduction is well-structured and clearly identifies the gaps in existing work, as well as the positioning of this study. This makes the contribution of the paper easy to understand.\n\n## Weakness\n1. Limited Practical Applicability: The real-world applicability of the proposed methods might be constrained. This limitation could be inherent in this type of approach. Specifically, under what circumstances would it be necessary to fine-tune the model for multiple languages? Wouldn't it be simpler to fine-tune a version of the model for each language separately and provide these versions to the relevant users? The authors are encouraged to propose a practical scenario where their methods would be advantageous.\n\n2. Experimental Scope and Task Variety: The experimental setup is limited to two cross-modal retrieval benchmarks. The authors should explain why the experiments were conducted exclusively in this context and whether it would be appropriate to extend the evaluation to other task types.\n\n3. Performance Concerns: The performance improvement brought by the proposed methods is not significant and, in some cases, slightly underperforms compared to PCL-base. While the time savings of approximately ten minutes are noted, such a minor difference in real-world scenarios is unlikely to be impactful. The authors are encouraged to provide further clarification regarding these aspects.", "was_revised": true, "final_justification": "Thank you for the rebuttal which cleared up all my confusion. I will adjust my rating to 4 accordingly.", "rebuttal": "**Q1 Why train a single multilingual model instead of multiple independent monolingual models?**\n\nA1: Thank you for this question. Compared with monolingual models, a single multilingual model offers advantages in training efficiency, inference efficiency, and operational simplicity, and therefore has broad real‑world applicability. Specifically, a multilingual model leverages cross‑lingual similarity to enable knowledge transfer across languages, which reduces training cost and improves performance on low‑resource languages. In multimodal retrieval, queries are naturally multilingual or even mixed‑language inputs (e.g., social‑media search). Additional language‑identification and routing stages would be required when deploying separate monolingual models, which struggle with mixed‑language inputs and introduce extra latency and failure modes. Moreover, language‑specific traffic fluctuates over time; maintaining many monolingual models complicates capacity planning and load balancing. A single multilingual model avoids these issues while providing consistent behavior across languages, making it the more practical choice in many real‑world scenes. We will more clearly clarify this in the final version.\n\n**Q2 Can the proposed methods be applied to other tasks?**\n\nA2: Thank you for this question. Our study focuses on the fine‑tuning mechanisms of multilingual CLIP. As multilingual cross‑modal retrieval is the primary application scenario for multilingual CLIP, we organize our main experiments on this task. To further assess the scalability of the proposed approach, Appendix A.3 reports results on the multilingual natural language inference task. We find that, on this task as well, our method achieves substantial training‑efficiency gains without loss in accuracy. These results verify the scalability of our approach. We will discuss this more clearly in the final version.\n\n**Q3 The practical impact of improving training efficiency.**\n\nA3: Thank you for this question. Our method reduces training time by 30%–50% while achieving the same accuracy as standard multilingual‑CLIP fine‑tuning methods (PCL‑base). This gain is proportional rather than a fixed offset. In real‑world settings, fine‑tuning datasets are often very large—for example, the full WIT dataset contains about 170M training samples, and a single fine‑tuning run with PCL‑base on 4 GPUs takes about 600 hours. Under the same setup, our approach could significantly cut training time by 180 to 300 hours. Moreover, in real‑world scenes, models are updated frequently as new data and new businesses emerge; thus the training‑efficiency improvements delivered by our approach have concrete practical value. We will discuss this point more clearly in the final version."}, {"review_id": "fVxJUV8Y93", "reviewer": "Reviewer_BWpp", "summary": "The paper proposes TaPCL and CiPCL strategies to address the problem of multilingual capability degradation during fine-tuning of Multilingual CLIP (MCLIP). TaPCL reduces redundant computation by reducing the sampling probability of languages with high token similarity, and CiPCL achieves precise knowledge transfer by translating key terms in the source language and inserting prompts. The experiments were verified on the MSCOCO₃₆ and XM3600 datasets, providing a new path for MCLIP to balance computational efficiency and multilingual robustness.", "questions": "1. After the data set is selected and processed, the amount of data is small, and the evaluation of the model's true capabilities may not be very accurate. There is also a risk of overfitting.\n\n2. Token similarity and key information extraction strategies may have certain limitations. \n\nFor example, token overlap cannot distinguish the semantic differences of polysemous words, such as the English word \"bank\" and the German word \"Bank\" (bench) are similar in form but unrelated in meaning. TaPCL may misjudge them as high-similarity language pairs, resulting in a lower sampling probability.\n\nCiPCL only extracts the top-1 key terms and ignores the semantic associations of multiple term combinations. For example, \"artificial intelligence\" needs to be translated into both \"artificial\" and \"intelligence\" to fully convey the semantics.\n\n3. Are all batch sizes set to the same? Because for retrieval tasks, the batch size also has a significant impact on the convergence speed and effect.\n\n4. Are the initial losses of different experimental settings in Figures 4 and 5 consistent? It seems inconsistent. Why?\n\n5. This paper only verifies the effectiveness of the experiment through cross-modal retrieval, and the generalization of multi-tasks is questionable, especially with such a small amount of training data.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nDynamically optimize sampling based on token similarity to avoid redundant training of highly similar languages, reducing the number of iterations by 30%-50%.\n\nOnly key terms are translated and inserted into the source corpus, without the need for full parallel data, reducing the amount of data while maintaining performance.\n\n**Weaknesses:**\n\n1. After the data set is selected and processed, the amount of data is small, and the evaluation of the model's true capabilities may not be very accurate. There is also a risk of overfitting.\n\n2. Token similarity and key information extraction strategies may have certain limitations. \n\nFor example, token overlap cannot distinguish the semantic differences of polysemous words, such as the English word \"bank\" and the German word \"Bank\" (bench) are similar in form but unrelated in meaning. TaPCL may misjudge them as high-similarity language pairs, resulting in a lower sampling probability.\n\nCiPCL only extracts the top-1 key terms and ignores the semantic associations of multiple term combinations. For example, \"artificial intelligence\" needs to be translated into both \"artificial\" and \"intelligence\" to fully convey the semantics.\n\n3. Are all batch sizes set to the same? Because for retrieval tasks, the batch size also has a significant impact on the convergence speed and effect.\n\n4. Are the initial losses of different experimental settings in Figures 4 and 5 consistent? It seems inconsistent. Why?\n\n5. This paper only verifies the effectiveness of the experiment through cross-modal retrieval, and the generalization of multi-tasks is questionable, especially with such a small amount of training data.", "was_revised": true, "final_justification": "I thank the author for their response, which addresses my concerns. I will raise my score.", "rebuttal": "**Q1: The dataset is small, risking overfitting and unreliable evaluation.**\n\nA1: Thank you for this question. Our data scale is sufficient for training and validating an adapter‑based multilingual CLIP fine‑tuning model. Specifically, on MSCOCO we sample 5,000 instances per language across ten target languages, yielding 50,000 training samples in total, as for XM3600, the total number of training sample is 28,800, which are adequate for robust evaluation. In addition, CiPCL essentially acts as a data augmentation method (injecting key terms into the original sentence), which increases diversity and is less prone to overfitting. Moreover, as shown in Table 1, we also provide an explicit overfitting check. The performance gap between the training and test sets is consistently ≤ 1.5%, indicating no memorization effects. Meanwhile, we adopt the adapter model to fine-tuning MCLIP, which is a parameter-efficient fine-tuning technique. Adapter does not update the original pre-trained parameters but instead adds LoRa residual layers, thus reducing the risk of overfitting. Therefore, our evaluation is reliable and overfitting is minimal. We will discuss this more clearly in the final version.\n\n**Table 1: Performance on training set and test set of MSCOCO$_{36}$**\n|Method|Training set R@Avg|Test set R@Avg|\n|:---|:---:|:---:|\n|PCL-base|76.48|75.28|\n|TaPCL|76.95|75.39|\n|CiPCL|76.56|75.15|\n||||\n\n**Q2 Token similarity and key term extraction may misinterpret polysemy or ignore multi-word semantics.**\n\nA2: Thank you for this question. In TaPCL, we measure cross‑lingual similarity using the language–level overall token overlap. Therefore, several polysemous cases do not dominate the decision. The role of token overlap in cross‑lingual transfer has been emphasized in prior work [1–3], and our empirical results in Figure 1 corroborate this. For CiPCL, as shown in Table 7 in the appendix, we compared multi‑key‑term and single‑key‑term prompts, where multi‑term prompts bring no significant gains while increasing training time, so we adopt the top‑1 scheme, which maintains effectiveness while reducing overhead. Moreover, we believe that a single key term can still serve as a strong semantic anchor (e.g., artificial → “人工” (Chinese) or “artificielle” (French)). Therefore, our similarity metric and term‑selection strategy are reasonable and effective in practice. We will discuss this point more clearly in the final version.\n\n[1] Telmo Pires, Eva Schlinger, and Dan Garrette. \"How Multilingual is Multilingual BERT?\" ACL 2019.\n\n[2] Vaidehi Patil, Partha Talukdar, and Sunita Sarawagi. \"Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages.\" ACL 2022.\n\n[3] Tomasz Limisiewicz, Jiří Balhar, and David Mareček. \"Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages.\" ACL Findings 2023.\n\n**Q3: Are batch sizes consistent across experiments?**\n\nA3: Thank you for this question. To ensure a fair comparison, all experiments, including all the baselines and our variants, use the same batch size of 128. Similar results can also be observed based on other batch sizes (e.g., 64 and 256). We will discuss this more clearly in the final version.\n\n**Q4: Why are initial losses in Figures 4 and 5 inconsistent?**\n\nA4: Thank you for this question. The initial loss difference is caused by the sampling strategies differ across methods; thus, the first mini‑batch contains different samples, leading to different starting losses in Figures 4 and 5, but their initial accuracy is the same. By comparing the early‑stage slopes of the loss curves across methods, we can assess convergence speed. From Figures 4 and 5, our approach shows a steeper initial slope, indicating higher convergence efficiency. Moreover, we will add accuracy‑vs‑iteration curves to further validate the effectiveness of the proposed approach in the final version. We will discuss this point more clearly in the final version.\n\n**Q5: Generalization ability for other tasks.**\n\nA5: Thank you for this question. Our study focuses on the fine‑tuning mechanisms of multilingual CLIP. As multilingual cross‑modal retrieval is the primary application scenario for multilingual CLIP, we organize our main experiments on this task. To further assess the scalability of the proposed approach, Appendix A.3 reports results on the multilingual natural language inference task. We find that, on this task as well, our method achieves substantial training‑efficiency gains without loss in accuracy. These results verify the scalability of our approach. We will discuss this more clearly in the final version. We will discuss this more clearly in the final version."}]

Author rebuttal:

Meta-review: This paper analyzed several factors impacting cross-modal retrieval and found that  only token overlap degree correlates well to drop in performance. The authors then proposed two strategies to improve training efficiency:
1) a weighted sampling based on the token overlap degree, and 
2) focusing on top-k critical word key in the source languages and only translating those into target languages.  

Experiments on 2 benchmarks show that the proposed approach reduces the number of iterations by 30%-50% while maintaining performance. Reviewers recognize that this paper is clearly written with well-described problem statements, methodology, and contributions. However, there is moderate training speed gain and no quality gain.
Decision: Accept (poster)