Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment
Authors: Zhen Zhang
Jialu Wang
Xin Eric Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: Cross-lingual Transfer
Vision and Language
Parameter Efficiency

Abstract: Pre-trained vision and language models such as CLIP have witnessed remarkable success in connecting images and texts with a primary focus on English texts. Despite recent efforts to extend CLIP to support other languages, disparities in performance among different languages have been observed due to uneven resource availability.
Additionally, current cross-lingual transfer methods of those pre-trained models would consume excessive resources for a large number of languages. 
Therefore, we propose a new parameter-efficient cross-lingual transfer learning framework that utilizes a translation-based alignment method to mitigate multilingual disparities and explores parameter-efficient fine-tuning methods for parameter-efficient cross-lingual transfer. 
Extensive experiments on XTD and Multi30K datasets, covering 11 languages under zero-shot, few-shot, and full-dataset learning scenarios, show that our framework significantly reduces the multilingual disparities among languages and improves cross-lingual transfer results, especially in low-resource scenarios, while only keeping and fine-tuning an extremely small number of parameters compared to the full model (e.g., Our framework only requires 0.16\% additional parameters of a full-model for each language in the few-shot learning scenario).

Parameter-Efficien Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment
Zhen Zhang", Jialu Wang Xin Eric Wang',
UC Santa Barbara, UC Santa Cruz
zhen zhang @ @ucsb.edu, ffaldict@ @uciceeucsceedu xwang366] ) @ Quusc.edu
Abstract
et al., 2018), dependency parsing (Ahmad et al., 2018), named entity recognition (Rahimi al., 2019). question answering (Lewis et al., 2019), and dialogue (Schuster et al., 2018), among many oth- ers. Recent advancements, such as XLM-R (Con- neau et al., 2019), mBART (Liu et al., 2020), and mTS (Xue et al., 2020), have extended the capa- bilities of large language models in multilingual manner, enabling them comprehend and process multiple languages concurrently.
Pre-trained vision and language models such CLIP (Radford et al., 2021) have witnessed remarkable success in connecting images and texts with primary focus on English texts. Despite recent efforts to extend CLIP to sup- port other languages, disparities in performance among different languages have been observed due to uneven resource availability, Addi- tionally, current cross-lingual transfer meth- ods of those pre-trained models would con- sume excessive resources for a large num- ber of languages. Therefore, we propose new parameter-efficient cross-lingual transfer learning framework that utilizes a translation- based alignment method to mitigate multi- lingual disparities and explores parameter- efficient fine-tuning methods for parameter- efficient cross-lingual transfer. Extensive exper- iments on XTD (Aggarwal and Kale, 2020) and Multi30K (Elliont al., 2016) datasets, cov- cring 11 languages under zero-shot, few-shot, and full-dataset learning scenarios, show that our framework significantly reduces the mul- tilingual disparities among languages and im- proves cross- lingull transfer results, especially low-resource scenarios, while only keep- ing and fine-tuning an extremely small num- ber of parameters compared to the full model (e.g., Our framework only requires 0616% addi- tional parameters of a full-model for each lan- guage the few-shot learning scenario). The codes are available at https: //gi com/ eric-ai- iic-ai-aab/PECTVV
Two-stream vision- language pre-trained model CLIP (Radford et al., 2021) has demonstrated markable performance in image-text retrieval (Cao et al., 2022) by encoding images and text into a shared representation space. However, it primarily focuses on English and cannot comprehend other languages. To address this limitation, Multilingual- CLIP (Carlsson et al., 2022) has been proposed to enhance the CLIP's ability to support multi- ple languages through cross-lingual transfer. Nev- ertheless, Multilingual-CLiP treats English pivot language. leading to performance dispari- ties across languages, especially low-resource lan- guages. While previous work (Wang et al., 2022) has accessed and highlighted this multilingual dis- parity, there is currently a lack of proposed solu- tions to address it.
Multilingual models often encounter a perfor- mance trade-off across different languages (Xin et al., 2022) in the sense that overfitting the model in one language may degrade its performance in another. This can be significant issue the need to train and maintain separate models for each lan- guage can become resource- insuureeinteesivee when dealing with a large number of languages.
1 Introduction
Cross-lingual transfer of models facilitates the transfer of learned representations or knowledge from one language to another. It plays vital role in enhancing the performance of target languages, where the availability of labeled data and linguistic resources are particularly limited. Cross-lingual transfer has found applications in various NLP tasks, including sentiment classification (Chen
The goal of this paper is to address the multi- lingual disparity in parameter-efficient manner. To achieve this, we introduce a framework that extends the capabilities of the Multilingual-CI. model. More specifically, within this framework, we propose a translation- alignment method

---

that effectively minimizes the distribution gap be- tween translated and natural language distributions. This alignment method plays a crucial role in sig-
2 Background
Multilingua-CLIP CLIP (Radford et al., 2021). proposed by OpenAI, is two-stream vision- language pre-trained model with textual and vi- sual encoder. It is trained a large scale image- text pair dataset using a contrastive loss encode the image and text into shared embedding space. CLIP calculate the cosine similarity between image and text features to measure their semantic similar- ity.
nificantly reducing the multilingual disparity ex- hibited by Multilingual-CIII. Artetxe et al. (2023) also point out the importance of machine transla- tion in classification tasks. Additionally, we adopt Parameter-Efficieer Fine-tuning (PEFT) methods (Houlsby al., 2019; Karimi Mahabadi et al., 2021; He al., 2022a; Rücklé al., 2020; Li and Liang, 2021; Guo al., 2020; Hu et al., 2021; Za- ken et al., 2021; Lester et al., 2021a) as solution to achieve parameter efficiency. Furthermore, we find that, in the zero-shot scenario, hard prompt can also reduce the multilingual disparity and improve mul- tilingual ability in addition to parameter efficiency. Compared with full-model fine-tuning each lan- guage, our framework mitigates the multilingual disparity and obtains higher average performance across all languages, using much fewer additional parameters than a single model.
Recently Multilingual-CUA (Carlsson al., 2022) extend CLIP to a multilingual version. This work replaces original English text encoder with a pre-trained multilingual language model such as M-BERT (Devlin et al., 2018) and trains it using teacher learning (Hinton et al., 2015). Although Multilingull-LLPP endowed CLIP with multilin- gual capabilities, the performance of Multilingual- CLIP in other languages is worse than in English due the limited amount of data available in low- resource languages, leading to insufficient training in these languages. Furthermore, training data for other languages are translated from English text, which can result in distribution gap during train- ing and practical application. Noticing this prob- lem, we aim reduce this multilingual disparity in this paper.
We conduct our experiments on XTD and Multi30K datasets covering 11 languages in zero- shot, few-shot, and full-dataset learning scenarios. Through extensive analytical experiments, we ver- ify the effectiveness our framework and provide answers to our research questions. Based on the results of our experiments, we conclude the follow- ing:
Parameter-Efficies Fine-tuning As the size of foundation models (Bommasani et al., 2021) in- creases, fine-tuning and saving the entire model becomes very resource-intensive Many parameter- efficient fine-tuning (PEFT) methods have been proposed to solve this issue. These approaches add additional parameters inside the model (Houlsby et al., 2019; Karimi Mahabadi et al., 2021; He et al., 2022a; Rücklé et al., 2020; Li and Liang, 2021), optimize small portion of the parameters their low-rank decomposition matrix(Guo al., 2020; Hu et al., 2021; Zaken al., 2021), or add train- able token embedding into the input (Lester et al.. 2021a). Moreover, He et al. (2021) and Ding et al. (2022) analyze and combine these approaches from unified perspective. Furthermore, Hu et al. (2022) and Zhang et al. (2022) propose automatic methods to search for an optimal combination of these PEFT methods for language models and visual models, respectively. Many works (Gao al., 2021: Zhou et al., 2022: Zhang et al., 2021: He al., 2022h) also apply PEFT methods to CLIP models. Never- theless, those PEFT methods have not been thor-
1. The Multilingual- CLII model can achieve bet- ter performance than the original CLIP model, but still suffers from a significant multilin- gual disparity. Meanwhile, we find machine translation can map the distribution of text em- bedding to a better initialization and reduce multilingual disparity. (Section 5.2)
2. Mapping the distribution of text embedding to a better initialization and approximating to natural pivot language distribution as a better target can significantly help reduce the multilingual disparity. (Section 5.3)
3. PEFT methods can address the excessive resource consumption of Multilingual-LLIP while maintaining acceptable performance degradation. Moreover, we find that hard prompt in English is very effective the zero- shot learning scenario and can be applied to all languages. (Section 5.4)

---

oughly explored for the Multilingual-LLIP model in the cross-lingual transfer setting. It important to note that PEFT methods often result decline in performance to varying degrees compared to full-model fine-tuning. Therefore, it is essential to conduct experiments to verify the effectiveness of these methods and determine the most appropriate approach for specific tasks and models.
language text pairs, which depict the same image in both the pivot (English) and target languages. Our alignment method has different combina- tions of routines and loss functions to be compared.
Alignment Routines. The term "alignment rou- tine" refers to which two distinct embedding types we are going to align. Given the pivot language (En- glish) and target language, there are three routines in the representation space narrow the gap be- tween embedding distributions as shown in the left part of Figure 1. To be specific, these routines are (1) aligning original English and target language text embeddings, (2) aligning translated English (to target) and original target language text embed- dings, as Multilingual-CIa CLPI is only pre-trained in target language in translation distribution where it performs better than natural language distribution, and (3) aligning original English and translated tar- get language (to English) text embeddings. We compare these three routines in the experiments and find routine 3 performs best. Note that these three routines do not apply to pivot languages (En- glish) and routine 3 still need machine translation in the inference process.
3 Framework
Our main contribution is a cross-lingual transfer framework for Multllnguul-CLIP (Figure 1). In this framework, we propose a novel translation- based cross-lingual alignment method that reduces the multilingual disparity and exploits parameter- efficient tuning methods solve the resource con- sumption problem in cross-lingual transfer.
3.1 Insights of Our Method
Our method is grounded in experimental results (Table 2) and motivated by the work of (Wang et al., 2022). Table 2 shows that machine transla- tion improves M-CLIP''s multilingual disparity and performance with English consistently performing the best. Wang et al. (2022) demonstrates that text embeddings more closely aligned in the transla- tion portion. Our hypothesis that English text can yield embeddings of better quality owing to rich language resource, thus become a good target for alignment. Additionally, translation can bring other languages' embedding to a better initialization for further alignment with English.
A Alignment loss functions. In addition to the alignment routines, alignment loss functions must also be considered. Mean Squared Error (MSE) loss and contrastive loss are two practical loss func- tions for narrowing the distance between embed- dings.
To be specific, the original contrastive loss be- tween image and text embeddings can be written as:
3.2 Translation-basee Cross-lingual Alignment
1 Lize
log
(I)
N
N j=1 (coo(t))t)/
In Figure 1(a). we present a diagram of the translation-based cross-lingual alignment method. The blue circles represent the distribution of natural language embeddings in the representation space. while the orange ones represents the embedding distribution of text generated by machine transla- tion. Since the pre-training of Multilingual-CI. involves aligning English text with translated tar- get text in other languages, there exists a dispar- ity in the distribution of text between training and real-world usage. This gap varies among different languages and contributes the multilingual dis- crepancy. In our approach, we aim minimize this distribution gap by employing machine translation to map one embedding distribution to another. We propose an alignment method using pivot-target
N
Li =
Σ
log
(2)
N
E coos(t,)))/"*
1=1
where o; is the visual embedding of the image in the i-th pair and ty represents the textual embed- ding of the text in the j-th pair. We use i2t and 12i to represent image-text and text-image matching. т is the temperature used to scale the cosine similar- ity. Following CLIP (Radford al., 2021), it is set to 0.01. N is the number of image-text pairs in the dataset. Pivot and target language text embedding can be obtained through text encoder:
p text text_enccoerr(extt
(3)
g text_ ccc.occcroeetcef
(4)

---

(a) Translation-based Cross-lingual Allgnment
(b) Multilngual-CLIP
(c) Transformer architecture and PET modules
Adapter
Aligmment Loss
Contruative Losa
ED,,
Adared
WTT on
Text Encoder
Compacter
PreiTuaing
en
[XX Modiiko
Image Encoder
Compacter
S
EDu
Adkeeer Weiihh
Nachine Translation
LoRA
W
W
LIKAA
Alignment
Pre-Training
60
lgt
Image
Wy
Prompten
Figure 1: Illustration our framework based on multilingual CLIP. We propose a translation-bart alignment method to narrow the distribution gap and adopt PEFT methods to achieve parameter efficiency. We also find hard prompt is very effective in the zero-shot scenario.
Note that texts can be translated from one to another:
usually designed for different tasks, while we in- stead use them for different languages achieve parameter efficiency.
te Transigt-ren (text
(5)
Adapter (Houlsby et al., 2019): Figure I(c) top right. Adapter adds a few trainable linear neural modules after every attention and feed- forward layer. it consists of down sampling matrix Wdown E Rdxr and up sampling matrix W up Rrxd with nonlinear activation function in the middle, where d is the dimention of the in- put x Rd of the adapter. There is also a residual connection and the output O can be written as:
textPivot + Transen-tgul (LextPivor).
(6)
These two different losses to regularize the dis- tance between parallel text embeddings can be rep- resented as:
C
MSE alignment
(ipioot farget
-= x + f down)) WWpp
(10)
e ( (oivas //T//T
1
conntaattive alignment
log
Compacter (Karimi Mahabadi al., 2021):
N
1=1 (onn(1) (enneteees) s/YT
Figure (cc) center right. An improvement work of Adapter. This work replaces the standard Adapter layer with a low-rank hypercomplex Adapter layer, which requires fewer parameters and yiclds compet- itive results. To be specific, Compacter decompose the W E d to the sum of & Kronecker products of matrix A; E Rk** and B; R B; is further decomposed to two low-rank matrices E R xrя and f; E RTax where TB represents the rank of B, Finally, the formula can be written as follows:
(o 1/th//
log
. cos(t 1/T
===
ghoot and & garget refers to the i-th text embedding in pivot (English) and target language. They are added to the contrastive loss between image and text with an alignment coefficient X:
= Liz + Lzzi + A Laligmment-
(9)
R Σ A.BB. = Σ A (1I)
3.3 Parameter--ffficient Cross-lingual
W Wdmon
Transfer Learning
Our framework utilizes Garameter-Efficien Finc- tuning (PEFT) methods to solve the resource con- sumption problem. We compare the following PEFT methods with full-model fine-tuning. When training these PEFT modules, we freeze the pa- rameter of Multilingual-CIn CLIP. PEFT methods are
W up is also decomposed in this way with shared A,
LoRA (Hu et al., 2021): Figure 1(c) bottom right. LoRA assumes the low-rank change of model weights W E kRdx K and then uses two trainable

---

rank-decomposition matrices WA E Edxr and WB E Rr to approximate the matrix change. Consequently, LoRA adds changes to the original output O from the input a::
Flueewak/PPL)
Alg
Kangel
Kangel
wis (ARCELIE1005))
63.4
57.42
475
16.00
65.00
0811
2.15
<0.45%)
59.94
67.50
66.73
1.30
datl-d
Mide (4FCLIZ,
64.67
58.57
4,83
16.05
76.10
74,33
0.70
1.55
0 WWWWB
(12)
(Ckrrs -0.4991)
$4.83
68.45
3.10
10A3
55
75,85
0.56
1.35
Table I: Results on XTD and Multi30K of Multilingual- CLIP (M-CLIP) with and without our framework. We report the Recall score and bold the best result in each scenario each dataset. " "Avg-.nn" represents the average score without English and "PPL" represents rameters per language. Statistical indicators standard de- viation (Std) and range used to evaluate multilingual disparity. PEFT methods used for zero-shot, few-shot and full-dataset scenarios are hard prompt, LoRA and Adapter, respectively.
Following the default setting of LoRA, we apply this method query and value projection matrices (W,, W,) in self-attention layers.
Hard and soft Prompt (Lester et al., 2021b): Figure 1(b) bottom. Hard prompt attaches text prompts to the front of the input text (c.g., "a photo of [TText""), which is manually designed and ex- plainable. CLIP uses multiple hard in the pre- training phase, so we are interested in whether it is applicable in cross-language transfer scenarios. We compare different combinations of the hard prompt and input text in different languages, and find En- glish hard prompt work well on average across all languages. Soft prompt, also known as prompt tun- ing, adds trainable token embeddings to the front of the input. We do not plot soft prompt in our frame- work as our experiments show it doesn't 't perform well.
is a Cross-lingual dataset for the image-text re- trieval task covering 11 languages. It only has test split with 1000 samples per language with the same images. For few-shot setting, we randomly split the original test split into train/dev/test sets with 50/50/900 image-text pairs.
Base Model and Translation Tool We use XLM- LLarge-ViT 1,114 (Carlsson et al., 2022) as our base model. The model fixes the original visual encoder of OpenAl ViT (Radford et al., 2021) and replaces the text encoder by LL.M-Roberta,.arge (Conncau et al., 2019) trained by teacher learning (Hinton et al., 2015). We use Google Translation! strong Neural Machine Translation (NMT) sys- tem, to translate between all the different languages. To reduce the computational overhead, we trans- late the dataset in advance, rather than when it is used. We give the results of the translation the code part. Analysis of different machine translation tools can be seen in Appendix B.
In our experiments, we also tune the linear head and layer-norm layer of the text encoder when we train Adapter, Compacter and LoRA and the num- ber of their parameters 0.44%, 0.05% and 0. 1666 of the text encoder, respectively. Hard prompts only require saving few words, while soft prompts re- quire storing several token embeddings, each of which takes up 1024 floating point numbers in stor- age space.
3.4 Optimization Objective
Our optimization objective is to get optimal pa- rameters of specific PEFT modules on each target language by minimizing the loss L.
5 Experiments and Analysis
4 Experimental Setup
Dataset We work with two datasets: (1)
In this section, we first present the overall re- sults of our framework with the optimal combi- nation and then conduct analytical experiments to demonstrate the effectiveness of machine transla- tion (Section 5.2), routine 3 and MSE loss is the best choice for alignment (Section 5.3) and hard prompt, LoRA and Adapter is respectively outper- forms in zero-shot, few-shot and full-dataset sce- nario (Secion 5.4). The details of our experimental configurations are Appendix A.
Flickr30K (Young et al., 2014) is an image cap- tioning datasets in English, split into train/dev/test datasets with the number of 29000/1024/1000. The Multi3K (Elliott et al., 2016) dataset extends cap- tions of Flickr30K dataset with human translated and independent German sentences. Elliott al. (2017) and Barrault et al. (2018) further translate English Flickr30k captions to French and Czech, respectively. (2) XTD (Aggarwal and Kale, 2020)

---

5.1 Cross-Lingual Transfer Results on XTD and Multi30K
ited multilingual capabilities, with the help of ma- chine translation, can achieve high multilingual capabilities to a certain extent and even surpass Multilnguul- However, Multilingual -LIP with machine translation obtain the best multilin- gual capability and the lowest disparity in both scenarios. We did not use the setting "M-CLIP (en-*ggt) comparison as it is not practical application scenario. On the other hand, there is large multilingual disparity for multilingual CLIP. For example, the difference between Japanese and English is up to 16% and the standard deviation is up to 4.7%. Aided by machine translation, the im- provement of Multllnguul-CLIP on multilingual differences is very obvious. Finally, using data translated from English, Multilingual-LLIP shows large improvement in other languages (mean improvement of 1.6%) but still lower than in En- glish. This may be because the model is pre-trained with text translated from English, making it more adapted to this situation. This also suggests that multilingual disparity are partly the result of dif- ferences in the quality of the datasets in different language instead of the ability of the model.
Table shows the results of Multilingual-CAn with and without our framework on XTD and Multi30K datasets in zero-shot, few-shot, and full- dataset scenarios. We report the Recall @ score on the English dataset and the average score across all other languages for evaluation of the multilin- gual performance and calculate statistical indica- tors, standard deviation, and range for evaluation of the multilingual disparity.
Compared to the vanilla MultlliguullCLiPP our framework outperforms in all zero-shot, few-shot, and full-dataset scenarios. It reduces range by more than 5 points and standard deviation by more than 1.5 points while achieving significant perfor- mance improvement both in English and on average across all other languages on the XTD dataset in both zero-shot and few-shot scenarios, which is a common application scenario of low-resource lan- guages. The improvement in the Multi30K dataset is also significant. In terms of the number of param- eters, our framework is also more efficient than full- model fine-tuning. In eleven languages, Adapter and LoRA only use 4.89% and 1.73% of the pa- rameters respectively, which far less than the parameters of models.
Remark. In terms of the multilingual representa- tion space, machine translation serves the purpose of mapping text from one embedding distribution to another. By mapping text into English, we achieve more favorable initialization of the distribution for subsequent alignment processes. While the embedding distribution of the translated text may differ slightly from that of natural language, this disparity is significantly smaller compared to the distinction between two distinct languages. Conse- quently, it becomes easier to optimize and narrow the gap between these distributions.
To sum up, our framework significantly reduces the multilingual disparity and enables parameter- efficient cross- lonsulln transfer, with a byproduct of improving multilingual performance.
5.2 Analysis on Multilingual Disparity of Multilingual CLIP
Considering that Multilingual-CLIP replaces the text encoder with multilingual version while keep- ing the image encoder of CLIP, we directly com- pare Multilingual-LLiP to the CLIP model equiped with machine translation as a strong baseline (Jain et al., 2021). We compare the performance of the CLIP and Multilingual CL.IP models on XTD dataset with the help of machine translation. we utilize machine translation to convert non- con-Esglish corpora into English. This translated version is then employed as input for both CLIP and Multilingual- CLIP. Additionally, we perform translations of the English dataset into each respective language and evaluate them using multilingual CLIP.
5.3 Analysis on How to Exploit Pivot Language
Datasets in low-resource languages are usually small, and we can obtain the corresponding pivot language (English) text from target language text through human annotation. In particular, for the image caption dataset, annotators can directly give high-quality English captions based on the image without mastering other languages. For (relatively) high-resource language, the parallel text is also a source obtain texts in different languages with the same meaning. We call these texts as pivot- target language text pair. Since the model has higher performance on pivot language and those
Result analysis. As shown in Table 2, we ob- serve that although the original CLIP has lim-
hhaps:/ttanssatt.goog..ccm

---

XID
Method
fr
ko
Avg.t
Avg...
Runge]
CLIP
62.06
25.33
32.94
31.28
25.00
0.56
5.67
1.72
4.50
1.39
6.83
17.93
13.52
19.36
61.50
CLIP&
60.56
66156
58.00
53.72
60.06
56.72
58.27
57.90
3.38
11.34
Zero shot
M-CLIP
63.44
59.94
60.05
58.90
60.72
51,00
61.50
56.11
59.28
59.28
47.44
57.97
57.42
4.75
16.00
M-CLIP
63.44
62.59
62.39
62.61
61.33
61.33
61.33
61.78
62.11
62.44
54.17
61.41
61.21
2.49
9.27
M-CLIP MTT
63.44
61.11
61.33
62.33
61.17
61.44
61.50
54.83
61.67
58.00
52.72
39.96
59.61
3.35
10.72
M-CLIP
64.67
61.83
60.67
61.33
62.00
52.22
62.61
56.33
60.39
59.72
48.61
59.13
58.57
4.83
16.06
M.CLIP: MT
64.67
61.89
62.00
62.17
61.44
61.00
63.00
56.11
62.00
59.50
53.83
60.69
60.29
3.14
10.84
Table 2: Recall @ in percentage image-text retrieval XTD dataset. We compare CLIP and Muliilingual-CT.IP with machine translation as a tool in zero-shot and few-shot scenarios. We average the score to get the overall performance across the languages, and evaluate the multilingual disparity with standard deviation and range. M- CLIP is short for Multilingual-CLII and MT is short for machine translation. We bold the best scores for zero-shot and few-shot respectively. "Avg.en" represents average score without English and "en-*tgt" means data in other languages is translated from English set.
Setting
XTD
Mahi3kk Sidl
This can be explained as that Multilingual-CLI uses MSE loss for text-text pairs in the pre-training stage, and natural English embedding distribution is a better distribution, which denotes an embed- ding with higher performance. Mcanwhile, ma- chine translation maps the target language embed- ding distribution to distribution close to optimal distribution where multilingual CLIP performs best, making it easier to optimize.
Avg-us
SML
Rungel
Arg.ai
Range]
Flo-slact
76.
16/6
74.91
0.70
MT
60.29
3.14
10.34
15.30
0.68
1.45
Scutinc-MMS
38.85
15.73
15.57
0.72
Routine2 MSSE
58.81
15677
75.68
0.55
Roulin:3MMF
6IL52
3.11
10.30
75.97
[U60
1.45
34Ctine344T (pinnttgg)))
60.30
3.12
10.73
75.47
0.98
2.15
KoutinSSCCL.iist-mmapp)
Table 3:
We compare different alignment routines for parallel corpus on XTD and Multi30K datasets. report the score in English once as these combinations cannot applies on English set. CL is short for contrastive loss. We bold the best results on each dataset.
5.4 Analysis on Serameter-Efficies Cross-Lingual Transfer Learning
pivot-target text pairs can provide more informa- tion, there must be better approach exploit pivot language for better cross-lingual transfer of Multilingual-CL..
In this section, we first evaluate the performance of hard prompt. Then we compare Adapter, Com- pacter, and LoRA, and discuss the feasibility of using these methods for cross-lingual transfer.
Comparison of different alignment routines and loss functions. In section 3.2, we mention three alignment routines and two alignment loss func- tions. We compare their different combinations with two baselines without alignment. In the first baseline, we directly apply contrastive loss between images and texts in target language in mini-batch. For the second baseline, we translate all texts to English previously on the basis of the first baseline. We conduct experiments XTD and Multi30K in zero-shot, few-shot, and full-dataset learning scenarios.
5.4.1 Hard Prompt
We primarily focus on the prompting method, particular the hard prompt method, which demon- strates remarkable capabilities in zero-shot learning scenarios where fine-tuming model parameters with domain-speciif data is unfeasible. In a multilin- gual setting, we must take into account two types of prompts. Firstly, prompts can be constructed in multiple languages, potentially leading to perfor- mance disparities across different languages. Sec- ondly, the text input can be automatically translated into any other language using machine translation. Therefore, we explore the following combinations: (1) Simply prepend an English prompt before the text. (2) Translate the optimal English prompt into target languages and append them before their spective texts. (3) Translate all the text inputs English and place English prompts in the front.
As shown in Table 3, we first compare different routines combined with MSE loss and find that rou- tine 3, translating the target language into English and doing alignment between natural and trans- lation English embedding distribution, performs best. Natural refers to "generated by humans rather than machine translation". Then we compare dif- ferent alignment methods with routine and find MSE loss performs best. Ultimately, routine3 com- bined with MSE loss performs best on all 3 metrics.
Result analysis. From results Table 4, can be found that the zero-shot performance increases by simply adding prompt in English, with both

---

Setting
de
ko
iP
Avg.
Arg.ea
Sid]
Range]
Zoro-shot
(1) English Hard Prompe
64,96
60.72
59.89
61.78
61.28
51,67
62,06
56,00
60.11
59.78
47.83
58.65
58.11
4.90
16.23
(2) Tarpet Lang Hand Prompo
64.06
60.83
60.33
62.06
61.23
49.00
61.00
56,00
59.30
59.33
48.06
58.
57.72
5.22
16.00
(3) English Hard Prompt MT
64.06
61,5
61.89
62.25
61.56
60.39
62.39
55.5
61.89
58.78
53.22
60.31
59.94
3.26
10.84
Row-shot
Soft Prompt (3 tokens)
63,94
61.17
62,00
62.17
61.56
60.50
62.44
55,44
61
58.72
53.33
60.28
59.92
3,12
10.61
Soft Prompt (20 tokens)
62.72
89
60.50
60.28
59.83
59.44
60.22
51.33
60.00
58.11
53.11
58.95
58.57
2.81
9.61
Table 4: Results on XTD dataset for comparison of different combinations of prompts and texts.
Satting
Updazed per Leag (90)
sbot)
MultiUK dataset)
de
26
30
pd
zb
jP
Avg.
C4
de
fr
Avg.
FT
100
64.67
01.83
60.67
61.33
62.00
52.22
62.61
56.33
00.39
59.72
48.61
59.15
76.10
74.55
74,80
75:45
75.25
Adapter
0.45
64.44
61.17
G0661
60.72
61.39
52.17
62.83
56.44
59.50
59.67
49.17
58.92
75.20
74.50
73,85
76,40
74,99
Compueter
63.67
60.61
60.41
52.06
62.67
55.91
59.39
58.67
74.25
72.40
73.35
73.55
73.51
LoRA
0.16
6483
61.10
61.94
56.67
60.50
72.30
73.65
74.35
Our famework
0.16/00.5
64.30
61.94
62.00
62.00
61.33
61,55
62.72
55.94
62.33
59.84
54.23
6082
75.33
75.40
75.15
76.40
75.58
Table 5: Comparison of different PEFT methods. We care about the degree of performance degradation caused by different PEFT methods. We bold the best score among the three PEFT methods and bold the score of our framework it is better than full-model fine-tuning. Our framework updates 0.16% and 0.45% parameters for few-shot and full-dataset scenario, respectively. The non-"Our framework" rows show results without translation alignment.
text input in target language and translated into English. We compare different hard prompts and find "a photo of" performs best. However, Translat- ing English prompt into target language makes the performance decrease slightly. Finally, we get the best performance by translating all the text inputs into English and adding the best English prompt to them.
parameter efficiency and outperform full-model fine-tuning. From the results shown in Table 5, we can find that Adapter performs the best in full- dataset scenario, which preserves 99.7% perfor- mance of fine-tuning and LoRA performs the best in few-shot scenario, which even achieve almost the same performance. We further adopt Adapter and LoRA with the best combination discussed in Section 5.3, which forms a part our final frame- work, and find this combination obtains better per- formance than full-model fine-tuning. It indicates that with the help of PEFT methods, our framework can achieve the parameter-cfficieney without losing too much performance.
Comparison with soft prompt. We also con- duct experiments on soft prompt based on the third combination: initiate prompt from the best prompt, which result in 3 trainable token embeddings, and randomly initiate 20 token embeddings. With 50 training instances in the few-shot scenario, the model obtains marginal improvement or even per- formance decreasing by utilizing these templates. As result, we do not incorporate soft prompt in our framework.
5.4.3 Summary of PEFT Methods
In summary, this section presents three primary contributions of PEFT methods: (1) We conducted comprehensive investigation of PEFT methods in cross-lingual transfer learning scenarios using the M-CLIP model. Through this study, we discov- ered novel insights. For instance, in multimodal and cross-lingual transfer settings, we found that adapters are often not the optimal solution, and employing English prompts tends to yield superior performance compared to multilingual prompts. (2) Our framework demonstrated improved parameter efficiency compared to full-model fine-tuning, pecially when considering the vast array of avail- able languages. (3) We achieved enhanced trans- fer learning performance, significant accomplish- ment considering that PEFT methods generally exhibit lower performance than full-model fine-
5.4.2 Other PEFT Methods
this section, we compare three popular PEFT methods, Adapter, Compacter, and LoRA, with full-model fine-tuning. Following He et al. (2022a), we unfreeze the linear head and assign the same learning rate as fine-tuning. The number of pa- rameters of Adapter, Compacter and LoRA is only 0.45%, 0.05% and 0.16% of Multilingual- -CLIP's text encode, respectively. Thus, even if we assign different parameters to each of the 100 languages, the total number of parameters smaller than that a single model. As a result, we utilized language- specific modules, yet our method can still achieve

---

tuning in other research works.
Wasi Uddin Ahmad, Zhisong Zhang, Xuezhe Ma, Ed- uard Hovy, Kai- Wei Chang, and Nanyun Peng. 2018. On difficulties of pross-lingual transfer with order dif- ferences: case study on dependency parsing. arXiv preprint arXiv: 1.00570.
6 Conclusion
In this paper, we propose a framework that sig- nificantly mitigates the multilingual disparity of Multilingual-CIIP in parameter-efficient manner in zero-shot, few-shot and full-dataset scenarios. Our framework uses translation-baal alignment method and adopts parameter-efficiett tuning meth- ods. Analytical experiments indicate that machine translation is effective for cross-lingual transfer; exploiting pivot language can help reduce the dis- parity; parameter-efficient tuning methods are ben- eficial for reducing resource consumption without too much performance degradation.
Mikel Artetxe, Vedanuj Goswami, Shruti Bhosale, An- gela Fan, and Luke Zettlemoyer. 2023. Revisiting machine translation for cross-lingual classification. arXiv preprint arXiv:2305. 14240.
Loic Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Find- ings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference Machine Translation: Shared Task Papers, pages 304-323.
Rishi Bommasani. Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bersstein, Jeannette Bobg, Antoine Bosse- lut, Emma Brunskill, al. 2021. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07255
7 Limitations
Our work primarily focuses on addressing multilin- gual disparity by improving the multilingual text encoder in the CLIP-liked framework. However, is very possible that the visual encoder can also be enhanced with image and text data from diverse cul- ture and languages. During the pre-training process, the original Hultilingual-CLIPss visual encoder is directly aligned with English corpora only, and con- nected with other languages by using English as a pivot language. We expect future work can ad- dress the multilingual disparity problem from the perspective of a more powerful visual encoder.
Min Cao, Shiping Li, Juntao Li, Liqiang Nie, and Min Zhang. 2022. Image-text retrieval: A survey on recent research and development. arXiv preprint urXiv:2203.14713
Fredrik Carlsson, Philipp Eisen, Faton Rekathati, and Magnus Sahlgren. 2022. Cross-lingual and multilin- gual clip. In Proceedings of the Language Resources and Evaluation Conference, pages 6848-6854, Mar- seille, France, European Language Resources Asso- ciation.
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2018. Adversarial deep aver- aging networks for cross-lingual sentiment classifica- tion. Transactions of the Association for Computa- tional Linguistics, 6:557-570.
Broader Impact
This work provides framework for cross-lingual transfer in few-shot learning setting. The deploy- ment of our method is potential to mitigate the per- formance disparity for state-of-the-ee art multimodal models for scarce-resource languages. However, we note that our method relies on the collection of parallel corpus, either collected from online ma- chine translation systems or native human speakers. Our work does not thoroughly scrutinize whether these parallel corpus contains implicit social biases in different dimensions, such as race, gender and religion. When these parallel corpus contains un- expected biases stereotypes, it is likely that the model learned from such data may perpetuate these biases that we did not foresee.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2019. Unsupervised lings-lingual representation learning at scale, arXiv preprint 191.00216.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2018, Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 110.00880.
Ning Ding. Yujia Qin, Guang Yang, Fuchao Wei, Zong- han Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2022. Delta tuning: A comprehensive study of parameter efficient meth- ods for pre-trained language models. arXiv preprint arXiv:2203.0699
References
Pranav Aggarwal and Ajinkya Kale. 2020. Towards zero-shot cross-lingual image retrieval.
Desmond Elliott, Stella Frank, Loic Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the

---

second shared task on multimodal machine transla- tion and multilingual image description. In Proceed- ings of the Second Conference on Machine Transla- tion, Volume Shared Task Papers, pages 215-233, Copenhagen, Denmark. Association for Computa- tional Linguistics.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. In Anmual Meeting of the Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint
Desmond Elliott, Stella Frank, Khalil Sima'an an, and Lu- cia Specia. 2016. Multi30k: Multilingual english- german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70-74. Association for Computational Linguistics.
Brian Lester, Rami Al-Rfou, and Noah Constant. 202 la The power of scale for parameter-efficien prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Peng Gao, Shijie Geng, Renrui Zhang. Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. 2021. Clip-adapter: Better vision- language models with feature adapters. arXiv preprint arXiv: 10.004444
Brian Lester, Rami Al-Rfou, and Noah Constant. 202 The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv: 08601.
Demi Guo, Alexander M Rush, and Yoon Kim. 2020. ceaumeter-efficienn transfer learning with diff prun- ing. arXiv preprint arXiv:2012.07463.
Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. MIqa: Eval- uating cross-lingual extractive question answering. arXiv preprint arXiv: 1910:7791.
Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Berg- Kirkpatrick, and Graham Neubig, 2021. Towards unified view of parameter-effiieet transfer learning. arXiv preprint arXiv:21 10.04466.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv: 200.0000.0
Xuehai He. Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. 2022a. Parameter-efficient fine-tuning for vision transformers. arXiv preprint arXiv:2203.16329
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- rions of the Association for Computational Linguis- rics, 8:726-742.
Xuehai He, Diji Yang. Weixi Feng, Tsu-Jui Fu, Ar- jun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, and Xin Eric Wang. 2022b. Cpl: Counterfactual prompt learn- ing for vision and language models. arXiv preprint arXiv:2210.1083 10362.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark. et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. 2015. Distilling the knowledge a neural network. arXiv preprint arXiv: 0. 2(7).
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In In- ternational Conference on Machine Learning, pages 2790-2799. PMLR.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Mas- sively multilinuual transfer for ner, arXiv preprint arXiv:1902.0019.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2020. Adapterdrop: On the effi- ciency of adapters in transformers. arXiv preprint arXiv:2010. 11988.
Edward Hu, Yelong Shen, Phil Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. 1021. Lora: Low-rank adaptation of large language models.
Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2018. Cross-lingual transfer learning for multilingual task oriented dialog. arXiv preprint arXiv:1810.13322
Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang. Yasheng Wang. Zhiyuan Liu, and Maosong Sun. 2022. Sparse structure search for delta tuning. In Advances in Neural Information Processing Systems.
Jialu Wang. Yang Liu, and Xin Wang. 2022. Assess- ing multilingual fairness in pre- pre-trained multimodal representations. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2681- 2695, Dublin, Ireland. Association for Computational Linguistics.
Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Ja- son Baldridge. 2021. Mural: multimodal, mul- titask retrieval across languages. arXiv preprint 1rXiv:2109.0515..

---

Derrick Xin. Behrooz Ghorbani, Ankush Garg, Orhan Firat, and Justin Gilmer. 2022. Do current multi-task optimization methods in deep learning even help? arXiv preprint arXiv:2209. 1177.
To find an optimal combination of hyperparam- eters, we conduct grid search on learning rate and guidance coefficient A. The learning rates lie within (3e-5, le-4, 3e-4) for parameter-efficient methods, and ( (1e-6, 3e-6, le-5) for fine-tuning the whole model. The guidance coefficients fall within large range from 0.001 to 10. The optimal λs vary from language language, but most of them distributed in smaller interval from 0.1 to
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.119: 11934.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions visual
B Analysis on Different Machine Translation Tools
denotations: New similarity metrics for semantic in- ference over event descriptions. TACL, 2:67-78.
Different machine translation models may have var- ious impacts the experimental outcomes. Our decision to Google Translate was based its widespread adoption and high-quality translation results, which reflect the general situation. Higher- quality professional translation tools could poten- tially map target language embeddings to an even better initialization for subsequent alignment with English, further reducing disparity.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Gold- berg. 2021. Bitfit: Simple ceameter-efficienn fine-tuning for transformer-bused masked language- models. arXiv preprint arx: 200. 10199.
Renrui Zhang, Rongyao Fang. Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2021. Tip-adapter: Training-free free clip- addater for better vision-language modeling. arXiv preprint arXiv:21/1.03930.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. 2022. Neural prompt search. arXiv preprint arXiv:2206.04673.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. 2022. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 16816-16825.
A Experimental Details
We use Adam (Kingma and Ba, 2014) as our opti- mizer with a cosine decay learning rate scheduler. For few-shot on XTD dataset, we train the model for 40 epochs and use the batch size of 10, which results in a total of 200 steps. When we train our model on the whole Multi30K, we set the number of epoches to 15 with batch size of 48 due to the memory limitation and the total steps is 9k. When the memory insufficient, we appropriately reduce batchsize to adapt. We evaluate the performance every 5/300 steps respectively and save the best checkpoint for test. For a more obvious compari- son, We report Recall @ score in all experiments. We froze the image encoder for fair comparison since the difference between languages is in the text input and tuning image encoder will introduce additional randomness. All the experiments are done on Nvidia V100 GPUs. We use the official code of CLIP and Multilingual-CCn to load and use pre-traine training parameters


Reviews:
[{"review_id": "svP4GA7EBS", "reviewer": "Reviewer_qRQE", "summary": "", "questions": "A. 9-12: why does cross-lingual transfer requires additional resources from a large number of languages? If we do zero-shot we don't need any resources in other languages. If we do few-shot we only need a few examples from other languages. \n\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Limited novelty:\n\nWhile aligning representations to English has been explored for text models, we thank the reviewer qRQE for acknowledging that our work is the first to study this for vision-language models in a multilingual setting. We agree that the novelty in directly applying existing methods is limited. However, this work surfaces unique challenges in cross-lingual transfer for multimodal models compared to prior work focused only on text. For example, the visual encoder trained only with English corpora exacerbates disparities when transferring to other languages.\n\nWe believe the key novelty lies in providing the first comprehensive benchmark on cross-lingual transfer techniques for CLIP-like models. We analyze the effects of different alignment routines and losses, yielding practical guidelines for reducing multilingual disparity. We also provide novel insights into optimal strategies for prompting multilingual CLIP, showing English prompts work best across languages. Further, we reveal that different parameter-efficient tuning methods are preferable depending on data availability (e.g. LoRA for few-shot, adapter for full dataset).\n\nRegarding the analysis of soft/hard prompts, prompting methods can behave quite differently for vision-language models compared to text models. Our systematic comparison of prompting techniques provides new insights specific to multilingual CLIP. While prior work [1] shows in-language results are competitive to English for text models, we find the improvement from English prompts is more pronounced for multimodal models.\n\n[1] Discrete and Soft Prompting for Multilingual Models (Zhao & Schütze, EMNLP 2021)\n\n## Explanation of our claim:\n\nAlthough XLM-R is pre-trained on multiple languages, it is only aligned to the English representation during the process of aligning to the visual encoder when pretraining M-CLIP. This is where the disparity arises. Our experimental results in Table 2 show the model's performance in English is the best, and translating to English significantly boosts performance. This indicates English is the optimal language to align representations with. We believe our existing experiments sufficiently support this claim.\n\n## Questions for the Authors:\n\nWe intended to highlight that naively fully fine-tuning separate models per language can become resource-intensive in few-shot and full-dataset scenarios. This motivates exploring parameter-efficient approaches like ours for a uniform multilingual model. We apologize for any confusion and will reword this point for greater clarity.\n\n## Missing References:\n\nThank you for pointing out these relevant missing references, which we will be sure to include in the camera-ready version.\n\n## PET vs PEFT:\n\nWe agree PEFT is a more standard abbreviation and will update to use PEFT instead of PET.\n\n## Minor Edits:\n\nThank you for catching these minor issues. We will fix the typos, remove the extra period, and address any other presentation issues you noted.\n\nWe sincerely appreciate you taking the time to provide thoughtful and constructive feedback. We hope our response has properly addressed your suggestions, and hope you will consider our rebuttal when making your final recommendation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper conducts comprehensive experiments under zero-shot, few-shot and full fine-tuning. All of the author's claims are sound and well supported by experiments. To my surprise, their method can even outperform full-model fine-tuning under certain scenarios, showcasing the effectiveness of their method.\n\n2. The author shows that by aligning representations of different languages to English, one can improve cross-lingual transfer in image-text models. Which I believe, is the first work on cross-lingual transfer on Vision-Language models.\n\n3. The paper benchmarks several PEFT methods, alignment methods as well as comparison between soft and hard prompts on vision-language models, which can be used to facilitate further research in this area. The author also presents insightful conclusions and recommendations for model trainers on which prompting method to use. ", "reasons_to_reject": "1. Limited Novelty: Although to the best of my knowledge this is the first work of cross-lingual transfer on vision language models, one of the main findings \"aligning to English can improve performance\" has been studied before in text [1, 2, 3]. Analysis between soft and hard prompts under multilingual setting have also been studies before but only in text [4, 5, 6, 7]. Directly applying existing methods to a new task without demonstrating new insights on the specific task offers limited novelty.\n\n2. One of the authors claim: The multilingual CLIP's encoder is only aligned to English representation is not well supported. Since the authors use XLM-R large, which is a multilingual model, then it should be not aligned to English. To strengthen this claim, the authors need to conduct translate-test/train to other languages as well to show that English is the best language to align or translate into as a \"pivot language\". There has been research as well showing that other languages can also be good pivot languages [8]. \n\n[1] Emerging Cross-lingual Structure in Pretrained Language Models （Wu et al., ACL 2020)\n\n[2] Learning Invariant Representations on Multilingual Language Models for Unsupervised Cross-Lingual Transfer (Xian et al., ICLR 2022)\n\n[3] A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning (Ding et al., COLING 2022)\n\n[4] Discrete and Soft Prompting for Multilingual Models (Zhao & Schütze, EMNLP 2021) \n\n[5] Enhancing Cross-lingual Natural Language Inference by Prompt-learning from Cross-lingual Templates (Qi et al., ACL 2022)\n\n[6] Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizer (Li et al., ACL Findings 2023)\n\n[7] Enhancing Cross-lingual Prompting with Dual Prompt Augmentation (Zhou et al., ACL Findings 2023)\n\n[8] Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer (Turc et al., 2021)", "questions_to_authors": "A. 9-12: why does cross-lingual transfer requires additional resources from a large number of languages? If we do zero-shot we don't need any resources in other languages. If we do few-shot we only need a few examples from other languages. \n\n", "paper_topic_and_main_contributions": "This paper proposes to use parameter efficient fine-tuning methods to reduce the disparity between different languages in vision-language pre-trained models (CLIP). The authors conduct comprehensive experiments under zero-shot, few-shot and full dataset scenarios. They propose to align the target language translated to english representation to english representations (this can be seen as improving the  the translation quality). They find that using parameter efficient fine-tuning methods can effectively reduce the multilingual disparity, resulting in better performance on image retrieval tasks over strong baselines including translate-test. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Highly relevant work but contemporary: Revisiting Machine Translation for Cross-lingual Classification (Artetxe et al., 2023) \n\nSee the references in \"reasons to reject\" section.\n"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "First, I would suggest to change \"PET\" to \"PEFT\" since \"PET\" usually refers to \"Pattern Exploiting Training\" in the literature [1]. \n\n627: an excessive period at the end\n\n[1] Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference (Schick and Schutze, EACL 2021)\n\n"}}, {"review_id": "zqPt0VeQb0", "reviewer": "Reviewer_WwLy", "summary": "", "questions": "- Table 5, are the rows other than \"Our framework\" represent normal training without using the proposed translation-based alignment method? \n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback on our paper. We appreciate you taking the time to provide detailed comments. Please see our responses below:\n## Novelty\nWhile our method builds on existing techniques like pivot language alignment and contrastive learning, we argue that the novelty comes from the effective combination and adaptation of these methods to the new problem of mitigating disparities in multilingual multimodal retrieval. Meanwhile, this work surfaces unique challenges in cross-lingual transfer for multimodal models compared to prior work. We show strong results from bringing them together and add more discussion in the paper to highlight this synthesis. We believe the specific combination and application to multilingual multimodal retrieval is a unique contribution. The translation-based alignment helps mitigate disparities, and efficient fine-tuning allows scaling up without sacrificing performance. We argue our main contributions are: (1) Providing the first comprehensive benchmark on cross-lingual transfer techniques for CLIP-like models, analyzing alignment methods and loss functions to offer guidelines for reducing multilingual disparity; (2) Novel insights into optimal prompting strategies for multilingual CLIP, demonstrating English prompts work best across languages; (3) Revealing different fine-tuning methods excel given data availability (e.g. LoRA for few-shot, adapter for full dataset).\n\n## Section 5.4 as a Technical Report:\nWhile section 5.4.1 contains some technical specifics, the broader analysis and discussion across section 5.4 suggest this work does not constitute a purely technical report. Therefore, we do not consider this to be entirely a technical report. However, we appreciate you for your suggestion, we will consider revising this section to make it better.\n\n\n## The relationships between translation-based alignment and PEFT\nOur translation-based alignment method enables the multilingual CLIP model to learn better multilingual representations. The PEFT techniques then allow efficient fine-tuning of the aligned model on downstream tasks. We leverage both alignment and efficient tuning to obtain strong performance on multilingual search with minimal computational overhead. The performance improvement is primarily attributed to the translation-based alignment, while PEFT is introduced to address resource consumption issues. Without PEFT, the feasibility of the translation-based alignment would be severely compromised.\n\n\n## Table 5 and Training Details\nThe reviewer is correct - the rows besides \"Our framework\" correspond to training without translation alignment, and serve as baselines. Our description in section 5.4.2 \"we compare three popular PET methods, Adapter, Compacter, and LoRA, with full-model fine-tuning\" is not very clear. We will update the caption to clarify that the non-\"Our framework\" rows show results without translation alignment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper conduct comprehensive experiments and analysis about different routines of translation-based alignment for multilingual-CLIP, and various PEFT methods. \n- The performance gain of the proposed framework is clear.", "reasons_to_reject": "- The weakness of this paper is that the proposed method is a little lack of novelty, specifically, it based on combinations of existing ideas (using pivot languages for alignment), objective functions (contrastive loss and MSE loss), and PEFT methods (Adapter, LoRA, etc). \n\n- Although there are a lot of experiments conducted in the paper, it is hard to see the relationships between the translation-based alignment and PEFT. \n\n- Section 5.4 is more like a technical report rather than an academic paper.", "questions_to_authors": "- Table 5, are the rows other than \"Our framework\" represent normal training without using the proposed translation-based alignment method? \n", "paper_topic_and_main_contributions": "This paper utilizes a translation-based alignment method to mitigate multilingual disparities in multilingual multimodal retrieval task, and apply various parameter-efficient fine-tuning methods on this task. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UGp06FLgyc", "reviewer": "Reviewer_tqnJ", "summary": "", "questions": "A. To my understanding, zero-shot evaluation does not involve any training, why do the results on English in Table 1 w/o and w/ the proposed framework differ? Don’t they use the exact same model?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable feedback on our paper. We appreciate you recognizing the contributions of our paper. Below we address your concerns and suggestions:\n\n\n## Question about Table 1\nThis part is in response to the second reason for rejection, and Question A.\nTable 1 provides an overview of the results of our translation-based alignment method with PEFT in different scenarios. Most of the results in Table 1 can also be found in other tables. For example, the results on XTD in the zero-shot setting are shown in the third row of Table 2 (results without prompting) and the fourth row of Table 4 (results with prompting). The results in the few-shot and full-dataset settings are detailed in Table 5. To directly answer Question A, the baseline is without prompting and our framework in English uses prompting. To clarify, we will add more explanation in the revised version and provide a detailed Table 1 in the appendix.\n\n\n## First reason for rejection: Too few datasets\nWe agree that using more datasets would strengthen our conclusions. Due to constraints on computational resources and the limited availability of suitable datasets, we were only able to experiment on a small number of datasets. For instance, XTD only has about 1000 instances, which we split into two parts for the few-shot experiments. We believe the existing results sufficiently demonstrate the performance and effectiveness of our method. As future work, we will conduct more experiments as more datasets become available.\n\n\n## Third reason for rejection\nWe acknowledge that the 0.3 performance improvement is not very large in absolute terms. However, this improvement is significant given the difficulty of the task and compared to other methods discussed in the paper. First, the model must choose the most relevant image out of 1000 for a given text, and vice versa - a challenging task where a vanilla M-CLIP achieves only 57.97 on average across languages. Second, prompting alone gains just a 0.68 improvement (3rd row of Table 2 vs. 1st row of Table 4). Therefore, the gains from our method are noteworthy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Improved average performance and narrowed range across languages compared with the original M-CLIP and the machine translation aided version.\n* Insights from various alignment routines.\n* Detailed dataset and hyperparameter setting.\n", "reasons_to_reject": "* Few-shot and full-dataset evaluations are only conducted on only one dataset respectively, which is not enough to support the conclusion.\n* Table 1 is too brief. Table 2 is much more informative. I suggest merging them or providing a detailed Table 1 in the appendix so that one can evaluate whether the improvement is consistent.\n* Compared with the baseline M-CLIP+MT, the proposed approach only obtains marginal improvement (less than 0.3 for Avg$_{\\rm -en}$).", "questions_to_authors": "A. To my understanding, zero-shot evaluation does not involve any training, why do the results on English in Table 1 w/o and w/ the proposed framework differ? Don’t they use the exact same model?", "paper_topic_and_main_contributions": "This paper aims at extending the monolingual vison-language model CLIP to the multilingual setting based on a previous work Multilingual-CLIP (M-CLIP in short). The authors claim M-CLIP suffers from performance disparities across languages and a gap between the translation distribution during pre-training and the natural language distribution during inference. To this end, they propose a translation-based alignment approach, making three main differences from M-CLIP. First, they use datasets with independent captions for each language rather than machine-translated captions from English; Second, the alignment is conducted within English space, i.e., between the original English captions and the back-translated captions from other languages; Third, they train separated language specific parameters for each language with parameter-efficient fine-tuning techniques to obtain better performance. Empirically results on XTD and Multi30K under zero-shot, few-shot, and full-dataset scenarios demonstrate the newly proposed approach is more effective than M-CLIP.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In L280 Equation 8, one of $i$s in the exponents of the denominator should be $j$."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a translation alignment objective to increase the performance of multilingual vision/language models. The reviewers all agree that the experiments in the paper are comprehensive and sound. However, the main complaint is the lack of novelty because the paper seems to combine several existing approaches together. Given that the paper conducts comprehensive study of these approaches on multilingual vision-language setting, this paper could still have value for readers interested in this direction.
Decision: Accept-Findings