Multi-Lingual Acquisition on Multimodal Pre-training for Cross-modal Retrieval
Authors: Liang Zhang
Anwen Hu
Qin Jin
Venue: NeurIPS 2022 Accept (2022)
Keywords: cross-lingual cross-modal retrieval
multilingual representation
multimodal representation

Abstract: Vision and diverse languages are important information sources in our living world. A model that understands multi-modalities and multi-languages can be applied to a wider range of real-life scenarios. To build such a multimodal and multilingual model, existing works try to ensemble vision-language data from multiple languages in pre-training. However, due to the large number of languages, these works often require huge computing resources and cannot be flexibly extended to new languages. In this work, we propose a MultiLingual Acquisition (MLA) framework that can easily empower a monolingual Vision-Language Pre-training (VLP) model with multilingual capability. Specifically, we design a lightweight language acquisition encoder based on state-of-the-art monolingual VLP models. We further propose a two-stage training strategy to optimize the language acquisition encoder, namely the Native Language Transfer stage and the Language Exposure stage. With much less multilingual training data and computing resources, our model achieves state-of-the-art performance on multilingual image-text and video-text retrieval benchmarks.

Multi-Lingual Acquisition on Multimodal Pre-training for Cross-modal Retrieval
Liang Zhang', Anwen Hu', Qin Jin 122+
School of Information, Renmin University of China
Key Laboratory of Data Engineering and Knowledge Engineering (MOE),
Renmin University of China (zhangliang00, anwenhu, qj @ruc.. wnw.rruueeuuueeeommmmm edu.
Abstract
Vision and diverse languages are important information sources in our living world. A model that understands multi-modalittee and aulti-languuges can be applied a wider range of real-life scenarios. To build such a multimodal and multilingual model, existing works try ensemble vision-language guages in pre-training. However, due to the large number of languages, these works often require huge computing resources and cannot be flexibly extended new languages. In this work, we propose a Multi-Lingual Acquisition (MLA) frame- work that can easily empower monolingual Vision- Vision-Language Pre-training (VLP) model with multilingual capability. Specifically, we design lightweight language acquisition encoder based on state-of-the-ar monolingual VI.P models. further propose two-stage training strategy to optimize the language acquisition encoder, namely the Native Language Transfer stage and the Language Exposure stage. With much less multilingual training data and computing resources, model achieves state-of-the performance on multilingual image-text and video-text retrieval benchmarks.
data from multiple lan-
Introduction
We live in a multimodal and multilingual world. The information we receive our daily lives may come from different modalities and lan- guages. Therefore, building multimodal and multilingual models to effectively understand such information has attracted much research attention [12 38 21 3]. Recently, Multilin- gual Vision-Language Pre-training (M-VLP) achieves convincing performance in various cross- lingual cross-modal tasks such as multilin- gual image-text retrieval [28]
Image Data
Text Data
Monolisgua Lasgon Pre trining
Fr
Image
IN
Multilinual Visionel angunge Pre-emining
(b) Multi-i ingual Acquisition
Comparison data usage between M- VLP and MLA. The size of circle reflects the amount of training data.
Figure 1;
[8) and multimodal machine translation 35 As shown
in Figure T ()). M-VLP models handle multiple languages and modalities simultaneously during pre-training. Despite their successes, M-VLP mod- els suffer from problems. First, pre-training vision and multilingual data consumes huge computing resources. For example, the state-of-the-at M-VLP model MURAL [18 is pre-trained on 128 Cloud TPUv3 for four days. It could support multimodal tasks on 100+ languages. However, considering there are 6,900+ languages worldwide [44), building such a single model to handle all
"Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

languages will be highly expensive. Second, M-VLP models cannot be flexibly extended to new languages. Additional training is required for M-VLP models to achieve satisfactory performance on a new language. However, this training process will cause performance degeneration of M-VLP models on the original languages due to the limited model capacity. For example, the limited model capacity even results in M-VLP models performing worse than their monolingual counterparts on English [28 44].
To build multimodal and multilingual models with low-cost and high-flexibility, we refer to our human learning habits when acquiring new languages. We humans normally learn our native language during childhood and practice it through interactions with the multimodal living environments. When learning a new language, we humans initially tend to align it with the native language, as we can easily map words in the native language to real-wort woold objects and concepts. After having a certain language foundation, we could further master by interacting with the environment directly using the new language. This is known as the language exposure [5] The whole learning process rarely degrades our native language capability.
Inspired by this, we propose a new framework, multimodal and multilingual models based on monolingual VLPs. The topology of the MLA-based multimodal and multilingual model is illustrated in Figure lb). Unlike M-VLPs, which handle data from multiple languages and modalities in a single model, MLA empowers monolingual VLPs with multilingual capability using much less training data through a language acquisition encoder. The language acquisition encoder realized by inserting our proposed lightweight language acquirers into the pre-trained monolingual encoder of the VI.P model. During training, original parameters in the pre-trained monolingual encoder are fixed, only multi-lingual embeddings and language acquirers for each new language are optimized. Following the human learning habits, we propose a two-stage training strategy to train the language acquisition encoder. In the Native Language Transfer (NLT) stage, the model is trained to establish the correspondence between the new languages with the native language. In the Language Exposure (LE) stage, the model is optimized to build cross-modal alignment between new languages and images. We apply our proposed MLA to the monolingual VLP model CLIP [1 and achieve state-of-the-ee results on both multilingual image- text and video-text retrieval benchmarks with much less training data and computing resources. Ablation studies demonstrate the effectiveness of our training strategy. Owing to the independence merit of the language acquirers, the MLA-based models without compromising the performance of their original languages. Since cross-modal mooaal retrieval is a
Acquisition (MLA), which constructs
be easily extended to new languages
fundamental task for multilingual and multimodal learning, and most M-VLP [28 44 [8] works
evaluated using this task, we focus on cross-modal retrieval in the paper. The application of MLA to other tasks will be explored in our future work.
The main contributions of our work are as follows: sition (MLA) framework that can casily empower monolingual VLPs with multilingual capability. 2) We propose a two-stage training strategy to optimize the MLA- models inspired by the language learning habits of humans. Ablation studies prove the effectiveness of the strategy. 3) We apply MLA to the monolingual VLP model CLIP and achieve the new state-of-the-ar results on both multilingual image-text and video-text retrieval benchmarks with much less training data and parameters.
1) We propose a lightweight Multi-Lingual Acqui-
2 Related Work
Vinion-Language Pre-training: There are increasing interest in building Vision-Language Pre- training (VI.P) models. From the perspective of how to interact between vision and language modalities, existing models can be divided into two categories: single-stream and dual-stream models. The single-stream models perform interaction on image and text directly with a cross- modal transformer 7 26 22]. In contrast, the dual-stream models encode image and text with two independent encoders and optimize via simple objectives like image-text contrastive learning BI 19 (3). Compared with the single-stream models, the dual-stream models are more efficient to utilize noisy image-text data harvested from the web [17], and thus achieve better performance on downstream tasks. Meanwhile, the dual-stream models are more flexible for extension. Since the dual-stream models process images and text through independent encoders, we can fix the vision encoders and focus on extending the text encoders to support new languages. Therefore, we focus on empowering dual-stream VLPs with multilingual capability in this work.

---

Multilingual Vision-Language Pre-training: To achieve both multilingual and multimodal capa- bility, many works try to learn the relationship between multiple languages and modalities simulta- neously through pre-training. M [28] introduces the multimodal code-switched training method to enhance multilingual transferability. UC 44 augments the English image-text data to other languages through machine translation and proposes fine-grained pre-training objectives to encourage alignment between image regions and multilingual tokens. More recently, MURAL [8] adopts the dual-stream structure. It is pre-trained with image-text and text-text contrastive objectives on multilingual image-texi pairs and translation pairs. M-VLP models significantly outperform previous non-pe non-rreraaining models [22 38 21 3 on multilingual image-text retrieval. Despite their success, these models typically consume huge computing resources and large-scale multilingual training data. Moreover, they fail to take full advantage of the cross- modal knowledge learnt in monolingual VLP, and building cross-modal cross-lingual representations from scratch can be very hard. In contrast, our MLA framework aims to empower VL.P models with multilingual capability and it builds multimodal and multilingual models with much less data and computing cost.
Multilingual Extension: Some works explore making pre-trained monolingual language models multilingual 32, 30 36). Reimers et al. [32 extend sentence embeddings from monolingual to multilingual by Multilingual Knowledge Distillation (MKD). Artetxe et al. (2] extend monolingual
models by training additional word embeddings.... MAD-X B3] extends multilingual pre-traine pre-training
models to support low-resoure, languages through adapters (15]. By extending state-of state-of-the-ate ---th---rt pre- pre-trained language models, these works have achieved impressive results in NLP tasks such as bitext retrieval 32. lross-lingual QA and NER 30 32]. However, few works focus making VLP models multilingual. OSCAR+ (29 extends single-stream model OSCAR (6 to multilingual for VQA tasks by adopting similar strategy with MAD-X 30 It trains language adapters with Masked Language Modeling (MLM) for each language, and replaces the English adapter with the target language one during inference. This strategy [30 29 shares the same idea with MLA that handles different languages through independent structures but with two core differences: i) It is designed for MLM-based VLPs and cannot be applied to dual-stream models such CLIP [T. Consequently, it is not suitable for cross-modal retrieval considering the gap between retrieval and MLM. ii) It generalizes poorly on other languages since MLM can only implicitly establish the correspondence between these languages and English, let alone vision correspondences. In contrast, MLA directly builds the connection between the other languages with English and then with vision through the two-stage training strategy. It thus achieves comparable results on these languages as on English in the downstream retrieval tasks.
3 Method
The Multi-Lingual Acquisition (MLA) framework is proposed to empower dual-stream monolingual VLP model with multilingual capability. We define the native language of VLP as its pre-training language. In this paper, we choose CLTB-VVT-B [3I] as the VLP model. is pre-trained with 400M image-text pairs in English 31 Note that MLA can also be applied to VLP models with different native languages.
Since the state-of-the-art VLP models can project vision and native language into shared multimodal space, we design language acquisition encoder to process non-native languages. We then simulate the learning habits of human beings and propose tto-stage training strategy to optimize the language acquisition encoder. We first introduce the architecture of the MLA framework in Sec Then, we describe our training strategy in Sec 3.2
3.1 Architecture
Figure 2 (a) illustrates the overview of the MLA framework, which consists of three modules: the pre-trained text encoder, the pre-trained vision encoder, and the language acquisition encoder.
Pre-trained Text Encoder. Given sentence S in the native language, the corresponding sentence representation s ((S; 0x) is generated through the pre-trained text encoder 4. To preserve the cross-model knowledge of VLP, 04 is keep fixed during training. As shown the top part of Figure 2 a). the pre-trained text encoder contains a native embedding block and / transformer layers 337. The native embedding block first tokenizes S with byte pair encoding
(BPE) [3. Then, [SOS] and [EOS] are special converts words into embeddings Es = [co= [eo=[sos] C1, ... e M= [Bc3]]
3

---

Pre trained Tesl Ercoder
man and his dogs.s.
Native Cyrbbedding
Transformrr byee1
Trandormer byyer
Stage
Language Acquirer
Langaage Acqurer
Shared
EXLT
Acquiition Frceder
Add Norm
Upper Projection
nathes
Traratorer lypen1
Transformer
Feed-forward
Stage two +
Add & Norm
Down Projection
Proiidired Visao bruoder V
Attertion
Image ennbedding
mransformr byye-1
Transformer bayend
(a)
(b)
Figure 2: Model illustration: (a) The overview of MLA framework. (b) The structure of language acquirer
tokens denoting the boundary of S. The word embeddings are then passed through the transformer layers:
H |eo= [s0s] ci....,, [M=[E09) + Epos
H TransformeeLaaerr(f 0%))
(2)
where H = [h6...., N is the hidden state of the layer i. 0g denotes the parameters of the layer i. Eppo is the positional encoding. Note that the causal self-attention mask is used in the transformer layers [31]. The last hidden state of the [EOS] token is chosen to generate the sentence representation:
W Wah
where 8 is the sentence representation of S, and W. denotes linear projection.
VVVV; of image V with the pre-trained vision encoder V. Similar with the pre-trained text encoder, By is also frozen. The pre-trained vision encoder is implemented of Figure a), it consists of a image embedding block and transformer layers. Given an image V, the image embedding block first divides V into patches V [uf..... I'yl following ( Then, they are linearly projected into patch embeddings E, [e(CLASs], W Wprh.-. W. Wy'') where e(cLASS) is special embedding for the whole image and Wp the linear projection. The patch embeddings are then fed into transformer layers:
Pre-trained Vision Encoder. We extract the representation
Vision Transformer [9]. As shown in the bottom part
z = [etCLASs], W Wou'l + Epos
(4)
z Transforerrlayer 0%,)
(5)
where 7 [ embedding o selected produce the representation of image V:
= ...,, the hidden state of the layer The last hidden state of the [CLASS]
Wia6
(6)
where v is the image representation of V, and We denotes a linear projection.
Language Acquisition Encoder. As shown in the middle part of Figure 2a), the language quisition encoder built upon the pre-trained text encoder. Suppose T is a sentence written in non-native language L, get the representation of T through language acquisition encoder t = 4'(T; 0g. 0emb, 01), where Og are fixed parameters of the pre-trained text encoder, 0emb refers to shared non-native embedding block and 0g represents specialized language acquirers for language L. Non-native sentence is first tokenized and processed into word embeddings E, [un= [sas] ... uM= [ens]] through the non-native embedding block. The word embeddings then encoded through the pre-trained transformer layers and language acquirers:
x [W_40= W. W1 W....... W.um==(os)) + Epos
(7)
H TransformerLayer(X"-1
(8)
X LA(H
(9)

---

[=) Enn is the hidden state of the layer i. We is linear projection to keep dimension consistency, 0% denotes the parameters of the i-th language acquirer for language As shown in Figure b)) the language acquirer is implemented as a bottleneck MLP with residual
where X
connection 14]
LA(X) = LW down X) + X
(10)
Similar with the pre-trained text encoder, the last hidden state of the [EOS] token is projected into the sentence representation t:
t = W
(11)
Eq 11 shares the same linear projection Wa with Eq 3 The main advantage of the language acquisition encoder is that it can extend the VLP models to support new languages without influencing the existing languages, it handles different languages with independent language acquirers.
3.2 Training Strategy
To simulate the language learning habits of humans, we optimize the model in two stages: the Native Language Transfer (NLT) stage and the Language Exposure (LE) stage.
Native Language Transfer. When learning new language, we humans initially tend to align with the native language. To simulate this learning phase, we align the non-native representations to the native representations during the Native Language Transfer (NLT) stage. Specifically, suppose {(S1, , (Sn, } are translation pairs, where S; is in the native language, and T, is in non-native language L. The objective in the NLT stage is minimizing the Mean Square Error (MSE) between the native representation s; = (S;; fa) and the non-native representation t; s' (T;; 84 Br, A emm)):
B LNLT Σ
|
(12)
where B is the batch size. Note that ls is loaded from the VLP model and is kept frozen. Og is trained for non non-native language L. Bemb is shared among non-nat non-native languages.
During the NLT stage, the non-native language correspondence with vision can be built by pivoting on the native language, since the correspondence between the native language and vision is well established through VL.
After the NLT stage, the model has built implicit connection between non-native languages and vision. However, due to the existence of synonyms, two same words in the native language may correspond different images. Thus, ambiguity may arise when learning non-native languages solely by relying on the native language. Actually, we can regard the language acquisition encoder after the NLT stage as a person with certain language foundation. He/She has learned the basic usage of a language through native language teaching. To master it, he/she may practice the non-native language by interacting with the multimodal living environments. Inspired by this learning phase, we directly establish the cross-modal alignment between non-native languages and vision during the Language Exposure (LF) stage. Given image-text pairs ((V,, Ti), (Vn, T where T; is in non-native language L, the sentence representation f; = ob' (T;; 0g 0L. Bemb) should be closer the aligned image representation v; (VVV; Bv), and away from the misaligned one v, = (Vy; j.j # i. This can be achieved by performing contrastive learning between non-native languages and images. For non-native sentence Ti, we treat the corresponding image V; a positive sample, and other images in the same batch Vj,j o i as negative samples. Vice versa for images. The objective in the LE stage is minimizing the NCE loss [13 defined as follows:
Language Exposure.
1 LLE
(f L12w)
(13)
2
B
exp(sim(e: (v, tT)/T)
L
log N
(14)
B
1= exp(sim( U., tx)/
i=1
B
exp(sim( (v, LT)
Liz0
log
(15)
B
1= exp(sim Uk. t/)/T)
i=1
5

---

where B is the batch size. sim(a, y) a temperature nyper-parameted to scale the logits. Note that though the image-to-tex loss Loze is optimized, the pre-trained vision encoder is kept frozen during training. Similar to NLT, the trainable parameters LE come from the language acquirers and the non non-native embedding block.
x is the cosine similarity between vectors. is xe 3r
Experiments
4.1 Dataset Description
We train our model with the Conceptual Captions (CC) dataset 34 and two translation enhanced versions of the CC image-text retrieval evaluation, and MSRVTT [39 16 for multilingual video-text retrieval evaluation. Conceptual Captions (CC) 34] contains 3.3 million image-text pairs in English crawled from the Web We also randomly select 300K image-text pairs denoted as CC300K for training our model to show the low-cost merit of MLA. For multilingual sentences, we leverage two translation augmented CC datasets: (1) CC6L 44] that translates all English captions of the CC into 5 languages (German French(fr), Czech(es). Chinese(zh)) 3) and (2) CC69L 44] that contains 27K captions in each of the 68 languages translated from English Considering the languages of the downstream datasets, we train the model with CC6L, for multilingual image-text retrieval, and with CC69L for multilingual video-text retrieval.
44 4
We use Multi30K (TO. MSCOCO 6
25 40) and XTD ① for multilingual
Multi30K I0] is built upon Flickr30K 41 The English(en) captions are manually translated into German(de), French(fr) and Czech(cs). It contains 3 31K images paired with 5 captions per image in English and German, and I caption in French and Czech. We use the standard train, dev and test splits defined by Young et al. [4]]
MSCOCO (6) contains 123K images with 5 English captions per image. Yoshikawa et al. 40
annotate 5 Japanese captions per image, and Li et al. [25] extends MSCOCO with Chinese captions for 20K images. We follow the standard train, dev and test splits for English and Japanese as in 20
only perform zero-shot evaluation the full splits have overlaps with English and Japanese splits.
For Chinese,
the test split defined by Li et al. 25 as
XTD ① provides captions in 11 languages (English(enn), German(de), French(fr), Chinese(zh), Japanese(ja), Italian(it), Spanish(es), Russian(ru), Polish(pl), Turkish(tr). Korean(ko)) for 1K MSCOCO images. Except for Japanese, all non-English captions are translated from the English caption directly. We use this dataset for zero-shot image-text retrieval evaluation only.
MSRVTT 39 is a video caption dataset with 10K videos, where each video is annotated with 20 English captions. Huang et [66 translates the English captions into 8 languages (German(de). French(fr), Russian(ru). Spanish(es), Czech(ez). Swahili(sw), Chinese(zh) and Vietnamese(vi)) via machine translation service. We follow the standard train/dev splits in [39, and evaluate on the 1K test split as described in 42
4.2 Implementation Details
We apply MILA on two VIP models: CLIP-VIT-B-32 and CLIP-ViT-B-16 [11, denoted as MI.AcıIP and MLACLIP16 respectively. The hidden dimension of the language acquirers is set to 256, and all language acquirers for each non-native language cost only 3.14 MB parameters. The non-native embedding matrix is initialized with M-BERT [8) It costs 92.2 MB and shared with all non-native languages. We train two separate models for multilingual image-text retrieval and video-text retrieval. For the image model, we train with CC6L 44 For the video model, we use multilingual captions from CC69L [4. For both models, we optimize multiple language acquirers iteratively with a batch size of 128. The NLT stage performs 157.150 steps with learning rate of Ie-4, and the LE stage performs 11 11,715 steps with a learning rate of 3e-6. The temperature is set to 0.01. For both stages, we use the Adam optimizer 23 with linear warm-up for the first 10% of steps. The whole training process takes about 12 hours to converge on 1 Nvidia V100 GPU.
We can only access 2255 million images due some broken URLs. 3 'Datsset released at wtwptt:tt//iiiie /ww/////////////me////// under MIT license. 'Released at https://githhu http:/twwww..oo/rtddeerraaaaaaae http:///eweaaaaaaaaaaaaaaaaaagaa ingual CLIP under MIT license. We remove captions of unaccessible images, leaving ~0K captions for each language.

---

Table I: Multilingual image-text retrieval results Multi30K and MSCOCO. TrTrain: Translate- train, FT-En: Fine-tune on English, FT-AIl: Fine-tune on All. o: Models re-implemented by us. t: Models trained with publicly unavailable datasets. I: Models fine-tuned COCO-CN [25], which has an overlap train split with the test split of English and Japanese. Best results are in bolded and second best are underlined.
Multi30K
MSCOCO IK MSCOCO. 5K
Method
Training Data
en
de
Cs
en
ja
en
ja
Unicoder-Vi VI.
CC3M (English only)
72.0
63.7
ALIGN
AT-en (English only)
84.3
80.0
M
CC3M+ Wiki
$7.9
36.8
27.1
20.4
63.1
33.3
Zero- shot
UC?
TrTrain(
66.6
62.5
60.4
55.1
70.9
62.3
MKDOLIP
TVTrain(CC3008
82.1
77.1
75.2
72.3
78.5
73.6
MURAL
TrTrain(CC300K
67.8
62.7
60.8
57.5
68.1
62.5
43.3
37.1
MURAL
TrTran(CC12M)-EOON
80,9
76.0
75.7
68.2
78.1
72.5
58.0
49.7
MURAL
AT+MBT
82.4
76.2
75.0
64.6
79.2
73.4
59.5
54.4
ML.ACLIP
(CC300)CC)300
84.4
78.7
77.7
70.8
79.4
74.9
60.5
54.1
MLACLIPIC
TCTrinnCC3000K
80.8
72.9
80.9
76.7
57.0
M
CC3M+
87.4
82.1
67.3
65.0
88.6
56.0
FT-Ea
UCa
TVTrain(CC3M)
87.2
83.8
77.6
74.2
88.1
71.7
MURAL
TTTriin(CC300S
82.8
76.5
70.1
84.5
74.6
$2.5
MLACLIP
(CC30K)
82.6
76.2
89.3
80.4
62.1
MLACLIPIC
(CC300)
94.5
86.4
87.3
79.5
91.3
82.6
79.4
65.5
M
CC3M+Wiki
87.7
82.7
72.2
88.7
87,9
UC
TVTrain(CC3M)
88.2
84.5
83.9
81.2
88.1
87.5
FT-AII
MURAL
TTTrain(CC300K)
83.4
80.3
79.4
76.8
84.8
83.2
62.9
MURAL
(C12))FFCC
91.0
87.3
86.4
82.4
89.4
87.4
73.7
71.9
MURALI
AT+MBT
92.2
88.6
87.6
84.2
88.6
88.4
75.4
74.9
MLACLIP
TCCr0k)(C3300K)
92.0
86.8
85,4
82.3
89.3
88.1
75.7
73.2
MLACLIPIC
TCCrai(CCC300K)
94.5
89.7
89.2
85.9
91.3
90.4
79.4
76.5
4.3 Evaluation Multilingual Image-Text Retrieval
In multilingual image-text retrieval, models are given a sentence in a certain language to find the most semantically relevant image from image database and vice versa. We compare model with state-of-the- multilingual directly evaluate the model without fine-tuning downstream datasets. (2) Fine-tune English: we first fine-tune the VLP model on downstream English data. We then insert the language acquirers and non-native embedding block into the fine-tuned model and evaluate other languages directly.
lasion- pre-training methods under three settings: (1) Zero-shot:
(3) Fine-tune on All: after (2), we fine-tune the language acquirers and non-native embedding block the downstream dataset and freeze other parts of the model. Following previous works (28 44 18).
we report Average Recall (AR), which is the average score over Recall Recall 5, and Recall @ 10 on two retrieval directions (image comparison, we re-implement MURAL [88 (marked with o in tables) and train it with the same data MLA.
ttxx, text, text-imagee). The results are shown in Table ! For fair
Under the Zero-shot setting, we observe that MLACLIP performs significantly better than state-of-thh-ar M-VLP models on English. This
Table 2: Comparison over trainable parame- ters (#TP) and computing costs (Costs).
Method
Langs
#TP
Costs
because MLACLIP could completely main- tain the strong English performance of CLIP. In contrast, M-VI.P models typically perform worse than their monolingual counterparts on English PP 57.9 vs. Unicoder-VI (24) 72.0,
Mºp
100
566 M
8xV100x<d
UC
6
478 M
8xV100x4d
MURAL
109
300 M
128xTPUV3X40
MLACIIP
6 108 M
IxV100x0.5d
MURAL 80.9 vs. ALIGN [19] 84.3). MLACLIP also outperforms M-VLP models on other lan- guages. For example, MLACLIP achieves 78.7 average recall on German by using 300K image-text pairs, outperforming MURAL pre-trained with the same data by 16.0% and CC12M+EOBT by 2.7%. It demonstrates that MLA is -igf-deta-effcciet method to empower monolingual VLP models with multilingual capability. Meanwhile, M-VLP models like MURAL require pre-training over a large amount of data to achieve convincing performance. Under the Fine-tune on English setting, MIA shows strong cross-lingual transfer capability. Under the Fine-tune on All setting, MLACLIP performs slightly worse than MURAL which was pre-trained on publicly unavailable datasct AT+MBT [8]. We consider the reason is that MURAL has larger transformer- based text

---

encoder (768 hidden dimensions, 12 heads) compared to MLACLIP (512 hidden dimensions, 8 heads). This makes easier for MURAL to the downstream datasets with a certain scale such as Multi30K and MSCOCO. MILIIII achieves tate-of-thhe-r results on all languages under three settings. These results suggest that MLA could probably reach better performance on multilingual image-text retrieval if stronger VLP models such as ALIGN-L2 [9] or Florence [43] are provided.
4.4 Evaluation on Multilingual Video-Texs Retrieval
In multilingual video-text retrieval, the model searches for the most semantically relevant videos given a text query certain language. Following 277, we first uniformly sample 12 frames from each video, and use the pre-tree pre-trained vision encoder to extract representations for each frame. We then perform mean pooling over frame representations to get the video representation. We also evaluate the models under three settings as in S.c We report the text Recall@ score in 3 Under Zero-shot setting, MLACIIP which is trained on CC69L without using any video data, achieves comparable or even better results than the fine-tuning results of the state-of-the-art M-VLP model XLM-R-MMP [T6] on several languages (de: 20.1 vs. 21.1; fr: 22.0 vs. 21.8; es: 20.2 vs. 21.9). Under the Fine-tune on English and Fine-tune on All settings, MLACLIP also outperforms XLM-R- -MMMP significantly. We consider the convincing performance comes from two reasons: 1) CLIP is strong VLP model that can generalize well video data. 2) The proposed MLA framework can well transfer the open-domain knowledge learned by CLIP to other languages. These results suggest that MLA could maintain the open-domain capability of the VLP model which generalizes well on different downstream data.
Table 3: Multilingual video-text retrieval results on MSRVTT. ZS: Zero-shot
Method
en
de
fr
es
zh
ru
Vi
SW
es
mean
FT-All FT-En zs
DuCLIP w/o LE)
30,8
18.3
18.9
14.5
18.6
12.6
7.2
10.2
19.3
16.7
Ours(MI ACLIP)
30.8
20.1
22.0
15.7
18.3
14.4
8.2
10.7
20.2
17.8
XLM-R-MMP [6
23.8
19.4
20.7
19.3
18.2
19.1
8.2
8.4
20.4
17.5
Ours(MLAccui)
42.5
26.1
26.7
20.5
25.3
18.9
12.9
12.6
27.2
23.6
XLM-R-MMP [6
23.1
21.1
21.8
20.7
20.0
20.5
10.9
14.4
21.9
19.4
Ours(MLAcLii))
42.5
33.1
34.5
30.5
31.6
28.9
16.9
24.3
33.5
30.6
4.5 Ablation Studies
A. Training Strategy
We conduct an ablation study in Table to val- idate the effectiveness of the proposed MLA training strategy. For those settings with NLT and LE at the same stage, we add the loss of the two objectives together during training. By comparing row 1 to row 2&3, we observe that LE at stage one leads to poor performance. This indicates that aligning with the native language is more important for the VLP model to acquire new languages
Table 4: Ablation study on training strategy.
Row
Stage
Stage
Multi3KK
MSCOCO IK
NLT
LE
NLT
LE
de
fr
es
ja
76.3
74.2
67.2
72.1
75.7
68.2
67.7
58.6
71.7
71.1
69.7
59,8
67.6
73.9
V
78.7
77.7
70.8
74,9
78.5
78.4
77.3
69.9
74.2
78.1
early stage. It is consistent with the learning habits of humans. By comparing row 1 and row 4, we see that I.E at stage two could bring improvements on the new languages. Additionally, comparing row 4 and row suggests that optimizing the model with NLT and LE together at stage two does not bring improvements.
B. Language Acquirers and Embedding Initialization
In order to validate the effectiveness of the pro- posed Language Acquirers, we remove the lan- guage acquirers and the M-BERT embedding initialization from the model respectively and evaluate on zero-shot multilingual image-text retrieval. As shown in Table 5 the performance on all languages drops significantly without lan- guage acquirers. Meanwhile, initializing the embedding with M-BERT ( only brings incremental improvements. It indicates that the language acquirers contribute most to the performance, and MLA does not depend much on the initialization of non-native embedding.
Table 5: Ablation study over Language Acquir- ers (LA) and Embedding Initialization (EI).
Multi30K
MSCOCO IK
Methods
de
fr
ja
zh
MLACLIP
78.7
77.7
70.8
74.9
78.5
MLACLIP wlo LA
76.1
74.9
65.7
70.3
76.5
MLACIIP w/o EI
77.9
76.2
69.4
74.6
78.1

---

C. Comparing with Machine Translation based method
We compare MLA with translate-based method CLIP+MT as follows. For non- English text queries in Multi30K and MSCOCO, CLIP+MT first translates them into English with strong commercial machine translation service Iflytel and then performs image-text retrieval. As shown in Table 6 the CLIP+MT performs
Table 6: Comparing with the machine translation based baseline.
Multi30K
MSCOCO IK
Method
fr
ja
zh
CLIP+MT
75.3
73.0
66.3
72.2
74.2
MLACLIP 78.7
77.7
70.8
74,9
78.5
significantly worse than MLA in all languages. This is because the translate-based method can introduce extra translation errors or ambiguity although the commercial translation service has been optimized with a large volume of multilingual corpus. By directly optimizing the alignment between vision and each language, MIA does not suffer from this issue.
D. Low-resource Languages
Image-text pairs may be rare for low-resource languages. To explore the performance of MLA under this situation, we further simulate low- resource scenario using XTD dataset. We fine- tune MLACLIP, UC² and MURAL with small amount of data from XTD in an unseen lan- guage. Both UC2 and MURAL are pre-trained on CC6L with 3M images. We randomly sample 600 pairs for finetuning, and the remained 400 samples are evenly divided for validation and testing. Korean is chosen to perform simulation as its script and language family are not covered by CC6L. Experimental results in Table 7 show that MLA can achieve competitive results with very small amount of text-text pairs only (row 3), and adding image-text pairs brings further improvement (row 4). It demonstrates that MLA still attractive method for low-resource languages even without any image-texi text pairs.
Table 7: Low resource performance on image- Korean retrieval.
Training samples
Row
Method
Data
100 200 600
UC
Img- Txi
47.0/60.17
MURAL
Both
53.2/ 60.8 73.3
3
MLAcuP
Txt-Txt
51.7 /2.8/ 78.7
4
MLACTaP
Buth
56.7 / 66.9/ 80.1
E. Amount of Training Data
We conduct experiments to control the numbers of image-text pairs used for each language. We train the models with CC6L and evaluate on MSCOCO IK and Multi30K under the zero-shot setting. The corresponding mean AR over non- English languages (de, fr, ja. zh) are drawn in Figure 3 We observe that MLA performs significantly better than MKD 32 in all cases.
75-
Mean Average Racall
10
65
60
Note that when the amount of training data is small, the advantage of MLA is more obvious, which could outperform MKD even without the LE training stage. Additionally, when training with only 30K image-text pairs per language, MLA outperforms UC², which pre-trained with 3M pairs per language. MLA is thus data-efficient method to build multilingual and multimodal models.
MLA w/o LE
UC2
30K
100K
300K
# of image-text pairs per language
Figure Mean AR vs. Number of data pairs.
F. Language Extensibility
Multilingual models often encounter the need to support new languages that do not occur in the training stage. We conduct language extension experiments to compare MLACLIP with M-VLP model UC2 [4 on the XTD dataset [1. XTD supports 11 languages, and 5 of them (en, de, fr, cs, zh, ja) are seen the pre-training stage of UC², while other 6 languages (it, es, ru, pl, tr, ko) are unseen. To make fair comparison, we first train MLACLIP with the same data UC² and then train both them on unseen languages with CC69L. The zero-shot image-text retrieval results on XTD are shown in Table 8 We observe significant performance degeneration on the seen languages for UC2 when training solely with unseen languages (row vs. row 2). Even keep training with the seen languages, the performance is still significantly reduced due to the limited model capacity (row I vs.
https://global. wtw.obaal..yuu...o

---

row 3). In contrast, as MLA decoupled multiple languages through acquirers, the performance of the seen languages is rarely affected (row vs. row 5). This suggests that MLA framework can build multimodal multilingual models that are suitable for supporting increasing numbers of languages.
Table 8: Language extention experiments on XTD dataset.
Seen languages
Unseen languages
Row
Method
en
de
fr
sh
it
Tu
pl
tr
ko
UC# w/o unseen lunguage training
71.8
67.5
68.4
61.9
51.5
2
UC w/ unseen language training
63.6
57.8
57.6
57.6
48.4
56.4
56.2
51.3
56.4
51.6
51.3
UC w/ all language training
65.2
59.3
59.7
60.1
50.5
57.7
56.5
50.9
55.3
53.2
50.2
MLLAcIIP unseen language training
75.9
72.6
73.7
MLAquap w/ unseen language training
76.0
72.6
72.9
73.8
67.2
64.7
62.8
58.1
63.0
56.5
57.3
Conclusion and Limitations
In this paper, we propose the Multi-Lingual Acquisition (MLA) framework that can empower multilingual capability on monolingual Vision-Language Pre-training models with low-cost and high- flexibility. MLA injects language acquirers and non-native embedding block into VLPs support new languages. Inspired by the language learning habits of humans, we propose a two-stage training strategy optimize the language acquirers and non-native embedding block. MLA applied on CLIP achieves state-of-thee afrtharrrt performances on multilingual image-text and video-text retrieval benchmarks with much less computing costs and training data. Extensive ablation studies demonstrate that MLA is a flexible, effective, and efficient method to empower multilingual capability on multimodal models.
Though MLA shows high data efficiency that achieves high performance with small amount of multilingual training data in our experiments, it has the following limitations, which also suggest potential research avenues: i) MLA relies bilingual corpora, which limits its application low-resource languages. Meanwhile, state-of-the-art models, such as UC² and MURAL, also rely on bilingual information. How to achieve comparable performance with monolingual corpora only is still an open problem. ii) MLA is currently limited to coarse-grained retrieval tasks. We will investigate the further application of MLA to Line-grain tasks like VQA in our future work. Furthermore, the majority of our training data is automatically constructed through machine translation, so the ethical prejudice from the machine translation service may potentially affect the behavior of multilingual models produced by MLA. One way to mitigate such concern is to use human annotated or reviewed data for training.
6 Acknowledgement
This work was partially supported by the National Key R&D Program China (No. 2020AAA0108600), the National Natural Science Foundation of China (No. 62072462), and the Large-Scale Pre-Training Program 468 of Beijing Academy of Artificial Intelligence (BAAI).
References
[1] Pranav Aggarwal, Ritiz Tambi, and Ajinkya Kale. Towards zero-shot cross-lingual image retrieval and tagging. CoRR, abs/2109.07622 2021.
[2] Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolingual representations. In 58th Annual Meeting of the Association for Computational Linguistics, pages 4623 4637.. Association for Computational Linguistics, 2020.
[3] Andrea Burns, Donghyun Kim, Derry Wijaya, Kate Saenko, and Bryan A Plummer. Learning scale multilingual representations for vision-language tasks. In European Conference on Computer Vision, pages 197-213. Springer, 2020.
[4] Fredrik Carlsson, Philipp Eisen, Faton Rekathati, and Magnus Sahlgren. Cross-lingual and multilingual CLIP. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6848-6854, Marseille, France, June 2022. European Language Resources Association.
10

---

[5] Dominic Castello. First language acquisition and classroom language learning: Similarities and differences. ELAL College of Arts & Law. pages 1-18, 2015.
[6] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325. 2015.
[7] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. European conference on computer vision. Springer, 2020.
[8] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLL (1), 2019.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[10] Desmond Elliott, Stella Frank, Khalil Sima' an, and Lucia Specia. Multi30k: Multilingual inglish-german image descriptions. In VL@ ACL, 2016.
[11] Hongliang Fei, Tan Yu, and Ping Li. Cross-lingual cross-moda pretraining for multimodal re- trieval. In 2021 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, pages 3644-3650, 2021.
[12] Spandana Gella, Rico Sennrich, Frank Keller, and Mirella Lapata. Image pivoting for learning multilingual multimodal representations. In EMNLP 2017: Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2017.
[13] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international con- ference artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition, 2016.
[15] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In 36th International Conference on Machine Learning, 2019.
[16] Yao Huang, Mandela Patrick. Junjie Hu, Graham Neubig, Florian Metze, and Alexander G Hauptmann. Multilingual multimodal pre-training for zero-shot cross-lingual transfer of vision- language models. In 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
[17] Yuqi Huo, Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, et al. Wenlan: Bridging vision and language by large-scale multi-modal pre-training, 2021.
[18] Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Jason Baldridge. MURAL: Multimodal, multitask representations across languages. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3449-3463, 2021.
[19] Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 4904 4916. PMLR, 2021.
[20] Andrej Karpathy and Li Fei-Fei. Deep visual-semante alignments for generating image descriptions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
11

---

[21] Donghyun Kim, Kuniaki Saito, Kate Saenko, Stan Sclaroff, and Bryan Plummer. Mule: Multimodal universal language embedding. In AAAI Conference on Artificial Intelligence. volume 34, pages 11254-11261, 2020.
[22] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and and lnn-lagg transformer without convolution or region supervision. In 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research. PMLR, 2021.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
[24] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1 1936--1144,, 2020.
[25] Xirong Li, Chaoxi Xu, Xiaoxu Wang, Weiyu Lan, Zhengxiong Jia, Gang Yang, and Jieping Xu. Coco-en for cross-lingual image tagging, captioning, and retrieval. IEEE Transactions on Multimedia, 2109):347-2360, 2019.
[26] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision. Springer, 2020.
[27] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval. Neurocomputing, 508:293-304, 2022.
[28] Minheng Ni, Haoyang Huang, Lin Su, Edward Cui, Taroon Bharti, Lijuan Wang. Dongdong Zhang, and Nan Duan. M3p: Learning universal representations via multitask multilingual multimodal pre-training. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3977-3997 2021.
[29] Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan- Martin Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. xGQA: Cross-lingual visual question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2497-2511, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[30] Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. Mad-x: An adapter-based framework for multi-task cross-lingual transfer. In 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 7654-7673, 2020.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Leanning transferable visual models from natural language supervision. In 38th International Conference on Machine Learning, volume 139, pages 8748-8763, 2021.
[32] Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In 2020 Conference on Empirical Methods in Natural Language 2020.
Processing (EMNLP), pages
[33] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, 2016.
[34] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2018
[35] Yuqing Song, Shizhe Chen, Qin Jin, Wei Luo, Jun Xie, and Fei Huang. Product-oriented machine translation with cross-modal cross-lingual pre-training. 29th ACM International Conference Multimedia, 2021.
12

---

[36] Ke Tran. From english to foreign languages: Transferring pre pre-trained language models. arXiv preprint arXiv:2002.07:306, 2020.
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages
2017.
[38] Jonatas Wehrmann, Douglas M Souza, Mauricio A Lopes, and Rodrigo Barros. Language- agnostic risual-semantic embeddings. In IEEE/CVF International Conference Computer Vision, pages 5804-5813, 2019.
[39] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[40] Yuya Yoshikawa, Yutaro Shigeto, and Akikazu Takeuchi. Stair captions: Constructing large-scale japanese image caption dataset. In 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2017.
[41] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[42] Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint sequence fusion model for video question answering and retrieval. In European Conference on Computer Vision (ECCV), 2018. 2018.
[43] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint 11.1v:2111.11432. 2021.
[44] Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, and Jingjing Liu. Uc2: Universal cross-lingual cross-modal iisoo-andd-language pre-training. IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 4155-4163 June 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec S
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec S
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Please check the supplementary material for the code and model.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec[4.2
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] S.24..2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See the footnote of page 6.
(c) Did you include any assets either in the supplemental material URL? [Yes] Please check the supplementary material for code and model.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "n8f6QVdzz3B", "reviewer": "Reviewer_uTRM", "summary": "### After author response\nThanks authors for running additional experiments, including implementing MURAL. I raise the score to 6.\nI think there's a merit in adapting monolingual models to multilingual ones, the approach in the paper is good, but it's overselling. I recommend the authors to reword a bit, saying what is reflected in the results. For instance, in the response the authors said MLA doesn't suffer from the curse of multilinguality based on a result of *only* 11 languages. I don't think you can extrapolate that to hundreds of languages. \n\n-----------------\n\n### Summary \nThe paper proposes a way to adapt English Vision-Language Pre-training (VLP) to other languages via two stages: Native Language Transfer stage and the Language Exposure stage.  In Native Language Transfer state, parallel data is utilized to adapt language encoder to the new languages using language acquisition encoder, a combination of embeddings for new languages and adapter.. In the Language Exposure stage, only Language Acquisition encoder is trained to match with images via contrastive loss. In the text-image retrieval evaluation, MLA shows strong performances compared to other multilingual VLP. The main contribution of the paper is to propose a lightweight approach turning monolingual VLP into multilingual one.\n", "questions": "### Questions:\n- For a low resource scenario (Table 6), since MURAL supports 110 languages, what is the performance of MURAL?\n- What is the training size of translation data CC6L and CC69L?\n- If you train your dual stream model from scratch for 5 languages in Multilingual image-text retrieval experiment, how long does it take? and what would be the final performance?\n\n### Suggestions:\n- I think the analogy to how human learning and human learning habit just makes the paper more difficult to read. From technical perspective, you want to retain as many as pre-trained parameters as possible, so you adjust the parameters of new languages. It makes sense. Maybe just say that.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- A lightweight approach for making monolingual VLP multilingual.\n- Evaluation on zero-shot, finetuning English and fine-tuning all shows substantial gains over previous work on multilingual VLP (e.g., MURAL, MKD, …)\n- On multilingual video-text retrieval, MLACLIP outperformed XLM-R MMP significantly.\t\n\n\n### Weaknesses\n- Adapting monolingual language pretrained models into multilingual ones has been studied extensively in the literature. For instance, an efficient method was proposed in [[1](https://arxiv.org/abs/2002.07306)] which uses only 1 gpu. I feel that the novelty aspect is a bit short in adapting monolingual VLP to multilingual one. Nevertheless, this is not a major weakness of the paper.\n- My main concern is the comparison with previous work. MLA adapts CLIP-ViT-B-32 and CLIP-ViT-B-16 to 4 languages (de, fr, cs, ja) in multilingual image-text retrieval. While MLA is compared with MURAL, it is worth pointing out that MURAL supports 109 languages. MURAL uses EOBT that consists of translations of 109 language pairs and Alt-Text (image-text pairs) covers 110 languages. Thus comparing MLA to MURAL is somewhat comparing an apple to orange. The curse of mulitlinguality tells us that adding more languages into the mix would degenerate the performance of high resource languages, a well known problem in multilingual machine translation. In my opinion, a fairer baseline would be the MURAL system but it is trained on 5 languages similar to MLA. In line 273, when the authors compare MLA to MURAL, note that MURAL has more parameters but they are mostly the embeddings of the rest 104 languages, which will never be updated during fine-tuning. So having more parameters doesn’t mean that it is easier for fine-tuning.\n- On computational cost, Table 2 shows that MLA need 1 V100 GPU and 0.5 day to train. This is rather optimistic since it uses pre-trained CLIP-ViT-B-32. I think the cost of training the CLIP model should be added into the comparison. \n- While conceptually MLA can support more languages, as I mentioned earlier, adding more languages could lead to performance drop on high resource languages. So the aspect of language extensibility remains to be questioned. Additionally, MLA requires translation of image captions to train the language acquisition encoder. This is problematic because it depends on machine translation systems, which often support a certain number of languages. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Jaosr1Th2v", "reviewer": "Reviewer_4qbz", "summary": "___________AFTER THE RESPONSE______________\n\nI would like to sincerely thank the authors for their additional experimental efforts, aiming to provide coherent responses to all my questions and suggestions. Including such experiments into the paper (and adding relevant discussions) would make the paper much stronger. If the focus of the work is on retrieval tasks, I strongly suggest the authors to reflect that fact in the title and also in the introduction, and add discussions focused on why they don't extent their framework to other tasks. The title and the introduction of the paper should be changed to reflect this and avoid 'overselling'. The paper should also make its connections to previous work on adapters more apparent, and clearly extract its core novelty in comparison to other related work in this field (e.g., such as the adapter-based approaches from the xGQA paper). I raise my score from 6 to 7 to acknowledge the wealth of extra experiments conducted for the response, and a very detailed response. However, this increase (as a note to the area chair as well) is conditioned on really incorporating all the promised changes and edits to the final camera-ready paper, if the paper gets accepted.\n++++++++++++++++++++++++++++++++++++++++++++\n\n\nThis paper proposes a novel framework for multilingual multimodal learning that: 1) bypasses the need for joint and very expensive multilingual multimodal pretraining (e.g., M3P); and 2) adapts to multilingual setups to parameter-efficient modules (called language acquirers in this work, where these acquirers are basically standard bottleneck adapters). Besides more cost-friendly training of the model, the new framework, termed MultiLingual Acquisition (MLA)*, yields strong results in several cross-modal video-text and image-text retrieval tasks; the underlying model is a standard dual-strema model: CLIP-ViT-B, which is already through pretraining design bound to excel in (monolingual) retrieval tasks. The paper also presents a series of ablation studies and side experiments, covering important aspects such as training strategy, embedding initialisation, simulating low-resource languages and scenarios, zero-shot evaluations (on unseen languages) and amount of training data. There are additional qualitative and quantitative analyses available in the appendix.\n\n*Minor: I would suggest a more precise name for the framework (that also includes multi-modality somewhere)", "questions": "- This is not a criticism but more a suggestion: conducting some experiments with other underlying models beyond CLIP (and/or CLIP-style models of different sizes) would offer other useful comparisons and points of reference for future work.\n\n- It would also be interested to try out language acquirers with pretrained M-VLP models like M3P. Can MLA help M3P to adapt/specialise for a particular source/target language? This would be a very useful addition to the paper.\n\n- Minor: I suggest to cite papers based on their publications in conference proceedings rather than arxiv versions.", "limitations": "The statement in limitations of learning 'coarse-grained' versus 'fine-grained' representations and limitations seems quite hand-wavy to me - what is meant by this exactly? I would also suggest to be more open with other limitations of the work (e.g., some of them are mentioned in this review - (i) there weren't any real low-resource languages in evaluation; those scenarios were mostly simulated; (ii) application and evaluation is currently tied only to retrieval tasks; (iii) there's no discussion of potential avenues for future work, etc.)", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n=========\n- The motivation for the work is very clean and clearly stated. The core methodology is soundly described, and the paper is very easy to follow in general.\n- The results in the standard cross-modal retrieval tasks are consistently positive. \n- The ablation study and additional experiments are also very well motivated and very comprehensive, and offer the readers many interesting analyses and points of discussion.\n- The methodology, along with the core hypotheses, is very sound, and the experimental setup aligns well with the methodology.\n\nWeaknesses:\n==========\n- The current MLA framework is tested and seems to work only on retrieval tasks. A more comprehensive analysis on other, structurall different multilingual multi-modal tasks would make the main findings (and consequently, the impact of the work) much stronger. I strongly suggest to extend the paper to more underlying models (also single-stream models), and evaluate on a carefully designed recent IGLUE benchmark https://arxiv.org/pdf/2201.11732.pdf). The paper should convince the reader that MLA is not intended only for retrieval tasks (or the other way around - if it doesn't work as well for other tasks, to narrow the scope of the work to retrieval only in the title and in the introduction).\n- One of the core components of the work are 'language acquirers': they are basically standard language adapters, examined to detail in the body of work on cross-lingual text-only transfer (e.g., see again the cited MAD-X work from Pfeiffer et al.). Having said that, the idea of 'language acquirers' is basically also very related to another work of Pfeiffer et al. (cited in the paper as \"[28] Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin O Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. xgqa: Cross-lingual visual question answering. arXiv e-prints, 2021.\"). More discussion is needed here so that the reader really understands the connections to language adapters and connections to the adaptation work done on multilingual multi-modal work in previous work. In my understanding, the idea is basically very similar, with two core differences: a) MLA is applied on a dual-stream model which is suitable for retrieval tasks but might not be suitable for e.g. VQA tasks; b) Pfeiffer et al., relied on MLM-ing for adaptation, while MLA relies on specific bilingual information to learn transformations from the target language to the language of the monolingual VLP model (i.e., English).\n- There is no comparison with standard translation-based baselines. A very basic question: why would one want to go through the MLA procedure if translation-based approaches can work better? Some comparison numbers must be provided.\n- The chosen retrieval datasets might paint a wrong picture about the difficulty of the task: I would suggest the authors again to check the IGLUE benchmark and also to evaluate on the recently proposed WIT dataset. IGLUE also covers some lower-resource languages and its tasks are generally more difficult than the standard overly optimistic evaluations on Multi30K, and MSCOCO-1k and 5k.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "J_M2sca7b2b", "reviewer": "Reviewer_spgr", "summary": "The paper try to solve the problem that existing vision-language pre-trained models require huge computing resources and cannot be flexibly extended to new languages. They mainly propose a two-stage training strategy to optimize the language acquisition encoder, namely the Native Language Transfer stage and the Language Exposure stage. The authors compare their model to different baselines on multilingual image-text and video-text retrieval benchmarks, and find that their model compares well to the current state-of-the-art.\n", "questions": "1.\tWhy use the average score over Recall@1, Recall@5, and Recall@10 as your evaluations? What's the idea behind it, and not Recall@1, Recall@5, and Recall@10, or something else?\n2.\tI want to know whether the Language Acquier is shared in different languages? This is very important to evaluate the size of parameters of model.\n3.\tI want to know the training strategy. Whether different languages are trained in parallel or serial ways? For a language, whether the model only is trained on this language? And, the model have not the multilingual ability?\n4.\tThe comparison of Table 2 is very unfair. The baselines are on trained on many different languages at the same time. However, the model of this paper is only trained on a language for some lingual applications and the baselines are suitable for different languages. Even though the model are trained on some languages as the papers, the trained number of languages is less than M3P.\n5.\tThe description of the model in Section 3 needs a lot of improvement. It is confusing and many times convoluted. There are too many variables being introduced.\n6.\tIt is not clear whether you do not introduce the equations for the enumerated items, or whether you do not use the enumerated items in your model architecture. Please clarify.\n7.\tThe paper writing is poor. I found a lot of grammar mistakes in the paper which need to be corrected.\n", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper main issue is the lack of clarity at certain crucial points, especially in Section 3 (the model description). The maths is at times too convoluted and should be simplified and improved. Moreover, the major concern I have about this paper is the experimental results. The work is relatively novel, although a deeper discussion of why the specific architecture used was not done and would have been important to\nunderstand the contributions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ITmH6TCZL6c", "reviewer": "Reviewer_9Y6f", "summary": "The paper proposes a Multilingual Acquisition (MLA) framework that leverages vision and language modality at a low cost. MLA empowers monolingual VLPs with multilingual capability using much less training data through a language acquisition encoder. The authors apply the proposed method to multilingual image-text and video-text retrieval benchmarks with much less training data and parameters. In the training process, they present a two-stage training strategy to train the language acquisition encoder: 1) Native Language Transfer (NLT) stage, the model is trained to learn the connection between the native and new language, then followed by 2) Language Exposure (LE) stage, the model to learn the alignment between new languages and images.", "questions": "- I was wondering why did you only choose English as the source language for the zero-shot setting other than English is the high-resource language. Have you investigated the zero-shot setting where English is not the source/pivot language? It would be interesting to find out whether other languages? Many of these evaluated languages are not closely related to English (e.g., Japanese, Chinese).", "limitations": "Yes, the authors address the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- A simple two-step approach to acquire a new language by leveraging monolingual VLP by aligning the languages and modalities. The approach is straightforward and effective.\n- This is a solid paper; it has comprehensive results and analysis\n- The proposed approach has strong zero-shot and few-shot capability,\n\nWeaknesses:\n- Didn't find any substantial weakness to report.\n\nTypo:\n- MultiLingual => Multilingual. It is not common to write multilingual with a capital L. Another way to write it is Multi-lingual.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept