mCLIP: Multilingual CLIP via Cross-lingual Transfer
Year: 2023

Abstract:
Large-scale vision-language pretrained (VLP) models like CLIP have shown remarkable performance on various downstream cross-modal tasks. However, they are usually biased towards English due to the lack of sufficient non-English image-text pairs. Existing multilingual VLP methods often learn retrievalinefficient single-stream models by translationaugmented non-English image-text pairs. In this paper, we introduce mCLIP, a retrievalefficient dual-stream multilingual VLP model, trained by aligning the CLIP model and a Multilingual Text Encoder (MTE) through a novel Triangle Cross-modal Knowledge Distillation (TriKD) method. It is parameter-efficient as only two light projectors on the top of them are updated during distillation. Furthermore, to enhance the tokenand sentence-level multilingual representation of the MTE, we propose to train it with machine translation and contrastive learning jointly before the TriKD to provide a better initialization. Empirical results show that mCLIP achieves new state-of-the-art performance for both zero-shot and finetuned multilingual image-text retrieval task.

heading: 1 Introduction
text: Recently, large-scale dual-stream vision-language pretrained (VLP) models, such as CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021) and their variants (Yao et al., 2021; Mu et al., 2021; Zhai et al., 2021), have shown remarkable performance on various downstream multimodal tasks. These models use separate encoders for the images and texts, and allow efficient inference in the image-text retrieval task because the image or text features can be computed offline. However, most current VLP models are biased toward English, due to the lack of sufficient high-quality multilingual multimodal datasets for direct large-scale pretraining.
Despite the lack of sufficient non-English imagetext pairs, previous methods attempt to create wordlevel code-switched image-text pairs by looking up
bilingual dictionaries (Ni et al., 2021), or sentencelevel augmented multilingual image-text pairs by translating the English text to other languages (i.e, translate-train pipeline) (Zhou et al., 2021). Then each image and its paired text are concatenated as a single sequence to train a single-stream Transformer-based model. Despite the good performance of these models (Ni et al., 2021; Zhou et al., 2021), they are less efficient than dual-stream models on large-scale image-text retrieval tasks, as the data from both modalities are intertwined to compute the self-attention and the unimodal features can not be pre-computed. Instead of creating word-level or sentence-level multilingual imagetext pairs, MURAL (Jain et al., 2021) extends the ALIGN (Jia et al., 2021) model with multilinguality by an additional text-text contrastive loss among hundreds of languages. However, MURAL is trained from scratch and requires large-scale training data with high computation cost to obtain strong performance on multilingual cross-modal retrieval tasks.
To tackle the aforementioned problems, we propose the triangle cross-modal knowledge distillation (TriKD) to learn a dual-stream multilingual VLP model mCLIP, which learns triangle alignment among the pretrained CLIP’s image encoder, CLIP’s text encoder and a pretrained Multilingual Text Encoder (MTE) through knowledge distillation. Specifically, to avoid catastrophic forgetting of the knowledge already learned in the pretrained CLIP and MTE, they are kept frozen. The triangle alignment is achieved by adjusting a linear projector on top of CLIP and a shallow Transformerbased X-projector on top of the MTE. Since the XLM-R (Conneau et al., 2020) used for initializing the MTE has unsatisfactory performance when directly used for retrieval tasks (Hu et al., 2020), before performing the TriKD, we propose to enhance the MTE via both the machine translation task and a contrastive loss to improve the token-
13028
and sentence-level multilingual representation. The proposed mCLIP is both parameter- and computation-efficient as only the projectors are trained, which accounts for only 3% of the total parameters of mCLIP. Empirical results of zeroshot and finetuned multilingual image-text retrieval on MSCOCO (Lin et al., 2014) and Multi30K (Elliott et al., 2016) show that the proposed mCLIP achieves better performance while being much more efficient in inference than single-stream baselines or using less training data than MURAL. The proposed method can also be extended to train a multilingual VLP based on a unimodal image encoder and the MTE, 89.4% performance retained.1

heading: 2 Related Work
text: Multilingual VLP Models. Monolingual visionlanguage pretrained (VLP) models (Radford et al., 2021; Yao et al., 2021; Jia et al., 2021; Li et al., 2022) trained with large-scale image-text pairs have shown remarkable performance on various downstream tasks like image-text retrieval. Recently, some attempts have extended VLP models to the multilingual scenario. The first line of work applies the translation method to create multilingual image-text pairs and then concatenates the multilingual text with its paired image as a single sequential input to a single-stream Transformerbased encoder. For instance, M3P (Ni et al., 2021) constructs a multilingual code-switched text by randomly replacing English words with translations of other languages, and UC2 (Zhou et al., 2021) directly translates a whole sentence into other languages. However, these single-stream models are inefficient for the image-text retrieval task as unimodal features cannot be pre-computed beforehand. MURAL (Jain et al., 2021) directly trains from scratch with both augmented multilingual imagetext pairs and parallel text corpus, which is expensive in both data and computation. Besides retrieval tasks, recent PaLI (Chen et al., 2022b) and ERNIEUniX2 (Shan et al., 2022) use the encoder-decoder architectures for multilingual multimodal generation tasks. In this paper, we introduce a data- and parameter-efficient knowledge distillation method to train a dual-stream multilingual VLP model by aligning a frozen English VLP and a frozen MTE.
Knowledge Distillation. Knowledge distillation (Hinton et al., 2015) is firstly proposed for model
1Our code is publicly available at https://github.com/ huawei-noah/noah-research/NLP/mclip.
compression. The knowledge in the output logits of a large teacher model can be transferred to a smaller student model without significant performance degradation. Besides the logits, the hidden states and attention outputs can also be used for knowledge distillation (Jiao et al., 2020; Hou et al., 2020). Recently, Tian et al. (2020b) propose to distill knowledge with contrastive learning, which maximizes the mutual information between the teacher and student models. For multimodal models, Wang et al. (2021) propose to train a dualstream VLP model with the knowledge distilled from a single-stream model for faster inference. Furthermore, VLKD (Dai et al., 2022) augments a dual-stream VLP model with a pretrained language model via vision-language knowledge distillation, enabling the multimodal generation ability without hurting the original NLP ability. However, to the best of our knowledge, knowledge distillation has not been studied for training multilingual VLP models, for which efficiency is an important factor due to the data scarcity issue. In this paper, we introduce a novel triangle cross-modal knowledge distillation method to efficiently align a multilingual text encoder to the multimodal space of a pretrained dual-stream VLP model.

heading: 3 Method
text: In this section, we first introduce the architecture of mCLIP in Section 3.1. It extends the monolingual VLP model CLIP to a multilingual one by aligning CLIP and a multilingual text encoder (MTE) to a shared space, through a novel triangle cross-modal knowledge distillation (TriKD) using English image-text pairs (Section 3.2). The performance of mCLIP on non-English image-text retrieval is highly dependent on the quality of the multilingual representation of the MTE. Thus in Section 3.3, we propose to first improve the token- and sentence-level cross-lingual representation of the MTE with the neural machine translation (NMT) task and contrastive learning (CTL).

heading: 3.1 Model Structure
text: The architecture of mCLIP is shown in Figure 1a. Like CLIP, mCLIP is a dual-stream model with separate image and text encoders. The vision encoder of mCLIP is the original CLIP ViT image encoder, while the text encoder is a multilingual one initialized from XLM-R (Conneau et al., 2020) with enhanced representations.
Updated version
CLIP. The image encoder of the pretrained CLIP (Radford et al., 2021) is aligned with the English text encoder, by contrastive learning over 400M English image-text pairs. The Vision Transformer (ViT) is used as a kind of CLIP image encoder, which takes image patches as input and generates the final feature through a Transformer-based model. An additional [cls] token is added before the image patches, and its output at the last Transformer layer represents the image’s global feature. The CLIP text encoder has a similar structure to the GPT (Radford et al., 2019) model. The final output of the [eos] token represents the global feature of an English sentence. Note that CLIP’s text encoder is only used during the training of mCLIP, but not inference.
Multilingual Text Encoder. Instead of using the original CLIP’s English text encoder, we use the multilingual encoder XLM-R (Conneau et al., 2020) with enhanced token- and sentence-level cross-lingual representations (Section 3.3.)
Our ultimate goal is to learn the triangle alignment among the CLIP’s image encoder, CLIP’s English text encoder and the multilingual text encoder (MTE) in a shared multilingual multimodal representation space. In Section 3.2, we propose triangle cross-modal knowledge distillation (TriKD) to achieve this goal while maintaining the already learned alignment between the image and English text of CLIP, as well as the multilinguality of the learned MTE. Specifically, as is shown in Figure 1a, to avoid destroying the pretrained alignment between CLIP’s image and text encoders, we freeze the parameters of both CLIP’s image and
text encoders, and use a shared linear projection (i.e., the CLIP-projector) on the top of them. On the other hand, to keep the learned multilinguality of XLM-R, we also freeze its parameters and align it to CLIP’s multimodal space by optimizing the learnable X-projector, which consists of two randomly initialized XLM-R Transformer layers (Huang et al., 2021). The input to the X-projector is the outputs of all positions from the MTE. The [eos] output representation after the X-projector is used as the global representation of the text.

heading: 3.2 Triangle Cross-Modal Knowledge Distillation
text: Contrastive learning is proved effective in both unimodal (Tian et al., 2020a; Gao et al., 2021) and cross-modal (Radford et al., 2021) representation learning. Here, we also consider using contrastive losses to learn the triangle alignment among CLIP’s image encoder, CLIP’s English text encoder, and the multilingual text encoder (MTE). Since the image and text encoders of CLIP are already aligned, the TriKD contains only (i) an image-text contrastive (ITC) loss to align the MTE and CLIP image encoder; and (ii) a text-text contrastive (TTC) loss to align the MTE and CLIP’s English text encoder (Figure 1).
In contrastive learning, the model parameters are optimized by letting the features of paired samples close and apart otherwise. Specifically, consider a training batch of N samples, where xi,yi are a pair of features from two views of the ith sample, e.g., the image and text features of an image-text pair; or the text features of the same text from two different
text encoders. We use in-batch negatives, i.e. for xi, yi is its positive, and all the other yj’s (where j ̸= i) are its negatives. Denote x = {xi}Ni=1, y = {yi}Ni=1, and the temperature parameter is τ . the contrastive loss can be written as
ℓ(x,y) = − 1 N
N∑
i=1 log exp(x⊤i yi/τ)∑N j=1 exp(x ⊤ i yj/τ) . (1)
Image-Text Contrastive Loss. For the ith imagetext pair in a training batch, denote the ℓ2normalized output of the [cls] token after the CLIP image encoder and CLIP-projector as hIi and the ℓ2-normalized output of [eos] after the MTE and X-projector as hXi . Denote h
I = {hIi }Ni=1 and hX = {hXi }Ni=1, the ITC loss LITC is formulated as the average of image-to-text (Li2x) loss and textto-image (Lx2i) loss:
LITC = 1/2(Li2x + Lx2i) = 1/2[ℓ(hI ,hX) + ℓ(hX ,hI)]. (2)
Text-Text Contrastive Loss. For the ith imagetext pair, suppose the ℓ2-normalized output of the [eos] token after the CLIP text encoder and CLIPprojector is hTi . Denote h
T = {hTi }Ni=1, the TTC loss is calculated as the average of contrastive losses in both directions:
LTTC = 1/2(Lt2x + Lx2t) = 1/2[ℓ(hT ,hX) + ℓ(hX ,hT )], (3)
where Lt2x,Lx2t are the contrastive losses of CLIP text features to XLM-R features and vice versa.
The training loss of the TriKD is the weighted sum of ITC and TTC losses:
LTriKD = LITC + λLTTC.
We use λ = 0.1 following Jain et al. (2021). For training with non-English image-text pairs, only ITC loss is applied as the CLIP text encoder does not support non-English languages.
Since the backbones of image and text encoders are frozen and only the additional projectors (3% of total parameters) are learnable, the training is efficient and allows a large batch size, which is shown to be crucial to the success of contrastive learning (Chen et al., 2020; Radford et al., 2021).
Through TriKD, though mCLIP learns only on English image-text pairs, it already implicitly has the ability to transfer to other languages through the multilinguality embedded in the frozen MTE. The
retrieval performance on non-English languages relies on both the English text-image retrieval performance and the cross-lingual transferability of the MTE. However, the original XLM-R is not directly optimized for retrieval and its cross-lingual ability for retrieval is not satisfactory (Hu et al., 2020), so in Section 3.3, we propose a two-stage training method to enhance the MTE before TriKD.

heading: 3.3 Multilingual Text Encoder
text: In this section, we propose to enhance the tokenand sentence-level alignment among different languages of XLM-R for retrieval tasks, with the neural machine translation (NMT) task and contrastive learning on the textual-only multilingual parallel corpus. Intuitively, an NMT decoder generates semantic-equivalent translation with token-level interactions with the encoder output, encouraging the encoder output to maintain fine-grained token-level information, which is required as the X-projector is trained over token-level inputs during TriKD. On the other hand, the contrastive loss benefits crosslingual transfer by explicitly aligning the sentencelevel representations of parallel sentences.
Note that XLM-R is only an encoder, to train with the NMT loss, we add a decoder with randomly initialized weights (Figure 2a). Inspired by Chen et al. (2021), we adopt a two-stage training schedule to avoid catastrophic forgetting of the strong multilinguality of the pretrained XLM-R encoder. Before joint training with the NMT and contrastive loss, we freeze the encoder and train this decoder with the NMT task on parallel text corpus at the first stage. Note that all embeddings are initialized with XLM-R and fixed all the time. With a slight abuse of notation, here we denote xi and yi as the ith source and target sentence in a batch of N paired sentences, and |yi| is the length for sentence yi, the NMT loss can be formulated as:
Lstage_1= LNMT
=− 1 N
N∑
i=1
|yi|+1∑
t=1
log p([yi]t | [yi]0:t−1,xi),
At the second stage, we tune both the XLMR encoder and the decoder with both NMT and contrastive loss (Figure 2b). Note that we do not tune the embeddings as no further improvements are observed empirically. Specifically, for the ith sentence pair, denote hSi ,h O i as the averaged representation of all the tokens of the source and target
[bos] A cat sat on the mat. [eos]
Multilingual Text Encoder
Input Embedding
Output Embedding
Multilingual Text Decoder
Eine Katze saß auf der Matte. [eos]
[bos] Eine Katze saß auf der Matte.
Output Embedding
Multilingual Text Encoder
Input Embedding
[bos] Eine Katze saß auf der Matte. [eos]
shared
shared
shared
CTL Loss MT Loss 𝒙
𝑚𝑒𝑎𝑛𝑚𝑒𝑎𝑛
𝒚
SG SG SG
SG
[bos] A cat sat on the mat. [eos]
Multilingual Text Encoder SG Input EmbeddingSG
Output EmbeddingSG Multilingual Text Decoder Eine Katze saß auf der Matte. [eos]
[bos] Eine Katze saß auf der Matte.
Output Embedding SG
MT Loss
sentences from the last encoder layer, respectively. Denote hS = {hSi }Ni=1, hO = {hOi }Ni=1, the conrastive loss and training l ss are:
LCTL = 1/2[ℓ(hS ,hO) + ℓ(hO,hS)], Lstage_2 = LNMT + αLCTL,
where ℓ(·, ·) is the contrastive loss defined in Equation 1, and α is the weight to balance the two loss terms, which is set as α = 2.0 in our experiments.
Note that when computing the contrastive loss during the second stage, we select the average representation over all tokens as the sentence-level text feature instead of the [eos] feature, as the former empirically performs better in the cross-modal retrieval task. We speculate this is because the Xprojector uses token-level outputs from the MTE instead of the [eos] representation for learning alignment between images and texts. After the two-stage training, the MTE is used to initialize mCLIP using the TriKD method in Section 3.2.

heading: 4 Experiments

heading: 4.1 Setup
text: Models and Pretraining Datasets. We train two models (i.e., mCLIP and mCLIP+) based on the officially released CLIP ViT-B/32 and XLM-R (Conneau et al., 2020) base models. For the vanilla mCLIP, the enhanced MTE in Section 3.3 is trained with the parallel text corpus MT6, which contains 120M parallel sentences between English and six languages and covers 12 language directions (Chen et al., 2022a). Then we perform the TriKD in Section 3.2 with the cross-modal dataset CC3M (Sharma et al., 2018). For mCLIP+, its MTE is trained with OPUS-100 (Zhang et al., 2020) dataset in addition to MT6, covering a total of 175M parallel sentences among 100 languages. The TriKD of mCLIP+ is performed with TrTrain(CC12M),
which is obtained by applying the translate-train method and translating the English captions of CC12M (Changpinyo et al., 2021) into Czech, German, Japanese and French with an in-house translator. Note that the TTC loss is removed during TriKD for non-English image-text pairs. More details about MT6 and OPUS-100 are in Appendix A.2. We use the XTD10 (Aggarwal and Kale, 2020) Spanish image-text pairs as the validation set to select the checkpoints, as we care more about the multilingual cross-modal performance.
Downstream Tasks and Evaluation Metrics. We test the efficacy of the proposed mCLIP on both multilingual image-to-text and text-to-image retrieval tasks, on the test sets of Multi30K (Elliott et al., 2016) and MSCOCO (Lin et al., 2014). We use the same data splits as Young et al. (2014) and Karpathy and Fei-Fei (2015). More details are in Appendix A.1. For both retrieval tasks, we compute the recall of top-K candidates (recall@K) with K=1, 5, and 10. The mean recall averaged over all these 6 scores is used as the evaluation metric. Following Ni et al. (2021), we evaluate the model’s zero-shot and finetuned performance. Under the zero-shot setting, the pretrained mCLIP is directly tested on multilingual retrieval tasks. We use three finetuned settings: (i) English-only Finetune: finetune the pretrained mCLIP with only English Multi30K or MSCOCO and test on each target language; (ii) Single-language Finetune: finetune with training data of target language and test; and (iii) All-language Finetune: finetune on training data of all languages and test on each language.
Compared Methods. We compare our proposed method against the recent multilingual multimodal models M3P (Ni et al., 2021), UC2 (Zhou et al., 2021) and MURAL (Jain et al., 2021). The results of these models are taken from their original pa-
pers. MURAL-base with the similar model size is compared. Note that UC2 does not report its zeroshot results. Its results on English and Japanese MSCOCO test sets are not directly comparable with the other methods, because they simplified the task by splitting the 5k images and 25k captions into five smaller test sets to calculate the scores. The training details and hyperparameters can be found in Appendix A.4.

heading: 4.2 Main Results
text: The zero-shot and finetuned cross-modal retrieval results on Multi30K and MSCOCO are shown in Table 1. As can be seen, finetuning and using more pretraining data improves the performance of our model. In particular, All-language Finetune has the highest mean recall score for both mCLIP and mCLIP+. We speculate this is because image-text pairs with diverse languages allow the projectors to learn the multilingual multimodal alignment explicitly, instead of relying on the implicit multilinguality embedded in the MTE.
Comparison with Baselines. Compared with M3P, our proposed mCLIP achieves 16.7 and 12.8 more mean recall scores in zero-shot and English-
only finetuned settings, respectively, despite that M3P uses more fine-grained code-switched imagetext pairs and more languages. Moreover, M3P is a single-stream model and can be less efficient for retrieval tasks. Compared with UC2 pretrained on 5x larger translation-augmented TrTrain(CC3M), mCLIP trained with only English CC3M achieves 2.1 higher mean recall scores on English-only Finetune. Again, UC2 is a single-stream model like M3P and also suffers from inefficient inference. Compared with MURAL, the mean recall of mCLIP+ is 1.8 (resp. 2.3) points higher under the zero-shot (resp. English-only Finetune) setting with about 1/3 parallel texts. This may be because the MTE of mCLIP+ learns strong multilinguality in Section 3.3 and the X-projector only needs to focus on the multimodal alignment rather than the multilingual alignment. In contrast, MURAL has to learn to align the multilingual texts from scratch with parallel texts. Besides achieving better performance with less training data, mCLIP is also parameter-efficient, i.e., the learnable projectors only account for 3% of the total parameters during the triangle distillation.

heading: 4.3 Ablation Study
text: In this section, we conduct ablation studies using mCLIP pretrained on CC3M and report results on zero-shot multilingual image-text retrieval tasks.
Components of Training Objectives. Table 2 shows the effect of different training objectives in training the enhanced multilingual text encoder (MTE) and during the triangle cross-modal knowledge distillation (TriKD). mCLIP−LNMT represents finetuning the XLM-R with only contrastive loss on parallel texts, while mCLIP−LCTL is to finetune XLM-R with the two-stage training scheme only on the NMT task. mCLIP−Lθ represents the mCLIP trained with original XLM-R from Conneau et al. (2020). As can be seen, in the TriKD, both image-text and text-text contrastive loss contribute positively to the performance and the image-text contrastive loss LITC is more crucial to the retrieval performance. In the learning of enhanced MTE, both the contrastive and NMT losses improve the performance in non-English languages. However, NMT loss improves English image-text retrieval while contrastive loss degrades it. This may be because the NMT loss allows the MTE to learn more fine-grained token-level textual representations, and facilitate the learning of the Xprojector which relies on these token-level inputs.
Design Choices of Locked Parameters. We compare different design choices of locked parameters of mCLIP in Table 3. As can be seen, the performances of all three languages drop when either CLIP or XLM-R is finetuned. Finetuning CLIP degrades the performance because the image encoder gradually forgets its learned knowledge from large-scale pretraining on 400M image-text pairs
and tends to overfit to the small CC3M dataset used for triangle distillation. When the XLM-R is finetuned, the ability of multilingual transfer degrades and the text encoder biases toward English. When both CLIP and XLM-R are locked, the knowledge embedded in these two models is maintained, contributing to the success of the cross-lingual crossmodal transfer. Yet another advantage of locking both backbones is the improved training efficiency, which allows the much larger batch size, as 97% parameters are frozen during training.

heading: 4.4 Discussion
text: Results on More Languages. Table 4 compares our model with baselines on more diverse languages of the cross-modal retrieval task in the IGLUE (Bugliarello et al., 2022) benchmark2. Following Bugliarello et al. (2022), we report the mean recall@1 score under the zero-shot setting. The results of M3P and UC2 are taken from Bugliarello et al. (2022). We do not compare with MURAL as it is not open-sourced and its original paper does not report results on IGLUE. As can be seen, mCLIP+ has the best performance among all languages, achieving 17.2 and 17.8 higher averaged Recall@1 score than M3P and UC2.
Different Image Encoder Backbones. Besides using the CLIP-ViT as the image encoder of mCLIP, we also try to use the Swin Transformer (Swin-B3) (Liu et al., 2021), a novel unimodal model trained with only the image classification dataset. We use the same setup as Section 4.1 except that we remove the TTC loss. Empirical results on zero-shot retrieval on Multi30K and MSCOCO in Table 6 show that using Swin Transformer has 89.4% mean recall scores of that using CLIP-ViT. This indicates that our proposed method can also be extended to align a unimodal image encoder and a multilingual text encoder into a multilingual multimodal model.
2https://github.com/e-bug/iglue 3swin_base_patch4_window7_in22k of timm toolkit

heading: 5 Analysis of the Representations
text: The training objectives of contrastive learning encourage the positive samples to stay closer (i.e., alignment) while the negative samples to scatter on the hypersphere (i.e., uniformity) (Wang and Isola, 2020). Similarly, a desired multilingual and multimodal model should also learn good alignment between images and multilingual texts, as well as uniform representations within each modality.
We analyze the quality of the learned representations with the uniformity and alignment scores introduced in Wang and Isola (2020). The alignment score ℓalign = E(||hIi − hXi ||2) measures the distance between the ℓ2-normalized features of the image-text pairs (i.e, hIi ,h X i for the i-th image-text pair), while the uniformity score measures how uniformly the representations are distributed: ℓuniform = −E(exp(−2||h∗i − h∗j ||2)), where h∗i ,h ∗ j with ∗ ∈ {I,X} are ℓ2-normalized features of different samples from the same modality. Smaller alignment and uniformity scores indicate higher alignment and uniformity, and thus better learned representations. We use mCLIP trained on English CC3M and analyze the learned representations of XTD10 (Aggarwal and Kale, 2020) test set with the two metrics. The uniformity score
is calculated for each of the three modalities: images, English text, and non-English text. We report results for non-English languages averaged over It, Es, Ru, Pl, Ko, Zh, and Tr.
From Table 5, both CTL and NMT losses improve the alignment and uniformity scores for nonEnglish languages, as well as the uniformity of the images. However, for English, the NMT loss improves both scores while the CTL loss cannot improve or even degrade them. This is consistent with the finding in Table 2 where CTL loss leads to worse English retrieval performance. This again affirms that the NMT loss learns more fine-grained token-level representation which benefits the Xprojector for aligning the English image-text pairs, thus rendering better alignment and uniformity scores for English. The NMT loss also reduces the burden of mCLIP projectors to learn multilingual alignment, which contributes to better uniformity of image features. To summarize, mCLIP relies on NMT for English image-text retrieval and CTL for further improvement on non-English retrieval (mainly on the uniformity of the image).

heading: 6 Conclusion
text: In this paper, we introduce mCLIP, a novel multilingual vision-language pretrained model which aligns CLIP and an enhanced multilingual text encoder through triangle cross-modal knowledge distillation. This distillation method is both parameterefficient with only 3% of the total parameters of mCLIP trained, and data-efficient with only English image-text pairs required. The performance of mCLIP can be further improved with more parallel text corpus from more languages and mul-
tilingual image-text pairs from the translate-train pipeline. Empirical results show that the proposed mCLIP+ achieves state-of-the-art performance in multilingual image-text retrieval tasks.

heading: 7 Limitations
text: This work only explores the multilingual VLP model for the image-text retrieval task. We leave the exploration of other multilingual vision-andlanguage downstream tasks such as visual question answering as future work. At the same time, our proposed method relies on a well-pretrained vision Transformer and a multilingual text encoder. Its performance is heavily influenced by the performance of the visual and textual backbones. This hinders the mCLIP from further improvements with the given backbones.

heading: 8 Ethical Considerations
text: We present a data- and training-efficient approach to build a multilingual VLP model mCLIP, by aligning the pretrained monolingual VLP model CLIP and a multilingual text encoder XLM-R to the same multimodal multilingual space. Despite the strong multimodal and multilingual abilities inherited from both models, the proposed mCLIP also inherits the societal impacts including some negative ones of the original CLIP and XLM-R, e.g., societal biases (Radford et al., 2021) and misuse of language models (Tamkin et al., 2021). The implicit biases are expected to be removed by debiasing either the dataset or the model (Meade et al., 2022; Zhou et al., 2022). Besides, our proposed method makes it simpler to retrieve malicious or offensive content (Welbl et al., 2021) from image-text pairs of different languages. Future explorations are needed to mitigate the misuse of VLP models.

heading: Acknowledgements
text: This project was supported by National Natural Science Foundation of China (No. 62106138) and Shanghai Sailing Program (No. 21YF1412100). Jia Pan and Wenping Wang are partially supported by Centre for Transformative Garment Production. We thank the anonymous reviewers for their insightful feedbacks on this work.

heading: A More Experimental Setup
text: A.1 Cross-Modal Dataset for Image-text Retrieval
Multi30K. The Flickr30K dataset (Young et al., 2014) contains 31k images in total, each of which has five English captions. The Multi30K dataset (Elliott et al., 2016) extends the Flickr30K dataset to three other languages. Each image has five German captions, one Czech caption and one French caption. Following previous works (Ni et al., 2021; Jain et al., 2021), we use the same train/validation/test splits as Karpathy and Fei-Fei (2015) for each language.
MSCOCO. The MSCOCO dataset contains 123k images, each of which has five English captions. Yoshikawa et al. (2017) manually create the Japanese descriptions for MSCOCO images. Li et al. (2019) extend MSCOCO with Chinese captions for 20K images. Following previous works (Ni et al., 2021; Jain et al., 2021), we use the same dataset splits as Karpathy and Fei-Fei (2015) for English and Japanese, and the test set of each language has 5k images and 25k captions. For Chinese, we use the same dataset split as Li et al. (2019), whose test set has 1000 image-text pairs.
A.2 Machine Translation Dataset
Training the enhanced multilingual text encoder (MTE) in Section 3.3 requires parallel sentences. Thus we create a dataset called MT6, which contains 120 million parallel sentences between English and six languages: Czech, German, Japanese, Russian, Spanish, and Chinese. The MT6 dataset is from WMT translation task4, CzEng 1.65, JParaCrawl v1.06 and CCAligned corpus7. For Es-En and Ru-En, MT6 uses the first 20M sentence pairs of the CCAligned corpus. The validation sets are from the development and test sets of the WMT translation task. More details are shown in Table 7. To compare with MURAL, we combine MT6 dataset and OPUS-1008 (Zhang et al., 2020) to train the enhanced MTE. All texts are tokenized by the sentencepiece (Kudo, 2018) tokenizer as used in the original XLM-R model (Conneau et al.,
4https://www.statmt.org/wmt19/ translation-task.html
5https://ufal.mff.cuni.cz/czeng/czeng16 6http://www.kecl.ntt.co.jp/icl/lirg/
jparacrawl/ 7http://www.statmt.org/cc-aligned/ 8https://opus.nlpl.eu/opus-100.php
2020). The source sentence length is limited to 512, which is the maximum source sentence length supported by XLM-R.
A.3 Language ISO code
The languages used in this paper are shown in Table 8.
A.4 Training Details
We first train the enhanced multilingual text encoder from XLM-R following Section 3.3. Adam (Kingma and Ba, 2015) is used as the optimizer. Each batch has 32,768 tokens. At the first training stage, the learning rate is warmed up to 0.0005 within 4,000 steps, and then decays to 0. At the second stage, the learning rate decays from 0.0001 to 0 without warmup. The model is trained for one epoch at the first stage and 0.5 epoch at the second stage. The training data of different language pairs are sampled following that of XLM-R: qi = p β i / ∑ j p β j , where β = 0.2 and pj is the percentage of each language in the training dataset.
Then we perform the TriKD in Section 3.2. We use the LAMB optimizer (You et al., 2020). The learning rate is linearly warmed up to 0.01 within the first 500 steps and then decayed to 0. The batch size is 16,384. The temperature for the ITC loss is initialized as 0.07 and then learned by gradient descent, while the temperature of TTC loss is fixed as 0.07 (Jain et al., 2021). The models are pretrained for 15 epochs when the smaller dataset CC3M is used, while for 3 epochs when CC12M is used.
When finetuning mCLIP on the downstream image-text retrieval dataset, we use the contrastive loss to finetune the projectors while keeping other parameters frozen. When one training image has multiple captions, all its paired captions are treated as positives.
For all experiments in all pretraining and finetuning stages, we use the inverse square root learning rate scheduler and conduct experiments on 8 NVIDIA V100 GPUs. We use the same dropout method as XLM-R (Conneau et al., 2020). The dropout ratios are set as 0.3. The detailed hyperparameters of different stages are listed in Table 9.

heading: B More Experimental Results
text: B.1 Comparison with Translate-test Method
The translate-test (Conneau et al., 2020; Ni et al., 2021) is another possible method for the multilingual cross-modal retrieval task. It first translates non-English texts into English and then completes the cross-modal retrieval task with an English vision-language pretrained model like CLIP. In this part, we compare mCLIP+ and the translatetest baseline (CLIP+TrTest) on the non-English languages of MSCOCO and Multi30K test sets under the zero-shot setting. For CLIP+TrTest, the captions are translated with the open-sourced m2m1009, a recent strong NMT model that is trained
9https://github.com/facebookresearch/fairseq/ tree/main/examples/m2m_100
with 7.5 billion parallel sentences. The translations are generated with beam size 5 using the 1.2B model checkpoint.
We compare the latency on both the text-toimage and image-to-text retrieval tasks. For the text-to-image retrieval task, we precompute the image features and report the inference time for one text query, which contains (1) the time to extract the feature of one text query, (2) the time of similarity calculation and ranking, and (3) for CLIP+TrTest, the time to translate the non-English text query into English. Similarly, for the image-to-text retrieval task, we precompute the text features and report the inference time for one image query, which contains (1) the time to extract the feature of one image query, and (2) the time of similarity calculation and ranking. Note that for CLIP+TrTest, the translation cost is not included in this latency as the text features are precomputed. All latency values are averaged on all test sets over ten runs using one NVIDIA V100 32G GPU.
From Table 10, mCLIP+ achieves 7.9% better mean recall score than CLIP+TrTest, with fewer model parameters and lower latency. Directly designing a multilingual cross-modal retrieval model like ours is more practical than the translate-test method: (1) The translate-test method has to deploy an additional NMT system, which introduces the storage and computation overhead. Although CLIP+TrTest can be improved with better translations, it usually comes with the cost of larger NMT models and longer latencies. (2) For the translate-test method, the translation process of every text query has to go through the encoder and decoder of the NMT model. The decoding process is usually in an autoregressive manner, which brings non-negligible computing overhead and latency. However, the text query of mCLIP+ only goes through a multilingual encoder, which is computed parallelly. (3) The translate-test method is not suitable for the image-to-text retrieval task. To
apply the translate-test method, every non-English image description has to be translated into English and stored in databases, which is infeasible for real billion-level cross-modal retrieval applications.
B.2 Comparison with English CLIP. For zero-shot English retrieval on MSCOCO, the mean recall of the original CLIP (ViT-B/32) model is 60.4 (Radford et al., 2021). On the other hand, though mCLIP gains zero-shot cross-lingual transferability on the non-English image-text retrieval, from Table 1, its mean recall score on English is only 53.2, accounting for only 87.2% of the original CLIP’s performance. This comparison reveals that one limitation of mCLIP is that the proposed method may slightly degrade the performance on the English cross-modal retrieval task. This performance degradation can be alleviated by using more pretraining data, i.e., mCLIP+ trained with more parallel corpus and multilingual image-text pairs retains 93.8% English retrieval ability of the original CLIP. In addition, since the image backbone of mCLIP initialized from CLIP is frozen during training, one can store an additional CLIP text encoder for English image-text retrieval task when the storage is allowed.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7
3 A2. Did you discuss any potential risks of your work? Section 8
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 and Section A of Appendix
3 B1. Did you cite the creators of artifacts you used? Section 4 and Section A of Appendix
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? They are widely used open-sourced data and tools.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The use is consistent.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We follow the previous works.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? They are covered in the original papers.
7 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. They are covered in the original papers.
C 3 Did you run computational experiments? Section 4 and Section B of Appendix
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section A of Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4 and Section A of Appendix
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? The computation cost of one experiment run is high. We follow previous work to report experimental results.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future explorations are needed to mitigate the misuse of VLP models