Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training
Year: 2023

Abstract:
In this paper, we introduce Cross-View Language Modeling, a simple and effective pretraining framework that unifies cross-lingual and cross-modal pre-training with shared architectures and objectives. Our approach is motivated by a key observation that cross-lingual and cross-modal pre-training share the same goal of aligning two different views of the same object into a common semantic space. To this end, the cross-view language modeling framework considers both multi-modal data (i.e., image-caption pairs) and multi-lingual data (i.e., parallel sentence pairs) as two different views of the same object, and trains the model to align the two views by maximizing the mutual information between them with conditional masked language modeling and contrastive learning. We pre-train CCLM, a Crosslingual Cross-modal Language Model, with the cross-view language modeling framework. Empirical results on IGLUE, a multi-lingual multi-modal benchmark, and two multi-lingual image-text retrieval datasets show that while conceptually simpler, CCLM significantly outperforms the prior state-of-the-art with an average absolute improvement of over 10%. Moreover, CCLM is the first multi-lingual multimodal pre-trained model that surpasses the translate-test performance of representative English vision-language models by zero-shot cross-lingual transfer.1

heading: 1 Introduction
text: Recently, the tremendous success of selfsupervised language model pre-training (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Radford et al., 2019; Dong et al., 2019; Raffel et al., 2019; Lewis et al., 2020; Brown et al., 2020) has been expanded to the
∗Equal Contribution. Work done at ByteDance. †Correspondence to: zengyan.yanne@bytedance.com
1The code and pre-trained models are available at https: //github.com/zengyan-97/CCLM.
multi-lingual (Conneau and Lample, 2019; Conneau et al., 2020; Pfeiffer et al., 2020; Chi et al., 2021) and multi-modal (Lu et al., 2019; Tan and Bansal, 2019; Su et al., 2020; Chen et al., 2020; Li et al., 2020) domain. Advances on multi-lingual pre-training enables cutting-edge language technology to benefit a much boarder group of users including non-English speakers. Similarly, multimodal pre-training makes pre-trained models applicable to a much larger set of tasks and user groups. Both of these directions make people’s lives in a multi-lingual multi-modal world easier. Therefore, a natural next step is to explore multi-lingual multi-modal pre-training which enables pre-trained models to solve multi-modal tasks expressed in non-English languages without the need of collecting training data in these languages, which can be very costly for certain low-resource languages.
While appealing, multi-lingual multi-modal pretraining has its own challenges. Unlike multilingual pre-training and multi-modal pre-training where relatively large amount of parallel data is available, there exists only a few multi-lingual multi-modal corpora and their language coverage is also limited. Two pioneering works, M3P (Ni et al., 2021) and UC2 (Zhou et al., 2021), propose to pivot either on English texts or images to align multi-lingual multi-modal representations. Both of them introduce a number of new objectives to make use of the anchor for alignment. However, a recent benchmark on multi-lingual multi-modal pre-training (Bugliarello et al., 2022) reveals that these multi-lingual multi-modal pre-trained models are still falling short: while achieving seemingly promising zero-shot cross-lingual transfer performance on some vision-and-language tasks, they still significantly under-perform “translatetest”, a simple baseline which translates the test examples into English and uses an English-only vision-language model for inference. This prevents existing multi-lingual multi-modal models to be
5731
applicable in real-world applications. In contrast, multi-lingual pre-trained models such as XLMR (Conneau et al., 2020) significantly outperforms the translate-test baseline in most languages and is widely used in practical applications.
This paper aims to fully exploit the potential of multi-lingual multi-modal pre-training. We point out two major limitation of current state-of-thearts. First, existing methods do not exploit parallel text corpora, which can be easily collected and are abundant for many language pairs. Instead, M3P performs masked language modeling with monolingual texts in different languages for multi-lingual alignment. However, parallel texts are shown to be more helpful according to multi-lingual pretraining literature (Conneau et al., 2020; Chi et al., 2021). Second, a number of new pre-training objectives involving specific architecture changes and different input-output formats are introduced for English or image pivoting, making it non-trivial to combine them together for better performance and scale to larger data.
In this work, we argue that multi-lingual and multi-modal pre-training are essentially achieving the same goal of aligning two different views of a same object into a common semantic space. Therefore, we believe these two seemingly different strategies can be combined into a unified framework. To this end, we introduce cross-view language modeling, a simple and effective framework that unifies cross-lingual and cross-modal pretraining with shared architecture and objectives. Specifically, we consider both multi-modal data (i.e., image-caption pairs) and multi-lingual data (i.e., parallel sentence pairs) as pairs of two different views of the same object. With either multimodal or multi-lingual data as input, we encode the two views with Transformer models and then fuse their representations with a cross-attention Transformer model shared for both cross-modal and cross-lingual fusion. We train the model to align the two views into a common semantic space by maximizing the mutual information between them with a conditional masked language modeling objective, a contrastive learning objective, and a matching objective. In this way, the cross-view language modeling framework unifies English pivoting and image pivoting schemes seamlessly and makes the best of both worlds.
To evaluate the effectiveness of our approach, we pre-train CCLM, a Cross-lingual Cross-modal
Language Model, with the proposed cross-view language modeling framework. Experimental results show that CCLM significantly outperforms prior state-of-the-art with an averaged absolute improvement of over 10% and 30% on multi-lingual visionlanguage understanding and retrieval tasks in terms of accuracy and R@1 on IGLUE (Bugliarello et al., 2022), a recently released multi-lingual multimodal benchmark. Notably, CCLM is the first multi-lingual vision-language model that surpasses the “translate-test” performance of mono-lingual vision-language models via zero-shot cross-lingual transfer, which we believe is a crucial step towards practical multi-lingual multi-modal pre-training. Since previous work used different pre-training datasets, making direct comparison difficult, we also conduct an in-depth ablation study to investigate the contribution of different parts in our framework. The results show that use of parallel sentence pairs helps to fully exploit the potential of language pivoting for multi-lingual multi-modal pre-training and also confirm the importance of unified architectures and objectives in CCLM.
Contributions. (1) We propose a cross-view language modeling framework that unifies multilingual and multi-modal pre-training with shared architectures and objectives. (2) CCLM advances the state-of-the-art of multi-lingual vision-language pre-training by a large margin. It also surpasses the translate-test baseline for the first time, demonstrating the potential of multi-lingual multi-modal pre-training. (3) We further scale up CCLM with massive pre-training data and larger model size. We will release our large-scale pre-trained multilingual multi-modal models to benefit a larger set of tasks and user groups and setup a strong and easily reproducible baseline for multi-lingual multimodal research.

heading: 2 Related Work
text: Multi-lingual Pre-training Multilingual BERT (Devlin et al., 2019) demonstrates that good cross-lingual transfer results can be achieved by performing masked language modeling on multi-lingual corpora with shared vocabulary and weight. Later, XLM (Conneau and Lample, 2019), XLM-R (Conneau et al., 2020), and Unicoder (Huang et al., 2019) introduce a number of new objectives including translation language modeling (TLM), cross-lingual word recovery, and cross-lingual paraphrase classification to improve
multi-lingual pre-training. More recently, MADX (Pfeiffer et al., 2020) and InfoXLM (Chi et al., 2021) further improve multi-lingual pre-training via adapter (Houlsby et al., 2019) and contrastive learning.
Vision-Language Pre-training Inspired by the success of language model pre-training, a number of work (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2020; Chen et al., 2020; Zeng et al., 2021; Wang et al., 2022; Yu et al., 2022) investigates vision-language pre-training on large scale imagecaption pairs and proposes a number of objectives to align vision and language representations, including masked multi-modal modeling, multimodal alignment prediction, RoI feature regression, image-text matching, to name a few. Visionlanguage pre-training has reshaped the landscape of vision-and-language research and pushed the state-of-the-arts on a wide range of vision-language tasks (Zhou et al., 2022). However, it is non-trivial to collect large scale image-caption pairs in other languages. As such, most existing vision-language pre-trained models are limited to English tasks.
Multi-lingual Multi-modal Pre-training Multilingual multi-modal pre-training aims to make multi-modal models applicable on non-English texts by cross-lingual transfer. In this paper we mainly consider multi-modal in the visionlanguage context. The key difficulty of multilingual multi-modal pre-training is the lack of non-English image-text pairs. Two representative works tackle the lack of non-English image-text pairs by pivoting on either English texts or images. Specifically, M3P (Ni et al., 2021) uses English as pivot and alternates between Englishonly vision-language pre-training and multi-lingual masked language modeling. UC2 (Zhou et al., 2021), on the other hand, translates English captions into multiple languages and considers images as the anchor, achieving state-of-the-art on various multi-lingual vision-language tasks. More recently, MURAL (Jain et al., 2021) collects large-scale image-text pairs in 110 languages and pre-trains a dual encoder model via contrastive learning. MURAL achieves new state-of-the-art on multi-lingual image-text retrieval tasks. However, the dual encoder architecture of MURAL makes it unable to perform multi-modal understanding tasks well.

heading: 3 Cross-View Language Modeling

heading: 3.1 Overview
text: Cross-view language modeling is a simple framework that unifies cross-lingual pre-training and cross-modal pre-training with shared architecture and objectives. CCLM consists of an image encoder, a cross-lingual text encoder, and a fusion model. All components are Transformer-based. Specifically, the image encoder (Dosovitskiy et al., 2021) first splits an image into non-overlapping patches, and then embeds these patches with transformer layers, yielding {v⃗cls, v⃗1, ..., v⃗N1}. For an image of resolution of 224x224 and patch size of 32x32, we have N1 = 49. Similarly, the crosslingual text encoder encodes a text input via transformer layers, yielding {w⃗cls, w⃗1, ..., w⃗N2}. N2 is the length of the text input. Then, the fusion model fuses text features with the corresponding image features or features of the translated text based on cross-attention, producing {x⃗cls, x⃗1, ..., x⃗N2}.
As illustrated in Figure 1, with either (text, image) pairs or (text, translation) pairs as input, we consider the paired input as two different views and train the model to align their representations in a common semantic space. This unified crossview perspective allows us to share input-output formats, architectures, and training objectives between cross-lingual inputs and cross-modal inputs. Specifically, we completely share the fusion model for both cross-lingual fusion and cross-modal fusion, and optimize the model by contrastive loss, matching loss, and conditional masked language modeling loss for both cross-lingual and crossmodal inputs. We select these objectives because they are universally effective in both cross-lingual and cross-modal pre-training literature (Chi et al., 2021; Li et al., 2021). We will show that the three loss maximize sequence-level and token-level mutual information between image-caption pairs or parallel sentence pairs. On the other hand, we empirically find that the three loss are more effective for cross-lingual cross-modal pre-training than certain task-specific loss such as masked regionto-token language modeling which is specially for multi-modal pre-training or translation language modeling for multilingual pre-training.

heading: 3.2 A Mutual Information Maximization Perspective
text: In this section, we explain our approach from an information-theoretic perspective. Formally, given
two random variablesA andB, mutual information I(A,B) measures dependencies between the two random variables. We define A = a and B = b as two different views of a data point, which can be either an image-caption pair or a parallel sentence pair. In this case, we will show that CCLM maximizes a lower bound of I(A,B) for crosslingual cross-modal pre-training by minimizing the InfoNCE loss (Oord et al., 2018) defined as:
Lnce = −Ep(A,B) [ log
exp(fθ(a, b))∑ b̃∈B̃ exp(fθ(a, b̃))
] , (1)
where fθ ∈ R is a function parameterized by θ and B̃ contains the positive sample b and |B̃| − 1 negative samples.
The contrastive loss between the image encoder and the cross-lingual text encoder is a symmetric version of Lnce:
Lcl = −1 2 Ep(A,B)
[ log
exp(fθ(a, b))∑ b̃∈B̃ exp(fθ(a, b̃))
+ log exp(fθ(a, b))∑
ã∈Ã exp(fθ(ã, b))
] , (2)
where |Ã| = |B̃| = N is the batch size, and we predict (a, b) pairs from in-batch negatives. fθ(a, b) = gv(v⃗cls)
⊤gw(w⃗cls)/τ given an imagecaption pair or fθ(a, b) = gw(w⃗acls)
⊤gw(w⃗bcls)/τ given a translation pair. v⃗cls and w⃗cls are the output
[CLS] embedding of the image encoder 2 and the cross-lingual text encoder. gv and gw are transformations that map the [CLS] embeddings to normalized lower-dimensional representations. τ is a learnable temperature parameter.
Similarly, the matching loss applied on the output [CLS] embedding of the fusion model (denoted as x⃗cls(a, b)) can also be viewed as a symmetric version of Lnce:
Lmatch = − 1
2 Ep(A,B)
[ log
exp(fθ(a, b))
exp(fθ(a, b)) + exp(fθ(a, bneg))
+ log exp(fθ(a, b))
exp(fθ(a, b) + exp fθ(aneg, b))
] ,
(3)
where we only sample a negative instance for each ground-truth (a, b) pair and predict whether a pair is matched (true or false). In this case, fθ(a, b) = v⃗ ⊤ truex⃗cls(a, b), where v⃗true is a parametric vector. The conditional MLM loss can also be interpreted as maximizing mutual information (Kong et al., 2020) between the context c = (â, b) (â denotes the masked text input, and b is the corresponding image or translated text) and the masked token wi in a:
Lmlm = −Ep(C,W ) [ log
exp(fθ(c, wi))∑ w̃∈V exp(fθ(c, w̃))
] , (4)
2Some vision transformers, e.g. Swin-Transformer, use the output of average pooling layer as the [CLS] embedding.
where fθ(c, wi) = ψ(wi)⊤x⃗i(â, b). x⃗i is the output vector at wi position of the fusion model. ψ(w) : V → Rd is a lookup function that maps a word token w into a parametric vector. V is the full vocabulary set.
Finally, the pre-training objective of CCLM is defined as: L = Lcl + Lmatch + Lmlm, where the contrastive loss and matching loss maximize sequence-level mutual information while the MLM loss maximizes token-level mutual information, which are complement of each other.

heading: 4 Experiment

heading: 4.1 Experimental Settings

heading: 4.1.1 Pre-training Datasets
text: We pre-train CCLM on the combination of imagecaption pairs and parallel multilingual texts. Appendix A.1 describes compared models in details. Multi-modal Data For image-caption pairs, we follow the practice of UC2 to make a fair comparison and use their released translation-augmented version of CC3M dataset. It contains the original CC3M image-caption pairs (Sharma et al., 2018) and machine-translated captions in five different languages (German, French, Czech, Japanese, and Chinese). This multi-modal dataset is widely utilized by previous work, including UC2, mUNITER and xUNITER. We denote this variant as CCLM3M. In additional to this setting, we leverage large-scale vision language pre-training by utilizing the pretrained weights of X2-VLM (Zeng, 2021; Zeng et al., 2022) which has been trained on more than 1B image-text pairs in English. Based on it we apply the proposed framework for multi-lingual multi-modal pre-training. Multi-lingual Data Previous work such as mUNITER, xUNITER, and M3P use large-scale monolingual texts in different languages, namely multilingual Wikipedia 101G dataset, for multilingual alignment. Differently, we propose to utilize parallel text corpus. We collect a subset of the WikiMatrix (Schwenk et al., 2021) dataset containing parallel texts between English and other languages in the IGLUE benchmark. Appendix A.2 shows the number of pairs per language. In total, the dataset consists of 19M parallel sentence pairs.

heading: 4.1.2 Implementation Details
text: CCLMbase consists of 12 Transformer layers for the image encoder and the text encoder respectively. CCLMlarge consists of 24 layers for each encoder.
The fusion encoder contains 6 Transformer layers for both CCLMbase (d = 768) and CCLMlarge (d = 1024). In total, CCLMbase and CCLMlarge consist of ∼ 420M and ∼ 970M parameters respectively. Following existing models such as M3P and UC2, we also utilize XLM-R (Conneau et al., 2020) as the text encoder. Concretely, CCLM3M is initialized with a pre-trained image encoder (Liu et al., 2021b) and XLM-R. CCLM is initialized with the pre-trained X2-VLM (Zeng, 2021; Zeng et al., 2022) and XLM-R.
In pre-training, the image encoder takes images of resolution of 224 × 224 as input for pretraining. During fine-tuning, we increase the image resolution to 384 × 384 and interpolate the positional embeddings of image patches following Dosovitskiy et al. (2021). The maximum sequence length is set to 30 and 64 for image captions and parallel multilingual texts respectively. We apply mixed precision for pre-training. We use the AdamW (Loshchilov and Hutter, 2019) optimizer with a weight decay of 0.02. We mix different types of data in a training batch. Following UC2, to make a fair comparison, we train CCLM3M for 30 epochs on 8 NVIDIA A100 GPUs and the batch size is set to 1024, which tasks ∼ 1.5 days. The learning rate is warmed-up to 1e−4 in the first 2500 steps and decayed linearly. We train CCLMbase and CCLMlarge for 40 epochs.

heading: 4.1.3 Downstream Tasks
text: We evaluate CCLM on the IGLUE benchmark (Bugliarello et al., 2022), a recently released benchmark for evaluating multi-lingual multimodal pre-training, and a multi-lingual image-text retrieval benchmark including the multi-lingual version of Flickr30K (Young et al., 2014; Elliott et al., 2016) and MSCOCO (Chen et al., 2015). Note that CCLM can also be applied on generation tasks such as image captioning by following the adaptation strategy of X-VLM (Zeng et al., 2022; Zeng and Nie, 2021). XVNLI: The Cross-lingual Visual NLI dataset is collected by combining SNLI (Bowman et al., 2015) with its multi-modal (Xie et al., 2019) and multi-lingual (Agić and Schluter, 2018) counterparts. It requires the model to predict if a texthypothesis “entails”, “contradicts”, or is “neutral” to an image-premise. xGQA: The Cross-lingual Grounded Question Answering task (Pfeiffer et al., 2021) is collected by manually translating the GQA (Hudson and Man-
ning, 2019) validation set into 7 languages. It requires a model to answer several types of structured questions about an image. We model GQA as a generation task following Li et al. (2021). MaRVL: The Multicultural Reasoning over Vision and Language dataset (Liu et al., 2021a) requires to determine whether a textual description is true or false about a pair of images. The MaRVL dataset is used for testing and the NLVR2 (Suhr et al., 2019) dataset is used for training. xFlickr&CO and WIT: The xFlickr&CO dataset is collected by combining 1000 images from Flickr30K and MSCOCO respectively and crowdsource image descriptions in 6 other languages. Similarly, the Wikipedia-based Image Text dataset (Srinivasan et al., 2021) is collected from Wikipedia in 108 languages. We follow the data preprocessing and splitting details in IGLUE for both datasets. Multi30K: This dataset (Elliott et al., 2016) extended Flickr30K (Young et al., 2014) from English (en) to German (de), French (fr) and Czech (cs). It contains 31,783 images and provides five captions per image in English and German, and one caption per image in French and Czech. Dataset splits are defined as the original Flickr30K. MSCOCO: This dataset extends the MSCOCO caption dataset (Chen et al., 2015) by translating the captions into Japanese (Yoshikawa et al., 2017) and Chinese (Li et al., 2019). The Japanese and Chinese subsets consist of 820k and 20k captions respectively. Following previous work, we use the same train, dev, and test splits for English and Japanese as defined in Karpathy and Li (2015). As for Chinese, we use the COCO-CN split (Li et al., 2019). For all retrieval tasks, we follow previous work (Li et al., 2021) and X-VLM (Zeng et al., 2021). During fine-tuning, we optimize Lcl and Lmatch. For inference, we first compute similarity for all images and texts, and then take the top-k candidates and calculate the final ranking scores using the fusion model.

heading: 4.2 Experimental Results

heading: 4.2.1 Results on IGLUE Benchmark
text: Table 1 shows CCLM performance on the IGLUE benchmark. First, for zero-shot cross-lingual transfer, we can see that CCLM3Mbase outperforms all compared models by a substantial margin while pre-trained on the same multi-modal data. Specifically, compared to UC2, the prior state-of-the-art, CCLM3Mbase obtains an average accuracy improvement of 11.4% on multi-lingual multi-modal understanding tasks including XVNLI, xGQA, and MaRVL, and an average R@1 improvement of
47.3% and 18.2% on multi-lingual multi-modal retrieval datasets including xFlickr&CO and WIT. This confirms that previous multi-lingual multimodal models fail to fully exploit the potential of multi-lingual multi-modal pre-training and our proposed cross-view language modeling framework can better align multi-lingual multi-modal representations with unified objectives.
We also find that the performance of our framework can be significantly improved by leveraging large-scale image-text pre-training in English (CCLMbase) and/or scaling up the model size (CCLMlarge). Notably, CCLM is the first multilingual multi-modal pre-trained model that substantially outperforms the translate-test results of representative English VLMs tested in the IGLUE benchmark. This, for the first time, proves the potential of multi-lingual multi-modal pre-training on building practical real-world applications involving vision-language tasks in different languages.

heading: 4.2.2 Results on Multi-lingual Retrieval
text: Table 2 gives the results on the multi-lingual imagetext retrieval benchmark. When pre-trained on the same multi-modal data, CCLM3Mbase substantially outperforms UC2, the prior state-of-the-art, with an averaged improvement of over 10% (in terms of averaged recall) across four languages on Multi30K. This confirms that our approach can better align multi-lingual multi-modal representations. CCLM3Mbase even outperforms MURAL. This is notable because MURALlarge is larger than our model and is pre-trained on much more data (∼ 450× more image-text pairs and 390× more parallel sen-
tence pairs). Moreover, we show that CCLM also outperforms MURAL without fine-tuning in Appendix A.5.
We also find that the cross-view language modeling framework yields better performance if leveraging large-scale pre-training on image-text pairs in English (CCLMbase) and/or scaling up the model size (CCLMlarge), which is consistent with the experimental results on the IGLUE benchmark. It confirms that the proposed framework is scalable to both massive data and larger model size.

heading: 4.2.3 Cross-lingual Transfer Gap
text: In addition to absolute cross-lingual transfer results reported in Table 1 and Table 2, we also compare the cross-lingual transfer gap of different models. We visualize the ratio of a model’s performance on non-English languages to its performance on English test set, in Figure 2. A larger radar chat indicates the model has a smaller relative transfer gap and can better transfer its performance to non-English test sets. We can see that CCLM’s relative cross-lingual transfer gap is consistently smaller than that of UC2 across all tasks in the IGLUE benchmark (a) and all languages in the multi-lingual retrieval datasets (b). The absolute cross-lingual transfer gap is even more significant. For example, in Appendix A.5, we can see that for M3P, the absolute zero-shot cross-lingual transfer gap between EN-CS and EN-JA in Multi30K and MSCOCO are 41.4% and 32.6% respectively. This indicates that masked language modeling on unpaired texts in multiple languages are not very effective for cross-lingual alignment of multi-modal models. The gap for UC2 is reduced to 13.2% and 16.4%, demonstrating the effectiveness of using machine-translated captions for multi-lingual multi-modal pre-training. CCLM3Mbase further reduces this gap to 5.4% and 4.4%. This confirms that the proposed cross-view language modeling
framework can effectively transfer multi-modal representations from English to other languages without language-specific fine-tuning. In addition, we also visualize the multi-lingual text representations and image representations in CCLM and a baseline approach in Appendix A.6, which clearly shows our approach can better align multi-lingual imagetext representations.

heading: 4.3 Ablation Study
text: Since previous work such as M3P, UC2, and MURAL all use different pre-training datasets, making direct comparison difficult, we conduct an in-depth ablation study to investigate the contribution of different design choices in the cross-view language modeling framework. We pre-train 5 ablated variants of CCLM where parallel sentence pairs, unified architecture, or unified objectives are ablated. All compared models are pre-trained with the same CC3M and WikiMatrix data (except that w/o parallel sentence pairs) for 15 epochs to ensure a fair comparison. The results are shown in Table 3.
First, we find that the use of parallel sentence pairs plays a very important role. This indicates that previous methods fail to fully exploit the potential of language pivoting for multi-lingual multimodal pre-training. On the other hand, CCLM variant trained without parallel sentences in Table 3 which uses the same pre-training dataset as UC2 still significantly outperforms previous models such as M3P and UC2.
We then compare other ablated variants which all utilized parallel sentence pairs. We find that separate parameterization of cross-attention and FFN modules for the cross-lingual and the crossmodal task in the fusion model leads to inferior results, especially for multi-lingual multi-modal
understanding tasks such as xGQA. Moreover, we conduct ablation study on loss functions. We mainly consider multi-lingual objectives because the multi-modal objective combination of itc+mlm+itm is the de-facto choice for multi-modal loss (Li et al., 2021; Zeng et al., 2021). We find that using common objectives in the multilingual pre-training literature underperforms our unified objective. These observations confirm the importance of unifying architectures and objectives for multi-lingual multi-modal pre-training.

heading: 5 Conclusion
text: In this paper, we introduce cross-view language modeling, a simple and effective framework that unifies cross-lingual and cross-modal pre-training. It considers cross-lingual and cross-modal pretraining as the same procedure of aligning the representation of two different views of the same object, thus using shared model architectures and training objectives for multi-lingual multi-modal pretraining. We train CCLM with the proposed framework and show that it advances the state-of-the-art on all downstream multi-lingual vision-language tasks by a large margin. Moreover, it surpasses the translate-test baseline for the first time, demonstrating the potential of multi-lingual multi-modal pre-training. Furthermore, the experimental results also confirm that the proposed framework is scalable to massive data and larger model sizes. We believe our model will become a foundation for future multi-lingual multi-modal research and serve as a strong baseline. Moreover, the cross-view language modeling framework also has the potential of unifying more modalities such as audio and video with the same architectures and objectives. We leave this for future work.
Limitations
In this paper, we pre-train CCLM with moderate multi-modal data, e.g. CC3M, to make a fair comparison with previous work such as M3P and UC2. We leverage large-scale vision language pretraining simply by utilizing the pre-trained weights of X2-VLM which has been pre-trained on billionscale image-text pairs in English. Collecting more image-text pairs in different languages will very likely lead to further performance improvements. Moreover, there exists larger public available multilingual datasets, such as MultiUN (Ziemski et al., 2016) and OPUS (Tiedemann, 2012). Leveraging more multi-lingual datasets for pre-training should also yield a more powerful multi-lingual multi-modal model.
As for social impact, multi-modal pre-trained models can be used in applications that help people with disability in one modality. Our work makes these applications applicable to minority people speaking non-English, and potentially low-resource languages. In sum, our work potentially enables deep learning technology to benefit more people, and is unlikely to have direct negative social impact.

heading: Acknowledgements
text: We would like to thank Hang Li, Jiaze Chen, and Huiyun Yang at ByteDance for insightful comments in technical discussions. We also thank Yaoming Zhu at ByteDance for his generous assistance in data collection and valuable feedback.

heading: A Appendix
text: A.1 Compared Models
mUNITER and xUNITER: A multi-lingual variant of the UNITER (Chen et al., 2020) model pretrained by Liu et al. (2021a). The model is pretrained by alternating between a batch of multimodal English data from CC3M with UNITER objectives and a batch of text-only multilingual Wikipedia data with the MLM objective. mUNITER and xUNITER differ in their initialization: mUNITER and xUNITER are initialized from mBERT and XLM-R.
M3P: A multi-lingual multi-modal model initialized from XLM-R and pre-trained with the combination of multilingual masked language modeling, multi-modal code-switched masked language modeling, multi-modal code-switched masked region modeling, and multi-modal code-switched visuallinguistic matching. The code-switched training method allows the model to explicitly align images with non-English languages. In each multi-modal batch, image-text pairs are fed to the model either fully in English or with code-switched words according to a given sampling ratio. Similar to mUNITER and xUNITER, the model is trained by alternating multi-modal and multi-lingual batches.
UC2: The state-of-the-art multi-lingual visionlanguage model which relies on (text-only) machine translation technologies to obtain CC3M data in five languages (Czech, French, German, Japanese, and Mandarin). The model is then pre-trained on multi-lingual multi-modal batches where a caption is sampled uniformly from the available languages for each image. As for pretraining objectives. In addition to conventional vision-language pre-training objectives, a visualconditioned translation language modeling objective is added to improve multi-lingual multi-modal alignment.
A.2 Details for Multi-lingual Data

heading: 609 509 455 412 353 281 269 241 61 51
text: A.3 Results on English Tasks
Table 5 reports CCLM performance that is pretrained on COCO, VG, SBU, and CC3M, on three common English multi-modal tasks. We can observe that CCLM also has very competitive performance compared to strong English multi-modal baselines.
A.4 Few-Shot Results on IGLUE
Table 6 gives results on IGLUE benchmark. For our models, mean and standard deviation (in brackets) of 3 different runs with different random seeds are reported. Results of compared models are directly copied from the IGLUE benchmark. In the few-shot setting, the English trained models are continually fine-tuned with a few labeled examples in a target language before evaluating on this language. We select exactly the same few-shot examples following IGLUE instructions to ensure our results are compatible with that reported in IGLUE. We omit few-shot evaluation on the WIT dataset because this setup is also omitted in IGLUE. We find that similar to existing models, CCLM can also benefit from few-shot learning with a few examples in the target languages.
A.5 More Results on Retrieval Tasks
Table 7 reports results on multi-lingual image-text retrieval of CCLM. We follow the practice of prior work and evaluate in three different settings including English-only fine-tuning, single-language fine-tuning, and all-language fine-tuning, where the model is fine-tuned on English data, target language data, and the combination of training data in all languages, respectively.
We also report multi-lingual image-text retrieval results without fine-tuning (zero-shot) in Table 8. M3P and UC2 do not report their zero-shot retrieval performances. We can observe that CCLM3Mbase outperforms MURAL which is pre-trained on much larger data. Besides, the performance gap on non-
English test sets of MURAL is larger, which shows our model has better cross-lingual transfer ability.
A.6 Visualization of Representations
Figure 3 visualizes several examples in xFlickr&CO test set in 2D space using tSNE (Van der Maaten and Hinton, 2008). The image representations and text representations are the output [CLS] embeddings of the image encoder and the cross-lingual text encoder respectively.
We can observe that CCLM’s text representations in different languages are more gathered and the distances between text representations and corresponding image representations are relatively shorter. This indicates our approach can better align multi-lingual image-text representations.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: section 6

heading: 3 A2. Did you discuss any potential risks of your work?
text: section 6

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? experiment section

heading: 3 B1. Did you cite the creators of artifacts you used?
text: experiment section
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? they are commonly used datasets
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? they are commonly used datasets
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? they are commonly used datasets
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? they are commonly used datasets
7 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. they are commonly used datasets
C 3 Did you run computational experiments? section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? section 4.1.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section 4.1.1, 4.1.2, and 4.1.3; we report our best hyperparameter values with code release.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix Table 5 and Table 6. For our model, mean and standard deviation (in brackets) of 3 different runs with different random seeds are reported.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We believe our model will become a foundation for future multi-lingual multi-modal research and serve as a strong baseline. Moreover, the cross-view language modeling framework also has the potential of unifying more modalities such as audio and video with the same architectures and objectives. We leave this for future work. Limitations
In this paper, we pre-train CCLM with moderate multi-modal data, e. g. CC3M, to make a fair comparison with previous work such as M3P and UC2. We leverage large-scale vision language pretraining simply by utilizing the pre-trained weights of X2-VLM which has been pre-trained on billionscale image-text pairs in English. Collecting more image-text pairs in different languages will very likely lead to further performance improvements. Moreover, there exists larger public available multilingual datasets, such as MultiUN (Ziemski et al. , 2016) and OPUS (Tiedemann, 2012). Leveraging more multi-lingual datasets for pre-training should also yield a more powerful multi-lingual multi-modal model. As for social impact, multi-modal pre-trained models can be used in applications that help people with disability in one modality. Our work makes these applications applicable to minority people speaking non-English, and potentially low-resource languages. In sum, our work potentially enables deep learning technology to benefit more people, and is unlikely to have direct negative social impact