Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages
Authors: Jinyi Hu
Yuan Yao
Chongyi Wang
SHAN WANG
Yinxu Pan
Qianyu Chen
Tianyu Yu
Hanghao Wu
Yue Zhao
Haoye Zhang
Xu Han
Yankai Lin
Jiao Xue
dahai li
Zhiyuan Liu
Maosong Sun
Venue: ICLR 2024 spotlight (2024)
Keywords: Large Multimodal Models
Multilingual Transfer
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Recently there has been a significant surge in multimodal learning in terms of both image-to-text and text-to-image generation. However, the success is typically limited to English, leaving other languages largely behind. Building a competitive counterpart in other languages is highly challenging due to the low-resource nature of non-English multimodal data (i.e., lack of large-scale, high-quality image-text data). In this work, we propose MPM, an effective training paradigm for training large multimodal models in low-resource languages. MPM demonstrates that Multilingual language models can Pivot zero-shot Multimodal learning across languages. Specifically, based on a strong multilingual large language model, multimodal models pretrained on English-only image-text data can well generalize to other languages in a (quasi)-zero-shot manner, even surpassing models trained on image-text data in native languages. Taking Chinese as a practice of MPM, we build large multimodal models VisCPM in image-to-text and text-to-image generation, which achieve state-of-the-art (open-source) performance in Chinese. To facilitate future research, we open-source codes and model weights at https://github.com/OpenBMB/VisCPM.

Published as a conference paper at ICLR 2024
LARGE MULTILINGUAL MODELS PIVOT ZERO-SHOT MULTIMODAL LEARNING ACROSS LANGUAGES
Jinyi Hu'
Yuan Yao!
Chongyi Wang
Shan Wang*
Yinxu Pan
Qianyu Chen'
Tianyu Yu²
Hanghao Wu
Yue Zhao?
Haoye Zhang'
Xu Han Yankai Lin
Jiao Xue
Dahai Li
Zhiyuan Liu Maosong Sun
Tsinghua University Beijing University of Posts and Telecommunications
3 Shanghai Artificial Intelligence Laboratory
4 Renmin University of China
5 Zhihu Inc. ModelBess Inc.
hu- mro-jy21 @ma ss. t iiguaa........ edu.cn cn
ABSTRACT
Recently there has been a significant surge in multimodal learning in terms of both image-to-text and text-to-image generation. However, the success is typically limited to English, leaving other languages largely behind. Building a competitive counterpart in other languages is highly challenging due to the low-resource nature of non-Englist multimodal data (i.e., lack of large-scale, high-quality image-text data). In this work, we propose MPM, an effective training paradigm for train- ing large multimodal models in non-English languages. MPM demonstrates that Multilingual language models languages. Specifically, based on a strong multilingual large language model, multimodal models pretrained on English-only image-text data can well gener- alize to other languages in a (quasi)-zero-shot manner, even surpassing models trained on image-text data in native languages. Taking Chinese as a practice of MPM, we build large multimodal models VISCPM in image-to-text and text-to- image generation, which achieve state-of-the-art (open-source) performance Chinese. To facilitate future research, we open-source codes and model weights at httpss//ggiuubb.oo//////BBB/B//
Pivot zero-shot Multimodal learning across
INTRODUCTION
With the rapid advancement of powerful models such as GPT-4 (OpenAl 2023) and Stable Diffusion
(Rombach et al., (022) in their multimodal capabilities. large multimodal models have emerged as the latest frontier in pursuing achieving Artificial General Intelligence (AGI). Generally, the multimodal generative capabilities across images and text can be divided into two categories: (i) In the field of image-to-text generation, prominent multimodal large language models like GPT-4 OOpenAl 2023). LLaVA (Liu et al., 2002a) and InstructBLIP (Dai et al., 2023) exhibit remarkable multimodal conversational and reasoning abilities based on images; (ii) In the field of text-to-image generation, models such as Imagen (Saharia et al., (022) and Stable Diffusion (Rombach et al.) 2022) excel in generating highly realistic and relevant images based text prompts. These models possess exceptional capabilities in processing images and text, profoundly reshaping the landscape of multimodal AI in both academia and industry.
However, the success of large multimodal models has mainly been achieved within the English community, while the multimodal capabilities in other non-Engee -on-Ennlish languages significantly trail behind. Bridging this gap is challenging due to the extensive image-texl pair data requirements for training multimodal models. For instance, the pretraining of BLIP-2 (Li et al.) 2023a) involves more than 100M high-quality image-text pairs, while Stable Diffusion (Rombach et al.) (022) utilizes more than 2B pairs. As a result of the paucity of such multimodal data resources in non- oon-English languages, the progress of multimodal research in these languages remains hindered.
To address this challenge, we propose MPM, an effective training paradigm for large multimodal mod- els in non-English languages. MPM utilizes the Multilingual language model to Pivot Multimodal
"ooresponding authors.

---

Published as a conference paper at ICLR 2024
High-resource English as Pivat
Op
Image-Text Pretrain
VisCPM-Chat
Instruction Tuning
Muttlingual
Alignment
Cat
Alignment
Multimodal
Visual Encoder
Imaga-to-Toe Text Model
CPA--Be Bilingual
3ith
Backbone LLM
Large- Image- Pairs
Non-Enge Jnn-Enngish
Generalization
Language
Image- Pretrain
VisCPM-Paint
Vauaa
Text-t-image Model
(a) Motivation of MPM based on the generalization of multimodal alignment across languages.
(b) Architecture of VISCPM.
Figure I: Overview of the motivation and architecture of MPM and VISCPM.
learning across languages and considers English, which contains substantial multimodal data re- sources, as a pivot between the visual signals and non- Non-English languages which commonly lacking in multimodal data. MPM draws inspiration from the Bilingual Dual-coditg Theory (Paivio) 2014]; Clark & Paivio 1991) that argue that visual semantics are largely amngugge-gnosste Intuitively, as portrayed in Fig. Ta multilingual learners can effectively align the visual semantics with newly ac- quired language based on established multimodal and multilingual alignment. Simulating the human learning process, MPM also divides the non- non-English multimodal learning into two consecutive stages: multilingual alignment and multimodal alignment. The former focuses on building multilingual model, while the latter culminates in a multimodal model spanning multiple languages.
Specifically, for multilingual alignment, MPM harnesses pretrained multilingual large language model (LLM) as the backbone language model, which can provide aligned representations for different languages. Next, for the multimodal alignment, MPM trains the visual modules based the multilingual model exclusively on English image-text pairs to align English and visual semantics. Similar to how humans learn, using the multilingual model pivot point, the resultant multimodal model naturally acquires zero-shot multimodal capability in other non-English languages.
Taking Chinese practical instance for MPM, we develop Chinese large multimodal models named VISCPM based on English-Chinese bilingual large language model CPM-Bee (Zhang et al. 202T). Notably, pretraining exclusively on English image-text pairs, the zero-shot performance of VISCPM in Chinese still surpasses that of existing Chinese multimodal models trained on image-text pairs in native Chinese. The promising performance of MPM in Chinese sheds light on its potential application in broader languages. Following the same training process, we further extend MPM to develop a multilingual multimodal conversation model supporting six languages based on the LLaMA (Touvron et al. 2023), including English, German, French, Spanish, Italian, and Portuguese.
In summary, the contributions of this paper are as follows: (i) We propose MPM, an effective training paradigm specifically designed for non-Englin Englisn languages lacking multimodal resources. Researchers worldwide can utilize MPM to rapidly adapt advanced multimodal learning methods from English to their respective languages. (ii) We develop series of Chinese large multimodal models VIsCPM as a practical application MPM, which achieves state-of-the-an performance among open-source Chinese multimodal models. (iii) We open-source the model weights of VISCPM and provide experimental details, serving as valuable reference for other researchers.
(iv) We validate the generalization capability of VISCPM in diverse languages and develop the multilingual multimodal conversation model across six languages.
RELATED WORK
Image-to-text Models. Traditional image-to-text generation models mainly focus on the task of image caption and visual question answering (Yuan et al.] 2021] Yu et al.) [022aa] Wang et al. 2022b). Recently, the mainstream of image-to-text has turned to multimodal LLM, focusing on rendering
LLM capable of multimodal interaction with users. These models connect the visual module and
LLM with perceivers, such as BLIP-2 (Li et al. 2023a) and InstructBLIP (Dai et al. (023) or linear projectors such LLaVA VA (Liu et al.) 2023a) and PandaGPT (Su al. (023). VPGTrans Zhang ct al. 2023) explores the transferability of visual modules across LLM. Meanwhile, many efforts
have been dedicated to building multimodal instruction following datasets. I.LaVA (Liu et al. 2023a
2

---

Published as a conference paper at ICLR 2024
and MiniGPT-4 (Zhu al. 2023) build visual content-related dialog by transferring image captions
into conversation data using GPT-4. InstructBLIP (Dai et al., 2023) and M (Li et al. 2023b)
incorporate downstream rision-language datasets to construct instruction data. LLa VA- (Sun
et al. 2023) and RLHF-v (Yu et al. 2023b) propose multimodal RLHF for trustworthy behavior.
Text-fo-image Models. In the early stages of text-to-image model development, generative adversarial
networks (Zhu et al.; 2019: Li et al.; 2019) and auto-regressive generation (Esser et al. (021)
are popularly chosen architectures for text-to-image synthesis models (Li al. 2019). More recently, large-scale diffusion-basse text-to-image models such as DALLE-2 L-2 (Ramesh et al. 2022).
Imagen (Saharia et al. 2022), and Stable Diffusion (Rombach et al. 2022 have taken center stage,
demonstrating exceptional generative capabilities.
Multilingual Multimodal Models. The extension of multimodal models to include more languages has become key research focus over the past few years. Researchers have made efforts to extend the powerful image-text model CLIP (Radford et al. 2021) to handle more languages using techniques of knowledge distillation (Carlsson et al.) [022]] Hu et al., 2023) contrastive learning (Bianchi et al.
2021 Chen et al. 2023). Other studies have aimed to create a universal framework for multilingual
Vision- laiiuag pertaining and simultaneously achieve multilingual and multimodel alignment (Ni cet al. 2021 Zhou et al., 202I; Zeng et al., 2023). In the era of LLMs, PaLI (Chen et al., 2022) develops IB multilingual sanguag--image model based on 10B image-text pairs spanning 100 languages. VL. (Li et al. 2023b) shows that instruction tuning in English can generalize to other languages. MultiFusion (Bellagente al., 2023) discover that the multilingual language model can help cross-lingual transfer in lext-to-image generation. In comparison, this work provides a more systematical formulation for training multilingual multimodal models and demonstrates that the zero-shot transfer performance of these models can surpass that of models trained on native-language multimodal data.
3 MPM TRAINING PARADIGM
In this section, we first present the formulation of multilingual multimodal learning and provide an overview of the training procedure of MPM. Following this, we detail the specific training procedures of MPM for both image-to-text and text-to-image generation.
3.1 PROBLEM FORMULATION AND OVERVIEW
Multimodal learning can be formulated as modeling the relationship between images, denoted as x, and text, denoted as y. in target language 4, In this context, the image-to-text generation, which can be roughly summarized as generating description for input images, is to learn the conditional distribution pe(y'1 parameterized by 0: the text-to-image generation, which to synthesize relevant images given input text prompts, is to learn (1 (x]ly") y parameterized by 6.
In the vanilla setting, these conditional distributions are typically trained using image-text pairs D, {(I,, af )) in the target language 4 (Radford et al. 2021 Yang et al. 2022). However, high-quality image-text pairs are extremely scarce in most languages. To mitigate the dependency on native image-text text pairs, we introduce the pivot language bn. which contains abundant multimodal pairs
D, = ( v/" M where M > N. Imitating the human learning mechanism that can naturally align visual concepts with various learned languages, MPM aims to transfer visual concepts learned in the pivot language to the target language.
MPM divides the multimodal learning process in target language I, into two consecutive stages: multilingual alignment and multimodal alignment. For the multilingual alignment, MPM aims to establish the cross-lingual alignment for 4 and lp. This achieved by directly leveraging a pretrained multilingual LLM, denoted as fa. which can provide close hidden representations for text pair y and y with similar semanties, i.c., fo (v/1) ss (z For the multimodal alignment, MPM utilize the sufficient multimodal resource D, in the pivot language and optimize the image-to-text objective ( 2)) and text-to-imaan objective pa p (rlg/r). In the following sections, we introduce the training process of multimodal alignment stage. It's worth noting that MPM is agnostic the specific model architecture and training method, which enables us to flexibly utilize existing highly effective model architectures and training techniques in each task.
3

---

Published as a conference paper at ICLR 2024
3.2 IMAGE-TO-TENT GENERATION
In image-to-text generation, we incorporate an image encoder module mi parameterized by E to pro- vide visual feature hg (x). These visual features z are then concatenated with the text embedding as input into the multilingual LLM. Following recent work to train multimodal conversation models (Zhu al. 2023 Liu al. 2023a). MPM's training process for image-to-text generation consists of two sub-stages: Multimodal Pretraining and Instruction Tuning.
Multimodal Pretraining In this sub-stage, we pretrain the visual module to align it with LLM large scale of image-text pairs using the language modeling objective:
M
(pe, Dp)
Σ
log pe (v) (v/ (he
Here, we fix the parameters of LLM (0 {E}) prevent the powerful capabilities of LLM from being influenced by short texts in the image-text pairs.
Instruction Tuning. To enhance models' capabilities in following human instructions, conduct instruction tuning on elaborately curated multimodal instruction tuning datasets built by blending the existing multimodal instruction tuning datasets in the pivot language and their translated version in the target language. We denote this multilingual instruction tuning datasets as DD = ( siy Va,k k=== where v is the instructions and Va is the response in certain language I. Both the visual module and multilingual LLM are fine-tuned, i..., 0 (5,0), o) by maximizing the probability of the response:
Cp D.) = Σ log pe ( (zg())), fo (o
Interestingly, we find a quasi-zero-shot transfer capability of multilingual multimodal models in this scenario. If excluding the translated variant in the target language and solely performing instruction tuning using the pivot language, when given an image z and a question or an instruction 2 in the target language, the resultant model responds accurately though mostly in the pivot language. This can be attributed to the close resemblance between the hidden representation of instructions in two languages provided by the multilingual LLM, i.e., Ja (w(') ~ fo (ol Consequently, we have
Poly ( (vlt)) = po(2 (hg (z), fo (o(4)) )). Since both the pretraining and instruction tuning stages employ text components solely in the pivot language, the LLM can understand the question in the target language but cannot calibrate the response in the same language.
To stimulate the model respond in the target language, MPM incorporates small number of translated pairs in the target language during instruction tuning. In this way, MPM simultaneously capability and calibrates the response language, ultimately
improves the model's instruction-followie realizing a multimodal chatbot in the target language.
3.3
TEXT-TO-IMAGE GENERATION
In the text-to-image generation, we adopt a similar architecture with Stable Diffusion (Rombach et al. 2022). It incorporates a denoising network gs with a UNet architecture (Ronneberger et al. 2015) parameterized by 8 as an image decoder to generate images given the input prompt. The LLM fo and image decoder Is are interconnected with cross-attention mechanism (Vaswani al. 2017).
Diffusion models (Ho et al. 2020 Song al. 2020 involve learning an iterative process of denoising Gaussian noise into data distribution. The denoise network is optimized to remove the noise of noised image Fr,, conditioned on the hidden states of text input provided by the L.LM. The training
objective is defined follows:
C(ps. D(p).Dp) -.. ff(()))))))
-=(13)
In this stage, o- = (5). i.e., the image decoder is trained to align with frozen LLM. In this way, when input with the unseen prompt in the target language y²t the multilingual LLM fo can in- herently provide representation fo (y²* close to the seen representation fo (1/² of the pivot language prompt with similar semantics. Therefore, the capability of text-to-image in the target language can be seamlessly transferred from the pivot language in a zero-shot fashion, illustrated as ga ga(#r; Jo (y²)), )) 98(x,, (a('),)
4

---

Published as a conference paper at ICLR 2024
4 VISCPM
As a practice of MPM, we develop a series of large-scale Chinese multimodal models called VISCPM. We use Chinese as the target language and English as the pivot language. The Chinese-English bilingual language model CPM-Bee (Zhang et al. (021) serves as the backbone multilingual LLM. We have two variations of the model: ViSCPM-Chal for image-to-text multimodal conversation and VISCPM-Paing for text-to-image synthesis. In the following sections, we begin by providing an overview of existing multimodal datasets in Chinese and then introduce the training procedure of VISCPM-Chat and VISCPM-
4.1 ARE CHINESE MULTIMODAL DATASETS ENOUGH To TRAIN A MULTIMODAL MODEL?
The largest publicly available multimodal dataset of native Chinese is Wukong (Gu et al., 2022), consisting of 100M image-text pair pairs. However, by visualizing the CLIP score com- puted with Chinese-CLIP (Yang et al. 2022)) as shown in Fig. 2 and manually inspecting, as introduced in Appendix[G; we discover that only minor fraction of image-text pairs in Wukong possess semantically matched content. The poor quality of the dataset escalates existing data re- source shortfalls. A straightforward method to enhance the count of Chinese image-text pairs is to translate the English image-text pairs into Chi- nese, which has been utilized in previous work (Wang et al. 2022a Qiu et al., 2022). How- ever, translation requires an external machine translation model, and translating large-scale dataset used in pretraining consumes substantial computing resources. Also, as discussed in Sec.
Histogram of CUP Score
Trequendy
500
645
5.32
121
4.90
3.0
Score
Figure 2: The histogram of CLIP score of 100M Chinese image-text pairs from Wukong dataset. We set 0.18 as moderate filtering threshold.
5.3. we practically find that incorporating translated image-text only has marginal improvement on the performance when using powerful multimodal L.MM backbone language model, which already possesses strong cross-lingual generalization capability. Based on this analysis, we argue that effectively utilizing the existing English data achieve knowledge transfer in multimodal alignment is the key to developing powerful Chinese large multimodal model.
4.2 6d VISCPM-CHAT
VisCPM-Chat is a Chinese- English bilingual multimodal chatbot capable of responding to users' instructions based on the input image. VISCPM-Chal utilizes the Muffin architecture (Yu et al.) 2023a as the image encoder. Specifically, Muffin directly leverages a pretrained vision-language model BEiT-3 (Wang al. 2023) as inherent bridge module between vision and language. In the multimodal pretraining stage, the visual module is trained on 100M image-text pairs to align with the frozen LLM for 180K steps. In the instruction tuning sub-stage, we utilize bilingual versions of LLaVA 150K (Liu et al. 2023a) and UniMM-Chat YYu et al. 2023a). and the English part of M 2023b) fine-tune the image encoder and LLM for 80K steps. The details of these datasets are presented in Appendix [B.2, Due to the quasi-zero-shot phenomenon in Chinese introduced in Sec.
L al.
we incorporate certain Chinese data by translating LLaVA 150K and UniMM-Char into Chinese
using machine translation
|
To demonstrate the effect of Chinese image-text pairs, we also train an additional version of VISCPM- Chat, which adds additional Chinese image-text pairs during pretraining, including 20M native Chinese image-text pairs filtered from 100M Wukong (Gu et al. 2022) and 20M Zero-Chiness (Xie) et al. 2022) dataset based on CLIP score threshold greater than 0.18 and 136M image-text pairs translated from Laion-COn COCO dataset. We name this model as VIsCPM-Chat+
'We employ CPM-Bee for machine translation. CPM- Bee can obtain 41.6 BLEU score on WMT 2017 test set (Bojar et al. (017). Except for special instructions, all translation processes in this work are carried out using the CPM-Bee model.
5

---

Published as a conference paper at ICLR 2024
Table Experimental results on LLaVA Test Set accessed by GPT-4. Con: Conversation, DD: De- tailed Description, CR: Complex Reasoning, AVG: the average score of three tasks. The best/second best results are marked in bold and underlined, respectively.
LLM Backbone
English
Chinese
Model
Con
DD
CR
AVG
Con
CR
AVG
MiniGPT-4 4 Zhu et al.
2023
Vicuna-13 13B
65.0
67.3
76.6
69.7
English Model
InstructBLIP
Dui et 2023
Vicuma-13B
81,9
68.0
91.2
80.5
LLaVA Liu al
Vicuna-133
89.5
70.4
96.2
85.6
mPLUG-Ow]
Ye ct al
2023
BLOOMZ-7B
64.6
47.7
80.1
64.2
76,3
61.2
77.8
72.0
VisualGLM
Du al. 2022
ChatGLM-6B
62.4
63.0
80.6
68.7
76.6
87.8
83.6
82.7
Ea-Zh
Bilingual
Ziya-Visuma
(Wang
2022
L-ya-IJaMA-II
82.7
69.9
92.1
81.7
85.0
74.7
82.4
80.8
Model
Qwen-VL. Vh.-hall (Ba
2023
Qwen-7B
76.9
91.9
VISCPM-Cha
CPM-Bee-1OF
81.4
69.2
93.1
81.4
90.0
87.4
95.0
90.9
VisCPM-Chat+
CPM-Bee-108
80.1
67.1
97.1
81.5
91.3
90.7
95.4
92.5
4.3
VISCPM-PAINT
VIsCPM-Paint is a text-to-image synthesis model that can accept prompts in both Chinese and English. VisCPM-Paine employs the UNet in Stable Diffusion (Rombach al. 2022) as the image decoder. To maintain the generative capability of UNet, the training process only involves the cross-attention layer of UNet and the linear transblock between the multilingual LLM and UNet. We optimize these parameters using extensive dataset of English image-text pairs Laion-2B (Schuhmann al. 2022 for 300K steps.
Similar to VisCPM-Chat+, we train an additional version of VISCPM which is fine-tuned on Chinese image-texi pairs. The component of these pairs are identical to that VISCPM-CHI -hCat+ uses. We name this model as VISCPM-Paint+.
5
EXPERIMENTS
5.1 EVALUATION OF VISCPM-CHAT
5.1.1
EVALUATION SETTING
Baselines. We compare VIsCPM-Chat with existing multimodal conversation models, which include
the English-only models: MiniGPT-4 (Zhu et al. 2023), InstructBLIP (Dai et al. 2023), and LLaVA
(Liu et af.) 2023a), as well as Chinese-English bilingual models: mPLUG-Ow Yee et al. (023). VisualGLM (Du et al. 2022). Risuali (Wang
2022a), and Qwen- VL-Chat (Bai
2023
All of these Chinese- Englishh models have performed large-scale training on both Chinese and English multimodal datasets. More details baselines are presented in Appendix C
Evaluation Benchmark. We assess the multimodal conversational abilities of the VisCPM-Chat in both English and Chinese on the LLaVA Test Set (Liu et al.) 2023a) and UniMMM-ennhh (Yu et 2023ah. We use CPM-Bee to translate them into Chinese and manually check the translation results. Specifically, LLaVA comprehensively evaluates the model's ss multimodal conversation capability from "conversation", "detailed description", and "complex reasoning". MhiMMMBennch drawn from commonly used visual question-answering datasets incluudnn OKVQA (Marino et al. 2019) AOKVQA (Schwenk et al. 2022), GQA (Hudson & Manning 2019), and VQAv2 (Antol et al. 2015) is designed particularly for evaluating multimodal conversation models on the abilities associated with reasoning, commonsense, and world knowledge. Considcring that multimodal conversation models commonly generate responses with complete sentences to answer the question, traditional werinngmatching--aee based metrics, such VQAScore (Antol et al. 2015), this context. Therefore, for the LLaVA Test Set and UniMM-BB -Bench, we follow Liu et al. (023a):: Yu al. (023a) and employ GPT-4 to rate the model-generated responses and the reference answers. Further details of the LLaVA Test Set and UniMM-Bench are presented in Appendix D.3
suitable in
"Specifically, the GPT-4-0314 version evaluate the responses.

---

Published as a conference paper at ICLR 2024
Table Evaluation of bilingual multimodal models on Table 3: Results of extension to UniMM-Bench in Chinese. See results in English at Table
more languages based on Vicuna.
Model
OKVQA
AOKVQA
GQA
VQAv2
AVG
Lang
Con
DD
CR
AVG
mPLUG-OWl
52.8
55.8
60.8
56.7
56.5
en
87.6
76.8
101.0
88.6
VisualGLM
53.7
57.5
64.8
62.5
59.6
de
90.8
80.8
93.6
88.7
fr
87.6
81.9
94.6
88,2
Ziya-Visual
59.2
58.1
61.9
59.1
59.6
Qwen-VL-Chat
58,5
58.5
70.6
72.0
64.9
es
87.1
79.6
102.3
89.8
84.5
79.5
93.3
85,9
VisCPM-Chat
62.8
64.9
68.3
71.8
67.0
pt
81.3
81.6
92.4
85.1
5.1.2 EXPERIMENTAL RESULTS
Quantitative Results. The evaluation results on the LLaVA Test Set and UniMM-Bench are presented Table and 2 In Chinese, VIsCPM-Chat achieves remarkable results and outperforms all baseline models on both benchmarks. It is worth noting that, unlike VisualGLM and Qwen-VL-Chat, which leverage a substantial number of Chinese image-text pairs during pretraining, VISCPM- Chat does not incorporate any Chinese multimodal data in its pretraining process. Despite this, VisCPM-Chat still presents superior performance on conversational abilities, commonsense, and world knowledge. Such results strongly demonstrate the effectiveness of MPM in transferring visual knowledge from English to Chinese. In English, the performance of VISCPM-Chal exceeds that of mPLUG-OWl, VisuaIGLM, LIP and MiniGPT-4, and is roughly on par with Eiya-Visuaa and Qwen- VL-Chat while remaining comparable to the strong baseline LLaVA.
Case Study. In addition acquiring broad world knowledge and commonsense, MPM training also implicitly stimulated VisCPM-Chat's competency in transferring knowledge unique to Chinese culture. For example, as shown in Fig. For instance, as illustrated in Fig. 3 VISCPM-Chat is capable of relating the Chinese classical poems present in Su Shi's "Water Melody" according to the full moon and pavilion. Although this image-text pair deeply related Chinese culture is not included in the training data, ViSCPM-Char successfully navigates such a non- trivial knowledge transfer through MPM training. For a more detailed analysis of cases involving VISCPM- Chat, please refer to Appendix B.
0000
30 A Rat H Vide 102222223300 MAREI
155 bia
PUS L2 thastta 20 ELES (
most
Chesos bulking and do puooardiin rgt. cruliia bosuull and server Adk the for wine donnt know the paince in the day ainerywart year
Figure 3: Multimodal conversation cases of VISCPM-Chat in Chinese.
5.1.3 GENERALIZATION TO MORE LANGUAGES
The remarkable results of VIsCPM on Chinese of MPM illuminate the potential broader application more diverse of languages. Specifically, leverage the multilingual LLM LLaMA (Touvron et al.) 2023) as the LLM backbone and consider German, French, Spanish, Portuguese, and Italian target languages. Following the same training procedure as in VISCPM- Chat, we develop a multilingual multimodal chatbot proficiently supporting six languages. We begin by pretraining the visual encoder with LLaMA achieve visual feature alignment English image-text pairs. Next, we employ M2M-100 (Fan et al., 2021 to translate the instruction training set of LLaVA into five target languages. The original English instruction training set and the five translated sets are merged and shuffled, then used for fine-tuning the visual module and LLM.
Table presents the evaluation results for English and five target languages on LLaVA Testset. See the results on IGLUE benchmark (Bugliarello et al. (2022) in Appendix D.5 Notably, for the relatively popular languages, such as German, French, and Spanish, the average scores exceed 88. Additionally, for Italian and Portuguese, the average scores above 85. These results are highly encouraging as they demonstrate the chatbot's s coherent and accurate responses to visual-related questions in all six languages, even though the five target languages are simply blended during instruction tuning. The results in these languages validate the generalization and robustness of MPM in building powerful multimodal models in diverse linguistic contexts.
7

---

Published as a conference paper at ICLR 2024
Overall
Alignment
Roelity
Table 4: Zero-shot FID on MSCOCO dataset.
36
45
55
FIDU
Model
GLIDE Nichol et al. 2022
12.2
Percent
Make-A-Scent (Gafm al.
2022
11.8
DALL-E-2 Ramnesh
2122
10.4
UniDiffuser Bao
9.7
Ding 2
al
Stable Diffusion
Rombach et al.
2022
8.6
AltDiffusion
Chen et al.
2023
17.2
16.1
TaiyiDiffusion
Wang
A122a
15.6
VssPP
Talyi
Alt
VicPM Talyi
Alt
VisCPM Taiyi
Alt
VIsCPM-Paing
9.5
10.9
Figure 4: Results of human evaluation for text-to-
VIsCPM-Paints
9.9
9.6
images on Chinese Drawbench.
5.2
EVALUATION OF VISCPM-PAINT
5.2.
EVALUATION SETTING
We compare VisCPM-Pain with several strong text-to-image model, which including English-only
models: GLIDE (Nichol et al. 2022), Make-A-Scene (Gafni et al. 2022). DALL-E-2 (Ramesh et al.
2022). UniDiffuser (Bao et al. 2023), and Chinese Chinese Chinese-English bilingual text-to-image models:
Cog View2 (Ding et al. 2022). AliDiffusion (Chen et al. 2023) and TaiyiDiffusion (Wang et al.
2022a). We mainly compare VisCPM-Paine with AltDiffusion (Chen al. 2023) and TaiyiDiffusion
Wang et al. 2022a More details of baselines are presented in Appendix
5.2.2
AUTOMATIC EVALUATION
For the text-to-image tasks, assess the zero-shot Frechet Inception Distance (FID) (Heusel et al. 2017) and the CLIP score (Ramesh et al. 2021) using the MSCOCO validation set (Lin et al. 2014). We sample 30K prompts from MSCOCO, and for the Chinese evaluations, the text prompts are translated from the original English captions. We maintain the same sampling configuration for VISCPM-Paint, AltDiffusion, and TaiyiDiffusion and grid search the optimal FID across eight separate classifier guidance scales.
We present the zero-shot FID on MSCOCO validation in Table In Chinese, VisCPM-Paint achieves the best FID performance. By solely training on English image-text pairs, VISCPM-Paint displays a significant advantage over AltDiffusion (Chen et al. 2023) and TaiyiDiffusion (Wang et al. 2022a). In English, the performance of VISCPM- is comparable to existing powerful text-to-image models, such as Stable Diffuson (Rombach al., 2022) and UniDiffuser (Bao et
2023). See more analysis of fidelity and alignment trade-off' in Appendix
E.2
5.2.3 HUMAN EVALUATION
Following previous work (Yu al. 2022b Chang al.) 2023), we perform human evaluation to have a more comprehensive understanding of model performance. Due to the lack of human evaluation benchmarks in Chinese, we create Chinese text-to-image human evaluation benchmark, named Chinese Drawbench. Chinese Drawbench consists of 174 prompts and evaluates text-to- image models' proficiency across different aspects. We conduct human evaluation involving VIsCPM-Paint, AliDiffusion (Chen et al. (203)). and 'TayyDiifuusinn (Wang et al. 2022al on Chinese Drawhench. We ask five independent annotators to judge the best image generated by three models for each prompt. Their evaluation criteria included Overall, Alignment, and Fidelity. More details of human evaluation and Chinese Drawbench are provided Appendix E.3
Figure preuenttsssntts the human evaluation results. The figure shows the preference shares of 5 evaluators for each generated image with different marginals over consensuses in three aspects. Notably, VisCPM-Painl receives the strongest preference across all three evaluation aspects. In each aspect, more than 40 percent images generated by VisCPM-Pain are the favored choice. Impressively, in the more objective metrics of Overall and Alignment, VISCPM-PAa -aint more than 20 percent 5/5
8

---

Published as a conference paper at ICLR 2024
Table 5: Performance in Chinese with different combinations of dataset languages in each training stage. IT means instruction tuning.
(a) Image-to-text:
Evaluation results on LLaVA Test Set.
(b) Text-to-image: zero-shot FID on MSCOCO.
Datasct Language
Dataset Language
Con
DD
CR
AVG
FID
Pretrain
Pretrain
Finetune
Native Chinese
Chinese
82.4
81.8
91.7
85.5
Native
15.1
English
Chinese
84.0
83.6
91.2
86.3
Chinese
English
Bilingual
85.4
81.4
96.6
88.0
English
10.9
Finlish++
English
Native Chinese w/o Filter
12.8
Bilingual
89.1
82.3
91.4
87.8
Native Chinese
English+
English
Native Chinese
10.7
Native Chinese+
Bilingual
90.3
81.4
92.1
88.2
English
Native Chinese
9.6
Translated Chinese
Translated Chinese
preference. These results strongly demonstrate the superior quality of VIsCPM- generated images compared to the two baseline models.
5.3 ABLATION STUDY
We conduct the ablation study of dataset languages to investigate the impact of varied dataset language combinations on multimodal models' performance in image-to-text and text-to-imaee tasks. For better efficiency, we only use LLaVA 150K for instruction tuning in the image-to-tess task. The detailed configurations of each experiment are reported in Appendix FF Based on the results presented in Table 5a and Table [b] We have the following observations: (i) Relying solely on a native Chinese dataset insufficieni for achieving good performance in both image-to-text and texl-to-image lasks. Models trained exclusively on native Chinese datasets yield worse scores, with the image-to-text obtaining an 85.5 average score and the text-to-ine sot-to-immae model attaining 15.1 FID. (ii) English data plays crucial role in improving the chat capability of the model in Chinese during the instruction tuning stage. When the image-to-text model is pretrained a large dataset of English but then fine-tuned using the monolingual Chinese instruction tuning dataset, its average performance experiences a decline from 88.0 86.3 compared to the model utilizing the bilingual instruction tuning dataset. (iii) The filtering process applied to the native dataset is essential for the Chinese performance. In the text- to-image task, after pretraining on English data and then fine-tuning with unfiltered native Chinese multimodal pairs, the FID worsens from 10.9 in the zero-shot scenario to 12.8. (iv) Incorporating the native Chinese dataset and translated Chinese dataset yields a marginal improvement the model performance. Specifically, adding native Chinese data into the pretraining stage of the image-to-text model or fine-tuning stage of the ext-to-image model only results in a 0.2 variation of metries, while further mixing translated datasets improves the FID from 10.7 to 9.6. The improvement from VISCPM -Chat to FisCPM-Chate more systematic exploration of the influence on the response language accuracy and content quality with different percentages of the Chinese instruction tuning datasets in Appendix [D4]
shown in Table also confirms the same results. We present a
6 CONCLUSION
In this work, we introduce MPM, an innovative training paradigm designed for effectively training large multimodal models in non-English languages. By utilizing multilingual L.M a pivotal intermediary between vision signals and target languages, MPM facilitates the efficient transfer of multimodal alignment knowledge across different languages. Based MPM, we develop series of open-sourced Chinese large multimodal models VISCPM, which show remarkable capability in Chinese image-to-text and text-to-image tasks. Our experimental results demonstrate that by solely relying on English multimodal data, VISCPM can achieve the SOTA performance among Chinese open-sourced multimodal models. We further scale the scope of language by constructing a versatile multimodal chatbot that supports six distinct languages. We believe that the effectiveness of MPM will contribute to the development of large multimodal models worldwide, thereby fostering the development of sophisticated multimodal models across various languages and cultures.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work is supported by the National Key R&D Program of China (No.20222D0110550))
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In Proceedings of ICCV, 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large lision-language model with versatile abilities. arXiv preprint arXiv:2308. 2023.
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in mudti-modal diffusion scale. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of ICML, volume 202, pp. 1692-1717. PMLR, 23-29 Jul 2023.
Marco Bellagente, Manuel Brack, Hannah Teufel, Felix Friedrich, Bjöm Deiseroth, Constantin Eichenberg, Andrew Dai, Robert Baldock, Souradeep Nanda, Koen Oostermeijer, al. Multifu- sion: Fusing pre-trained models for multi-lingual, multi-modal image generation. arXiv preprint arXiv:2300. 15296, 2023.
Federico Bianchi, Giuseppe Attanasio, Raphael Pisoni, Silvia Terragni, Gabriele Sarti, and Sri Lakshmi. Contrastive language-image pre-training for the italian language. arXiv preprint arXiv:2108.08688 2021.
Ondřej Bojar. Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, et al. Findings of the 2017 conference machine translation (wmt17). Association for Computational Linguistics, 2017.
Emanuele Bugliarello, Fangyu Liu, Jonas Pfciffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulić. IGLUE: A benchmark for transfer learning across modalities, tasks, and languages. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2370-2392. PMLR, 17-23 Jul 2022.
Fredrik Carlsson, Philipp Eisen, Faton Rekathati, and Magnus Sahlgren. Cross-lingual and multilin- gual clip. In Proceedings of LREC, pp. 6848-6854, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsual Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation 2023.
via masked generative transformers. arXiv preprint
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of CVPR, pp. June 2021.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. In Proceedings of ICLR, 2022.
Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Qinghong Yang, and Ledell Wu. АRtLIP: Altering the language encoder in CLIP for extended language capabilities. In Findings of ACL, pp. 8666-8682, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2223. findings-acl.552.
Schuhmann Christoph, Köpf Andreas, Veneu Richard, Coombes Theo, and Beaumont Romain. Laion COCO: 600M synthetic captions from Laion2B En, 2022. URL http://aaa....../a/.aaaaa
a co-cocoo
10

---

Published as a conference paper at ICLR 2024
James M Clark and Allan Paivio. Dual coding theory and education. Educational psychology review, 3:149-210, 1991.
Wenliang Dai, Junnan Li. Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general models with instruction tuning, 2023.
seneral-purpose purpose vision-language
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog. In Proceedings CVPR, 2017.
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. NeurIPS, 35:1 18900-1602,,, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 16 words: Transformers for image recognition at scale. Proceedings of ICLR, 2020.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of ACI, pp. 320-335, 2022.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of CVPR. pp. 12873-12883, 2021.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, al. Beyond english- centric multilingual machine translation. JMLR, 22(1):4899-4886, 2021.
Oran Gafni, Adam Polyak. Oron Ashual. Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a- Scene: Scene-based lext-to-image generation with human priors. In Proceedings of ECCV, pp. 89-106. Springer, 2022.
Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Minzhe Niu, Hang Xu, Xiaodan Liang, Wei Zhang, Xin Jiang, and Chunjing Xu. Wukong: 100 million large-scale chinese cross-modal pre-training dataset and foundation framework, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. NeurlPS, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33: 6840-6851, 2020.
Jinyi Hu, Xu Han, Xiaoyuan Yi, Yutong Chen, Wenhao Li, Zhiyuan Liu, and Maosong Sun. Ef- ficient cross-lingual transfer for chinese stable diffusion with images as pivots. arXiv preprint arXiv:2305.11540 2023.
Drew Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of CVPR, pp. 6700-6709, 2019.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. If you use this software, please cite it as below.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.026433 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123:32-73, 2017.
11

---

Published as a conference paper at ICLR 2024
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image generation. NeurIPS, 32, 2019.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of ICML, volume 162, pp. 12888-12999 PMLR, 17-23 Jul 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of ICML, volume 202, pp. 19730-19742. PMLR, 23-29 Jul 2023a.
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. M IT: A scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint arXiv:2306.04388 2023b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Proceedings of ECCV, pp. 740-755. Springer, 2014.
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. Visually grounded reasoning across languages and cultures. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10467-10485, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08448 2023a.
Shanyuan Liu, Dawei Leng, and Yuhui Yin. Bridge diffusion model: bridge non-english language- native rext-to-image diffusion model with english communities. arXiv preprint a0X0022309.00952. 2023b.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding DINO: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05430 054999 2023c.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. Proceedings of CVPR, 2019.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji. Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning, 2023.
Minheng Ni, Haoyang Huang, Lin Su, Edward Cui, Taroon Bharti, Lijuan Wang, Dongdong Zhang, and Nan Duan. M3P: Learning universal representations via multitask multilingual multimodal pre-training. In Proceedings of CVPR, pp. 3977-3986, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of ICML, volume 162, pp. 16784-16804, PMLL. PMLR, 17-23 Jul 2022.
OpenAl. GPT-4 technical report. arXiv preprini urXiv:2303.08774. 2023.
Allan Paivio. Bilingual Dual Coding Theory and Memory, pp. 41-62. Springer New York, New York, NY, 2014. ISBN 378-1-4614-9218-4 doi: P10. 1ww.100997----4411-2220
12

---

Published as a conference paper at ICLR 2024
Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. xGQA: Cross-lingual visual question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computationnal Linguistics: ACI. 2022, pp. 2497-2511, 1, Dublin, Ireland, May 2022. Association for Computational Linguistics.
Chen Qiu, Dan Oneatä, Emanuele Bugliarello, Stella Frank, and Desmond Elliott. Multilingual multimodal learning with machine translated text. In Yoav Goldberg, Zoriissa Kozareva, and Yue Zhang (eds.), Findings of the Association for Compulational Linguistics: EMNLP 2022, pp. 4178-4193, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 1148-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), Proceedings of ICML, volume 139, pp. 8821-8831, PMLR, 18-24 Jul 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06 06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of CVPR, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In MICCAI, pp. 234-241, Springer, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, (1.36479-36499 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400M: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Christoph
Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5B: An open large-scale dataset for training next generation image-text models. NeurIPS, 3525278-252944 2022.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-OKVQA: A benchmark for visual question answering using world knowledge. arXiv, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, pp. 2565, 2018
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proceedings of ICLR,, 2020.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. PandaGPT: One model to instruction-follow them all. arXiv preprint 23Xiv:2305.16355, 2023.
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang. Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. /4525, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
13

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 30, 2017.
Junjie Wang, Yuxiang Zhang, Lin Zhang, Ping Yang, Xinyu Gao, Ziwei Wu, Xiaoqun Dong, Junqing He, Jianheng Zhuo, Qi Yang, Yongfeng Huang, Xiayu Li, Yanghan Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, Kunhao Pan, Rui Wang, Hao Wang, Xiaojun Wu, Zhongshen Zeng, Chongpei Chen, Ruyi Gan, and Jiaxing Zhang. Fengshenbang 1.0: Being the foundation of chinese cognitive intelligence. CoRR, abs/2209. 02970, 2022a.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: Unifying architectures, tasks, and modalities through learning framework. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of ICML, volume 162. pp. 23318-23340, PMLR, 17-23 Jul 2022b.
simple
Wenhui Wang, Hangho Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for vision and vision-language tasks. In Proceedings of CVPR, pp. 19175-19186, 2023.
Chunyu Xie, Heng Cai, Jianfei Song, Jincheng Li, Fanjing Kong, Xiaoyu Wu, Henrique Morimitsu, Lin Yao. Dexin Wang. Xiangzheng Zhang. Dawei Leng, Xiangyang Ji, and Yafeng Deng. Zero and R2D2: A large-scale chinese cross-modee benchmark and a vision-language framework, 2022.
An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, and Chang Zhou, Chi- nese CLIP: Contrastive vision-language pretraining in chinese. arXiv preprint 23X0v22211.01335, 2022.
Yukang Yang. Dongnan Gui, Yuhui Yuan, Haisong Ding, Han Hu, and Kai Chen. Glyphcontrol: Glyph conditional control for visual text generation. 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mPLUG-OWI: Modularization empowers large language models with multimodality, 2023.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. arXiv preprint arXi::2205.01917 2022a.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. TMLR, 2022b. ISSN 2835-8856. Featured Certification.
Tianyu Yu, Jinyi Hu, Yuan Yao, Haoye Zhang, Yue Zhao, Chongyi Wang, Shan Wang, Yinxv Pan, Jiao Xue, Dahai Li, al. Reformulating dision-languse foundation models and datasets towards universal multimodal assistants. arXiv preprint arXiv:23 10.000000000 10.00230 2023a.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. arXiv preprint arXiv:2312.0088 00849, 2023b.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:211i.11432. 2021.
Yan Zeng, Wangchunshu Zhou, Ao Luo, Ziming Cheng, and Xinsong Zhang. Cross-view language modeling: Towards unified cross- loosullingual cross-modal pre-training. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 5731-5746, Toronto, Canada, July 2023. Association for Computational Linguistics.
14

---

Published as a conference paper at ICLR 2024
Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua. Transfer visual prompt generator across llms. arXiv preprint arXiv:2305.01278, 2023.
Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye. Yujia Qin, Yusheng Su, Haozhe Ji, Jian Guan, al. CPM: A large-scale generative chinese pre-trained language model. Al Open, 2:93-99, 2021.
Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, and Jingjing Liu. UC2: Universal cross-lingual cross-modal ussionaadd-lnggagee pre-training. Proceedings of CVPR, pp. 4155-4165, 2021.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. DM-GAN: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of CVPR, pp. 5802-5810, 2019.
15

---

Published as a conference paper at ICLR 2024
APPENDIX
A CONTRIBUTIONS
The authors' contributions can be outlined as follows:
In the preparation of the project, Jinyi Hu and Yuan Yao design the model architecture. Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun offer invaluable guidance in refining the model's architecture. Shan Wang, Chongyi Wang, and Jinyi Hu take charge of collecting and processing the extensive multimodal dataset required for pretraining. Additionally, Hanghao Wu, Yue Zhao, Haoye Zhang, and Yuan Yao collaborate on constructing the instruction tuning data. Jinyi Hu, Chongyi Wang, Tianyu Yu, Qianyu Chen, and Shan Wang jointly implement the training codebase.
In model training, Chongyi Wang, Tianyu Yu, and Yinxu Pan babysit the VisCPM-Cher training: Jinyi Hu, Shan Wang and Qianyu Chen take care of the VisCPM-Paine training.
In model evaluation, Jinyi Hu and Yuan Yao design the evaluation framework. Chongyi Wang and Tianyu Yu evaluate VISCPM-Char Jinyi Hu and Shan Wang execute the automatic evaluation of VisCPM-Paine and organize the human evaluation of VisCPM-Paint.
In paper writing, Jinyi Hu and Yuan Yao write the main paper, Yankai Lin, Zhiyuan Liu, and Maosong Sun provide suggestions to polish the writing.
For public usability, Jinyi Hu, Yinxu Pan, Chongyi Wang, Shan Wang, and Yuan Yao promote the open-source of VISCPM; Yinxu Pan develops the online demo and API of VISCPM; Chongyi Wang and Yinxu Pan implement the low-resource inference of VISCPM.
Throughout the project, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun provide invaluable technical guidance and advice.
B DATASET
B.1 PRETRAINING DATASET
COCO (Lin et al. 2014) The COCO dataset is a meticulously compiled image caption dataset that encompasses everyday scenes with common objects. It includes training set of 1 118.287 images, each complemented by five unique captions. For each pass through the dataset, we randomly select one caption, resulting 591,435 image-text pairs, five times the total number of images.
Visual Genome (Krishna et al. 2017) Visual Genome stands as meticulously labeled image dataset, enriched with detailed annotations for objects. Our employed training set features approximately 100K images, each accompanied by an average of 8 unique captions.
CC3M Sharma al., 2018) Also known Conceptual Captions, the CC3M dataset contains approximately 3.3M image-text pairs. CC3M is collected from the web and carefully processed to achieve the cleanliness and informativeness of captions. Due to some broken image links, the final successfully downloaded dataset contains approximately 2.8M pairs.
CC12M (Changpinyo al.) 2021) The CC12M dataset is an extension of the CC3M (Sharma et al., 2018), hosting nearly 12M image-text pairs. A more relaxed collection pipeline was used for its creation. After downloading, our dataset comprises approximately 6M pairs.
Laion2B (Schuhmann et al. 2022) Laion-2B is massive dataset populated with image data sourced from publicly accessible areas of the internet. Our successful download yielded a vast collection of around 1.3B images.
Laion-COCO (Christoph et al. 2022) Laion-COCO is a subset of the Laion-2B dataset which includes 600M image entries. These images have been captioned using the BLIP (Li al. 2022 to generate high-quality descriptions mimicking the MS COCO style.
Wukong (Gu al. 2022) Wukong is 100M image-text pair in Chinese, where the images filtered according to the image size, and the text is filtered according to its language, length, and frequency.
16

---

Published as a conference paper at ICLR 2024
Table 6: Evaluation of bilingual multimodal chat models on UniMM-Bench in English.
Model
OKVQA
AOKVQA
GQA
VQAv2
AVG
mPLUG-OWl
66.7
62.5
63.0
66.0
64.6
VisualGLM
57.6
62.8
58.1
63.9
60.6
Ziya- Visual
66.1
68.7
67.4
67.3
67.4
Twen-VL-Chhal
71.4
77.9
68.6
77.9
74.0
VISCPM-Chat
65.4
75.5
71.5
76.6
72.3
Table Summary of multimodal chat models' architecture, parameters, and training data. Here, the size of mPlug-Owll Oww's training data and the proportion of Chinese and English in the Ziya- Liya-lisual's training data are reported in their model cards.
Model
Visual Module
LLM
Training Data
mPlug-Owl
VIT-L/14 (0.3B)
BLOOMZ-7B
VisualGLM
Q-Former (1.6B)
ChatGLM-6B
English: 300M; Chinese 30M
Ziya-Visual
Q-Former (1.IB)
Zyy-LLaMAA-IB--I
20M
Owen-VL-CCre
ViT-bigG (1,9B)
Qwen-7B
English: I.IB; Chinese: 300M
VISCPM-CHI Chat
Muffin (0.7B)
CPM-Bec- 10B
English: 140M: Chinese: 1M
Zero (Xie et al., 2022) Sourced from search engine, the Zero dataset comprises 20M images and corresponding textual descriptions, selected from pool of 5B image-text pairs based user click-through rate.
B.2 INSTRUCTION TUNING DATASET
LLa -A---Asttuet-1100 Liu al. 2023a) LLaVA-Instruct-150K is a set of multimodal instruction- following data generated by GPT4 its creation, image captions and corresponding bounding boxes harnessed to encode images into textual sequences for the text-only GPT4 model. LLaVA- Instruct-1508 divides the dataset into three types: Conversation, Detailed Description, and Complex Reasoning. For each type, manually designed examples incorporated into prompts for in-context learning of GPT4.
MT Li et al. 2023b) The MT dataset collection of large-scale multimodal instruction tuning datasets curated by leveraging downstream datasets covering diverse vision-language tasks, including captioning, reasoning, and visual question-answeriig The datasets have been reformulated into unified image-text schema. Certain constructed instances, derived from key datasets, are translated into total of 100 languages. It important to note that we do not proceed with additional translations for non- Con-Chieese portions of the dataset. It's worth noting that we do not proceed with additional translations for non-Chinese portions the dataset.
UniMM-Chat (Yu et al. 2023a) UniMM-Chat is bespoke knowledge-intensiv multimodal conver- sational dataset. UniMM C-Chtt utilizes images from the MSCOCO (L.in al., 2014) and associated labeled datasets as the foundation for data generation. The related datasets incl include VQAv2 (Antol cct al., 2015). Visual Dialogue (Das et al. 2017). OKVQA (Marino et al. 2019) and AOKVQA (Schwenk et al., 2022). The initial step involves merging the annotation information across different datasets for the same image, including pairs of questions and answers, dialogues, rationale informa- tion, and image captions. The varied text annotations provide diversified view of the image content, enabling the comprehensive interpretation of the image by ChatGPT.
Table 8: Summary of text-to- -ottioiageeea models' architecture and training data.
Model
Visual Module
Text Encoder
Training Data
AltDiffusion
UNet
ARCCLIP
TaiyiDiffusion
UNet
Taiyi- CLIP-
English: O: Chinese: 20M
ViSCPM-Pain
UNet
CPM-Bec
English: 1.3B; Chinese:
17

---

Published as a conference paper at ICLR 2024
Table 9: Desription and examples of different categories Chinese Drawbench
Category
Description
Example
Relation
Ability to generate objects with specific interaction
"Ht-12 M
relationship between them
(A hat hanging on hanger.)
Compositional
Ability multiply kinds objects
"- is we 19 Ri ap 4.
(There yellow pears purple apples plate
Ability to accurately generate objects with given attributes, such as size, color, and action.
Attribute
blue velvet ovening pown)
Ability to generate unusual scene against common
1 # (1 Rai flies spaceship in space
Counterintuitiv
sense
Rare Word
Ability to understand rare expression
IA gladiolus on the windowsill)
Count
Ability to generate given numbers of objects
= this Ri
bread with butter.)
-R
Long Input
Ability understand long complex input
"E -
(On the calm lake, the boarmun rowed his pudile ncross the lake, and
appling, bird flew over
fish.)
I9F, 3.2H:UU
Chinese Culture
Ability to generute scene reluted to Chinese Culture
(As the bright moon shines over the sea,
From
share this with
Table 10: Details of image-text datasets used in VisCPM-Chat's pretraining, where "bil" means the mixed version of English and translated Chinese.
Datasets
Size
Weight
Epoch
VIsCPM-Char
COCO Visual Genome
626K
12.50%
27.60
CC3M CC12M
8.4M
25.00%
16.40
Laion Laion-COCO
390M
62.50%
0.35
VISCPM-Chate
COCO + VG
626K
12.50%
15.33
CC12M
5.6M
25.00%
3.43
Zero + Wukong
20M
12.50%
0.48
CC3M-bil
5.6M
12.50%
1.72
Laion-COCO-bil
780M
37.50%
0.04
B.3 CASE STUDY
Fig. 7 and Fig. g vividly depict VisCPM-Chat's capabilities, embodying wide-ranging global knowledge assimilation. As illustrated in the two case studies Fig. VisCPM-Chat can identify the Mona Lisa painting adapted in surreal style and recognize a stained map of New York City, further interpreting its real-world meaning. Demonstrating well-balanced capacity for Chinese- English multi-modal conversation and robust text recognition skills, exemplified in Fig. X VisCPM-Chat can engage fluent multimodal conversations on varying topics in English and effectively identify the words "Starbucks", "Avengers: Endgame", and its release date, "April 26th", from visual inputs.
C DETAILS OF BASELINES
The details existing Chinese-English bilingual multimodal chat models are summarized in Table and introduced as follows:
mPLUG-Owl (Ye et al. 2023): mPLUG-Ow/ consists of a vision encoder, vision abstractor module, and BLOOMZ Muennighoff et al. 2023) as language model backbone, is trained on LAION-400. COYO, CC3M, and MSCOCO.
18

---

Published as a conference paper at ICLR 2024
Table 11: Performance of MPM on IGLUE benchmark.
We compare MPM with mUNITER,
XUNITER (Liu et al. 2021
UC2
(Zhou et al. 2021 and M (Ni et al. 2021
XNLI
XGQA KOR
Model
ARB
SPA
FRA
RUS
BEN
DEU
IND
POR
RUS
CMN
MUNITER
46.7
57.0
59.4
51.7
3.1
24.0
9.4
4.2
13.7
8.5
7.0
XUITTER
52.0
58.9
63.3
59.7
10.8
34.8
33.7
12.1
22.1
18.8
19.6
UC
56.2
57.5
69.5
64.9
20.0
42.9
28.7
21.4
30.4
31.0
31.2
M
55.2
58.9
56.4
62.5
18.6
33.4
32.5
25.1
31.4
27.5
28.7
MPM
55.8
75.9
78.1
75.0
6.6
51.0
37.1
36.9
49.8
47.9
46.1
2022): VisualGI.M employs Q-Former as image encoder and ChatGL-M-BB (Du al., 2022) as language model backbone. VisualGLM-6B's pretraining incorporates 30M high-quality Chinese image-text pairs and 300M filtered English image-text pairs. In the fine-tuning phase, VisualGLM trained long VQA data.
VisualGI.M Du et al.
Ziya-Visual Visual (Wang et al. 2022a): Ziya- Visual leverage Q-Former (Li et al. 2023a) as image
encoder and Ziya-LLaMA- 13B-v1 as language model backbone. They utilize 20M Chinese image-
text pair, which is built by cleaning high-quality data from open source data, translating English
datasets, and extracting coarse- graiined information from captions using BLIP (Li et al. 2022 and
Grounded SAM (Kirillov et al. 2023 Liu et al.) 2023c).
Qwen-VL--Catt (Bai et al. 2023 Qwen-VL-CCat utilizes a large ViT (Dosoviskkiy et al. 2020)
with 1.9B parameters initialize from Openclip's 's bigG (Ilharco et al. 2021 as image encoder and Qwen-7B as language model backbone. The image encoder and language model are connected by
a one-layer cross-attention module augmented with positional encodings. Qwen-VL-Chal p
The details of existing Chinese-English bilingual text-to-image models summarized in Table and introduced as follows:
AltDiffusion (Chen et al. 2023 AltDiffusion is a Chinese Chinese-English bilingual text-to- text-to-image model based on Stable Diffusion and bilingual vision-language encoder AltClip (Chen al. 2023). The training data are collected form Laion (Schuhmann et al. (021).
TaiyiDiffusion (Wang et al. 2022a): TaiyiDiffusion is a Chinese text-to text-to-image model that adapts Chinese text encoder into Stable Diffusion. The visual part is frozen during training. The training datasets include 20M filtered Chinese image-text pairs.
D DETAILS OF VISCPM-CHAT
D.1 MULTIMODAL PRETRAINING
In the visual feature alignment pretraining, train the visual encoder mix of image-text pair datasets, including CC3M (Sharma et al. 2018)), CC12M (Changpinyo et al.] [202I) COCO (Lin et al. 2014), Visual Genome (Krishna al. (017), and Laion-COCO (Christoph et al. 2022). We train VisCPM-Chat and VisCPM-Chat+ for 180K and 480K steps, respectively, with a batch size of 768 and learning rate of le-5. The distribution of each dataset used in pretraining for VISCPM-CO and VisCPM-Chat+ are shown in Table I0 Only the parameters of the visual module are optimized this stage.
D.2 INSTRUCTION TUNING
During the instruction tuning phase, the optimization process involves the visual module and the LLM. We train VIsCPM-Chat on the mixed dataset from LLaVA-Instruct-1500 UniMM-Chat, and S where LLLVA-Insturn -aaecctt-10000 150K and UniMM- -hat are Chinese- -nnllse bilingual versions. The instruction tuning lasts for 80K steps with batch size of 64. The training configuration of VIsCPM-Chin is identical to VIsCPM-Chat uses beam search decoding with beam size of 3.
this stage. In the inference phase, VisCPM-Chat
19

---

Published as a conference paper at ICLR 2024
Compartson of Respense Language Acouracy and Contert Quality
202.0
67.5
4
Teacome Contase Quattt
92.
3
00.0
675
05.0
025
75 Ama
Figure 5: The trend of response language accuracy and content quality when using different percent- ages of translated Chinese data during instruction tuning.
D.3 EVALUATION BENCHIARK
LLaVA Test Set (Liu et al. 2023a) LLaVA Test Set consists of 90 instances, each containing an image, a question, and an answer, which comprehensively evaluates the model's performance in conversation, detailed description, and complex reasoning. Following LLaVA (Liu et al. 2023a), we use 4PT-4 to rate the model -enerregee answers and reference answers in a range of 1-10. We norm the average score of each instance to 1-100.
UniMM-Bench (Yu et al. 2023a) UniMM-Bench consists of 400 test instances, uniformly sampled from 4 commonly used VQA benchmarks, including OKVQA (Marino et al.) 2019), AOKVQA (Schwenk et al.] 2022). GQA (Hudson & Manning 2019), and VQAv2 (Antol et al. 2015), whose annotations has undergone meticulous check. In the traditional evaluation of visual question answer- ing, the metrics are to compute whether the model-generated answers are exactly matched to the reference answers. However, considering that L.MS typically generate responses with one complete sentence to answer the question. UniMM-Bence is built to comprchensively evaluate multimodal chat models' capabilities in the context of visual question-answering including reasoning, commonsense, grounding, and world knowledge (Schwenk et al. 2022). Given the cost of using GPT-4 for evaluation, 400 test instances are sampled from reliable VQA benchmarks.
D.4 THE TREND OF USING DIFFERENT NUMBER OF CHINESE TRANSLATED DATA
To investigate the influence of varying amounts of Chinese data the model's ability to respond in the target language during the instruction tuning, surpassing the quasi-zero-sho transfer threshold, systematically varied the percentages of translated Chinese data. Our analysis, depicted in Fig 5 elucidates the trends in both response language accuracy and content quality as a function of the Chinese data percentage utilized during instruction tuning. Remarkably, introducing mere 0. % of translated Chinese data suffices to achieve over 80% accuracy in response language accuracy, signifying the model's high sensitivity even minimal language-specits percentage to 1% results in a scenario where all responses are rendered in Chinese. Regarding content quality, our findings suggest that a range between 25% to 50% of translated Chinese data represents an optimal balance, yielding high-quality content responses. We leave the exploration of determining a more precise optimal proportion of translated Chinese data for maximizing both response content quality in future work.
data inputs. Elevating this
D.5 ADDITIONAL RESULTS ON IGLUE
We evaluate the effectiveness of MPM multilingual image understanding on two benchmarks of IGLUE: x VNLI (Bugliarello et al. 2022) and xGQA (Pfeiffer al. 2022). We choose model
We use GPT-4 0314 version evaluate the answers.
20

---

Published as a conference paper at ICLR 2024
cfg-scales: [1.5 2.0 3.0 4.0 5.0 6.0 7.0 8.0]
TalyiDiaasion
AllDilfusion
VisCMM-Pait
VisCPM-Paint-
20
FID Score
18
14.
20
CLIP Score (VIT-L/14)
Figure 6: The curves of FID 4 CLIP score with different classifier guidance scales.
pre-trained exclusively on English image-text pairs with LLaMA as the backbone language model, fine-tune it with IGLUE's English training set, and conduct zero-shot evaluation following the official setting of IGLUE.
As Table TT shows, the performance of MPM demonstrates notable improvement over baseline models. Specifically, when using Chinese- Chinese-English English bilingual LLM CPM-Bee as the backbone language
model, MPM attains 56.05 score on xGQA for CMN (Chinese Mandarin). Based on the above results, we can observe that for languages where multilingual LLMs demonstrate proficiency, such as Spanish, French, Russian, and German, the multilingual capability can be effectively generalized to achieve strong multimodal capabilities in the target language.
E DETAILS OF VISCPM-PAINT
TRAINING AND SAMPLING
VISCPM- consists of a UNet, initialized from Stable Diffusion, and bilingual LLM CPM-Bee (Zhang et al., 2021). We pretrain VisCPM-Pain
English image-text pairs dataset Laion-2B Schuhmann et al.) 2022) for 300k steps with batch size 4096. The 300K step consists of 200K-step training 256 256 resolutions and 100K training 512x512 512 resolutions. We only optimize the cross-attention layer of UNet and the linear transblock between L.M and UNet. Linear transblock consists of linear layer and layer-norm layer, where the linear layer converts the dimension of LLM's hidden states into UNet's dimension. After pretraining on Laion-2B, we continue to train VISCPM Paintt+ on 20M filtered native Chinese image-text pairs and 136M translated image-text pairs in
(Christoph et al. (022). We sample the generated image with DDPM scheduler (Ho al., (2020) with 50 steps. For each prompt, we run three models generate four images and select the one with the highest CLIP score evaluated by Chinese- YYang et al. 2022).
E.2 TRADE OFF OF FIDELITY AND ALIGNMENT
To visualize the trade-off between fidelity and alignment, Fig. plots the curves of the FID against the CLIP score under different classifier guidance scales. The results indicate VISCPM-Paint and VisCPM-Painte deliver good overall performance regarding the balance between image quality and semantic alignment. While the CLIP score of AltDiffusion slightly outperforms that of VISCPM- Paint and VISCPM-Paint+, AltDiffusion' 's FID falls significantly short. The sub-optimal quality of generated images will affect the practical use, shown in the human evaluation below.
21

---

Published as a conference paper at ICLR 2024
E.3 HUMAN EVALUATION
To better understand the model performance on text-to-image generation in Chinese, we create comprehensive set of prompts in Chinese, named Chinese Drawbench. Chinese Drawbench contains 174 prompts and consists of 8 categories, including Relation, Compositional, Attribute, Counterintuitive, Rare Word, Count, Long Input, and Chinese Culture. The descriptions and examples of each category are shown in Table [ During the construction of Chinese Drawbench, we asked 7 annotators create 25 prompts per person conditioned on the given categories. To diversify the prompts, we also restrain the field of 20 prompts ranging from 9 common classes, such as animal, plants, food, space, etc, and leave five prompts with no restrain. After this, we double-check the prompts and polish them minor.
We invite 5 independent human evaluators to assess the performance of VISCPM-Paint, TaiyiDiffu- sion [Wang et al. 2022a), and AltDiffusion Chinese Drawbench. For each instance, three random shuffled images and the input prompt are shown to the human evaluators. We ask them to select the best one among three images from 3 aspects: Overall, Alignment, and Fidelity. Alignment measures the consistency between the generated image and the input prompt. Fidelity measures the clarity, aesthetics appeal, and object realism of the image. Overall measures the comprehensive quality of generated images. Figure 10 provides detailed results of each category of Chinese DrawBench.
F DETAIL OF ABLATION STUDY
We introduce the detailed experimental configuration in the ablation study.
For image-to-text tasks shown in Table 5a the native Chinese dataset consists of 20M image-text pairs filtered from Wukong (Gu et al. 2022) and Zero (Xie et al. 2022). For the "Pretrain" column Table 5a "Native Chinese" stands for the 20M native Chinese image-text pairs filtered from Wukong GG et al. 2022) and Zero (Xie et al. 2022). "English" corresponds to the dataset used in VIsCPM-Chat i... COCO, Visual Genome, CC3M, CC12M, and Laion-COCO. "Translated Chinese" consists of 390M Chinese image-text pairs whose captions are translated from English in Laion-COCO. In instruction tuning, i.e., supervised fine-tuning (SFT), we only utilize LLaVA and its translated Chinese version to reduce the computation time in the ablation study, Specifically, "Chinese" in the "SFT" column of Table 5a means translated Chinese LLaVV instruction tuning dataset. "Bilingual" means the mixed version of English and translated Chinese LLaVa.
For text-to-image tasks shown in Table 5b. "English" stands for Laion-2B. The other setting is the same as the image-to-text tasks introduced above.
G DETAIL OF CLIP SCORE
We use Chinese-CLIP Yang et al. 2022) to compute the CLIP score in Chinese image-text pairs. Chinese CLIP models have been used in several Chinese text-to-image -i--geg models for data filtering and text encoding (Wang et al. 2022a Liu et al. 2023b Yang al.] 2023). We referred to Taiyidiffusion's ''s criteria for data volume and filtering standards (including CLIP score) in training Chinese text-to- image models. We manually check the caption in the Wukong dataset. We observe that the overall quality text content in Wukong is unsatisfactory. Here are some findings about low CLIP scores:
Broken Images: Some image URLs are broken or redirected to the same default image for downloading.
Unrelated captions: Some content of captions are totally unrelated to image content.
Incomplete sentences: Some captions are truncated or missing punctuation.
22

---

Published as a conference paper at ICLR 2024
DoDi
A 18 ItUm?
(Please appreciate this painting)
30 7- EA A A A # f
N (This paining depicts a portrat cat with long whiskers and large eyes. The cat staring straight at the viewer, giving it sense of curtosity or playfulness. green wall can seen in the background, adding splash of color the picture. The composition artietic playful, capturing
Wi 1300 7 REE. MWW EAR
expressive cheracteristice of cats.)
A A AME 2o/t???
(The image of this cat is related the characters in which world famous painting? Why?)
13 th (iillisds 4adt A.SASAMEEN Bi E - Kiitt TEEERII M
T iil (0 MER. APPEINE
(The image of the cat similar to the image of the woman in Leonardo da Vincis famous painting "Mona Lisa", as it depicted its
woman with be doing the startttttnggg--eeeeeeeeeeeeee wide-ayed and intently at viewer. This simiertty could due both having distinctive facial features displaying an emotonn and stmosphere similrr those captured in da arttirk
The painting
expressive wachat, the to
(1 N27F?
Do Do
(Which city map?)
+4
(This is map of New Yark City.)
(What different color the picture moan?)
the various boroughs New York each with These color-coding systems help people quickly easily idenafy distinguish between different neighborhoads neighbomoods within a city. By using this coding method, residents and tourista can easily find where they want to go and betler understand the layout and organization the city.)
(The different colored blocks in the map
color.
Figure 7: Multimodal conversation cases of VIsCPM- Phatttttiii in Chinese. VIsCPM-Chal can recognize artwork like the Mona Lisa, albeit adapted in surrealistic style, and a detailed, stained map of New York City.
23

---

Published as a conference paper at ICLR 2024
AVENGERS
Do Do
Which movie is this poster from and which characters appear the poster?
This poster is for the movie "Avengers: Endgame". which is set to release April
26th. The characters featured in the poster are Thor, Hulk, Iron Man, Captain America,
3v
Black Widow, and Hawkeye.
Who about the villain?
The villain in the movie is Thanos, a powerful and dangerous character from the Avengers Movic.
STARBUCK COFFEE
Do Do DO
What place picture?
The place in the picture is a Starbucks coffee shop, with large green and white sign on side that reads "Starbucks".
What can buy here?
In image, there is Starbucks coffee shop sign hanging on side a building. This suggests location. The presence the Starbucks logo also indicates that they offer various
buy beverages, such coffee
products and services related coffee, including coffee-related merchandise like mugs, T-shirts, and other accessories.
Figure& 8: Multimodal conversation cases of VisCPM-Char in English. VISCPM-CHE -Caat can conduct fluent multimodal conversations on different topics in English and recognize the words "Starbucks", "Avengers: Endgame" and its release date "April 26th" in the image.
24

---

Published as a conference paper at ICLR 2024
Figure 9: Generated images of VIsCPM-Paint.
25

---

Published as a conference paper at ICLR 2024
Tayyl
VIGOPM
100
60
40
Relation
Compostional
Atbroute
Counterintuiive
Rare Word
Count
Long Inpur
Chinase Culture
(a) Overall
100
80
60
Reiatinn
Compoctiinal
Altriutt
Countenntultiv
Rare Word
Count
Chinese Culture
Long
(b) Alignment
80
60
40
20
Reiadin
Compostionall
Attrbuta
Counterintultvv
Rare Word
Count
Long Input
Chinese Culture
(c) Fidelity
Figure 10: Human preference rate on Chinese Drawbench in different kinds of aspects.
26


Reviews:
[{"review_id": "gCGXmcOO1X", "reviewer": "Reviewer_scbv", "summary": "The paper presents a method for training a multilingual and multimodal model without needing to rely on large amounts of data in all of the target languages. Instead, the method relies on multilingual and multimodal pivoting, using a high-resource language to provide the target distributions for the lower-resource language. The method is realized in an image-text and text-to-image model named VisCPM-Chat/Paint, respectively. The model is trained by aligning the low-resource Chinese data against the high-resource English data using the CPM-Bee bilingual Zho/Eng language model; the method is also shown to generalize to LLaMA.", "questions": "1. Why do you think Chinese can be considered a low-resource language? See W2 for more details on my concerns.\n2. In Section 5.1, why is it methodologically sound to use GPT-4 to evaluate model output?\n3. In Figure 2, why is a CLIPscore threshold of 0.18 used to define high-quality? How was this determined?\n4. In Section 4.2, what was the dataset of 100M image-text pairs used to align with the frozen LLM for 180K steps?\n5. Did you also use M2M-100 to translate the 136M examples in the COCO-LAION dataset?\n6. Which version of M2M-100 did you use for translation?\n7. In Section 5.2.2, you describe an FID of 9.5 or 9.9 \"comparable\" to Stable Diffusion FID is 8.6. Is this a reasonable claim?\n8. How does your model perform on a larger set of languages in the multilingual multimodal benchmark IGLUE by Bugliarello+ ICML 2022?\n\nBugliarello, E., Liu, F., Pfeiffer, J., Reddy, S., Elliott, D., Ponti, E. M., & Vulić, I. (2022, June). IGLUE: A benchmark for transfer learning across modalities, tasks, and languages. In International Conference on Machine Learning (pp. 2370-2392). PMLR.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "S1: Conceptually simple approach to learning a multilingual and multimodal model. Makes good use of the vast amount of English language resources.\n\nS2: Extensive experiments with relevant benchmark datasets.\n\nS3: The approach is shown to generalize to more than one language model (LLaMA and CPM-Bee)", "weaknesses": "W1: There are some inconsistencies in the argumentation used throughout the paper. For example, in Section 4.1, the authors argue that translating a large-scale pretraining dataset will consume substantial computing resources, which is a fair argument. However, in Section 4.2, the authors describe that they train a version of their model called VisCPM-Chat+ using 1\n36 million translated examples from LAION-COCO. The authors could improve their argumentation if they clarify whether it\nis or is not challenging to create such translated examples. Bear in mind that other researchers have already translated\npretraining datasets, e.g. Thapliyal+ EMNLP 2022.\n\nW2: The motivation for this paper is to create multimodal models for low-resourced languages via the proposed pivoting method but the main languages used in the experiments are Chinese and English, neither of which could ever be described as low-resource languages. The remaining languages used for the experiments in Section 5.1.3 are also hardly low-resource: German, French, Spanish, Italian, and Portuguese. Using these languages and claiming that the method applies to low-resource settings affects the credibility of the conclusions that one can draw for actual low-resource languages. See, for example, Joshi+ ACL 2020 for a discussion on low-resource languages.\n\nJoshi, Pratik, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. \"The State and Fate of Linguistic Diversity and Inclusion in the NLP World.\" In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6282-6293. 2020.\n\nThapliyal, A. V., Tuset, J. P., Chen, X., & Soricut, R. (2022, December). Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 715-729).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EKpij2HEFH", "reviewer": "Reviewer_mPhK", "summary": "This paper studies a simple idea to use multilingual language models as the pivot to connect different languages for multimodal applications. The idea is based on the assumption that visual semantics are language agnostic, thus aligning a visual modal to a pre-trained (and sometimes frozen) multlilingual language model could support multilingual applications even without aligned multimodal data.", "questions": "See the above sections.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper is an engineering style paper, with great qualitative examples. The idea itself is simple and intuitive (but however the execution and writing is a bit complicated, see the next section). \n2. This paper promised to open-source the code and model weights. This is going to have a positive impact on the community thanks to the open sourced model weights.", "weaknesses": "1. The contribution of the MPM training paradigm is unclear. Taking the image-to-text generation as an example, in the multimodal pre-training step the proposal is to freeze multilingual LLM and only tune the visual module; in the instruction tuning step the proposal is to finetune everything on the datasets from pivot language and the target language. The multimodal pre-training step is quite straightforward as it aligns a trainable vision model to a pre-trained and frozen LLM; the instruction tuning step is a standard finetuning setup. If the contribution of this paper is to use a pre-trained multilingual LLM to support multilingual downstream tasks, then much more in-depth analysis should be done to justify the point. Examples include understanding the MPM design choices, how freezing LLM affects results, how translated pairs affect instruction tuning results, how many translated pairs are enough and what the trend looks like. \n2. Most of the tables present absolute comparisons, showing how great the proposed method is. However that might be less interesting from a reader's perspective without proper baselines and ablation tests as discussed in the above section (due to different dataset mixtures and tuning tricks used). It's high recommended to take a few tables, and perform multiple in-depth analysis and comparisons with controlled study, to make sure there are enough take aways and confidence in the current MPM design choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9obN1Bsfyh", "reviewer": "Reviewer_T6ab", "summary": "Text-to-image and image-to-text generation efforts have primarily been focussed on English only due to lack of large-scale, high quality data. This paper proposes MPM, an effective training paradigm for training large multimodal models in low-resource languages. They show that this technique enables competitive zero-shot performance on Chinese language as compared to models trained on the language data. This leverages the Bilingual Dual-coding Theory which states that visual semantics are largely language agnostic. MPM divides the non-English multimodal learning into two consecutive stages: multilingual alignment and multimodal alignment. The former focuses on building a multilingual model by using a pretrained multilingual large language model, while the latter culminates in a multimodal model spanning multiple languages on top of the multilingual model.", "questions": "None.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper shows a simple yet effective technique of using a mulitlingual LLM as a pivot to transfer multilingual image-text alignment across different languages. \n- The model is trained on English data only but still manages to perform better on Chinese tasks than models trained on Chinese data. This is highly encouraging in making the state-of-the-art techniques on image-to-text and text-to-image generation available in other languages.", "weaknesses": "None.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LYYpPYojvw", "reviewer": "Reviewer_BZ1h", "summary": "This paper introduces MpM, a framework to train multilingual multimodal models for image-to-text and text-to-image generation (mostly English+Chinese). For image-to-text, the model (VisCPM-Chat) is typically pretrained on English image captions by freezing the LM, and then fine-tuned on instruction data translated into the target language(s). The model shows competitive performance when evaluated on machine-translated versions of LLaVA Test Set and UniMM-Bench as evaluated by an off-the-shelf model (GPT-4). For text-to-image, both the text model and the image decoder are frozen, and cross-attention layers are trained using English data. The resulting model (VisCPM-Paint) achieves a lower FID score than previous models on 30K samples from COCO Val obtained through machine translation.", "questions": "1. The low CLIP scores in Wukong might be related to misunderstanding of Chinese captions from CLIP. You say that you perform manual inspection, can you elaborate more on your findings and their relation to CLIP scores?\n2.  It would be useful for the reader to have more information about the baselines, so that one can easily understand what changes among models when comparing their results.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed framework, MpM, corroborates and extends the line of work on multilingual transfer through frozen language models, showing it to be more effective than previous models on the chosen tasks according to the evaluation measures adopted by the authors.\n2. The plot in Figure 2 is interesting, showing how most of the Wukong text data might be poorly aligned with the corresponding images.\n3. Human evaluation on Chinese Drawbench shows how the proposed model is often preferred to the baselines.\n4. It is interesting to see how much English data helps during fine-tuning in the ablation study, perhaps because it bridges the gap between the style and language axes.", "weaknesses": "1. The paper claims to “generalize to other languages in a zero-shot manner.” This is not true, and very misleading. In fact, the chat model had to be fine-tuned on target language data, otherwise it would generate text in English. Calling the behavior “quasi-zero-shot” is also (i) different from the literature, and especially (ii) misleading: the model cannot generate text in the target language without supervision. Such claims, while valid for the text-to-image model, should be removed.\n2. The paper claims to propose “an effective training paradigm for training large multimodal models in low-resource languages.” However, none of the languages tested were low-resource. There is a significant difference in the representations of the LM model between high- and low-resource languages, as well as in the quality of translations that an MT system can provide for those languages. None of these things are true for Chinese. Any claims about performance and applicability to low-resource languages need to be removed as they are never verified.\n3. The paper also claims to be multilingual. Yet, most of the experiments are done in a bilingual setup (English+Chinese) or in a few, high-resource European languages. These languages share the same script, similar topology. Claims about multilinguality should be supported by a better selection of languages that increase diversity in scripts, topology, and resource availability.\n4. My other major concerns are related to the experimental setup.\n- (4a) The model is evaluated on unusual benchmarks for multilingual multimodal modeling. These datasets are obtained through machine translation of the corresponding English dataset. The authors claim to fix minor errors (only for Chinese, and not even for all the other 6 European languages), yet do not provide any indication of the quality of the resulting test data. \n- (4b) The evaluation scores are based on GPT-4, which might not understand languages beyond English well. This is an hypothesis that might or might not hold true, but the fact that such concern exists, I believe it is enough to undermine the evaluation setup used by the authors.\n- (4c) There exists multilingual multimodal benchmarks. For instance, IGLUE [1] tests for 4 multimodal tasks in 20 languages, with data *manually* collected by native speakers. MaRVL [2] and XM3600 [3] even include images sourced from the countries where a language is spoken. These benchmarks are adopted in the literature and completely missing (even as part of the related work) in this paper.\n5. There is a lack of discussion of relevant related work. Models like {m,x}UNITER [2], CCLM [4] and TD-MML [5] are missing. The latter in particular shows how machine translation data can be very helpful during pretraining if filtering out poor translations, which provides a contrasting point to the one made by the authors in this paper. A discussion of these findings and how they differ is important to guide the community.\n\n---\n\n[1] Bugliarello et al. IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages. ICML’22.\n\n[2] Liu et al. Visually Grounded Reasoning across Languages and Cultures. EMNLP’21.\n\n[3] Thapliyal et al. A Massively Multilingual Multimodal Evaluation Dataset. EMNLP’22.\n\n[4] Zeng et al. Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training. ACL’23.\n\n[5] Qiu et al. Multilingual Multimodal Learning with Machine Translated Text. EMNLP’22.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)