MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
Authors: Sen Xing
Muyan Zhong
Zeqiang Lai
Liangchen Li
Jiawen Liu
Yaohui Wang
Jifeng Dai
Wenhai Wang
Venue: ICML 2025 poster (2025)
Keywords: Image Generation
Primary area: applications->computer_vision

Abstract: In this work, we explore a cost-effective framework for multilingual image generation. We find that, unlike models tuned on high-quality images with multilingual annotations, leveraging text encoders pre-trained on widely available, noisy Internet image-text pairs significantly enhances data efficiency in text-to-image (T2I) generation across multiple languages. Based on this insight, we introduce MuLan,
Multi-Language adapter, a lightweight language adapter with fewer than 20M parameters, trained alongside a frozen text encoder and image diffusion model. Compared to previous multilingual T2I models, this framework offers: (1) Cost efficiency. Using readily accessible English data and off-the-shelf multilingual text encoders minimizes the training cost; (2) High performance. Achieving comparable
generation capabilities in over 110 languages with CLIP similarity scores nearly matching those in English (39.57 for English vs. 39.61 for other languages); and (3) Broad applicability. Seamlessly integrating with compatible community tools like LoRA, LCM, ControlNet, and IP-Adapter, expanding its potential use cases.

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
Sen Xing 12*
Muyan Zhong' 1*
Zeqiang Lai 3* Liangchen Li Jiawen Liu' Yaohui Wang? 2 Jifeng Dai 12 Wenhai Wang?3
Abstract
popular image-genen sogge-genrration models, Stable Diffusion (Rom- bach et al., 2022), and its successors (Podell et al., 2023; Esser et al., 2024) only supports English and a few Latin- based languages. This language barrier restricts the model's performance in multilingual contexts and hinders its ap- plicability worldwide across diverse cultural and linguistic backgrounds.
In this work, we explore cost-effective frame- work for multilingual image generation. We find that, unlike models tuned on high-quality images with multilingual annotations, leveraging text en- coders pre-trained on widely available, noisy In- ternet image-tex nexte-textppe pairs significantly enhances data efficiency in text-to-image (T2I) generation across multiple languages. Based on this insight, we introduce MuLan, Multi- Language adapter, a lightweight language adapter with fewer than 20M parameters, trained alongside a frozen text encoder and image diffusion model. Compared to previous multilingual T2I models. this frame- work offers: (1) Cost efficiency. Using readily accessible English data and off-the-shelf multi- lingual text encoders minimizes the training cost; (2) High performance. Achieving comparable generation capabilities in over 1 10 languages with CLIP similarity scores nearly matching those in English (39.57 for English vs. 39.61 for other lan- guages); and (3) Broad applicability. Seamlessly integrating with compatible community tools like LoRA, LCM, ControlNet, and IP-Adapter, ex- panding its potential use cases.
The community primarily adopts two approaches to achieve multilingual text-to-image (T2I) generation.
(1) The first is translation- based methods where input content is tem- porarily translated into English before generating the im- age. However, this approach often leads inference delays, translation errors, and notable issues when handling slang or culturally nuanced content. (2) The other approach is native multilingual T21 models (Zhang et al., 2022; Rad- ford et al., 2021; Arkhipkin et al., 2023; Shing & Akiba; Li et al., 2024b; Team, 2024), which is trained directly on high-quality images captioned in the target language. While this native approach improves image generation quality for non-English languages, it relies on extensive, carefully cu- rated image- gengerainn data in the target language, making it data- and resource-intensive As a result, exploring more efficient and generalizable approach to achieve strong multilingual generation capabilities remains challenging.
On the other hand, thanks to the development of computa- tional power and dataset scale, many existing language mod- els have achieved strong multilingual capabilities through training on large-scale internet data. For example, mod- els trained on text data (e.g., BERT (Devlin et al., 2019), the GPT series (Brown et al., 2020; OpenAI, 2023), and LLaMA (Touvron et al., 2023; Dubey et al., 2024)) and those trained on image-text pairs (e.g., CLIP (Radford et al., 2021), ALIGN (Jia al., 2021). and InternVL. (Chen et al., 2023b)) demonstrate outstanding performance in multilin- gual understanding. Since the multilingual capability of T21 generation models is closely tied to their text encoders, it be- comes essential to explore how these powerful multilingual text encoders can be leveraged to enable existing generative models to achieve multilingual capabilities more efficienly and effectively.
1. Introduction
Recent diffusion models (Esser al., 2024; Li et al., 2024a;b; Team, 2024; Wu al., 2024; Zhang et al., 2022) for content generation have attained stunning advancements in terms of both aesthetic quality and text-content align- ment. However, these models still face substantial limita- tions in multilingual support. For instance, one of the most
'Equal contribution 'Tsinghua University *OpenGVLab, Shanghai AI Laboratory "The Chinese University of Hong Kong "Johns Hopkins University. Correspondence to: Wenhai Wang <wangwenhai @ pjlab.org.cn>>
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this work, we deeply explore the application of multi-

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
Prompt: tiger lab cost with 1980s Miami vibe, turning well oiled science content machine, digital ort (English)
Prompt: equarell des sommnunlergangs (German)
Prompt: dinskka, stralomoeská mode- Prempt: الر كاطس web witd (Aribii)
Prompt: edifici de gran algoda (Catzlan)
má kuchyna velkýmii oknami (Solvak) Translation: dog
Tranlation: watercolor the sunsel
Translatio:: Dunish, Maditernmean modemn kitchen with large windows
Translation: highorise building
Prompt:
Prompt: Primitivii lesy (Czech)
Prompt: 1.37-/J J.F.FF 2/2 7
Prompt: Orada çax yasil bitkilor olam gossal bir vanúr (Acerbujani)
Prompt: d MA (Chinese) Translation: Cyherpunk robots
I.. 1 ((Krraa))
Transtation: primitive forest
2 (Japanese)
Translation: woman
Transtation: LEGO Amold Schwarrzneeger
Translation: there boautiful house with many green plans
wearing a hat
Prompt: a single f bloom (English emaji)
Prompt: the clear lake reflects the (Enggish anagi)
Prompi: MR3,, T
Prompt: a quiet lake at down, * S M (Finlish Chinese) Translation: quiet take at dawn, ink painting style
Prompt: a vibrant + 555 15 puinting on (English Japanese) Translation: vibraii tribal painting on canvas
(Chinese amoji) Transtatin:: falling down, covering the peaceful path
Bujvai L.
Figure 1. Images generated by MuLan with different backbones, such as Dreamshaper 8, Dreamshaper XL Lightning, and Pixart- Alpaa, using variety of languages mixed language prompts.
lingual semantic alignment image generation from the perspective of language and image-text alignment. We also reveal that text encoders trained large-scale multilingual image-text datasets with noisy data demonstrate remarkable data efficiency in multilingual image generation. Based on this insight, we introduce Mul.an, a lightweight Multi- Language adapter that connects text encoders to diffusion models with low-cost training, facilitating native support for hundreds of languages. Specifically, we employ plug-and- play language adapter with fewer than 20 million parameters to bridge a frozen multilingual text encoder with frozen diffusion model, which exhibits outstanding zero-shot per- formance on multilingual T2I generation tasks.
large-scale pre-trained multilingual text encoders, reducing the need for extensive multilingual datasets. As a result, the model can efficiently adapt to over 110 languages using only small amount of English training data, achieving generation quality comparable to English (e.g., CLIP sim- ilarity scores: 39.57 for English, average 39.61 for other languages). Additionally, as shown in Figure 4, MuLan plug plug-and-ple and-play architecture seamlessly integrates with exist- ing model architectures and frameworks, greatly enhancing its compatibility with various community-driven tools and models, such as LoRA (Hu al., 2021), LCM (Luo et al., 2023), ControlNet (Zhang et al., 2023), and IP-Adapter (Ye et al., 2023b).
Our model offers advantages terms of not only multi- lingual performance, but also training cost and adaptability compared to previous works. In nutshell, MuLan leverages
In summary, our contribution three folds:
(1) We investigate different possible ways equip monolin- gual text-to-image models with multilingual ability, among
2

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
which we demonstrate that using a text encoder properly trained from noisy web-scale data (such InternVI. (Chen et al., 2023b)) is of great data efficiency.
PEA-Diffusion (Ma et al., 2023) focus on fully utilizing the current T2I model by employing lightweight networks to align the semantics of other languages with English using low-cost parallel data. However, this often involves complex distillation loss, making the training process less stable. In our work, multilingual image generation capabilities can be generalized simply by training with the commonly used SD loss on English image-text pairs.
(2) We propose MuLan, a plug-and- lightweight lan- guage adapter that can be combined with any community models and tools for native multilingual generation in 110+ languages. Our method requires only English image-text pairs and avoids problems in previous attempts, includ- ing preparing datasets in restricted number of languages, heavy computing budget, and inflexibility when combined with various community models/tools.
(3) We demonstrate our method's effectiveness and effi- ciency through thorough quantitative and qualitative experi- ments. Notably, it requires only about 12 hours of training on 8 A100 GPUs and 17M English data to perform very well on 110+ languages, with very close CLIP similarity scores of 39.57 (English) v.s. 39.61 (average of other languages).
Multilingual Language Model. Language models have evolved through various modeling approaches, reflecting a range of training objectives and capabilities. Early models, such as BERT (Devlin et al., 2019), BART (Lewis et al., 2019), focus on understanding sentence structure and con- textual relationships by predicting masked tokens within a sentence. More recent large language models (LLMs), such as LLaMA series (Touvron et al., 2023; Dubey et al., 2024), T5 (Raffel et al., 2020) and GPT series (Brown et al., 2020; OpenAL. 2023), build on this foundation with improved context understanding and generation abilities, allowing them to handle a wide range of tasks with nuanced language comprehension and open-ended text generation. Building on these foundational text encoders, recent works, such as CLIP (Radford al., 2021), ALIGN (Jia et al., 2021), and Intern VL (Chen et al., 2023b), incorporate visual alignment through paired image-text data, creating joint representa- tions that bridge language and vision for cross-modal tasks.
2. Related Work
Multilingual Diffusion Models. Recent advancements have seen the rise of diffuusionbbased models, which have sig- nificantly improved image generation quality and diversity. Popular models such as SD series (Rombach et al., 2022: Esser et al., 2024; Podell et al., 2023), DALL-E (Ramesh et al., 2022; 2021), Imagen (Saharia et al., 2022) and Glide (Nichol et al., 2022) demonstrate photorealistic gener- ation capabilities, yet they are primarily trained on English data and thus struggle with multilingual image generation. Although diffusion models using CLIP text encoders can generalize somewhat to Romance languages (e.g., French), their performance significantly degrades for languages out- side this family, particularly East Asian languages such as Chinese. This limitation arises from the models' reliance on English-centric text encoders, such as CLIP (Radford et al., 2021) and T5 (Raffel et al., 2020), and the predominantly English training data.
Previous works such as mBERT (Devlin et al., 2019) and LLaMA3 (Dubey et al., 2024) have shown strong multi- lingual capabilities by pre-training on large multilingual corpora, enabling these models to understand and generate text in variety of languages. IntemVL-LLaMA (Chen et al., 2023b) achieves powerful multilingual cross-modal capabilities by aligning a text encoder with a vision transformer (ViT) on the multilingual image-text dataset LAION(Schuhmann et al., 2022), enabling effective image- text contrastive learning. However, due to the scarcity of large-scale multilingual image-text datasets, some ap- proaches (Chen et al., 2022; Carlsson et al., 2022) have resorted to using translated datasets and distillation learning to align multilingual text features. Despite these advances, an open challenge remains inefficiently leveraging multi- lingual text encoders for text-to-image (T2I) generation. Previous works, such as Taiyi (Wu al., 2024) and AltD- iffusion (Ye al., 2023a), have adopted multilingual text encoders and fine-tuned them on multilingual image-text pairs for T2I tasks. In contrast, our method takes advantage of Intern VL-LLaMA's multilingual capabilities, requiring only a small amount of English data and without the need to fine-tune the SD model weights, achieving sfate-of-the th-the--rt performance in multilingual T2I generation.
Recent efforts have explored multilingual image generation by incorporating multilingual text encoders and datasets to overcome the limitations of English-centric models. One approach involves building models entirely from scratch with non-English data. For instance, Hunyuan-DiT (Li et al., 2024b) and Kolors (Team, 2024) incorporate Chinese text encoders and extensive Chinese datasets to enhance their support for culturally specific concepts and improve gen- eration quality in Chinese. Alternatively, some methods attempt to adapt existing models by replacing or fine-tuning the text encoder. Models such as Taiyi (Wu et al., 2024), PanGu (Lu et al., 2023), and AltDiffusion (Ye et al., 2023a) replace the text encoder in SD and then fine-tune it with multilingual data, thus reducing the overhead compared to training from scratch. GlueGen (Qin et al., 2023) and
3

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
Text Encoder
Architecture
Supervision Method
Aligned
name
num
type
lang-num
quality
-BBON-5B
5B
T1 pairs
100+
Noisy
BERT
encoder only
MLM
PixArt
15M
Tl pairs
1
High
T5
encoder-decnder
MLM
X
JourneyDB
4M
TI pairs
1
High
CCAligned
100M
Text Parallel
137
Noisy
LLaMA
decoder only
NTP
XX
CCMatrix
69B
Text Parallel
80+
Noisy
CLIP
encoder only
CL
InterVL--LLMMa
decoder-only
NTP&CL
Table 2. Mainstream Dataset Types. The existing data used for T21 training mainly includes two formats: Text-Imape pairs (TI pairs) and Text Parallel data.
Table 1. Common Text Encoders. The common language model architectures and their supervision methods, "Aligned" indicates whether they have been aligned with images. In "Supervision Method", "MLM" is Masked Language Modeling. "NTP" is Next Token Prediction, and "CL" is Contrastive Learning.
multilingual text corpora, multilingual text-image pairs, and high-quality text-to-image datasets-offer valuable re- sources for multilingual image generation. Large-scale datasets like Laion-400M/5B (Schuhmann et al., 2022) contain noisy, lower-quality data but provide multilingual text-image pairs, supporting diverse languages. Meanwhile, multilingual translation corpora (e.g.. CCMatrix (Schwenk et al., 2020)) lack image-text pairs but offer cross-language alignments. High-quality text-to-inn rext-to-image datasets (c.g.. Jour- neyDB (Pan et al., 2023)) supply high-resolution images but mainly support English and few mainstream lan- guages. Thus. leveraging low-cost, internet-base text- image datasets and multilingual translation corpora to en- hance multilingual T2I model training remains a promising direction.
3. Proposed Method
In this section, we first revisit the mainstream text-to-image generation framework, analyzing its underlying structures and presently available data resources. Based on these in- sights, we introduce MuLan, a cost-effective multilingual generation framework designed to improve cross- langula adaptability and generation quality.
3.1. Revisiting Text-to-Image Generation
Given an input text prompt and ground-truth image y from the training dataset D, mainstream text-to-image (T2I) generation model G(),, which consists of a language model L(-) and a visual generator (-), can be defined as follows:
3.2. MuLan: Toward Multilingual T2I Generation
Overall Architecture. To facilitate efficient and effective multilingual T21 generation, MuLan incorporates two key designs: (1) the multilingual semantic alignment through easily accessible large-scale data, and (2) a language adapter trained on limited set of English T2I data. These designs enable Mulan operate without the constraints of mul- tilingual T2I data, allowing for more efficient training by leveraging existing models and datasets.
07,05 arg min ((z,))>D [[G((; Ot, 0)), 0,,0,
(1)
where 9r and 6r represent the parameters of the language model L(-) and the visual generator (-),, respectively, and f. denotes the generation loss, represents the optimal solu- tion for function optimization. It can be observed that the primary components related to multilingual processing are the training dataset D and the language model L(-). There- fore, in the following sections, we focus on the two modules, exploring pathways for building an efficient text-to-image generation model.
Multilingual Semantic Alignment. While many existing language models demonstrate robust multilingual capabili- ties, not all are well-suited for the multilingual T2I genera- tion task needed for the Mulan framework. In this work, we emphasize the importance of maintaining consistent vec- tor space across languages for multilingual T2I generation. Specifically, given two text prompts, æ1 and x2, that share the same meaning but are in different languages, and a text encoder L(-). the representations of these prompts, L(x1) and L(x2), should closely align. This alignment ensures that the conditional inputs to the image decoder remain consistent across languages, thereby preserving consistent image generation quality. Here, we mainly consider two alignment approaches: (1) Image-centered alignment; (2) Language-centered alignment.
Language Model. Existing T2I models typically employ pre-trained language models as text encoders. For instance, the SD series (Rombach et al., 2022; Podell et al., 2023; Esser et al., 2024) leverages CLIP (Radford et al., 2021) or T5 (Raffel et al., 2020). Recently, decoder-only LLMs like GPT (Brown et al., 2020; OpenAI, 2023) and LLaMA (Tou- vron et al., 2023; Dubey al., 2024) have gained attention for their superior NLP performance, multilingual capabili- ties, and word prediction training. As summarized in Table 1, a diverse range of language models is available, yet the optimal choice for multilingual T2I generation remains an open question.
(1) Language-Centtn Centered Alignment. A simple approach to multilingual semantic alignment leveraging large-scale
Dataset. As shown in Table 2, existing bitasets-incluce
4

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
Wrappinng
After aligning different languages using Eqn 2 Eqn 3, we can achieve multilingual T21 generation by training only this adapter. The key to low-cost implementation lies in freezing the language model and visual generator while training only the language adapter By on small-scale English data Den-
LoRM//CM(Optional
Multiinguul Text Encodor
Mulan Adapter
Prompt
Image Decoder(UNet, DIT)
Image
Original Text Encader
These adapters are used to re-project the high-dimensional representations of text prompts from different languages into unified space. We adopt different adapter designs for different diffusion models. In detail, we can achieve good results using simple MLP architecture for Pixart-cx (Chen et al., 2023a). However, we find MLP could not properly deal with SD models (Rombach et al., 2022; Podell et al., 2023). Instead, we choose to use one layer decoder transformer with a set of learnable queries for extracting embeddings from InternVL outputs. For SDXL (Podell et al., 2023), we use two transformers to project embeddings for two text encoders that SDXL adopts, and one attention pooling layer for extracting pool embeddings.
Notey
Depth Imagge
Optional
Canny Image
Ab ControlNet
Pose image
Figure 2. Overview of of MuLan. We use language model as the Multilingual Text Encoder, which has undergone Multilingual Semantic Alignment stage. Wc only train the Language Adapter, while all other modules are frozen.
translation data to align other languages' vector spaces with the well-supported English vector space. By conducting distillation training with translation data alone, this can be achieved: we designate the SD's language encoder as the teacher encoder, indicated by (x) and any multilingual encoder [1, 0.)) as the student encoder, aligning their features using MSE Loss. This alignment can be written as:
Bridge Together. As shown in Figure 2, after the Multi- lingual text encoder is trained with Multilingual Semantic Alignment, the Language Adapter can be trained with min- imal English training data at a low cost while enabling multilingual image generation capabilities for broad text-to- image diffusion models. The Multilingual Semantic Align- ment stage can be undergone with different strategies and base models, which could result in different performance on subsequent image-generation fine-tune saee-tuning. More details are elucidated in Section
0- arg min (.,))) (MSELLL,,, Le(ra)) (2)
0,
(2) Image-Centered Alignment. In the image-centered align- ment approach, CLIP maximizes the similarity between positive text-image pairs and minimizes it for negative pairs through contrastive learning. This training uses text-image pairs, and when the text includes multiple languages, the language encoder aligns different languages in the vector space naturally around the image. In this case, the objective function can be written as:
4. Experiments
4.1. Experimental setup
Datasets. We primarily use subset of LAION-EN (Schuh- mann et al., 2022) with all samples that have aesthetic scores larger than 5.8 for training base models and Pix (Chen et al., 2023a) dataset for aesthetic models.
0 = arg min B(a.))~ www.-//coosieeeeee 0t), Er)], 6:
(3)
Here, O refers to the parameters of the language model L()), Es refers to the image feature. This method, how- ever, is resource-intensive and requires large multilingual text-image pairs, which are challenging to obtain and re- quire substantial storage. Using pre-trained models can help reduce these costs.
Implementation details. We use InterVV--LLan (Chen et al., 2023b) our text encoder, which has undergone Image-Centered Alignment on multilingual image-text pair dataset. Then, we trained the MuLan Adapter for dif- ferent image decoders using the architecture described in Section 3.
Language Adapter. After getting the aligned language model, to achieve cost-effective multilingual text-to-image generation, we propose MuLan. This model incorporates a lightweight language adapter L' that bridges a multilingual- aligned language model with a visual generator, enabling generalization to multiple languages after training on a small amount of English text-to-image generation data Den- So the Eqn 1 could be rewritten
All Mulan adapters are trained with Adam W (Loshchilov Hutter, 2017) optimizer and 128 batch size. We use constant learning rate le-5 for SD 1.5/2.1 (Rombach et al., 2022), 1e-6 for SDXL (Podell et al., 2023), and 2e-5 for Pixart- a (Chen et al., 2023a). For SD 1.5, we train the adapter for 50k steps at the resolution of 512x512, and for SD 2.1 we adjust the resolution to 768 768. For SDXL, we first train the adapter for 100k steps at the resolution of 512x512 and finetune it for another 1k steps at the resolution of
07- arg min E5 ( De. ([GG(z0rr),))
(4)
0,,
5

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
x024x 1024. For Pixart-a, we train the adapter for 118k steps at the resolution of 512x512. We randomly drop text conditions at the rate of 10% and use min-SNR (Hang et al., 2023) to accelerate training.
As shown in Table 3. we compared our model with other multilingual image generation models. On the XM12 dataset, Miee-VL-MuLan-Pixae sion (Ye et al.. 2023a) and GlueGen (Qin et al.. 2023) in CLIP score, highlighting its strong multilingual understand- ing. However, Intern VL-MuLan-PixAAr fell behind AltD- iffusion in FID. further evaluate image quality, we used Laion's 's aesthetic predictor. Intern Ve-MuLaLMLa--DD VL-MuLL-
outperformed AltDiffu-
The training process was conducted on 8 NVIDIA A100- 80G GPUs. For SD 1.5 and SD 2.1, it took 12 hours, and for SDXL and Pixart-a, it took two days.
and achieved average aesthetic scores of 6.31 and 6.67, respectively, surpassing AltDiffusion's s score of 6.05. This highlights the superior quality and visual appeal of images generated by our model.
Evaluation Metries. We use XM3600 (Thapliyal et al., 2022) to assess the model's capabilities across 12 main- stream languages (denoted as XM12 hereafter). To evaluate the model's Is generalization to additional languages, we tested multilingual versions of the COCO2014 (Lin et al., 2015) validation set. We translated the prompts into 85 languages using Google Translate for these datasets. The model's 's per- formance was compared with an ad-hoc translation-bases SD 1.5 model and other multilingual T2I models (Yan et al., 2024). Regarding evaluation metrics, we employed FID and CLIP Score (SIM) calculated by Intern VLL-LLLA (Chen
To further evaluate our model's capability in the broader range of minority languages, we evaluated our model on the COCO2014 (Lin et al., 2015) validation set (85 lan- guages) and computed CLIP Score, as shown in Figure 3. Our model achieved performance comparable to AltDiffu- sion in mainstream languages while substantially surpassing AltDiffusion in less common languages.
et al., 2023b).
Our model achieves top performance across languages, reaching SOTA levels in mainstream languages and ex- celling low-resource ones, all with minimal training costs. VL-MLan SD15 requires just 0.5 GPU-days, and Intern VL-Mulan Art only 2x8 GPU-days. In compar- ison, GlueGen costs 5 GPU-days yet performs worse than the low-cost VL. 55- while AltDiffusion demands at least 19x64 GPU-days. This highlights the efficiency and cost-efffetiveeness of our approach.
4.2. Quantitive Results
IntermVL-Mulan Mu1an-S015
AtDiffusion
et Sn fi br uk ca cabg i el do
gl
It
sv
pl
ga
zh
R
Comparison of MuL.n and Translate API We compared MuLan with the approach of using Google Translate and NLLB (NLLB Team et al., 2022) to convert non-English languages into English for input into image generation mod- els (Sun et al., 2024). Non-English languages in XM12 were translated into English, as input of SD15 (Rombach et al., 2022) and PixArt-a (Chen et al., 2023a), with results compared direct input into MuLan. As shown in Table 3, MuLan outperforms the approach that uses the open-source model NLLB as translation tool and achieves performance comparable to using Google Translate. It demonstrates even better results in low-resource languages where translation tools often struggle to provide accurate translations.
hi
es
ko
mt
de
ms
ru
te
ml
kn
gu
y1
mk
bn
ta
hy
Iv
10 wh yo smsg is swin st ne ur
Figure 3. Comparison of CLIP Score of InternVL-MuLann SD15 and AltDiffusion across hundreds of languages on COCO2014 val. Our model achieved performance comparable to AliDiffusion in mainstream languages while substantially surpass- ing AltDiffusion less common languages.
Comparison of MuLan and "as-is" Baseline We eval- uated Stable Diffusion v1.5 and PixArt-a by directly in- putting prompts in 11 languages from the XM12 dataset, using Intern VL-MLLLMM to compute CLIP scores, shown in Table 5. These "as-is" results, where the prompts in different languages were fed into the T2I models without translation, were then compared with our MuLan-adapted models. We found that MuLan adapters consistently im- proved performance across all languages. Notably, even for languages close to English- such French, Spanish, and German our method still achieved clear gains over the
Multilingual T2I Comparison. We integrated Intern VL- LLaMA (Chen et al., 2023b) into the adapter's model, which we call VLnMuan. Specifically, for each image de- coder, we train a separate adapter model, which we re- fer to as Intern Ww-tnVLLLLLL--S--III InterVV--MuLan-SDD2) Intern Intern VL-MMulan an-MuulanSSDXXX SDXI. and Intern an- -rtttt.
6

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
avg.
cn
es
ja
de
ko
ru
ar
pl
SIM
35.6
39.6
35.9
35.5
35.0
35.3
36.3
X
X
X
X
X
X
GlueGen*
FID
17.4
21.4
17.1
18.5
17.6
17.6
16.1
X
X
X
X
X
X
SIM
38.8
40.0
40.0
39.4
39.7
39.0
39.4
34.1
39.0
38.5
38.6
38.3
39.2
AltDiffusion
FID
9.1
8.1
8.2
9.4
10.4
9.4
9.4
10.2
8.2
8.9
9.7
8.9
9.1
SIM
36.5
39.6
37.6
37.4
37.2
36.2
36.2
32.7
37.7
35.9
36.8
33.9
36.3
SDI5(NLLB)
FID
25.0
21.4
24.9
25.0
25.0
25.5
24.9
28.2
24.5
25.4
24.6
26.6
24.4
SIM
36.9
39.6
38.2
38.0
37.8
36.6
36.7
33.1
38.4
36.4
37.4
34,4
36.7
SD15(Google)
FID
21.6
21.4
21.6
23.9
22.4
19.1
20.5
19.6
22.1
21.7
21.6
22.9
21.8
SIM
38.3
39.0
39.7
39.4
39.2
38.1
38.6
34.4
40.3
37.7
39.3
35.6
38.7
PixArt(NLLB)
FID
13.2
12.6
13.2
14.6
12.5
13.0
13.1
12.4
12.9
12.4
13.0
14.3
14.7
SIM
39.7
39.0
41.2
40.6
41.0
39.9
40.8
34.7
41.5
39.0
40.3
39.0
39,9
PixArt(Google)
FID
13.7
12.6
13.7
15.5
12.9
13.8
13.9
12.3
12.9
13.2
12.9
15.1
15.4
SIM
37.7
38.6
38.0
37.7
37.8
37.7
38.0
35.6
38.0
37.6
37.9
38.2
37.1
Mulan-SD15
FID
14.4
13.9
14.7
16.5
13.5
17.0
13.7
13.0
13.8
12.8
13.8
14.0
16.5
SIM
39.5
39.6
40.2
39.6
40.0
39.3
40.5
37.2
40.5
39.3
39.6
39.1
39.1
Mulan-PixArt
FID
11.8
12.8
12.2
13.6
11.7
10.6
10.7
13.4
12.4
8.4
10.5
11.9
13.2
Table 3. Comparison of Multilingual Generation Capabiliites on XM12. GlueGen and Altiiffusion are other multilingual T2I methods, where the imputs for SD15 and PixArt-a are translated using Google Translate and NLLB. GlueGen is equivalent to the original SD15 when accepting English prompts. Its performance in English was not included in the average calculation.
lang Id
Model
SIM
Comparison of MuLan and Specialized Models We com- pared the multilingual MuLan adapter with two language- specific models: Taiyi (Wu et al., 2024), trained on mil- lions of Chinese image-text pairs, and the Japanese SDXL model (Shing & Akiba), fine-tuned on high-quality Japanese image-text pairs. Our model performs better on both lan- guages, demonstrating its competitiveness with specialized models.
Moyyi-SDL-33.1
35,44
Taiyi-SD- SD-i-SDDII IB Chinese
36.84
zh
InternVL-Mulln-SDII
37.84
JapaneseSDXL 38.00
ja
InternVL-Mulan-SDD.
38.01
Table Comparison of CLIP Score on Chinese and Japanese with specialized models XM3600. Intern VL--Mulan LMMSLa surpasses specialized Chinese and Japanese models.
Comparison of Multilingual Semantic Alignment Meth- ods In this study, we trained the MuLan Adapter on mul- tilingual language models with and without Multilingual Semantic Alignment, using different alignment methods, and evaluated their performance on five non-English lan- guages on the XM12 dataset. The results are shown Table LLama (Tou-
as-is baseline, demonstrating its broad effectiveness.
Alternative CLIP for Image-Text Similarity Image-text similarity typically computed using the industry-standar CLIP-VIT models (Radford et al., 2021). However, these models are primarily trained on English data, making their similarity scores unreliable for non- non-English inputs. In this work, we employ VL-LLLV-- a multilingual model, to compute the image-text similarity scores. To further ensure the objectivity of the similarity evaluation, we addi- tionally use the multilingual CLIP-VIT model released by Laion (LAION, 2023). The results shown in Table 5 demon- strate that our model remains robust, and the observed trends are consistent with those reported in Table 3.
6. We first selected the multllingual-supporinn vron et al., 2023), mT5-xl (Xue et al., 2021) and XLM-R Large (Conneau, 2019) models as text encoders and trained MuLan without the Multilingual Semantic Alignment stage. Next, for Image-Centered Alignment, we choose Intern V- LLaMA (Chen et al., 2023b) and Mul-OpenCLIP (Con- neau, 2019) as text encoders, both of which have be trained on multilingual image-text pairs such as Laion-5B (Schuh- mann et al., 2022). For Language-Centnn Centered Alignment, we selected MultiLang- CLIP (Carlsson et al., 2022) and AltClip-m18 (Chen et al., 2022), models trained on par- allel datasets like CCMatrix (Schwenk et al., 2020), We also trained an XLM-R Large on CCMatrix for compari- son. Among them, Mul-OpenCLIP, MultiLang- and AltClip-m m18 all use the L...M--bbera---- Large architecture, but with different training methods or datasets. All image decoders used are based on SD 1.5 (Rombach et al., 2022).
Fine-grained Text-Image Alignment Evaluation While CLIP Score is useful for evaluating the presence of main sub- jects, is limited in capturing object-level details and spatial relations-especiee
for compositional prompts in XM12. To address this, we adopt VQAScore (Lin et al., 2024) with GPT-4o as the evaluator, enabling reliable multilingual as- sessment. As shown in Table 5, our model achieves strong fine-grained aligmment, outperforming GlueGen and Alt- Diffusion, and matching or surpassing translation-bart baselines across most languages.
The results are shown in Table 6. We found that multilingual text encoders trained solely on multilingual corpora, such
7

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
Metric
avg.
en
fr
es
it
zh
ja
ru
ar
pl
SDI5(as-is)
SIM
29.1
39.6
34.2
34.2
31.8
26.6
26.9
26.8
31.9
24.6
23.0
24.5
24.8
PixArt(as-is)
SIM
29.8
39.0
38.2
36.9
36.0
24.2
22.1
24.0
36.8
22.4
28.9
22.2
27.0
Mulan-SD15
SIM
37.7
38.6
38.0
37.7
37.8
37.7
38.0
35.6
38.0
37.6
37.9
38.2
37.1
Mulan-PixArt
SIM
39.5
39.6
40.2
39.6
40.0
39.3
40.5
37.2
40.5
39.3
39.6
39.1
39.1
GlueGen
SIM( Laion)
21.1
22.3
21.3
20.6
19.6
21.5
21.0
X
X
X
X
X
X
AldDiffusion
SIM(Laioo)
23.1
24.2
23.6
23.2
23.1
24.3
23.0
21.1
24.3
22.5
22.3
22.7
23.2
SDoo(G)
SIM(Laion)
22.3
22.3
23.8
23.4
22.8
22.6
21.3
20.8
24.5
21.7
22.6
19.5
21.9
Piogre(Google)
SIM(Laion)
24.3
24.1
24.2
23.8
24.5
26.4
25.1
21.6
26.5
23.1
24.6
23.3
23.9
Mulan-SD15
SIM(Laion)
23.0
21.8
23.6
23.2
22.7
23.6
24.2
22.7
24.5
21.8
23.0
22.3
22.9
Mulan-PixArt
SIM(Laion)
24.2
24.4
23.9
23,4
24.2
25.7
24.8
23.2
25.8
23.4
24.6
23.1
23.4
GlueGen AltDiffusion
VQAScore
0.53
0.81
0.51
0.52
0.45
0.41
0.50
X
X
X
X
X
X
VQAScore
0.58
0.71
0.60
0.60
0.59
0.50
0.53
0.62
0.49
0.56
0.51
0.64
0.62
(Do5(Go)
VQAScore
0.61
0.81
0.66
0.71
0.61
0.57
0.50
0.67
0.50
0.70
0.68
0.41
0.52
PixArt(Google)
VQAScore
0.75
0.85
0.78
0.80
0.82
0.71
0.71
0.82
0.71
0.76
0.76
0.59
0.69
Mulan-SD15
VQAScore
0.64
0.80
0.74
0.76
0.68
0.60
0.52
0.53
0.57
0.63
0.75
0.45
0.57
Mulan-PixArt
VQAScore
0.74
0.88
0.81
0.83
0.81
0.71
0.71
0.61
0.73
0.74
0.79
0.59
0.72
Table 5. Multidimensional Comparison of Multilingual Image Generation on XM12. Part 1: Comparison between English-only T21 models using as-is prompts and Mul am; Parr 2: Using the multilngnaal-cppble Laion's 's CLIP-ViT (L.AION, 2023) for image-text similarity; Part 3: Evaluating fine-grained text-image alignment using VQAScore (Lin et al., 2024).
Model
Align Method
avg SIM
through the image, which acts as a bridge between the lan- guage embeddings and the visual space. As result, the semantic space of different languages is already aligned with the image prior to training with MuLan, allowing Mu- Lan to further refine this alignment. In contrast, Language- Centered Alignment relies entirely on MuLan to establish the connection between language embeddings and the im- age feature space, making the MuLan adapter training more challenging and resulting in lower performance.
LLaMA2-7B
None
X
mT5-xl
None
X
XLM-R Large
None
X
MultiLang-CLIn -LIP
LC
33.2
AlIClip-m18
LC
33.3
XLM-R Large*
LC
34.7
Mul-OpenCLiI
IC
36.1
IntemVL-LLLMA
IC
37.8
Table 6. CLIP Score five non-English languages of Dif- ferent Text Encoders trained with MuLan Adapter. Mul- OpenCLIP stands for
In summary, MuLan innovatively uses image-centered aligned language model for multilingual image generation. Previous methods like GlueGen (Qin et al., 2023) and AltD- iffusion (Ye et al., 2023a), which rely on language-centered alignment, face inherent limitations due to their alignment method. GlueGen aligns non mon-English embeddings with the CLIP text encoder used in SD,, with SD kept frozen, but as shown in Table 3, its performance non-English languages is significantly worse than that on English. AltDiffusion, by contrast, retrains both text encoder and UNet on billions of multilingual pairs, incurring high training costs. In compar- ison, our approach achieves strong, balanced multilingual generation with minimal cost, requiring only small adapter while keeping the SD frozen. In practice, the use of uni- fied, image-centered alignment pre-trained model allows easy adaptation to various SD models in the community with minimal overhead.
Shon5B-s13B-b900 (LAION, 2023). "LC" and "IC" represent Sanguage-Centered Aligament and Image-Centered Alignment, respectively; XLM-R Large* is trained by the authors using CC- Matrix.
as LLaMA, mT5-xl, and XLM-R Large, fail to understand non- mon-English prompts entirely. This is because these models have undergone multilingual alignment, their multi- lingual embeddings not aligned. After connecting these language models with the MuLan adapter and training with English text-image pairs, only the English embeddings are aligned with the visual feature space, preventing understand- ing of non non-English prompts. However, with either of the two alignment methods, the multilingual embeddings are first mapped to a unified feature space, which the MuLan adapter then aligns with the image space using only English image-text pairs, enabling multilingual image generation.
4.3. Qualitative Results
The results also suggest that Image-Centered Alignment outperforms Language- Centered Alignment by a signif- icant margin. We speculate that this is because, in Image-Centered Alignment, language alignment is achieved
Our model can generate high-quality images across multiple languages. It also supports multilingual mixed input and can recognize certain emojis. In addition to simple T2I tasks, there are numerous downstream applications of SD
-enttrge-Centered
8

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
MVDream
BE AU cool Transformer)
Dreamshaper
Realistic Vision
Cartoonmix
3D Animation
LoRA (Lego)
LoRA (MidJourney)
ControlNet
LCM
Turbo
Lightning
Figure 4. Examples of MuLan integrates with community tools. MuLan seamlessly integrates with MVDream (Shi et al., 2023) for 3D model generation and is fully compatible with community tools like LoRA (Hu et al., 2021), ControlNet (Zhang al., 2023), and LCM (Luo et al., 2023).
within the community, including LoRA (Hu et al., 2021),
of original models. A promising extension would be to alle- viate the need for paired data and make original capabilities intact. Furthermore, MuLan currently focuses on improving multilingual generation capabilities, but it would be inter- esting extend it to improving prompt understanding and following under multilingual context.
LCM (Luo et al., 2023), and ControlNet (Zhang et al., 2023). These tools played crucial roles enhancing model adapt- ability, control over outputs, and finetuning for specific tasks. Our model does not require finetuning SD, allow- ing it to seamlessly integrate and be compatible with these counuunity-developee SD applications in plug-and-play manner. We show some examples in Figure 4.
Impact Statement
5. Conclusion
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences our work, none which we feel must be specifically highlighted here.
We introduce language adapter MuLan that could equip image/video/3D diffusion models with multilingual genera- tion abilities. MuLan shows strong zero-shot capabilities for up to 110 different languages, even if the adapter is solely trained on English data. MuLan also can be trained with a frozen text encoder and diffusion denoising model, which makes applicable for many downstream models, such as LoRA (Hu et al., 2021), ControlNet (Zhang et al., 2023), LCM (Luo al., 2023), and etc., without any additional finetuning. MuLan is currently trained with paired data, and it could inevitably bring in bias and cause distribution shift

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
References
weighting strategy. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7441- 7451,
Arkhipkin, V., Filatov, A., Vasleev, V., Maltseva, A., Azizov, S., Pavlov, I, Agafonova, J., Kuznetsov, A., and Dim- itrov, D. Kandinsky 3.0 technical report. arXiv preprint arXiv:23/2.03511 2023.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0966 2021.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33; 1877-1901, 2020.
Jia, C., Yang, Y., Xia, Y., Chen, Y.TT., Parekh, Z., Pham, H., Le, Q V., Sung, Y., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision, 2021. URL https://arxi>>
Carlsson, F., Eisen, P. Rekathati, F., and Sahlgren, M. Cross-lingual and multilingual clip. In Proceedings of the Language Resources and Evaluation Conference, pp. 6848-6854, Marseille, France, June 2022. Euro- pean Language Resources Association. URL https: //aclantholog log.. wwt..orgg/22.22 1rec-1 1. 739.
LAION. hitpriich---fffffoeexerrrrrrrrtd s s13b-b90k, 2023, Accessed: 2024-11-15,
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mo- hamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. Bart: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and comprehen- sion, 2019. htl.httss:///arae org/ abs 1910. 13461.
Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J.. Luo, P., Lu, H., and Li, Z. Pixart-o: Fast training of diffusion transformer for photorealistic text- to-image synthesis, 2023a.
Chen, Z., Liu, G., Zhang, B.-W. Ye, F., Yang, Q. and Wu, L. Altelip: Altering the language encoder in clip for extended language capabilities. 2022. doi: 10.48550/ARXII.22111 1.06679 URL https: /ttes///rxiii arxiv. org/abs/ 2211. 06679.
Li, D., Kamko, A., Akhgari, E., Sabet, A., Xu, L., and Doshi, S. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint 2rXiv:2402.
2024a.
Chen, Z., Wu, J., Wang, W., Su, W. Chen, G., Xing, S., Muyan, Z., Zhang, Q. Zhu, X., Lu, L.,, et al. In- ternvl: Scaling up vision foundation models and align- ing for generic risual-linguisee tasks. arXiv preprint arXiv:2312. 14238, 2023b.
Li, Z., Zhang, J., Lin, Q., Xiong, J., Long, Y., Deng, X., Zhang. Y., Liu, X.. Huang, M., Xiao, Z., Chen, D., He, J., Li, J., Li, W., Zhang, C., Quan, R., Lu, J., Huang, J., Yuan, X., Zheng, X., Li, Y., Zhang, J., Zhang, C., Chen, M., Liu, J.. Fang, Z., Wang, W., Xue, J., Tao, Y., Zhu, J., Liu, K., Lin. S., Sun, Y., Li, Y.. Wang, D., Chen, M.. Hu, Z., Xiao, X., Chen, Y., Liu, Y., Liu, W., Wang, D., Yang, Y., Jiang. J., and Lu, Q. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding, 2024b.
Conneau, A. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116 2019.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding, 2019. URL https://arxiv. org/abs/1 1tahss/811 810 04805.
Lin, T.-Y,, Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., and Dollár, Microsoft coco: Common objects in con- text, 2015. URL 0312.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al- Dahle, A., Letman, A., Mathur, A., Schelten, A., al. The llama 3 herd of models. ArXiv, abs/2407.21783, 2024. URL http:::/app..emaaticcaseeesessss Ttg/Corrussi II..puss:: 21:15111
org/ abs/1405.
Lin, Z.,, Pathak, D., Li, B., Li, J., Xia, X., Neu- big. G. Zhang, P., and Ramanan, D. Evaluat- ing text-to-visual generation with image-to-text gener- ation. In European Conference on Computer Vision, 2024. URL https: /ppi. www.ntiisccchiinieeeee
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint 2rX 2024.
Hang, T., Gu, S., Li, C., Bao, J.. Chen, D., Hu, H., Geng, X., and Guo, B. Efficient diffusion training via min-snr
Loshchilov, and Hutter, Decoupled weight decay regu- larization, arXiv preprint 171 2017.
10

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
Lu, G., Guo, Y., Han, J., Niu, M., Zeng. Y., Xu, S., Huang, Z.,, Zhong, Z.,, Zhang, W., and Xu, H. Pangu- draw: Advancing resource-efficient text-to-image syn- thesis with tmeedeccouled training and reusable coop- diffusion. arXiv preprint arXiv:2312 16486, 2023.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (1 (140):1-67, 2020.
Luo. S., Tan, Y., Huang, L., Li, J., and Zhao, H. La- tent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, Zero-shot text- to-image generation. In International Conference Machine Learning, pp.
PMLR, 2021.
Ma, J., Chen, C., Xie, Q.. and Lu, H. Pea-diffusion:
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.00 202220....,,, 2022.
Parameter-efficien adapter with knowledge distillation in non-english text-to-image generation. arXiv preprint arXiv:2311.17088 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P. McGrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022. URL https: 10741.
Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
//arxiv. abs////////////
Saharia, C., Chan, W., Saxena. S., Li, L., Whang. J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 1147, 2022.
NLLB Team, Čosta-jussà, M. R., Cross, J., Çelebi, O., HI- bayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., Sun, A., Wang, S., Wenzek, G., Youngblood, A., Akula, B., Barrault, L., Mejia-Gonzalez, G. Hansanti, P., Hoffman, J., Jarrett, S., Sadagopan, K. R., Rowe, D., Spruit, S., Tran, C., Andrews, P., Ayan, N. F., Bhosale, S., Edunov, S., Fan, A., Gao, C., Goswami, V., Guzmán, F., Koehn, P., Mourachko, A., Ropers, C., Saleem, S., Schwenk, H., and Wang, J. No language left behind: Scaling buman-centered machine translation. 2022.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv: 10.00840, 2022.
Schwenk, H., Wenzek, G., Edunov, S., Grave, E., and Joulin, A. Cematrix: Mining billions of high-quality parallel sentences on the web, 2020. URL bttps://arae o org aaa//b//1100 /1911. 04944.
OpenAl. Gpt-4 technical report, 2023.
Pan, J., Sun, K., Ge, Y., Li, H., Duan, H., Wu, X., Zhang, R., Zhou, A., Qin, Z., Wang, Y., Dai, J., Qiao, Y., and Li, H. Journeydb: A benchmark for generative image understanding, 2023.
Shi, Y., Wang, P., Ye, J., Mai, L., Li, K., and Yang, X. Mvdream: Multi-view diffusion for 3d generation. arXiv:2308. 16512, 2023.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
Shing, M. and Akiba, Japanese stable diffusion xl. URL Iattps://hugiingfaace haatsss///uuggnnaace...oooaaaaa http:/faccc.o/taaiitt.....a / fapanese-stablle--dffffiooo----e (https: (httpss japanese-stable-
Qin, C., Yu, N., Xing, C., Zhang, S., Chen, Z., Ermon, S., Fu, Y., Xiong, C., and Xu, R. Gluegen: Plug and play multi-modal encoders for x-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23085-23096. 2023.
Sun, J.-M., Wu, T., and Gao, L. Recent advances in im- plicit representation based 3d shape generation. Visual Intelligence, :1) Mar 2024. ISSN 2731-9008. doi: 10. 10.100774442670004400044..1 URL https:/ /doi. org 0tlg//0.107//00
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Team, K. Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis. arXiv preprint, 2024.

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
Thapliyal, A. V., Pont-Tuset, J., Chen, X., and Soricut, R. Crossmodal-3600: massively multilingual mul- timodal evaluation dataset. ArXiv, abs/2205.12522, 2022. URL 249062751.
org/Corpus ID:
Touvron, H., Martin, I.., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(Nov): 2579-2605, 2008.
Wu, X., Zhang, D., Gan, R., Lu, J., Wu, Z., Sun, R., Zhang, J. Zhang. P., and Song, Y. Taiyi-diffusion- xl: Advancing bilingual text-to-image generation with large vision-language model support. arXiv preprint arXiv:2401. 14688, 2024.
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. mt5: A massively multilingual pre-trained text-to-text transformer, 2021. URL https:tp/// /aaaaaivvvv///////a//aaa org/ abs/2010 11934.
Yan, Y., Zhou, Z., Wang, Z., Gao, J., and Yang, X. Di- aloguenerf: towards realistic avatar face-to-face con- versation video generation. Visual Intelligence, 2(1):
24, Aug 2024. ISSN 2731-9008. doi: 10.1007/
$44267-024400055--8. URL 1007/s44267-024-000
Ye. F., Liu, G.. Wu, X., and Wu, L. Altdiffusion: A mul- tilingual text-to-image diffusion model. arXiv preprint wxXiv22308.09911, 2023a.
Ye, H., Zhang, J., Liu, S., Han, X., and Yang, W. Ip-adapter: Text compatible image prompt adapter for text-to-ine text-to-image diffusion models. 2023b.
Zhang, J., Gan, R., Wang, J., Zhang, Y., Zhang. L.,, Yang. P. Gao, X., Wu, Z., Dong, X., He, J., Zhuo, J., Yang. Q., Huang, Y., Li, X., Wu, Y., Lu, J., Zhu, X., Chen, W., Han, T., Pan, K., Wang, R., Wang, H., Wu, X., Zeng, Z., and Chen, C. Fengshenbang 1.0: Being the foundation of chinese cognitive intelligence. CoRR, abs/2209.02970, 2022.
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
12

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
A. More Experiments
requires significantly less data and computational cost at every stage.
In this section, we provide supplementary analyses to further evaluate and understand the performance and efficiency of our proposed model. These studies delve into key aspects such as training efficiency, semantic alignment methods, and the model's generalization across multiple languages. By examining the impact of various design choices and alignment strategies, we aim to shed light on the model's multilingual capabilities and resource efficiency.
A.2. Impact of Semantic Alignment on Multilingual Features
In this section, we conduct a preliminary analysis of the multilingual output features produced by various text en- coders to reveal the effects of the two image alignment train- ing methods introduced in Section 3.2. The text encoders we selected for this analysis include XLM-R Large (Con- neau, 2019), CCMatrix trained XLM-R Large* in Table 6, Mul-OpenCLIP (LAION-5B (Schuhmann et al.. 2022) pre- trained XI.MRRBBERTa- 7B (Touvron et al., 2023), and IntemVL-LLLMA (Chen et al., 2023b).
A.1. Training Efficiency
Our model leverages the multilingual capabilities of the VL-LLALL text encoder, enabling effective training with minimal cost. this section, we examine the changes in the model's 's performance when data volume and training iterations are reduced.
(Conneau, 2019)), LLaMA2-
We employ t-SNE (van der Maaten & Hinton, 2008), a non- linear dimensionality reduction technique, to visualize the aggregation effects of multilingual features produced by these text encoders. t-SNE particularly suited for preserv- ing the local structure of tigh-dimensional data in low- dimensional space. Ideally, for text encoders trained with multilingual alignment methods, the features corresponding to semantically equivalent prompts in different languages should exhibit aggregation effects when projected into a lower-dimensional space.
A
# of samples
Avg SIM
1
17M
35,80
1/2
8.5M
35.46
1/4
4.25M
35.40
1/8
2.1M
35.23
1/16
IM
35.17
1/64
250k
35.13
1/256
63k
35.08
1/1024
16k
33.49
For this analysis, we randomly sampled 20 captions from the COC02014 4 (Lin et al., 2015) validation set and translated them into 8 languages using machine translation, resulting in total of 160 textual inputs. These inputs were then encoded using the selected text encoders to obtain their corresponding pooled feature representations.
Table 7. Impact of dataset size on multilingual performance of the model. We evaluated the model's performance using different proportions of the original dataset, ranging from full size (A = 1) to 1/1024, and measured the average CLIP Score across lan- guages. The results show that the model maintains relatively strong performance even with significantly reduced data sizes, demon- strating its efficiency and robustness under resource-constraing settings.
The results are shown in Figure As shown in Figures (a) and (d), the sample points in the t-SNE (van der Maaten & Hinton, 2008) plots exhibit scattered distribution for text encoders that have not undergone image alignment train- ing. This indicates that although LoBERTa (Conneau, 2019) and I.LaMA2 (Touvron et al., 2023) possess multi- lingual representation capabilities, their output features for synonymous multilingual prompts are not closely located in the Euclidean space. In contrast, as shown in Figures (b)( the sample points exhibit clustered distribution after alignment training with images, with each cluster cor- responding to the 20 different language versions of a single prompt. Through alignment training with images, the model minimizes the semantic discrepancies between multilingual representations, ensuring that the embeddings of synony- mous prompts converge in the vector space.
Setting. We choose Stable Diffusion 1.5 (Rombach et al., 2022) as the backbone model, using the AdamW optimizer for training. The model was trained to convergence with learning rate of 2e-5 and batch size of 128, without employing any training tricks. Additionally, the dataset size was reduced, and we set multiple data proportion levels ranging from 0.5 to 0.001 to evaluate the impact of data size on the model's performance.
Results. As shown in Table 7, we found that the model's multilingual performance decreased as the data volume was reduced; however, still maintained relatively strong per- formance until we reduced the data to 1/1024 of the default data volume. This level of data volume and training cost is highly frienddl. requiring only 48 GPU hours to achieve decent multilingual text-to-image image (T2I) capabilities for the model. Compared to existing multilingual T2I mod- els, such as AltDiffusion (Ye et al., 2023a), our approach
A.3. Detailed Results on COC02014 validation set
In Section 4.2, we evaluated Intern VL-Mulan-SDDi on the COC02014 (Lin al., 2015) validation set (85 languages)
13

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
@XLM-R--L
(b) CCMatrix Pre-trained
(c) LAION Pre-trained
(c) LL
( IntenVL-LLLaM
XLM-R-L
XLM-R-L
Figure 5. t-SNE analysis embeddings of 9 prompts in 20 languages produced by text encoders. (a) XLM-RoBERTa-Laagg (Con-
neau, 2019) (b) CCMatrix pre-trained -LM-RoBERTa-
(Conncau, 2019) in Table (c) Laion-5B (Schuhmann et al., 2022)
pre-trained XLM RoBERTa-Large (Conneau, 2019) (d) LLaMA2-7B (Touvron al., 2023) (e) Intern MieeWV-LLLMMA (Chen et al., 2023b). Points of the same color represent embeddings corresponding to the same prompt translated into different languages.
Id SDIS
SD21
SDXL
Id
SDIS
SD21
SDXL
SDI5
SD2I
SDXL
af
35.39
34.54
34.68
33.83
1d
35.67
35.55
35.49
31.53
ps
30.74
30.87
31.41
28.61
39.00
38.20
38.13
38.77
ig
30.65
30.80
32.10
30.09
37.17
36.88
36.93
30.51
az 34.19
33.67
33.66
30.95
30.68
30.96
31.97
29.17
38.11
37.86
37.63
38.51
be 35.24
35.23
35.20
30.74
37.13
36.62
36.56
37.62
30.61
30.20
30.26
27.93
bg 38.76
38.40
38.38
33.60
iw
38.52
38.07
38.17
26.84
sk
36.12
36.12
36.02
29.09
bn 33.58
32.56
33.61
30.06
38.05
38.14
37.82
39.43
36.83
36.28
36.22
27.21
bs 37.20
36.89
36.86
28.10
ka 35.70
35.53
35.80
30.38
29.94
30.24
32.44
27.94
ca 37.74
37.43
37.22
kk 32.18
31.70
31.44
sm
29.61
29.71
30.74
29.97
co 35.03
34.50
34.39
33.95
km
25.82
26.71
28.43
24.81
SO
29.67
29.05
31.38
29.05
cs 35.30
34.85
34.71
28.78
kn 28.94
27.42
29.65
28.59
bq
35.08
34.41
34.77
29.06
da 36.25
35.99
35.77
31.78
ko 38.06
37.21
37.09
38.27
37.77
37.90
37.91
31.09
de 36.79
36.98
36.87
38.16
ku 29.86
29.23
30.41
27.50
29.76
30.00
32.24
29.63
35,39
35.49
35.24
25.36
ky 32.49
32.08
32.37
Su
32.65
32.00
32.91
30.62
eo
31.17
30.81
30.99
28.92
la 30.25
30.32
30.39
29.71
SV
35.91
36.22
36.07
31.83
37.38
37.41
37.37
38.41
lb 30.96
30.59
30.93
30.28
SW
30.39
30.30
30.77
32.23
34.43
34.85
34.76
28.22
36.02
35.94
35.77
27.00
31.00
29.69
32.20
32.47
38.22
37.99
37.92
28.58
35.08
34.91
34.56
27.03
28.07
27.22
29.82
27.21
37.16
37.16
36.74
28.71
mi
30.34
30.60
32.75
29.54
tg 30.84
30.43
31.29
28.34
fr 37.38
37.57
37.41
38.81
mk
38.17
38.29
38.41
31.66
th 36.95
36.83
36.76
37.56
fy 32.73
31.97
31.90
31.64
ml
31.75
29.67
32.78
29.72
35.89
35.65
35.57
36.17
ga 27.56
26.96
28.26
28.27
33.39
33.91
34.63
30.04
uk 37.62
37.22
36.98
37.94
gd 27.95
27.52
28.69
mi 35.14
34.12
31.52
ur 34.82
34.30
34.10
29.01
gl 37.06
36.80
36.59
36.08
mss 35.25
35.04
35.01
30.61
UZ 30.18
29.50
30.24
28.91
gu 28.25
27.83
29.09
31.06
mt 29.97
29.22
30.64
29.87
vi 38.13
37.67
37.60
38.31
haw 31.31
31.52
33.81
31.37
my 28.15
28.10
29.56
27.90
29.22
29.01
29.49
27.86
hi 36.94
36.47
36.28
36.92
ne
34.37
33.53
34.34
32.72
yo 30.32
29.95
31.68
28.65
hr 37.25
36.94
36.93
28.00
36.85
36.64
36.39
37.91
zh 38.85
39.26
39.50
40.30
hu 36.38
36.08
35.98
26.44
nо 35.94
35.80
35.73
31.65
by 33.36
32.66
32.98
26.73
pl 35.97
35.82
35,30
37.12
Table CLIP Score the COCO2014 validation set. 'ld' indicates the key of different language. SD15/21/XL (Rombach al., 2022; Podell et al., 2023) represent our model implemented on the corresponding three T2I backbones, while AD refers to the AliDiffusion (Ye al., 2023a). Our model also demonstrates strong text-to-image (T2I) capabilities wider range languages.
14

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
and compared it with AltDiffusion (Ye et al., 2023a). More results are shown in Table 8. It be observed that our model generalizes well to wider range of languages and delivers impressive performance.
B. More Qualitative Results
In this section, we present more examples of multilingual generation by the model, well examples of its interac- tion with existing community models and tools.
B.1. Robustness to Multiple Languages
As shown in Figure 6, model supports multiple lan- guages, allows prompt inputs that combine different lan- guages, and even recognizes emojis. For example, We can use the car emoji as a prompt (the first image in Figure 6), and the model can generate an image of a car.
B.2. Plug and Play on Different Visual Generator
Our model can be seamlessly integrated into existing fine- tuned models, such as DreamShaper, Realistic Vision, and others. Figure 6, we present additional examples, all generated by existing models interacting with Mulan. These examples use prompts in multiple languages, demonstrat- ing support for various language combinations inputs. Our model also supports some existing tools based the Stable Diffusion series developed by the community. Here, we showcase several popular models, including LoRA (Hu et al., 2021) models, ControlNet (Zhang et al., 2023) mod- els, and IP-Adapter (Ye al., 2023b) models. The model's multilingual capabilities naturally come into play in these applications.
15

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
Prompt:
Prompt: Kozmikus
Prompt: Stunning botanical ス -
Prompt: Astronauten rijden paarden
Translation: car
máglyaködhen rekedt kalózhajó cnasslation:iiratt ship stranded in cosmic bonfire nebula
Translation: Stunning botanical, watercolor style, white background
in een schetsstijl
Translation: Astronauts riding horses sketch style
Prompt: Basket buahnya
Prompt: kadın büyüčüüsü Translation: witch
Prompt: byzylyk me muaza kristali
Prompt: Lick
ada di meja
Translation: landscape painting
Translation: The fruit basket is on table
Translation: crystal bead bracelet
Prompt: R
Prompt: Thit bō Translation: Beef
Prompt: une scule photo un coucher de solcil sur
Prompt: 'Eva; naιayιλoς noo ςрорйιι youλıá nyáoo
F-9 rabbit
Translation: A rabbit
la mer
Translation: A parrot
wearing
Translation: single photo of a sunset over the sea
wcaring sunglasses
Figure 6. Multilingual generation results. Our model supports multilingual Maaguu-g inputs.
Prompt: "otnm/==""-~/>>>>>>>/>>
Prompt: - REUARB 1A
Translation: Wearing sunglasses the beach
Translation: polar bear sitting chair drinking milkshake
Figure 7. IP-Adapter Results. Our model enables multilingual style transfer by integrating with the IP-Adapter (Ye et al., 2023b).
16

---

MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
Prompt: style kung tiger
Prompt: 3d style 2 المومية Asi!!
Prompt: KE - A
Prompt: pixel Ā R
Translation: style Egyptian Queen
Translation: Pixel style-a cute corgi
Translation: Pixel style-a cute
cat
Prompt: vector art, / Translation: vector art, eagle
Prompt: vector art, 97 | At
Prompt: papercut-subjece シ x
Prompt: papercut-subject- ;
Translation: vector art, astronaut
Translation: papercut -subject-fox
Translation: papercut- subject-river and sunset
Figure Lora Results. Our model can naturally support multilingual input when using LoRA (Hu et al., 2021).
Depth-ControlNG
Prompt: CH
Prompt: A
Prompt: 7 ターマン
Translation: spiderman
Translation: spiderman
Translation: spiderman
OpenPose-ControoNett
Prompt: - N Translation: dancing girl
Prompt: Un mannequin international les podiums
Prompt: スーツビ > L
Translation: An international
Translation: Handsome
model on the catwalks
man in a suit
Figure 9, ControlNet Results. Our model can utilize existing ControlNet (Zhang et al., 2023) models, enabling multilingual image generation with conditional inputs such as depth maps and pose images.
17


Reviews:
[{"review_id": "3gy9CmgH4u", "reviewer": "Reviewer_BvZv", "summary": "This paper introduces MuLan, a lightweight and plug and play language adapter that enables multilingual text to image generation for diffusion models with minimal computational cost. The central idea is that multilingual text encoders can be used to enable multilingual image generation without the need for extensive labeled datasets in multiple languages. Instead of training a full diffusion model on multilingual text-image pairs, the model freezes the text encoder and diffusion model and introduces a small adapter module that bridges the two. This approach allows model to support over 110 languages while requiring training only on English-language image-text pairs.\n\nThe main claims are that the model achieves strong multilingual image generation performance (on par with models explicitly trained on multilingual text image pairs) and that it requires significantly less computational cost and training data (by using leveraging pretrained multilingual text encoders and training only a small adapter) and that it Is flexible and integrates with existing models and community tools (e.g. LoRA, Control Net).\n\nThe authors evaluate the model on multilingual benchmarks and compare it against other approaches from the literature and translation-based approaches. The results show strong performance in both high-resource and low-resource languages, with significant efficiency gains in terms of compute and data requirements, suggesting that the proposed model is both computationally efficient and broadly applicable.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer BvZv,\n\nThanks a lot for your insightful reviews and support for our work! We hope our responses can address your questions.\n\n**Q1. Lack of a cross lingual consistency evaluation**\n\n**A1:** We appreciate the reviewer’s suggestion to evaluate cross-lingual conceptual consistency. Our work focuses on enabling multilingual text-to-image generation by leveraging a pretrained multilingual text encoder and English image-text pairs. While our current method demonstrates some degree of cross-lingual consistency, it may struggle with culture-specific corner cases, which fall outside the scope of this work and would require human curation or high-quality data to resolve. We will acknowledge this limitation in the main paper and plan to address it in future work. As a potential direction, we may explore collecting data with explicit culture-specific concepts and developing a dedicated benchmark for evaluating cross-lingual consistency in multilingual T2I generation.\n\n**Q2. Clarification on hyperparameter settings**\n\n**A2:**\nThe training hyperparameters for MuLan-SD15 and MuLan-PixArt used in Table 3 are already provided in Section 4.1. Here we provide additional details for completeness: we use the AdamW optimizer with $\\beta = (0.9, 0.999)$ and a weight decay of 0.01. For SD 1.5, SD 2.1, and PixArt, we follow the original models’ resolutions (512x512 or 768x768). For SDXL, we adopt a two-stage training strategy with different resolutions to ensure stability. We adopt classifier-free guidance by randomly dropping text conditions, following [a1].\n\nIn Table 5, we only replace the language model; all other training settings are identical to those of MuLan-SD15.\n\nWe will include these details regarding the selection of experimental hyperparameters in the updated manuscript.\n\n**Q3. Ablation study on the model size**\n\n**A3:**\n\n**For the size of the adapter**\n\nIn the main paper, our proposed adapter uses a lightweight one-layer Transformer encoder-decoder structure (see Section 3.2). To further investigate the relationship between model size and performance, we have conducted additional ablation experiments on Stable Diffusion 1.5, testing adapters with 2, 4, and 6 Transformer layers. The experimental setup follows Appendix A.1. We calculated the average CLIPScore across seven languages from the XM12 dataset.\n\n|layers|1|2|4|6|\n|-|-|-|-|-|\n|**Avg CLIPScore**|35.8|35.8|35.5|35.4|\n\nWe observe that increasing the number of Transformer layers in the adapter does not lead to improved performance; in fact, it **slightly degrades performance**. This result aligns with previous findings in works such as LLaMA-Adapter [a2] and MiniGPT-4 [a3], which demonstrate **that adapter-based tuning for LLMs typically does not require large parameter counts or massive training data**. Our lightweight adapter design is consistent with these community practices, suggesting that a compact architecture is sufficient and may even be more stable and efficient for adaptation.\n\n**For the size of the language model**\n\nOur approach relies on a series of pretrained multilingual language models, making it difficult to perform strictly controlled ablation studies on the size of the language model. These models differ not only in parameter count but also in architecture, tokenizer, training corpora, which complicate direct comparisons.\n\nAlthough a strictly controlled study is not feasible, a coarse-grained observation from Table 5 reveals a trend suggesting that the size of the language model may impact the performance upper bound. Specifically, the first four models—MultiLang-CLIP (33.2), AltClip-m18 (33.3), XLM-R Large* (34.7), and Mul-OpenCLIP (36.1)—all use **XLM-R Large(335M)** as the language model. In contrast, **InternVL-LLaMA(7B)**, which achieves the highest score of **37.8**.\n\nSince our approach builds upon existing pretrained multilingual models, conducting strictly controlled comparisons on model size is non-trivial. As part of future work, we plan to explore how our adapter performs when integrated with increasingly stronger language models, which could further reveal its potential in real-world multilingual generation.\n\nReferences\n\n[a1] Ho, Jonathan. “Classifier-Free Diffusion Guidance.” https://arxiv.org/abs/2207.12598\n\n[a2] Zhang, Renrui et al. “LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention.” https://arxiv.org/abs/2303.16199.\n\n[a3] Zhu, Deyao et al. “MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models.” https://arxiv.org/abs/2304.10592", "claims_and_evidence": "The paper makes several strong claims about the model. Overall, the claims seem well supported by the presented evidence.\n\nFirst, they claim that the proposed model achieves multilingual performance comparable to English trained models while requiring only English data. This is supported by quantitative results, where the CLIP similarity score for English (39.57) is nearly identical to the average score across all other languages (39.61). The authors also show visual examples demonstrating the model's ability to generate accurate images from prompts in diverse languages, including low resource languages. Second, they claim that in comparison to other multilingual diffusion models, the model reduces training costs. They argue that because the model only trains a small adapter, it avoids the high cost of training a full diffusion model on multilingual datasets. The claim is backed by the reported training time being orders of magnitude lower than the competing models that MuLan is benchmarked against, and by a comparative cost analysis against. Another claim is that the model generalizes across a large number of languages, which is supported by an evaluation on multilingual datasets showing that the model is comparable to or sometimes better than translation-based methods.", "methods_and_evaluation_criteria": "The method essentially involves training a multilingual adapter on top of diffusion models. The diffusion model itself is frozen, and the adapter is trained to connect it to a pretrained multilingual text encoder. This is quite appropriate for the context and enables text to image generation from a wide variety of languages while keeping the underlying generation model fixed with savings in terms of required compute. The adapter learns to map multilingual text embeddings into the same space as the english trained diffusion model. The paper explores 2 different strategies to achieve this: language centred alignment (aligning multilingual embeddings to English using parallel translation datasets and a distillation loss) and image centred alignment (using contrastive learning to align multilingual embeddings with image embeddings, ensuring the text prompts in different languages produce similar image representations). The experiments show that the image centred approach performs better - likely because the alignment is done directly between language and image space as opposed to relying on noisier translation datasets. \n\nThe evaluation metrics are the CLIP similarity score (text to image alignment) and FID score which measures image quality. These metrics are quite appropriate in this context and (especially in the case of FID) widely used across the literature. Furthermore, the model is benchmarked on XM3600 (12 languages) and COCO2014 (85 languages) and compared against a number of strong baselines (e.g. translation based stable diffusion) which strengthen the claims made by the authors. \n\nA minor remark here: it would be interesting to have a more detailed evaluation of cross lingual consistency e.g. whether a given concept is represented similarly across these languages and the degree to which similar results are obtained. It is understood however, that this is rather challenging and perhaps wouldn't strengthen the claims of this paper significantly enough.", "theoretical_claims": "not applicable.", "experimental_designs_or_analyses": "As discussed, the methodology and evaluation of this paper are strong. The methodology for language and image centered alignment is sound and aligns with common practice in the representation learning literature. The evaluation and benchmarking provide a broad test of generalization across low and high resource languages. The translation based baselines compare whether the direct multilingual generation is better than translation pipelines. The analysis of computational efficiency is thorough and well documented, illustrating very clearly the contrast with other multilingual models in the literature. The comparison of image vs language centred alignment also seems sound and empirically supports the claim that image centred alignment is the superior choice.\n\nAs before, a minor remark would be the lack of a cross lingual consistency evaluation. One can imagine that when testing across so many different languages, stylistic/cultural nuances could affect the results, and there could be some failure cases. Finally, as another minor remark, there is information about experimental design to aid reproducibility but it is not clear how some of the hyperparameter settings were arrived at or whether they are just default values.\n\nOverall there are no real comments from me on this, and methodologically and experimentally this paper is quite strong and comprehensive.", "relation_to_broader_scientific_literature": "The papers's key contributions align with several existing research areas in NLP and image generation. Adapter based methods for multilingual NLP are quite are a widely explored concept to enable multilingual capability without retraining whole models. The paper under discussion extends the idea to text to image generation. Indeed, due to the extensive commercial applications, text to image models have also been quite a popular area of research recently, so this paper is very timely in both respects.", "essential_references_not_discussed": "I would have expected acknowledgement of works like MAD-X [1] for adapter based cross lingual transfer. Other than that, several relevant works are cited (and of course directly compared against) and it is acknowledged that this work touches several disciplines. \n\n[1] https://arxiv.org/abs/2005.00052v2", "other_strengths_and_weaknesses": "The paper is well written and clearly presented. The methods are not very novel, but they are timely and of considerable impact. Nothing further to report beyond what was already discussed.", "other_comments_or_suggestions": "not applicable", "questions_for_authors": "not applicable", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q3zCnGiwYI", "reviewer": "Reviewer_RQVZ", "summary": "This paper proposes a simple yet effective way to handle multilingual text input in text-to-image generation. By utilizing a pre-trained multilingual text encoder and introducing a light-weighted adapter, the resulting model is shown to handle multilingual input well.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer RQVZ,\n\nThanks very much for your valuable comments. We hope our responses can address your concerns and clarify our contribution.\n\n**Q1. Comparison with translation baseline.**\n\n**A1:** In fact, previous multilingual text-to-image generation works (e.g., AltDiffusion) typically compare against open-source translation models such as NLLB [a1]. As shown in the following table, we also compared the results obtained using NLLB as a translation tool, and in 11 non-English languages of XM12, **our performance exceeded the baseline of NLLB as a translation tool**. We can also observe that using translation tools to handle non-English input is highly dependent on the performance of the translation tool, being very sensitive to its performance. \n|Model|open/closed source|avg|fr|es|it|zh|ja|hi|de|ko|ru|ar|pl|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|SD15 (NLLB)|open|36.17|37.6|37.4|37.2|36.2|36.2|32.7|37.7|35.9|36.8|33.9|36.3|\n|SD15 (Google Translate)|closed|36.70|38.2|38.0|37.8|36.6|36.7|33.1|38.4|36.4|37.4|34.4|36.7|\n|MuLan-SD15|open|**37.60**|38.0|37.7|37.8|37.7|38.0|35.6|38.0|37.6|37.9|38.2|37.1|\n|PixArt (NLLB)|open|38.27|39.7|39.4|39.2|38.1|38.6|34.4|40.3|37.7|39.3|35.6|38.7|\n|PixArt (Google Translate)|closed|**39.82**|41.2|40.6|41.0|39.9|40.8|34.7|41.5|39.0|40.5|39.0|39.9|\n|MuLan-PixArt|open|39.49|40.2|39.6|40.0|39.3|40.5|37.2|40.5|39.3|39.6|39.1|39.1|\n\n**Google Translate** is a strong commercial system with high development and data costs, and has undergone long-term refinement to reach its current performance, **it remains a closed-source tool, making it unsuitable for offline or privacy-sensitive scenarios**.\n\nIn contrast, **MuLan is a low-cost baseline that achieves multilingual image generation by training on English image-text pairs and is entirely based on open-source suites**. In terms of the **FID** metric, our model generally **achieves better results**. On CLIPScore, our results are already comparable to those obtained using Google Translate, and we show clear advantages in less common languages. These results highlight the potential of achieving multilingual image generation through training, without relying on external translation systems. We will include these points in the main paper in future revisions.\n\n**Q2. Adapt to the new SoTA model.**\n\n**A2:** One of our main contributions is the low adaptation cost (e.g., **MuLan-SD15 training only requires 96 GPU hours**). As demonstrated by the visual results in Figure 1 and Figure 4, the Adapter trained for Mulan-SD15 can be effectively applied to other fine-tuned variants of SD15 (e.g., Dreamshaper), and also works well with plugins such as LoRA and ControlNet. The same holds for models like SD21, SDXL, and PixArt-$\\alpha$. This indicates that our method exhibits strong generalizability across this family of models, with minimal additional cost. For newly emerging SoTA models, our method can still be employed to support them and their related plugins and variants with similarly low adaptation cost.\n\n**Q3. More analysis of performance differences in table 5**\n\n**A3:** Based on the experimental results in Table 5, we can draw two main conclusions.\n\n**1. Only the aligned text encoder can support multilingual image generation through MuLan.**\n\nBy comparing the models in Rows 1–3 and Rows 4–8 of Table 5, it is evident that without Language-Centered Alignment (LC) or Image-Centered Alignment (IC), the models are unable to support multilingual image generation through MuLan. We provide visualizations of the feature distributions for some of these models in Figure 5 of the appendix. As shown, models without alignment—such as LLaMA2-7B (Figure 5(d)) and XLM-R Large (Figure 5(a))—produce scattered features when processing inputs with the same meaning but in different languages. In contrast, aligned models—such as XLM-R Large* (Figure 5(b)), Mul-OpenCLIP (Figure 5(c)), and InternVL-LLaMA (Figure 5(e))—can cluster features of semantically equivalent inputs across different languages. This is a key reason why our method, trained solely on English image-text data, is still able to support multilingual image generation.\n\n**2. Image-Centered Alignment outperforms Language-Centered Alignment.**\n\nBy comparing the models in Rows 4–6 and Rows 7–8, we observe that LC (Language-Centered Alignment) yields inferior results compared to IC (Image-Centered Alignment). This is primarily because LC relies on translated data, which may introduce noise due to inaccuracies in translation. Moreover, IC aligns multilingual semantic features through the shared image feature space, effectively providing MuLan with a prior that facilitates further refinement of the alignment. In contrast, LC relies entirely on MuLan to establish the connection between language features and image features from scratch. These factors collectively contribute to the inferior performance of LC compared to IC.\n\nReferences\n\n[a1] https://huggingface.co/facebook/nllb-200-3.3B", "claims_and_evidence": "The paper claims to propose a method which can handle multilingual text input well in text-to-image generation. According to the results in the paper, it obtains good results than some previous methods.", "methods_and_evaluation_criteria": "The idea of utilizing a pre-trained multilingual text encoder is reasonable, because it has already been pre-trained on multilingual text and should be able to better align different languages into a single embedding space.", "theoretical_claims": "There is no theoretical claims in the paper.", "experimental_designs_or_analyses": "The experimental design seems reasonable. The authors compared the proposed method with related methods, and also included some important baseline such as directly using Google translate to process multilingual input.", "relation_to_broader_scientific_literature": "The difference is that the paper utilizes a pre-trained multilingual text encoder, while previous methods either fine-tuned the diffusion model on multilingual text-image dataset (may lead to image quality drop and potential bias) or align different languages with light-weighted network.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "According to the results in Table 3. The simple baseline, which directly use Google translate to process multilingual input obtains comparable results with the proposed method:  SD 1.5 baseline outperforms the proposed SD1.5 version model on 5/12 cases, and PixArt baseline outperform corresponding proposed method on 8/12 cases. People may choose to use baseline method because it doesn't require any training, and can choose arbitrary translation models and T2I models. Meanwhile, the proposed method has to be trained again if one want to apply it to a new SoTA model.", "other_comments_or_suggestions": "None", "questions_for_authors": "What is the reason causing the performance difference between different alignment methods in Table 5?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RHyR6eOUjo", "reviewer": "Reviewer_fvJg", "summary": "This paper proposes MuLan, a text-encoder adapter that equips T2I models that are pre-trained with data dominant in English now with multilingual capabilities. With MuLan, T2I modes may take in prompts in purely non-English terms and  \ngenerate images in quality on par with those from English prompts.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer fvJg,\n\nThanks so much for your constructive comments and support for acceptance. We hope our responses can address your concerns.\n\n**Q1: Lack of as-is baseline performances from English-only T2I backbone models.**\n\n**A1**: We thank the reviewer for emphasizing the importance of “as-is” baseline performances. In response, we evaluated Stable Diffusion v1.5 and PixArt-α by directly inputting prompts in 11 languages from the XM12 dataset, using InternVL-LLaMA to compute CLIP scores. These as-is results were then compared with our MuLan-adapted models.\n\nWe found that **MuLan adapters consistently improved performance across all languages.** Notably, even for languages close to English—such as French, Spanish, and German—our method still achieved clear gains over the as-is baseline, demonstrating its broad effectiveness.\n\n|Model|avg|de|fr|it|es|pl|hi|ru|zh|ja|ko|ar|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|SD15 (as-is)|28.1|31.9|34.2|31.8|34.2|24.8|26.8|23.0|26.6|26.9|24.6|24.5|\n|MuLan-SD15|**37.2**|38.0|38.0|37.8|37.7|37.0|35.6|37.4|36.6|36.7|36.4|38.2|\n|Δ (SD15)|**+9.1**|+6.1|+3.8|+6.0|+3.5|+12.2|+8.8|+14.4|+10.0|+9.8|+11.8|+13.7|\n|PixArt(as-is)|29.0|36.8|38.2|36.0|36.9|27.0|24.0|28.9|24.2|22.1|22.4|22.2|\n|MuLan-PixArt|**39.5**|40.5|40.2|40.0|39.6|39.1|37.2|39.6|39.3|40.5|39.3|39.1|\n|Δ(PixArt)|**+10.5**|+3.7|+2.0|+4.0|+2.7|+12.1|+13.2|+10.7|+14.9|+18.4|+16.9|+16.9|\n\n**Q2: Confusion over the CLIPScore specification.**\n\n**A2:** While the industry-standard CLIP-ViT models are indeed widely used, they are primarily trained on English data and thus provide unreliable similarity scores for non-English inputs. In contrast, InternVL-LLaMA has been trained on multilingual image-text pairs and possesses better understanding of non-English languages. Therefore, we chose InternVL-LLaMA as the surrogate model for computing CLIPScore in our paper.\n\nTo further ensure the objectivity of our CLIPScore evaluation, we additionally used the multilingual CLIP-ViT model released by LAION to compute similarity scores. The results, shown in the table below, demonstrate that **our model still performs strongly and that the observed trends are consistent with those reported in Table 3 of the paper.**\n\n|model|avg|en|fr|es|it|zh|ja|hi|de|ko|ru|ar|pl|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|GlueGen|21.05|22.3|21.3|20.6|19.6|21.5|21.0|-|-|-|-|-|-|\n|AltDiffusion|23.13|24.2|23.6|23.2|23.1|24.3|23.0|21.1|24.3|22.5|22.3|22.7|23.2|\n|SD15(Google Translate)|22.27|22.3|23.8|23.4|22.8|22.6|21.3|20.8|24.5|21.7|22.6|19.5|21.9|\n|PixArt(Google Translate)|24.26|24.1|24.2|23.8|24.5|26.4|25.1|21.6|26.5|23.1|24.6|23.3|23.9|\n|MuLan-SD15|23.02|21.8|23.6|23.2|22.7|23.6|24.2|22.7|24.5|21.8|23|22.3|22.9|\n|MuLan-PixArt|24.15|24.4|23.9|23.4|24.2|25.7|24.8|23.2|25.8|23.4|24.6|23.1|23.4|\n\n**Q3: Limitations of CLIPScore.**\n\n**A3:** The prompts in XM12 predominantly feature compositional text prompts. While CLIPScore is useful for evaluating whether the main subject in the image aligns with the prompt, it falls short in assessing object-level details and spatial relationships. As you suggested, we additionally evaluated our model using VQAScore to better measure fine-grained text-image alignment.\n\nSince the default VQAScore evaluation model (clip-flant5-xxl) does not support multilingual prompts adequately, we adopted GPT-4o as our evaluation model to enable multilingual VQAScore assessment. Results show that our model maintains strong performance, significantly outperforming GlueGen and AltDiffusion, and even surpassing or matching translation-based baselines across most languages.\n\nOur model effectively leverages the capabilities of existing MLLMs, demonstrating strong generalization in multilingual image generation. In the future, we plan to further integrate the native multilingual, contextual, and reasoning abilities of MLLMs into the image generation process.\n|model|avg|en|fr|es|it|zh|ja|hi|de|ko|ru|ar|pl|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|GlueGen|0.533|0.81|0.51|0.52|0.45|0.41|0.5|-|-|-|-|-|-|\n|AltDiffusion|0.581|0.71|0.6|0.6|0.59|0.5|0.53|0.62|0.49|0.56|0.51|0.64|0.62|\n|SD15(Google Translate)|0.612|0.81|0.66|0.71|0.61|0.57|0.5|0.67|0.5|0.7|0.68|0.41|0.52|\n|PixArt(Google Translate)|0.750|0.85|0.78|0.8|0.82|0.71|0.71|0.82|0.71|0.76|0.76|0.59|0.69|\n|MuLan-SD15|0.635|0.80|0.74|0.76|0.68|0.60|0.52|0.53|0.57|0.65|0.75|0.45|0.57|\n|MuLan-PixArt|0.744|0.88|0.81|0.83|0.81|0.71|0.71|0.61|0.73|0.74|0.79|0.59|0.72|\n\nReferences\n\n[a1] https://huggingface.co/laion/CLIP-ViT-H-14-frozen-xlm-roberta-large-laion5B-s13B-b90k\n\n[a2] Evaluating Text-to-Visual Generation with Image-to-Text Generation. https://arxiv.org/pdf/2404.01291 ECCV 2024", "claims_and_evidence": "**Issue #1: Lack of as-is baseline performances from English-only T2I backbone models.** Although the authors have shown comparisons with other multilingual T2I models such as AltDiffusion or GlueGen as in Table 3, what are the fundamental baseline performances that you simply feed in prompts in the target language as-is to an English-only T2I model, such as SD1.5? This as-is baseline setup is assumed to bring poor performance in ClipScore, certainly. But without showing these baselines, we may not have a solid idea how using a multilingual text encoder can make a significant difference in the first place. \n\nAs the authors have already implicated, the pre-training data used in the vanilla SD1.5 may have already had a language bias towards non-English Western languages. Thus, SD 1.5 should have higher as-is baselines in languages like Spanish or German. Will adding MuLan raise the performances across all languages by a universal increase, or will it instead selectively improve some languages than others over their as-is baselines? I don't think the robustness of the MuLan adapters can be truly demonstrated, if it is being presented without the as-is baselines.", "methods_and_evaluation_criteria": "**Issue #2: Confusion over the ClipScore specification.** According to Line 288, the CLIPScore/SIM used in the paper uses InternVL-LLaMA as the surrogate model. However, CLIPScore is mostly widely calculated with CLIP-VIT variants, such as in Dall-E 3 [1]. How do the MuLan adapters perform if measured in industry standard metrics?\n\n**Issue #3: T2I Metrics Other than ClipScore?** CLIPScore has already been proven to struggle at compositional text prompts as in [2]. Since the prompts in XM12 mostly feature compositional attributive phrases like those in Figure 1, how well does MuLan improve then, if measured in fine-grained text-image-alignment metrics such as VQAScore [3]?", "theoretical_claims": "There is no major deviation from mainstream theories.", "experimental_designs_or_analyses": "Please refer to my concerns in the Claims and the Method sections above.", "relation_to_broader_scientific_literature": "The potential of MuLan can be high, as it works in a plug-and-play / training-free style that can be easily integrated into established T2I pipelines, such as those implemented in the Huggingface framework.", "essential_references_not_discussed": "References that are mentioned in my concerns in the sections above.\n\n- [1] Improving Image Generation with Better Captions.  https://cdn.openai.com/papers/dall-e-3.pdf 2023\n\n- [2] Text encoders bottleneck compositionality in contrastive vision-language models. https://arxiv.org/abs/2305.14897 EMNLP 2023\n\n- [3] Evaluating Text-to-Visual Generation with Image-to-Text Generation. https://arxiv.org/pdf/2404.01291 ECCV 2024", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None", "questions_for_authors": "Please find my 3 major issues in the sections above. Out of all, Issue #1 has the highest severity and will greatly affect my impression of this work if left unaddressed. \n\nAfter all, I really like the potential application of MuLan, but I would like to push its presentation over the high-standard bar of ICML. I am open to updates and would like to engage with the author further.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper focuses on multilingual text-to-image generation and introduces a lightweight adapter to enable that on diffusion-based models with minimal computational cost. During training, only the adapter is optimized, while the text encoder and diffusion model remain fixed. This approach supports over 100 languages, although only English-image pairs are used for training. Reviewers appreciate the comprehensive experiments and efficiency improvements demonstrated in the paper. AC agrees with the reviewers in recommending acceptance and encourages the authors to incorporate the authors-reviewers discussions into the final paper and apply the method to more recent image generation models (*e.g.*, Flux).
Decision: Accept (poster)