Translation-Enhanced Multilingual Text-to-Image Generation
Year: 2023

Abstract:
Research on text-to-image generation (TTI) still predominantly focuses on the English language due to the lack of annotated imagecaption data in other languages; in the long run, this might widen inequitable access to TTI technology. In this work, we thus investigate multilingual TTI (termed mTTI) and the current potential of neural machine translation (NMT) to bootstrap mTTI systems. We provide two key contributions. 1) Relying on a multilingual multi-modal encoder, we provide a systematic empirical study of standard methods used in cross-lingual NLP when applied to mTTI: TRANSLATE TRAIN, TRANSLATE TEST, and ZERO-SHOT TRANSFER. 2) We propose Ensemble Adapter (ENSAD), a novel parameter-efficient approach that learns to weigh and consolidate the multilingual text knowledge within the mTTI framework, mitigating the language gap and thus improving mTTI performance. Our evaluations on standard mTTI datasets COCO-CN, Multi30K Task2, and LAION-5B demonstrate the potential of translation-enhanced mTTI systems and also validate the benefits of the proposed ENSAD which derives consistent gains across all datasets. Further investigations on model variants, ablation studies, and qualitative analyses provide additional insights on the inner workings of the proposed mTTI approaches.

heading: 1 Introduction and Motivation
text: Text-to-Image Generation (TTI) is an emerging yet rapidly growing area, owing its recent progress to ever-growing deep generative models, largerscale multi-modal datasets, and increasing computational resources. The success of recent TTI work is impressive; e.g., it is possible to synthesise not only high-resolution complex scenes (Ramesh et al., 2022; Rombach et al., 2022), but also sur-
∗This work has been done during the author’s internship at Amazon Alexa AI.
realist and ‘aesthetics-aware’ paintings (Gallego, 2022).
However, current models are made and deployed almost exclusively for the English language (EN). This is primarily due to the lack of annotated imagecaption data in other languages, which might result in inequitable access to TTI technology in the long run, especially for low-resource languages (Blasi et al., 2022). Hiring human annotators to write high-quality image descriptions is time-consuming and expensive; ‘gold standard’ data, if it exists at all, is thus typically used for evaluation purposes only (Lan et al., 2017; Aggarwal and Kale, 2020).
Even if we put the crucial concerns of data scarcity aside, training state-of-the-art (SotA) TTI models from scratch for each language is technically infeasible and impractical: it would consume massive computational resources, exceeding the capabilities of many research labs (Ramesh et al., 2021; Saharia et al., 2022) and raising concerns of its environmental impact (Schwartz et al., 2020).1 Therefore, in this work, we focus on multilingual TTI (mTTI) through the optics of NLP’s crosslingual transfer learning methods, leaning on the reasonable assumption of having abundant imagetext pairs in English (and/or a pretrained EN TTI model), but only limited gold-standard data for fine-tuning and evaluation in a target language.2
In particular, we investigate the role of crosslingual transfer and (neural) machine translation (MT) in bootstrapping mTTI, and we focus on two crucial research questions. (RQ1) Are standard MT-based cross-lingual transfer methods feasible for mTTI, and how do they compare with standard
1For instance, DALL-E (Ramesh et al., 2021) is trained on 1, 024 × 16GB NVIDIA® V100 GPUs for a total of 430,000 updates. DALL-E Mega, an attempt to reproduce DALL-E’s results, reports an estimated emission of 18, 013.47-kg CO2equivalents, training on a TPU v3-256 (128×TPU v3 chips) for 56 days. The estimation is based on a publicly available machine learning emissions calculator (Luccioni et al., 2019).
2A more detailed discussion on data sources, data availability and scarcity is provided in Appendix B.
9174
zero-shot cross-lingual transfer methods? (RQ2) Is it possible to enhance zero-shot cross-lingual transfer relying on (ensembles of) MT-generated output for improved mTTI?
Our experiments and core findings are based on several mTTI benchmarks. First, we use the standard and publicly available COCO-CN (Li et al., 2019) and Multi30K (Elliott et al., 2016), and we also build a new dataset for Finnish as a lower-resource language from LAION-5B (Schuhmann et al., 2022). Regarding RQ1, we then conduct a systematic empirical study comparing the standard cross-lingual transfer methods: TRANSLATE TRAIN, TRANSLATE TEST, and ZERO-SHOT TRANSFER. Our main results indicate that TRANSLATE TRAIN achieves the best performance, followed by ZERO-SHOT TRANSFER which outperforms TRANSLATE TEST.
Regarding RQ2, we aim to combine MT-based and zero-shot cross-lingual transfer via fast and parameter-efficient fine-tuning. Inspired by the speech processing literature where a list of Automatic Speech Recognition (ASR) hypotheses can be jointly considered for downstream tasks (Ganesan et al., 2021; Liu et al., 2021) to alleviate the misrecognition of ASR systems, we propose a module within our mTTI framework termed Ensemble Adapter (ENSAD). It fuses the text encodings of ‘non-English’ text input and a set of its translations to English. Additionally inspired by Ponti et al. (2021), the idea is to combine the knowledge from multiple translations to mitigate potential translation errors, and that way boost cross-lingual transfer for mTTI.
Our proposed method derives robust gains across all evaluation datasets. Besides offering SotA mTTI performance, the introduced ENSAD component also adds only 0.1% dedicated extra parameters (relative to the full mTTI model size) per each supported target language. Put simply, the use of ENSAD increases the portability of our mTTI framework through quick and parameter-efficient adaptation to new languages. The resources of our work are available at https://www.amazon.sci ence/code-and-datasets/translation-enh anced-multilingual-text-to-image-gener ation.

heading: 2 Related Work
text: Text-to-Image Generation. There are generally two categories of standard TTI setups: 1) a super-
vised setup, where gold standard training and test data are from the same domain (e.g., both from MS-COCO); and 2) a zero-shot setup, where there is a domain difference between the training data (typically large-scale noisy Web-crawled data) and the high-quality test data (typically manually constructed). GAN-based models are common in supervised TTI setups (Reed et al., 2016; Xu et al., 2018; Zhu et al., 2019): they still hold the SotA results, offering smaller model sizes and faster image generation speed (Zhang et al., 2021; Tao et al., 2022; Zhou et al., 2022). GigaGAN (Kang et al., 2023), a recent attempt to scale up GAN models, achieves fairly strong and competitive zero-shot TTI performance. However, in the zero-shot setup, large Vector Quantised Variational Autoencoder (VQVAE)-based models (Ramesh et al., 2021; Crowson et al., 2022; Gafni et al., 2022) and large diffusion models (Nichol et al., 2022; Ramesh et al., 2022; Saharia et al., 2022) play the leading role and offer the best performance.
Multilingual and Non-EN TTI. Research on mTTI and non-EN TTI is currently limited and only in its infancy. Cogview is a large VQVAE-based Chinese TTI model with training data partly from crawling Chinese websites and social media platforms, and partly from translating EN data (Ding et al., 2021). ruDALL-E is a VQVAE-based Russian TTI model recreating DALL-E (Ramesh et al., 2021) with training data translated from EN data.3
To the best of our knowledge, there are only two existing papers attempting multilingual or crosslingual TTI. Zhang et al. (2022) align two monolingual text encoders, one for the source and the other for the target language, with a fixed image generator pretrained on the source language (i.e., EN). Jung et al. (2022) take a step further, relying on a multilingual text encoder that supports more languages simultaneously.
We note several crucial differences to the prior work. 1) The two papers are based on earlier TTI models (Xu et al., 2018), which are now largely surpassed by recent SotA models (Zhou et al., 2022). 2) Their model designs are tied to the model of Xu et al. (2018) and cannot be easily adapted to the latest SotA TTI models. 3) They use traditional LSTM text encoders enhanced by mono-modal BERT features, while SotA TTI models (Zhou et al., 2022; Saharia et al., 2022; Rombach et al.,
3https://rudalle.ru/; ruDALL-E has not released an accompanying paper yet, but a technical blog is available.
2022) use the multi-modal CLIP model (Radford et al., 2021). Therefore, we neither adopt them as baselines nor try to adapt them for our use, also taking into account the difficulty of replicating the prior work as no code has been released to date. In contrast, our work relies on the mCLIP text encoder (Carlsson et al., 2022), the multilingual version of CLIP, and is developed based on LAFITE (Zhou et al., 2022), a SotA TTI model. In fact, as shown later in our work, training an English TTI model using mCLIP without any further tuning can already realise zero-shot mTTI, similar to what has been attempted by Jung et al. (2022).
Translation-Based Cross-lingual Transfer. Machine translation (MT) at both lexical level and sentence level has been successfully used for crosslingual transfer learning in NLP, where TRANSLATE TRAIN and TRANSLATE TEST usually serve as strong baselines for downstream tasks (Conneau et al., 2018; Glavaš et al., 2019; Hu et al., 2020; Ponti et al., 2021; Li et al., 2022a,b). In addition, MT is used to generate sentence pairs for training multilingual multi-modal models (Zhou et al., 2021; Carlsson et al., 2022). However, MT is still largely underexplored and underutilised for mTTI. In this work, we analyse the potential of MT to enhance multilingual and cross-lingual TTI.

heading: 3 Methodology
text: In what follows in this section, we first introduce our base mLAFITE model and three baseline approaches for mTTI (§3.1). Next, we propose an Ensemble Adapter module that can work in synergy with the pretrained mLAFITE model to improve mTTI performance (§3.2). Finally, we describe how we train our Ensemble Adapter and formulate our loss functions (§3.3).

heading: 3.1 mLAFITE and Baselines
text: For easier deployment and comparison of different cross-lingual transfer methods, our work focuses on the relatively lightweight GAN-based models, which are faster to train and evaluate compared with VQVAE-based models and large diffusion models (see §2). In particular, we adopt LAFITE (Zhou et al., 2022), a SotA GAN-based English TTI model, as our starting point. To unlock its multilingual capabilities, we replace its Englishonly CLIP text encoder (Radford et al., 2021) with mCLIP (Carlsson et al., 2022), which is already pretrained to align the sentence representation spaces
of 68 languages.4
There are three common categories of crosslingual transfer approaches which we apply to mTTI and adopt as our principal baselines:
TRANSLATE TRAIN. We translate all the captions from the English training set (e.g., COCO) into a (non-EN) target language (L) relying on an MT system. We then train a LAFITE TTI model in the target language from scratch, relying on mCLIP as the text encoder.5 At inference, an L sentence is directly fed into the target-language TTI model.
The other two approaches instead rely on a TTI model pretrained with English data, and they do not require further tuning with captions in the target languages. As our first step, we pretrain an mCLIPbased LAFITE model (we call it mLAFITE for brevity) from scratch.
TRANSLATE TEST. At inference, we first translate a caption in L into EN via MT and the EN translation then serves as mLAFITE’s input.
ZERO-SHOT TRANSFER. Since mCLIP is a multilingual sentence encoder, text in L can be directly fed to our mLAFITE for TTI without any extra fine-tuning.

heading: 3.2 mLAFITE with Ensemble Adapter
text: We now propose an attention-based Ensemble Adapter (ENSAD) module that aims to improve mTTI via leveraging knowledge from multiple translations of the same input. The full pipeline and how ENSAD extends the base mLAFITE model are illustrated in Figure 1. Given an input sentence in language L, L̸=EN, we first use any (N)MT system to sample a set of EN translations. We then deploy the ENSAD module between the mCLIP text encoder and the TTI generator to fuse the mCLIPextracted embeddings, bridging the EN-L language domain gap. The adapter can be trained with only a small set of image-L text pairs while mCLIP and the TTI generator networks are kept frozen.
4mCLIP is derived by fine-tuning a pretrained XLM-R model (Carlsson et al., 2022; Conneau et al., 2020), and it does not directly depend on parallel corpora or multilingual imagetext data. The work uses NMT to generate ‘silver’-quality EN∗ sentence pairs and then directly aligns the CLIP-extracted EN representations and mCLIP’s multilingual sentence representations of the NMT-generated data. Both CLIP and mCLIP use a shared CLIP visual encoder.
5We use mCLIP rather than monolingual CLIP since it is infeasible for most languages. Only several high-resource languages have publicly available monolingual models. For fair cross-language comparisons, we leverage the same mCLIP text encoder in all our experiments.
Formally, we use x0 to denote the L input text, while {x1, x2, ..., xm} is a set of m EN translations of the L input text. The fixed mCLIP encoder extracts their respective (l2-normalised) ddimensional sentence embeddings, yielding the matrix H = (h0,h1, ...,hm) ∈ Rd×(m+1). Then, our proposed ENSAD learns to fuse these sentence encodings from H . We define the query (q), key (K), and value (V ) inputs of our attention as:
q = h0, (1)
K = (h1,h2, ...,hm), (2)
V = (h1 − h0,h2 − h0, ...,hm − h0). (3)
Note that {h0,h1, ...,hm} are all close to each other in the mCLIP representation space. Therefore, to focus on the ‘additional information’ contained in the EN translations, we take the difference between hi, i > 0 and h0 as in Eq. (3).6 The calculation of attention scores is then based on the standard additive attention (Bahdanau et al., 2015):
A = W qq1T +W kK +W vV + b1T, (4)
sT = softmax(W p(tanh(A))). (5)
ENSAD’s hidden size is dhid; W q,W k,W v ∈ Rdhid×d are respective mappings for query, key, and value inputs; b ∈ Rdhid is the bias, and W p ∈ R1×dhid is a final projection matrix for deriving the attention scores. Then, the context vector is an attention-guided summarisation of V . ENSAD’s
6We adopt the simple mean pooling of {h0,h1, ...,hm} as an additional baseline with results in §6.2. We also tried multi-head self-attention (Vaswani et al., 2017), where Q = K = V = H , which, however, showed inferior performance in our preliminary experiments.
final output is the linear combination of h0 and the context vector, computed as follows:
V o = (1− α)V + α · tanh(W oV ), (6) c = V os, (7)
h̃ = ENSAD(H) = (1− α)q+ α · c, (8)
where W o ∈ Rd×d is the output mapping, and α is an interpolation hyperparameter. We also l2normalise the outputs of Eqs. (3), (7), (8), as well as the tanh(W oV ) term in Eq. (6).

heading: 3.3 Contrastive Adversarial Training
text: Our Generator (G) and Discriminator (D) network structures and the pretraining process of the base mLAFITE model all follow LAFITE’s original implementation for supervised TTI. As illustrated in Figure 1, we take the pretrained mLAFITE and insert the ENSAD between mCLIP and G. We then adversarially train ENSAD and D iteratively while mCLIP and G are kept frozen.7 Additionally, we propose to optimise a novel contrastive objective aligning the D-extracted real image and fake (synthesised) image features in adversarial training.
The (m)LAFITE GAN framework is adapted from the popular unconditional StyleGAN2 framework (Karras et al., 2020b) which features a redesigned adaptive instance normalization mechanism (Huang and Belongie, 2017) in G: it enables the unconditional channel-wise ‘style information’ (e.g., pose, lighting, background style) to control G’s image synthesis backbone (convolution and upsampling layers). The ‘style information’ is derived
7We also tried freezing D but this results in inferior performance in our preliminary investigation.
as follows: a random noise z is sampled from the standard Gaussian distribution N (0, I) and transformed into a so-called unconditional StyleSpace, which is proven to be a well-disentangled intermediate latent space (Wu et al., 2021).8 LAFITE further proposes to inject text-conditioning information into the StyleSpace via a series of non-linear and affine mappings. In our pipeline, G takes our ENSAD-gathered feature h̃ and noise z, and it then outputs a fake image: Ifake = G(h̃, z).
The discriminator has a characteristic ‘twobranch’ design: 1) D is in essence a convolutional image encoder, producing fD(I), a d-dim image feature for any real or fake (i.e., synthesised) input image I; 2) D also predicts if I is real or fake based on both I and h̃, where the prediction (a scalar output) is denoted as D(I, h̃) = Ds(I) + h̃T fD(I). This is realised via adding two affine transformations on top of a shared visual backbone for deriving fD(I) and Ds(I), respectively. We then define the adversarial (AD) losses for ENSAD and D following LAFITE:
LENSADAD = − 1
n
n∑
i=1
logσ(D(Ifakei , h̃i)), (9)
LDAD =− 1
n
n∑
i=1
logσ(D(Ireali , h̃i))
− 1 n
n∑
i=1
log(1− σ(D(Ifakei , h̃i))). (10)
n is the batch size, and σ(·) is the sigmoid function. We propose an auxiliary contrastive loss, aligning the discriminator-extracted Ifake and Ireal features, computed as follows:
si,j =cos(fD(Ireali ), fD(Ifakej )), (11)
LCL =− 1
n
n∑
i=1
log exp(si,i/τ)∑n j=1 exp(sj,i/τ) . (12)
cos(·) calculates the cosine similarity, and τ is the temperature.
In the original LAFITE paper, there are already two auxiliary contrastive losses: 1) LGCL aligns CLIP-extracted image features of Ifake and the input text embedding, i.e., h̃ in our case; 2) LDCL aligns fD(I) with its associated h̃.9 In our preliminary experiments, we found that LGCL was not
8The transformation includes a shared 8-layer MLP and a dedicated affine mapping per each generation layer. We refer the reader to the original work for further technical details.
9As with LAFITE’s original implementation, fD(I) is fD(Ifake) in LENSAD and fD(Ireal) in LD .
useful for ENSAD, so we completely remove it.10 Our final losses for training ENSAD and D are as follows, with two hyperparameters λ1 and λ2 controlling the weights of contrastive losses:
LENSAD =LENSADAD + λ1 · LCL + λ2 · LDCL, (13) LD =LDAD + λ1 · LCL + λ2 · LDCL. (14)
The full training process is also summarised in Algorithm 1, available in Appendix C. Note that the use of ENSAD introduces only up to 0.1% extra parameters per each target language relative to the full model size. This parameter efficiency boosts the portability of our mTTI framework, enabling quick and efficient adaptation to new languages.

heading: 4 Datasets
text: mLAFITE pretraining is based on the MSCOCO (Chen et al., 2015) training set comprising 82, 783 images, where each image is associated with 5 EN captions. 10% of the training set is held out as our dev set, and the rest is used for training. MS-COCO also provides a validation set (40, 504 images), frequently used for TTI evaluation.
For mTTI, we choose evaluation datasets that satisfy the following criteria: a) no overlap between images in the test set and images used in pretraining; b) the test set includes at least 5K images;11 c) the captions are human-written descriptions and not (manual or MT-derived) translations from EN captions.12 Based on these requirements, we select three ‘non-EN’ datasets, outlined in what follows.
COCO-CN (Li et al., 2019) provides Chinese (ZH) captions (i.e., human descriptions) for 20, 341 MSCOCO images. 6, 748 of them are from the COCO validation set not seen during mLAFITE pretraining; we thus use them as our test set. We randomly sample 20% of the rest as our dev set (2, 718), and the training set has 10, 875 images. Each image has
10The equations for the other two CL losses are similar to Eq. (12). For brevity, we skip the details and refer the reader to the original LAFITE paper.
11Previous work proved that small test set sizes result in biases and unreliable TTI evaluation (Chong and Forsyth, 2020); therefore, TTI work typically adopts test sets with more than 5K images (Zhou et al., 2022; Ramesh et al., 2021). For instance, the most common EN TTI data for evaluation is the MS-COCO validation set that contains 40K images. The smallest general-domain test set in Zhou et al. (2022) is LN-COCO (Pont-Tuset et al., 2020) containing∼ 5K images.
12Human-written descriptions are more realistic for realworld non-EN users, and translations from EN captions can cause unexpected ‘translationese’ bias (Elliott et al., 2016; van Miltenburg et al., 2017; Bugliarello et al., 2022).
only one ZH caption. COCO-CN additionally offers 5, 000 ZH sentences manually translated from EN captions; we only use the corresponding ENZH sentence pairs to calculate BLEU scores for comparing different MT systems.
Multi30K Task2 (Elliott et al., 2016, 2017) has 5 German (DE) captions (human descriptions) for each of 31, 014 Flickr30K (Young et al., 2014) images. We randomly sample and keep one caption per each image.13 We randomly split the data into train, dev, and test sets spanning 10, 000, 2, 000, and 19, 014 images, respectively.
LAION-5B (Schuhmann et al., 2022) is a largescale Web-crawled vision-language dataset with 5 billion image-text pairs covering 100+ languages. We focus on Finnish (FI) as a lower-resource language for our evaluation. Unlike carefully annotated COCO-CN and Multi30K, LAION-5B’s data are noisy, so we rely on massive filtering to select relatively high-quality data. The full data creation process for FI is provided in Appendix D.
The final dataset comprises training, development and test portions with 10, 000, 2, 000, and 18, 000 image-text pairs, respectively. Our manual inspection of the final dataset indicates that it is of acceptable quality although having its own characteristics (Appendix D) but the quality in general still cannot match COCO-CN or Multi30K. We use the data in our main experiments 1) as an initial trial to extend TTI evaluation to ‘non-COCO-style’ captions and another language and 2) for comparative analyses with COCO-CN and Multi30K.
Supplementary Dataset: IGLUE. In order to further widen the set of target languages, we also experiment with IGLUE xFlickr&CO (Bugliarello et al., 2022). It provides 2K images, where one half comes from the MS-COCO validation set and the other half from Multi30K with associated human descriptions in 5 additional languages: Spanish (ES), Indonesian (ID), Japanese (JA), Russian (RU), and Turkish (TR). Since IGLUE does not offer a training set, we use it only for RQ1-related experiments. Although IGLUE does not comply with our criterion b) above, we use it to extend our empirical analyses to more languages.
Table 6 in Appendix A provides a full and sys-
13All our non-EN TTI datasets uniformly have one caption for each image. This setup is also more realistic since realworld users only need to input a single sentence into a TTI model. For mLAFITE pretraining, however, as with all other related work, all the COCO captions are used.
tematic overview of languages and data statistics used in this work.

heading: 5 Experimental Setup
text: In what follows, we outline our experimental setups and choices related to the two core RQs from §1. We also show details concerning our mLAFITE pretraining, side experiments (most are RQ2-related), and evaluation metric.
mLAFITE Pretraining. All methods for mTTI are implemented based on our pretrained mLAFITE model, which is trained with 8×16GB V100 GPUs for 75 hours (i.e., 40 million data points sampled from the training set). Contrastive loss weights and other hyper-parameters follow the original LAFITE setup (Zhou et al., 2022).14 For fair comparisons, we use the same mCLIP text encoder for all our RQ1 and RQ2 experiments.
RQ1 Experiments. On COCO-CN, we compare four widely used MT systems: Amazon Translate15, a SotA commercial MT software, and three SotA Transformer-based NMT models developed in an academic context including Marian (Tiedemann and Thottingal, 2020; Junczys-Dowmunt et al., 2018), mBART50 (Liu et al., 2020; Tang et al., 2021), and M2M100 (Fan et al., 2021). We leverage them to generate the 1-best translations for TRANSLATE TRAIN and TRANSLATE TEST, and we also compare the BLEU scores of the MT systems against the TTI performance. Note that training a TRANSLATE TRAIN TTI model from scratch for each of the MT systems also takes 75 hours; our TRANSLATE TRAIN experiments thus do not extend to other datasets beyond COCO-CN due to the high computational cost.
Given the considerations above along with preliminary evaluations on COCO-CN which showed that Marian outperforms mBART50 and M2M100, for the other datasets we focus on comparing the Marian-based TRANSLATE TEST with ZEROSHOT TRANSFER.
RQ2 Experiments. RQ2 further studies the effectiveness of the proposed ENSAD module; see §3 and Figure 1. We select Marian as the NMT back-
14The original LAFITE model is based on EN CLIP and is trained with 25 million samples on 4×16GB V100 GPUs. Our mLAFITE model uses mCLIP, which is of the same dimensionality as CLIP and keeps the values of all the other relevant hyper-parameters, such as batch per GPU and learning rate, from the original work.
15https://aws.amazon.com/translate/
bone16 and sample m EN translations per each input sentence in the input language L.17 To compare with ENSAD (with the frozen mLAFITE generator), we also propose and experiment with several insightful and simple baselines (without the use of ENSAD) in addition to the RQ1 baselines: 1) we try standard mean-pooling as a simple ensembling baseline directly on mLAFITE; 2) we finetune G using the original non-EN captions;18 3) we fine-tune G using mean-pooled text features. Finally, we also investigate variants which combine ENSAD with the tunable generator G to check if further gains can be achieved.19
Training for RQ2 experiments is conducted on 8×V100 GPUs with a batch size per GPU of 16 for about 7 hours (i.e., a total of 2 million data points sampled from the respective training sets). We use Adam optimiser (Kingma and Ba, 2014) with a learning rate of 5e-4 and betas of (0, 0.99). For the generator-tuning baselines, their contrastive loss setups completely follow the original LAFITE (Zhou et al., 2022). In our ENSAD experiments, λ1=4 and λ2=2. Other hyper-parameters are as follows: the NMT beam size is 12, NMT temperature is 2.0, images are scaled to resolution 256× 256, m=12, d=512, dhid=256, and τ=0.5. In addition, we fuse 10% and 1% standard Gaussian noise into h0 and hi(1 ≤ i ≤ m) respectively as a data augmentation ‘trick’. The hyper-parameters are tuned on our dev split of COCO-CN with details in Appendix G. The same set of hyper-parameters is also adopted for the other two datasets.
Side Experiments. Besides the main RQ1 and RQ2 experiments, we also conduct a series of side analyses focused on ENSAD. They span 1) the impact of the number of EN translations m, 2) the impact of the interpolation hyperparameter α, and 3) robustness tests. We also conduct 4) ablation studies to validate the effectiveness of different components, and 5) present generated images and ENSAD attention scores.
16Amazon Translate’s API can only return a single 1-best translation; it thus cannot be used for ENSAD experiments.
17We adopt the common ‘beam search multinomial sampling’ for sampling EN translations.
18G is tuned adversarially following the original training setup of (m)LAFITE.
19We first fine-tune G and then train ENSAD with the fine-tuned G but still use the discriminator of our pretrained mLAFITE to alleviate its overfitting (Karras et al., 2020a). We also tried 1) training ENSAD first and then fine-tuning G and 2) training ENSAD together with G, but they both derive suboptimal results in our preliminary investigation.
Evaluation Metric. Following Zhou et al. (2022) and Ramesh et al. (2021), we report the Fréchet Inception Distance (FID) (Heusel et al., 2017) computed with 30, 000 synthesised images generated using randomly sampled test set texts against test set ground-truth images, which is the most authoritative machine evaluation metric for TTI so far.20

heading: 6 Results and Discussion
text: The main results are structured around the two central RQs from §1, discussed in §6.1 and §6.2.

heading: 6.1 RQ1: Results and Analyses
text: Comparison of Three Baselines. The results of TRANSLATE TRAIN, TRANSLATE TEST, and ZERO-SHOT TRANSFER on COCO-CN are summarised in Table 1. While all three methods use mCLIP, TRANSLATE TEST and ZERO-SHOT TRANSFER are based on a pretrained EN mLAFITE and do not require any further tuning. TRANSLATE TRAIN achieves the best FID scores; however, it requires training from scratch with translated L captions (see §3.1 and §5). Since MS-COCO provides ground-truth human-written EN captions for COCO-CN images, and Multi30K Task2 also provides EN human descriptions, we directly feed the EN captions to mLAFITE and report the FID scores as an upper bound (see the first row of each of Tables 1 and 2).21
The scores in Tables 1 and 2 show that ZEROSHOT TRANSFER outperforms TRANSLATE TEST, demonstrating the strong capability of the multilingual mCLIP text encoder. TRANSLATE TEST compares unfavourably to other methods, revealing the gap between EN translations and the ground-truth EN human descriptions (e.g., translation errors, ‘translationese’ bias). We further extend the comparison to five more languages from the IGLUE dataset, and the results from Table 7 in Appendix E corroborate the finding that ZERO-SHOT TRANSFER generally outperforms TRANSLATE TEST.
Comparison of MT Systems. We compare the performance of the four MT systems on COCO-CN and also report their BLEU scores on the additional
20Inception Score (IS) (Salimans et al., 2016) is another common evaluation metric for TTI, which is, to some extent, superseded by FID (Heusel et al., 2017; Borji, 2022). Moreover, IS gives misleading results when applied to datasets other than ImageNet (Barratt and Sharma, 2018), and is especially non-fitting for unannotated LAION-5B images.
21For fair comparisons, we keep only one ground-truth EN caption for each image.
5K sentence pairs. Table 1, as expected, reveals that the commercial Amazon Translate system offers much stronger MT performance than the three academic NMT systems in terms of BLEU. Concerning mTTI, Amazon Translate is the best system with the TRANSLATE TEST approach category and ranks second with TRANSLATE TRAIN. Interestingly, there are some salient discrepancies between BLEU-based versus TTI-based system rankings. For example, Marian ranks second in TRANSLATE TEST and is the best system with TRANSLATE TRAIN, although its MT performance underperforms both Amazon Translate and mBART50. We speculate that this might be due to the pretraining specifics of mCLIP, where Marian-generated pseudo-parallel sentence pairs were used (Carlsson et al., 2022).
In TRANSLATE TEST, M2M100 obtains the lowest ZH→EN BLEU score and also achieves the worst TTI performance. However, mBART50 and M2M100 have close EN→ZH BLEU scores in TRANSLATE TRAIN, and a small edge in BLEU cannot guarantee a better TTI performance. We additionally compare Marian and Amazon Translate for TRANSLATE TEST in Tables 2 and 7 (Appendix E) on other languages and datasets, which further validate the core findings.

heading: 6.2 RQ2: Results and Analyses
text: Effectiveness of ENSAD. The main results are summarised in Table 3. For all methods except
‘Ground-Truth EN Captions’, the language gap (with EN captions for mLAFITE pretraining) always exists since the text input is in language L. When there is no image domain gap (i.e., for COCO-CN), ENSAD without tuning G achieves the best score, surpassing also the TRANSLATE TRAIN baseline (cf. Table 1), and the absolute score also mitigates the gap to the upper-bound set by ‘Ground-Truth EN Captions’. With image domain gap present (i.e., DE and FI), training ENSAD (with frozen G) still shows a small edge over fine-tuning G (without ENSAD) for DE; however, for the noisier LAION-5B data, fine-tuning G is more useful. However, for both DE and FI, the best results are always achieved when ENSAD is leveraged, validating its usefulness combined with parameter efficiency. For example, ENSAD with G frozen consistently outperforms ZERO-SHOT TRANSFER while introducing only 0.1% extra parameters. Our robustness tests repeating ENSAD (Frozen G) experiments on COCO-CN with different random seeds further corroborate these findings (the deviation of FID is 0.04), with a short summary in Appendix F.
Variants of ENSAD. We further investigate the impact of crucial design choices and hyper-parameters in ENSAD such as m, α, and V (see Eq. (3)) respectively on the final TTI performance. The results of different variants are provided in Table 4. They indicate that increasing the number of translations m seems to be conducive to downstream TTI performance. In addition, when V = K, the FID score worsens, demonstrating the usefulness of the V variant as formulated by Eq. (3). Finally, the TTI performance deteriorates when α > 0.2, showing that h0 should still be the main component of h̃, and ENSAD provides auxiliary information (i.e., a translation-based enhancement).
Ablation Study. We now study the usefulness of two used contrastive losses: 1) our proposed LCL and 2) LDCL inherited from LAFITE. The results in Table 5 show that removing LCL causes a noticeable performance drop (increased FID). However, removing LDCL has only a minor impact on the FID score. When removing both CL losses, the adversarial losses alone produce an FID score of 14.82. We also additionally try the CL loss setup of the original LAFITE and find that the setup is detrimental to the training of ENSAD, producing a worse FID score than using the adversarial losses alone.
TTI Examples and Attention Scores. Finally, we refer the reader to Appendix H where we present images synthesised with TRANSLATE TEST, ZEROSHOT TRANSFER, and our ENSAD models and where we also show the ENSAD attention scores. The differences between images are subtle and we were unable to find a clear pattern that links high attention scores with particular translations.

heading: 7 Conclusion
text: This work is one of the first investigations of multilingual and cross-lingual text-to-image generation (TTI), with a particular focus on investigating the use of machine translation (MT) for the task. We systematically compared standard cross-lingual transfer approaches TRANSLATE TRAIN, TRANSLATE TEST and ZERO-SHOT TRANSFER in the context of TTI and also studied the differences over MT systems. We then proposed a novel Ensemble Adapter (ENSAD) method that leverages multiple translations to further improve the TTI
performance, with strong and consistent gains reported across a series of standard TTI benchmarks in different languages.
Limitations
First, we again emphasise that the lack of highquality non-English image-caption pairs is a primary obstacle to wider-scale multilingual and cross-lingual TTI investigations. We hope that researchers in the future can construct and release more high-quality vision-language data for different languages, especially for low-resource ones.
Second, our work uses 512-dim ‘XLM-R Large Vit-B/32’ mCLIP22 and is based on the StyleGAN2 framework (Karras et al., 2020b). Since the main focus of our work is to realise multilingual and cross-lingual TTI and enable fair comparisons across different models and approaches, we compare all proposed and baseline methods with the same mCLIP text encoder and the GAN framework. However, for readers and potential users interested in ‘chasing’ stronger absolute FID scores, we speculate that the larger 640-dim ‘XLM-R Large VitB/16+’ mCLIP text encoder and the more recent StyleGAN3 (Karras et al., 2021) can be helpful.
Third, we notice that in addition to LAFITE, several state-of-the-art large diffusion models such as those from Saharia et al. (2022) and Rombach et al. (2022) also use CLIP to condition image generation on text input. This means that we could be able to derive multilingual diffusion models for mTTI also by replacing CLIP with mCLIP and enhance the mTTI performance with our proposed ENSAD (of course, we would need to redesign our loss functions). However, due to limited computational resources, we leave it to future work.
Fourth, the ENSAD boosts cross-lingual transfer for TTI by combining the knowledge from multiple translations, which can mitigate potential translation errors. Our work does not demonstrate if ENSAD is applicable and adaptable to downstream cross-lingual tasks besides TTI. It is because 1) downstream tasks other than TTI are out of the scope of this work and 2) adapting ENSAD to different tasks will require redesign of model structures and losses catering to the characteristics of each downstream task, making us believe it is not proper to expand the topic and include everything in a single piece of work. Therefore, we also leave this to future work.
22https://github.com/FreddeFrallan/Multilingual-CLIP
Ethics Statement
The datasets involved in our experiments are publicly available and widely used, and it is quite common to train text-to-image generation models on publicly available data. To the best of our knowledge, the ethical risk is minimal. For privacy concerns, we do not present images with human faces and captions with real human names in the paper, and we will not release material that may contain any sensitive information.

heading: Acknowledgements
text: We would like to thank 1) all members of the Amazon Alexa Translations Science Team for helpful discussions and valuable comments during the weekly group meetings, 2) Yufan Zhou, the author of LAFITE, who kindly responded to our questions concerning LAFITE’s technical details on Github, and 3) the anonymous reviewers for their feedback.
Ivan Vulić is supported by a personal Royal Society University Research Fellowship ‘Inclusive and Sustainable Language Technology for a Truly Multilingual World’ (no 221137; 2022–).

heading: A Data Statistics and Languages
text: In Table 6, we summarise the data statistics and languages covered in our experiments.

heading: B Additional Discussion on Data Sources
text: Even without human-annotated image descriptions, there are two possible ways to derive captions for a target language L.
First, we could translate EN captions into L manually (still costly) or via machine translation. Our TRANSLATE TRAIN baseline (see §3) derives training data via machine translation and trains an L TTI model from scratch. One main disadvantage of this approach is that it incurs huge training costs. While translations can be used as training data, we are conservative about using translated captions for TTI evaluation which can cause unexpected bias (Elliott et al., 2016; van Miltenburg et al., 2017; Bugliarello et al., 2022).
Second, it is possible to use cheaper but noisy Web-crawled visual-language data. For example, the recently released LAION-5B dataset (Schuhmann et al., 2022) has 5 billion image-text pairs for 100+ languages. There are previous examples that successfully trained SotA EN TTI models with Web-crawled data, such as large VQVAE-based models and diffusion models. The models described in Ramesh et al. (2021), Nichol et al. (2022) and Ramesh et al. (2022) are trained on EN largescale Web-crawled data, but are eventually also tested on the gold-standard MS-COCO validation set. In our work, in addition to two gold-standard datasets, we also try to build on our own a smallscale dataset for both training and evaluation by filtering relatively good-quality image-text pairs from a subset of the noisy LAION-5B data (details in §4). Training non-EN TTI models from scratch with large-scale Web-crawled data such as LAION-5B is out of the scope of our work, and we focus on crosslingual transfer learning setups with limited L data. As mentioned in §1, this is to a large extent due to concerns about huge computational costs for training TTI models. Moreover, there are circa 7, 000 languages worldwide (Lewis, 2009), and for lowresource languages not covered in LAION-5B’s 100+ languages, cross-lingual transfer learning approaches would still be the first choice. Furthermore, the number of EN texts in LAION-5B is more than the total amount of texts from its 100+ non-EN texts. Making full use of the huge amount of EN image-text pairs via cross-lingual transfer learning
Algorithm 1 Supervised Training of Ensemble Adapter 1: Input: An image-text dataset {x0i , Ireali }Ni=1 2: Derive Hi for each x0i with NMT and mCLIP 3: while not converge do: 4: Sample mini-batch {Hi, Ireali }ni=1; 5: Sample random noise {zi}ni=1 ∼ N (0, I); 6: ENSAD forward pass h̃i←ENSAD(Hi); 7: Synthesise fake image Ifakei ←G(h̃i, z) 8: Feed (h̃i, Ireal) and (h̃i, Ifake) to D respectively; 9: Update ENSAD with Eq. (13); 10: Update D with Eq. (14); 11: end while
might be beneficial for other languages. Therefore, we think that cross-lingual transfer learning in relatively low-resource setups for multilingual TTI is a critical and valuable research topic.

heading: C The Detailed Training Process of
text: ENSAD
We summarise the training process of our ENSAD method (see §3) in Algorithm 1.

heading: D Deriving LAION-5B Dataset for Finnish
text: We download circa 5.1 million image-caption pairs from the FI category of LAION-5B. Since the Webcrawled data are noisy, we apply several filtering steps: 1) since our images will be scaled to resolution 256 × 256, to avoid distortion we keep only images with their width-height ratio between 0.5 and 2; 2) we keep captions with a minimum length of 8 words, which is also a requirement of MSCOCO (Chen et al., 2015) in its data annotation; 3) we use the langdetect library23 to remove texts misclassified into the LAION-5B FI category and make sure the texts left are actually in Finnish; 4) we keep captions with one ending period ‘.’.24 After these steps, 239K pairs are left, and we calculate mCLIP scores (cosine similarities between mCLIP-extracted text and image features) for all the pairs and keep the 30K highest-ranking pairs as the final dataset. We randomly split the data into training, development and test portions with 10, 000, 2, 000, and 18, 000 pairs, respectively.
We ‘sanity-check’ 50 randomly sampled instances from our filtered data and find that, in most cases, the text matches the image content. But there are a small number of exceptional cases where the
23https://pypi.org/project/langdetect/ 24In our initial trial, we found that, among the highest mCLIP-scored 30K pairs, most captions which do not end with ‘.’ are noisy short ads.
text contains extra information beyond the image content itself (e.g., event descriptions). Overall, the quality of our FI data still cannot match MS-COCO or Multi30K. Another interesting finding is that LAION-5B captions often use real and concrete names such as ‘Messi’ and ‘the national stadium’ to describe the image content, while MS-COCO and Multi30K tend to use general words such as ‘a man’/‘a football player’ and ‘a stadium’/‘a building’.

heading: E RQ1: Results on IGLUE
text: Table 7 shows additional TTI results on five languages from IGLUE, comparing TRANSLATE TEST (with Marian and Amazon Translate) and ZERO-SHOT TRANSFER baselines.

heading: F Robustness of ENSAD
text: We train the ‘ENSAD (Frozen G)’ model on COCOCN 6 more times (7 times in total) with different random seeds, and for each saved model we run TTI evaluation three times.25 Finally, we get 21 FID results, with min 14.47, max 14.62, mean 14.55, and standard deviation 0.04. Even the worst score of 14.62 outperforms all other baselines on COCO-CN.

heading: G Reproducibility Checklist
text: • TTI Data: the datasets used in our work are all publicly available including MS-COCO26, COCO-CN27, Multi30K Task228, LAION-
25DE and FI have larger test set sizes, so their evaluation results should be more reliable (Chong and Forsyth, 2020).
26https://cocodataset.org 27https://github.com/li-xirong/coco-cn 28https://github.com/multi30k/dataset
5B29, and IGLUE30.
• Parameter Counts: the number of parameters is 655, 873 for our ensemble adapter network, 44, 997, 026 for the generator network, 29, 126, 785 for the discriminator network, 560, 415, 232 for the mCLIP text encoder ‘M-CLIP/XLM-Roberta-Large-Vit-B32’31, and 87, 849, 216 for the CLIP visual encoder ‘ViT-B/32’32.
• Computing Infrastructure: we run our code on an Amazon EC2 P3.16xlarge Instance with 8×16GB Nvidia® Tesla® V100 GPUs, 64×2.30 GHz Intel® Xeon® E5-2686 v4 CPU cores, and 488GB RAM.
• Software: Python 3.7.0, PyTorch 1.12.1, and Transformers 4.21.0.
• Hyperparameter Search: our hyperparameters are tuned on our dev split of COCO-CN. The same hyper-parameters are used for Multi30K and LAION-5B (we also conduct minimal tuning on their dev sets and find that the hyperparameters tuned on COCO-CN are already (near-)optimal in our initial investigation). The learning rate is selected from {5e−5, 2.5e−4, 5e−4, 2.5e−3, 5e−3}, λ1 and λ2 which are weights for contrastive losses from {0.5, 1, 2, 4, 5, 10}, α the interpolation hyperparameter from {0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.5}, and dhid from {32, 64, 128, 256, 512}.
29https://laion.ai/blog/laion-5b 30https://github.com/e-bug/iglue 31https://github.com/FreddeFrallan/Multilingual-CLIP 32https://github.com/openai/CLIP
• Runtime: it takes 75 hours to train an mLAFITE TTI model or a TRANSLATE TRAIN model from scratch, 7 hours to train an ENSAD based on a pretrained mLAFITE, 7.5 hours to fine-tune G (without ENSAD) based on a pretrained mLAFITE, and about 4 minutes to run FID evaluation for our TTI model with ENSAD (data preprocessing, NMT, and mCLIP feature extraction excluded). All experiments and measurements were conducted on 8×16GB V100 GPUs.
• Other Technical Details: we adopt the ‘exponential sharpening’ for all contrastive losses as specified in LAFITE’s supplementary material.33
• Carbon Footprint: we estimate that 1) training an mLAFITE TTI model or a TRANSLATE TRAIN model from scratch can cause the emission of circa 56∼67-kg CO2 equivalents; 2) training an ENSAD model would result in about 5∼6-kg CO2 equivalents. These estimations are based on our computational infrastructure and a publicly available ‘machine learning emissions calculator’ (Luccioni et al., 2019).34

heading: H TTI Examples and Attention Scores
text: H.1 TTI Examples We compare images generated with TRANSLATE TEST, ZERO-SHOT TRANSFER, and our best ENSAD model in Figure 2, where for each TTI method we present two images generated with different random noise inputs as introduced in §3. The ‘Best’ model here refer to our ENSAD model that achieve the best FID scores (bold numbers) in Table 3 respectively for each language, i.e., ‘ENSAD (Frozen G)’ for ZH and ‘ENSAD + Fine-Tune G (L Text)’ for DE and FI. The differences between images generated with different TTI methods are very subtle.
33https://openaccess.thecvf.com/content/CVPR2022/supplemental/Zhou_Towards_LanguageFree_Training_CVPR_2022_supplemental.pdf
34https://mlco2.github.io/impact/#compute
H.2 Attention Scores Table 8 includes the original L input text, EN translations, and their associated ENSAD attention scores (in descending order) corresponding to the images in Figure 2. We did not identify any salient pattern concerning the type of EN translations to which higher ENSAD attention scores are attached.
H.3 Can ENSAD Incorporate Manually Added Information from Translations?
To better understand what kind of information ENSAD extracts from EN translations, we also try to manually add additional information to EN translations (the additional information does not appear in and is not added to the original L input). Of course, this section is for probing purposes only since MT systems are not likely to produce the same translations. We found that when the additional information is added to only several of the 12 EN translations, it can hardly get reflected in the generated image. Here, we show two COCO-CN test set examples in Figure 3 where we add the new information into 12 EN translations simultaneously. In its first and second rows, the original L input is ‘An open laptop is on the table.’ and ‘It’s a clean, but crowded kitchen.’ respectively (translated from the original Chinese captions). We manually add new objects ‘roses’ and ‘fruits’ respectively to all their EN translations as in Table 9. As seen in Figure 3, the roses and fruits do appear in the generated images.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: The Limitations section.

heading: 3 A2. Did you discuss any potential risks of your work?
text: The Ethics Statement section.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1 Introduction and Motivation.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 Methodology, Section 4 Datasets, Section 5 Experimental Setup, and Appendix G Reproducibility Checklist

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 Datasets, Section 5 Experimental Setup, and Appendix G Reproducibility Checklist

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 4 Datasets, Section 5 Experimental Setup, the Ethics Statement section, and Appendix G Reproducibility Checklist.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4 Datasets, Section 5 Experimental Setup, the Ethics Statement section, and Appendix G Reproducibility Checklist.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The Ethics Statement section.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4 Datasets, Section 5 Experimental Setup, Appendix A Data Statistics and Languages, and Appendix G Reproducibility Checklist.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4 Datasets, Section 5 Experimental Setup, and Appendix A Data Statistics and Languages
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 5 Experimental Setup, and Section 6 Results and Discussion

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix G Reproducibility Checklist

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5 Experimental Setup, Appendix G Reproducibility Checklist

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 6 Results and Discussion, Appendix F Robustness of ENSAD

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5 Experimental Setup, Appendix G Reproducibility Checklist
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We hope that researchers in the future can construct and release more high-quality vision-language data for different languages, especially for low-resource ones. Second, our work uses 512-dim ‘XLM-R Large Vit-B/32’ mCLIP22 and is based on the StyleGAN2 framework (Karras et al. , 2020b). Since the main focus of our work is to realise multilingual and cross-lingual TTI and enable fair comparisons across different models and approaches, we compare all proposed and baseline methods with the same mCLIP text encoder and the GAN framework. However, for readers and potential users interested in ‘chasing’ stronger absolute FID scores, we speculate that the larger 640-dim ‘XLM-R Large VitB/16+’ mCLIP text encoder and the more recent StyleGAN3 (Karras et al. , 2021) can be helpful. Third, we notice that in addition to LAFITE, several state-of-the-art large diffusion models such as those from Saharia et al. (2022) and Rombach et al. (2022) also use CLIP to condition image generation on text input. This means that we could be able to derive multilingual diffusion models for mTTI also by replacing CLIP with mCLIP and enhance the mTTI performance with our proposed ENSAD (of course, we would need to redesign our loss functions). However, due to limited computational resources, we leave it to future work. Fourth, the ENSAD boosts cross-lingual transfer for TTI by combining the knowledge from multiple translations, which can mitigate potential translation errors. Our work does not demonstrate if ENSAD is applicable and adaptable to downstream cross-lingual tasks besides TTI. It is because 1) downstream tasks other than TTI are out of the scope of this work and 2) adapting ENSAD to different tasks will require redesign of model structures and losses catering to the characteristics of each downstream task, making us believe it is not proper to expand the topic and include everything in a single piece of work. Therefore, we also leave this to future work. 22https://github