The Plug and Play of Language Models for Text-to-image Generation
Authors: Can Qin
Ning Yu
Chen Xing
Shu Zhang
Stefano Ermon
Yun Fu
Caiming Xiong
Ran Xu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Text-to-Image Generation
Language Models
Efficiency

Abstract: Text-to-image (T2I) models enable controllable image generation through user-provided captions. A text encoder is typically used to map captions to a latent space, and it has been shown to be critical for model's performance. However, replacing or upgrading the text encoder in a T2I model is challenging due to the tight bond between the current encoder and the image decoder. It requires training the model from scratch, which can be prohibitively expensive. To address this problem, we introduce a more efficient approach to align a pre-trained language model with the latent space of an existing T2I model. We propose a Model Translation Network (MTN) and a new training objective to align the representation spaces of the two text encoders using only a corpus of unlabeled text. We empirically find that MTN can be trained efficiently and can boost the performance of existing T2I models by upgrading their text encoder. Moreover, we find that MTN can align multilingual language models such as XLM-Roberta, thus allowing existing T2I models to generate high-quality images from captions beyond English. 

Under review as a conference paper at ICLR 2023
THE PLUG AND PLAY OF LANGUAGE MODELS FOR TEXI-GO-IMAGE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Text-to-image (T2I) models enable controllable image generation through user- provided captions. A text encoder is typically used to map captions to a latent space, and it has been shown to be critical for model's performance. However, re- placing upgrading the text encoder in a T2I model is challenging due to the tight bond between the current encoder and the image decoder. It requires training the model from scratch, which can be prohibitively expensive. To address this prob- lem, we introduce a more efficient approach to align a pre-trained language model with the latent space of an existing T21 model. We propose Model Translation Network (MTN) and a new training objective to align the representation spaces of the two text encoders using only a corpus of unlabeled text. We empirically find that MTN can be trained efficiently and can boost the performance of existing T2I models by upgrading their text encoder. Moreover, we find that MTN can align multilingual language models such as XLM-Roberta thus allowing existing T2I models generate high-quality images from captions beyond English.
INTRODUCTION
Text-to-image (T2I) generative models have made great progress in the last few years thanks to algorithmic advances and the availability of large-scale paired training datasets (Ramesh et al., 2022; Yu et al., 2022a; Saharia et al., 2022; Rombach et al., 2022). Diffusion-baat based T2I generative models in particular have achieved remarkable results in terms of image quality (Ho & Salimans, 2022; Nichol et al., 2021). Despite these strong results, controllable generation for these methods is still challenging: generated images often not faithful the captions, compositional capabilities are lacking, and prompt engineering is often required to achieve the desired results (Parsons, 2022). Moreover, most large-scale models have only been trained on English captions, greatly limiting their use across the world.
To improve T2I models, recent research suggests that the ability of their text encoders understand and represent text is critical and is a bottleneck for their image generation performance (Saharia al., 2022; Croitoru et al., 2022). Text encoders in existing T2I models are often trained only on short image captions, and their performance on complex prompts can be largely constrained by the quality of the features extracted by the text encoder (Rombach et al., 2022). However, upgrading the text encoder for an existing T21 model is challenging because the representation spaces of the text encoder and image generator are tightly coupled (Rombach al., 2022; Ramesh et al., 2022). Training only the text encoder alignment, hindering the final image generation performance. Training the entire T2I model from scratch (perhaps with higher quality image-caption pairs) would be prohibitively expensive (Ed- wards, 2022).
more complex and representative text corpus would break this
To solve this problem, we propose method that can efficiently align off-the-shed pre- ire-trained lan- guage models with image encoders of existing biffusion-based T2I models. With this method, the existing text encoder of T2I models can be replaced with more powerful language model even one for a language other than English as illustrated in Fig. 1. Crucially, the representation alignment between text and image encoders maintained without retraining from scratch. We refer to our pro- posed method as Model Translation Network (MTN). MTN follows an encoder-decoder structure. Specifically, given trained T2I model and pre-trained language model that we want to replace, the
TThe cost of training Stable Diffusion model is around 600K USD.

---

Under review as a conference paper at ICLR 2023
lange
VTN Eneder
Generator
75-L
New Ennader
Alganng
Ulwradee
ONd Test Encoder
XUM
Elcodar, Le. Bert
Illustration of our desired modularized T2I generation. With the proposed Model Transla- tion Network (MTN), the existing image generators can be bridged to off-the-shelf language models to expand their functionalities, iee, multilingual generation, within a limited budget.
Figure 1:
encoder of MNT firstly aligns the representation space of the pre-trained language model with that of the T2I model's image generator, by minimizing both dlmenttwwise and global-win wise discrepancy. Then, the decoder of MNT takes the aligned text representations as inputs, and maps them back to the original representation space of the pre-trained language model by minimizing the reconstruc- tion loss. The reason why we need this decoder during training is that some recent research reveals that training to align existing models inevitably decreases feature discriminability (Chen et al., 2019; Cui et al., 2020). Therefore we need to preserve rich semantics captured by the pre-trained language model during the alignment training, ensuring that decoder recover the original representation space of the pre-trained language model from the aligned one. The entire training of MTN requires only corpus of unlabeled text. At inference time, only the encoder of MTN is applied on top of the pre-trained language model for representation alignment.
To verify the effectiveness of the proposed framework, we have applied stronger language model, i.e, T5-3B (Raffel et al., 2020), to upgrade existing text encoders of the Latent Diffusion Model (Rombach et al., 2022). The competitive improvements in the FID score and user study ranking reveal the benefits of our model over the baselines. Furthermore, our model has the po- tential to bring new functionalities into existing T2I models, such as multilingual generation. We empirically find that MTN can align a multilingual language model such as XLM- Roberta-L (Conneau et al., 2019) with existing T2I models. enables the existing image generator to understand text beyond English, such as French and Chinese, and to generate high-quality images accordingly.
to-image
Our contributions can be summarized as follows:
To the best of knowledge, this is the first work to consider the problem of efficiently aligning a pre-trained language model with a pre-trained T2I diffusion model.
Extensive experiments on text-to-image generation benchmarks demonstrate the superi- ority of our model over the baseline LDM method on both image quality and language controllability.
Our framework also enables text-to-image generation beyond English prompts without the need of multilingual image-text pairs for retraining.
RELATED WORKS
2.1 TEXT-TO-IMAGE GENERATION
Generative Adversarial Nets (GANs) (Goodfellow et al., 2014) is one of the major frameworks in text-to-image generation. The method (Reed et al., 2016) was an early stage approach use GAN to train text-to-image generation network. Since then, other GAN based methods, e.g, Stack-GAN (Zhang et al., 2017), Attn-GAN (Xu et al., 2018) and SD-GAN (Yin et al., 2019), have obtained promising results. In addition, recent works showed more improvements the generation quality. DM-GAN (Zhu et al., 2019) improved text-to-imant performance by introducing a dynamic memory component. DF-GAN (Tao et al., 2022) designed fusion module to fuse text and image features. LAFITE (Zhou et al., 2021) took advantage of CLIP's model to construct pseudo image- text pairs, and proposed GAN model to learn text-image pairs.
2

---

Under review as a conference paper at ICLR 2023
rego-seeressive transformer is another major framework in text-to-image generation. DALL-E (Ramesh et al., 2021) and CogView (Ding et al., 2021) adopted an autoregressive transformer (Vaswani et al., 2017) to train the correspondences between visual tokens and text tokens. Parti (Yu et al., 2022a) used a powerful visual encoder, ViT-VQGAN (Yu et al., 2022b), to improve re- sults upon DALL--. The method in (Gafni et al., 2022) is similar Cog View and DALL-B, while introducing additional controlling elements to improve the tokenization process, NÜWA (Wu et al., 2022b) and NUWA-infinity (Wu et al., 2022a) trained autoregressive visual synthesis model to sup- port both text-to-image and text-to-video generation tasks.
Concurrently, diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Song et al,, 2020) became a main research focus. In such methods, noise is added to an image, and a score network is trained to denoise and recover the input image. GLIDE (Nichol et al., 2021) performed guided inference with and without a classifier network to generate high-fidelity images. DALL- (Ramesh et al., 2022) and Imagen (Saharia et al., 2022) set new state-of-the-ar results in the text-to-image generation. In DALL-E 2, prior to produce CLIP image embeddings conditioned on text was learned, and diffusion model was used to decode the image embeddings to an image. In Imagen, frozen large-scale text encoder T5-XXL (Raffel et al., 2020) was adopted generate embeddings, and a cascade of conditional diffusion models was used to map these embeddings to images of increasing resolutions. Latent diffusion model and stable diffusion model (Rombach et al., 2022) are state-of-the-art methods that apply the diffusion model on the latent embedding space as in (Sinha al., 2021; Vahdat et al., 2021). These methods are computationally friendly while achieving impressive results.
2.2 KNOWLEDGE DISTILLATION AND CROSS-LINGUAL ALIGNMENT
Knowledge distillation is technique to guide the training of student network with reference teacher network (Buciluă et al., 2006). (Hinton et al., 2014) applied probabilities of the teacher as target for the student to learn. Since then, many methods around the idea of "what knowledge to be distilled" have been proposed. Popular definitions include distance of features (Romero et al., 2015), map attention (Zagoruyko & Komodakis, 2017), distribution-wiss distillation (Passalis & Tefas, 2018) and inter-sample distance structure (Tung & Mori, 2019), etc. Despite the great success, they mainly focus on the classification task with discrete outputs and pay less attention to representation protection. Translating dense and high-dimensional features between large-scale language models requires new solutions.
Cross-lingual alignment methods attempt to align the representation across different languages based on a multilingual text model (Kulshreshtha et al.; Cao et al., 2020). (Kulshreshtha et al.) has com- prehensively explored the different alignment methods, including rotation and finetuning aligmment. (Cao et al., 2020) has modeled the contextual token-wise embeddings to aggregate richer informa- tion for alignment. Our paper considers another problem in aligning different text models with the same or parallel data. This is essentially different from cross-lingual alignment, with more compli- cated and challenging mismatches such as feature dimensions and variant token lengths.
3 PRELIMINARY
3.1 LATENT DIFFUSION MODEL
The Latent Diffusion Model (LDM) (Rombach et al., 2022) and its extension, i.e, Stable Diffusion, which are variants of Denoising Diffusion Probabilistic Model (DDPM) (Ho et al., 2020) family, are selected as our baseline models due to their excellent balance in efficiency and visual quality.
In general, LDM composed of two stages. Firstly, there is an auto-encoder (AF) pre-trained by enormous images with the regularization in either KL-divergence or vector quantization (Van Den Oord et al., 2017; Agustsson et al., 2017). An encoder network, i.e, E, is applied for latent feature extraction
Fxx), which can be mapped back to image space by decoder network. The input images x and reconstructed images ğ are almost identical x == â.
Secondly, LDM trains diffusion model in the latent space (Sinha al., 2021; Vahdat al., 2021). follows the standard DDPM (Ho et al., 2020) with denoising loss and uses U-net (Ronneberger et al., 2015) as the image decoder as in (Song& & Ermon, 2019). To enable generative controllability
3

---

Under review as a conference paper at ICLR 2023
Xiar
Asor
Stur
2OG aoodo
Mivemttek
Alinnig
Trangoraation
dOoOb
Ner
Xalign
b
Rar
Flodt Use
New Tri kreakr
Eacader
Chord Conotoeryy
(a)
(b)
Figure 2: (a): illustration of features transformation throughout the model translation. (b): the general pipeline and learning objectives of our proposed model translation network (MTN).
LDM has applied multiple conditioning signals (y) such text, mask, or layout, encoded aside and injected into the U-net, with the help of cross-attenting layers. This can be formulated as:
LLOM E(1) [lee E0(=,, (v))
where represents the time step, is the noise corrupted latent tensor at time step f, and 20 E(x). 8 the unscaled Gaussian noise, Ga is the conditioning network parameterized by & and Eo the Unet-like denoising network (a.k.a, image decoder). The parameters of both conditioning and denoising networks, i.e, 0, o, are trained by the LDM loss. During inference, clean images can be generated via classifier- free guidance (Ho & Salimans, 2022) as:
(zxlv)) eo (z)) + (#o(2,, co (v)) ==()))),
where s is the guidance weight to balance text controllability and image fidelity.
The basic T2I LDM model is trained Laion-400M (Schuhmann al., 2021) dataset. The Stable Diffusion Model (SDM) was trained with more epochs and training data, i.e, Laion2B-oc and Laion- high-resolution.
3.2 LANGUAGE MODELS FOR T2I
LDM uses Bert (Devlin al., 2018) as the text encoder jointly trained with the image decoder using the DDPM loss. SDM uses pre-trained CLIP (Radford et al., 2021) text encoder frozen during training. It has been found that increasing the size of the text encoder very helpful for performance (Saharia et al., 2022). In order to upgrade existing models, our goal is to be able to efficienlty plug in more advanced language models such T5-3B (Raffel al., 2020) with more parameters or Xobertoberta (Devlin et al., 2018) for multilingual T2I.
MODEL TRANSLATION NETWORK (MTN)
The text encoder is one of the key components of T2I models, as generation requires precise and fine- grained text embedding for guidance. The overall performance can be greatly boosted by increasing the size of the text encoder (Saharia et al., 2022), however, upgrading an existing text model to more powerful one is challenging. Because existing T2I models are not modular, directly replacing (or augmenting with) another pre-trained language model does not work as seen in Fig. 9 (a). The key technical challenge is the mis-alignment between the new language model and the old image decoder. Moreover, joint finetuning also falls short due to catastrophic forgetting occuring when updating well trained parameters.
Considering these two different text models as source and target domains, apply neural network to learn to align. As shown in Fig. 2, to efficiently upgrade the text encoder and simultaneously protect existing knowledge, this paper has presented general framework called Model Translation Network (MTN). MTN works as an additional module to bridge the new language model and the old image decoder seen in Fig.
4

---

Under review as a conference paper at ICLR 2023
4.1 OBJECTIVES
Given a set of text corpus X' - {miyli 1, 1,.., M,j 1,.., N] with M as length of tokens and N as total quantity, the source language model F and a target text encoder F have mapped the raw text data into embeddings denoted as S = { j = I,..,,M,, M..j j- = 1, .... .N} where Say = F (Iaz) E R' and T = {tyllt = M,,
j = I,...,A N where by = F (mi)) E R4. Due to the different
models, there is severe distribution mismatch between the source and target features p(s) * p(t). The MTN is an autoencoder whose encoder M-,, ①Mr) learns to map the new text features from source language model, i.e, & E S, to align with the current target text encoder, i.e, 1 E T, where
p(M M)) p(t). Therefore, this translation (i.e, F 9M F enables the image generator to understand the new text features coming from the new language model without finetuning. To achieve this aim, we consider it as domain adaptation (Ganin al., 2016; Qin al., 2019; Long et al.. 2018) problem and apply both element weisn and distribution-wir wwie alignment losses, which includes the minimization of the mean square error (MSE) loss, i.e, Lmoe. and the adversarial loss, Le, Ladv. measured over discriminator network, i.e. [c, OD):
Lusse (QN) = E x X (11F M(F (I..); 1l))
(3)
Cade((p)) 2M = xx Dog (F 0 + E xx,,, [lg (M.) 0p)l
(4)
where X, and X, denote the source and target inputs respectively and they can be the same prompts in English (e.g. LDM-Bert + T5-3B (Raffel al., 2020)) or in bilingual but parallel content for multilingual T2I generation.
Some insightful investigations in transfer learning reveal that the cross-domain alignment will in- evitably decrease feature discrimination (Chen et al., 2019; Cui et al., 2020). To project the rich semantics necessary for model upgrading, we further apply a decoder network, ie, NN. Ox), for feature reconstruction:
(ree(OM, QN) == -
(5)
MTN is trained in an end-to-end manner and the details of which can be referred to in Algorithm and Sec. (Ma et al., 2022) introduced that universal learning engine should seek a compressive closed-loop transcription with good property in pruuctre-preservvie requires similar functionality. MTN desires stability after a loop whose input and output should keep almost equivalent, i.e, æ Rs & Rs å ss ..., due to the constraint of reconstruction loss. Therefore, inputting the reconstructed data i to MTN is expected to be consistent with the previous loop. This is helpful for accelerating MTN training.
Our cross-model mapping also
4.2 ARCHITECTURE OF MODEL TRANSLATION NETWORK
The input data to alignment can be repre- sented as the sequence of tokens: X,, (Ii,1, -1,,, Fi,, where X,, E X' and Jij E IRC, sequence length S and to- ken dim C. Inspired by the methods of image super-resolution (Lim et al., 2017; Ahn et al., 2018) densely regress target data under different input-output dimen- sions, the translator network stacks three sub-nets in sequence, as seen in Fig. 3. The head net only used for simple fea- ture transformation. The body net has many residual modules for fine-grained representation learning. The tail network is appended and is mainly employed for dimension conversion to match the target tensors. Every residual module is implemented based on the MLP-based token mixer (Tolstikhin et al., 2021; Ma et al., 2021) with high efficiency and strong ability in representation learning. Within the MLP- mixer block, it consists of Token Mixer and Sequence Mixer for orthometric purposes. The
Roschal Modale
MI P Black
LN
Tull Net
Linear
Toay Ra
Token MLP
GELU
LNN
Residaal Medule
Tranpoejddni diei)
GELU
Hoal Nes
Soq ML.P
LN
Figure 3: Detailed architecture of Model Translation Network.
5

---

Under review as a conference paper at ICLR 2023
Token Mixer learns the representation of each token with shared MLP, and the Sequence Mixer is designed for learning the same channel in sequential tokens:
TokMixer: U., = (2 * (X).), 1...,
(6)
SoqMixer: Xj.- =X,, + (Wo (U))))))
(7)
where X. is the feature, LN,,8 denotes layer normalization parameterized by y and B, and o indicates the non-linear operator such as GELU (Hendrycks & Gimpel, 2016).
5 EXPERIMENTS
In this section, we comprehensively evaluate proposed framework for the monolingual and mul- tilingual text-to-image generation upon upgrading the text encoder pre-trained models. More details can be referred to in Appendix.
5.1 EXPERIMENTS SETUP
Upon the shoulders of giants, this paper applies the Latent-Diffusion Model (LDM) and Stable Diffusion Model (SDM) (Rombach et al., 2022) as the baselines with a similar model but different text encoders and scalabilities. LDM uses Bert (Devlin et al., 2018) as the text encoder that is jointly updated with the image decoder based on DDPM loss, while SDM uses a frozen pre- trained CLIP text encoder during training. The LDM is designed for the images in 256 x 256 and SDM targets 512 x 512 images. Considering our limitations in data and computation resources, the experiments monolingual T21 based LDM with lower cost of text encoder upgrading than that of SDM. We implement the multilingual T2I generation upon the SDM.
Implementation. The MTN is implemented as the Fig. 3 with encoder M and decoder N in symmetrical architecture consisting ~55M parameters. For simplicity in denotation, we refer MTN encoder MTN in the following experiments, whose body net consists of 5 residual modules as de- fault. We take the A (Loshchilov & Hutter, 2017) as the optimizer based on py PyTorch (Paszke al., 2019). The learning rate assigned as I x x10- for MTN training and I xx10 for LDM-T5- MTN finetuning. The MTN model is trained on Nvidia-A 100-40GB, requiring to 5 GPU days.
Language Models. We apply the T5- 3B (Raffel et al., 2020) to upgrade the text encoder of LDM Bert. T5-3B has 6x more parameters than LDM Bert and trained by an enormous text corpus be- yond the image captions. Furthermore, to enable multilingual T2I generation, we choose the -.M--Roet (Devlin et al., 2018) as the new text encoder to let SDM image decoder understand languages be- yond English.
Image Quality
Text Controllability
33.6%
34.0%
32.7%
35.7%
32.4%
31.7%
Datasets. We have collected the text data for MTN training. In monolingual T2I. only English texts are required, and we extract 18M captions from Laion-400M as training corpus, and more data will be more helpful. Multilingual MTNs require parallel texts in different languages. These data are col- lected by Wikimarix (Schwenk et al., 2019) and Googletrans with ~2M sentences for each.
Figure 4: User study of text controllability and im- age quality with the prompts from Drawbench (Saharia et al., 2022) and Winoground (Thrush et al., 2022).
Ablations. T5+MTN refers to the LDM model with the stacking of T5 and MTN as the text encoder where the LDM image decoder unchanged. T5+FT represents finetuning of the LDM image decoder with DDPM loss based on text conditions from fixed T5. T5+MTN+FT is the final version, including both MTN and joint finetuning with the image decoder. The finetuning takes ~ 100 GPU days of 116M image-text pairs filtered from Laion-400M by BLIP score (Li et al., 2022).
5.2 MONOLINGUAL TEXT-TO-IMAGE GENERATION
Monolingual T2I generation focuses on the comparison with LDM given the English text prompts. We have decomposed our pipeline into stages: alignment and finetuning. MTN helps to align

---

Under review as a conference paper at ICLR 2023
a puccing by gaar coople.
aggn sape image
danh vader playing-etts saiing
person wakking
a perezs walkng feld yellow
getting
LDM Original
TO IMAGE"
TS+PT
IN AL
13+MIN
Text Texx Irage'
T5+MIN+FT
IMAGE'
Figure 5: Monolingual generation in resolution 256 256 with guidance weight 7.5 and DDIM steps 200. The details of each method can be referred to in Sec. 5.1.
the new language model whose potential can be better exploited by further finetuning of image text pairs. This section verifies the benefits our vanilla and finetuned models in both quantitative and qualitative evaluations.
5.2.1
QUANTITATIVE EVALUATION
Tab. 1 reported the zero-shot FID) score (Heusel et al., 2017) on COCO dataset (Lin et al., 2014) based on the pyorch-fid (Seitzer, 2020) package. Our re-implementation score of LDM is slightly higher than the paper's one (Rombach et al., 2022), which may because of different implementations. The table shows the minor drop of T5+MTN from LDM since our current MTN model still leaves some incorrect alignments, which could be fixed by either joint finetuning or collecting more text corpus to train MTN. With the 100 GPU-days finetuning of subset of the Laion dataset (i.e, 116M image-text pairs), the overall performance has been considerably improved, outperforming the LDM by certain margins.
We have also conducted a user study com- pare the image quality and text controllabil- ity in Fig. 4 by Amazon Turk (Buhrmester et al., 2011). There are ~1K prompts col- lected from Drawbench (Saharia al., 2022) and Winoground (Thrush et al., 2022), which comprehensively cover various scenarios and numerous difficult cases. We report the per- centages users' votes for the three meth- ods in Fig. 4. The study of image quality is designed to evaluate the resolution, cleanness,
Table 1: FID COCO.
indicates implemented results. ZS denotes zero-shot.
Method
FID,
ZS
CogView (Ding et al., 2021)
27.10
LAFTTE (Zhou et al., 2021)
26.94
X
GLIDE (Nichol et al., 2021)
12.24
X
Make-A-Secne (Gafni et al., 2022)
11.84
LDM (Rombach et al., 2022)
12.63
LDM*
13.55
TS+FT
23.30
and smoothness of object boundaries, where our method shows fragile superiority. In the user study of controllability, the LDM by 3%, which is a non-trivial improvement. Moreover, the p-value 2 is computed as 0.04
TS+MTN
14.32
T5+MIN+FT
12.05
outperformed
with three repeats of user study that is statistically significant according the criteria of <0.05.

---

Under review as a conference paper at ICLR 2023
Eeglish portaa of out painzo by Piaaso
Chuese: #1 (RA10R
Geman Purtrik cincr Kazze von Pausso gesull
Ukainiin: noprper NomI, nammarrch Iliaace
Iuiian eit ato gatto diiitt da Ficesso
Hludoo
Figure 6: Multilingual generation results in resolution 512 512 of XLM-Roberta + MTN SDM decoder (sd-v1-4) with the same caption, i.e, "portrait of a cat painted by Picasso". With the help encoder, the SDM decoder can support different languages including Japanese, Ukraine, German, Italian, Korean, Greek, Chinese, French and Spanish. The guidance weight assigned 7.5 and PLMS (Liu et al., 2022) sampling steps are 50.
of different MTNs and multilingual
5.2.2 VISUAL COMPARISON
In Fig. 5, we have exhibited some generated images of four different methods. By comparing T5+FT and T5+MNT+FT with similar finetuning costs (100 GPU-days), we can conclude that the proposed MNT helps save much effort in plugging a new language model due to the high-quality images. Moreover, the T5+MNT+FT shows stronger controllability in many cases compared with LDM, such as "a cube made of brick" and "a horse getting wet",, whereas the LDM's 's results fail to display concepts of "cube" and "wet".
5.3 MULTILINGUAL TEXT- CONT-TOOIMAGEE GENERATION
Our proposed framework is very general and generic to many language models, including the mul- tilingual backbones. We choose the recent SDM as our base model with the pre-trained cli--vit- large-patch14 14 as the text encoder to be upgraded by the multilingual one, i.e, XLM-Roberta-L. The MTN helps to align such two backbones with the parallel corpus as training data. Thus, the old SDM image decoder can successfully understand the text embeddings of other languages encoded by MTN and LLLM-Robrrt--L without any additional parameters updating except MTN.
5.3.1 VISUAL COMPARISON
Fig. 6 shows the example of multilingual generation results with the prompt "portrait of a cat painted by Picasso" many languages. Most of the results reveal the content "portrait cat painted".. The "Picasso" is challenging notion that rarely appears in our training data. Thus, more high- quality parallel data will be greatly helpful. Compared with those finetuned over multilingual image text pairs, our solution is far more efficient and cheaper.
5.3.2 HYBRID MULTILINGUAL GENERATION
The hybrid multilingual generation is more challenging that both the original SDM and third-party translation toolbox cannot address. Our proposed MTN has surprising capability in this problem. According to the results in Fig. 7, the multilingual backbone bonded with MTN gives the precise guide the SDM decoder even with the prompts in three different languages and randomly mixed.
5.4 ALIGNMENT ANALYSIS
8

---

Under review as a conference paper at ICLR 2023
*B Fサ セカング E sit table, "5
- 8 + ,,,,,, crash into buildiig"
Stable Diffasion
NrMM-RobertsMTNN
Stable Diffusion
M-Reberta-MTN
(a)
(b)
Figure 7: Hybrid multilingual generation in resolution of 512 512. There three-different- language texts in the input caption including Chinese, Japanese and English. The caption of (a) is "colorful, cat painted by Picasso, sit on table, is eating food" and the caption of (b) is "a white, sedan, crash into building". With our MTN infused ahead, the XLM-Roberte can guide SDM decoder generate reasonable results where the original SDM fails work.
120
300
MIN (a)
50-th MIN
MIN
300-th MIN
500-th MIN
Iorations
(b)
(C)
Figure 9: Alignment analysis. (a): Initial result of direct replacement of new text encoder, Le, LDM-Bert -> T5-3B. (b): Results of MTN different iterations with the new text backbone. The prompt is "an astronaut riding horse pencil drawing". (c): Curve of Lmse and Cree in training.
To understand how MTN maps the cross- model features, we have visualized its tra- jectory with the generative results in dif- ferent iterations in Fig. 9. In the begin- ning, the old image decoder could understand the new text embeddings at all. Then, certain patterns such as "pen- cil drawing" appear after 300 iterations of training with a significant decrease in losses. Such gap can finally be sealed at the end where both Emee and Cree con-
MAl MTN
MIN-SRM
MTN-HRMM
Figure 8: t-SNE (Van der Maaten & Hinton, 2008) cross-model feature map visualization.
verged to small values.
5.5 T-SNE FEATURE MAPS WITH DIFFERENT MTN SIZES
Fig. illustrates the feature maps of different text encoders. Source features are collected from LDM-Bert and target ones are achieved by T5-3B with the shared prompts from Drawbench and Winoground. RMs represents the number of the residual modules. We can see that MTNs' features precisely aligned with those of the source text model. There is much difference between larger and smaller MTNs. Considering efficiency, we choose MTN-5RMs as the default model.
CONCLUSION
Infusing the pre-trained language models into the existing T2I image generator is an existing di- rection toward more powerful Al system. However, the current text encoder cannot be easily upgraded due to the tight match. This paper attempts to break such strong constraint of the corre- sponding image-text models towards flexible modularization and efficient upgrading. To address the severe misalignment, we have proposed the model translation network (MTN) with both objectives for cross model alignment and originality protection. Empirically, is beneficial for the overall performance and enables versatile functionalities such as multilingual generation within limited budget. We wish this work to be inspirable the community in large-scale Al system design.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Luc Van Gool. Soft-to-hard vector quantization for end-to-end learned compression of images and neural networks. arXiv preprint arXiv: 170.v:/704000445 3, 2017.
Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. Fast, accurate, and lightweight super- resolution with cascading residual network. In ECCV, 2018.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil, Model compression. In SIGKDD, 2006.
Michael Buhrmester, Tracy Kwang, and Samuel Gosling. Amazon's mechanical turk: A new source of inexpensive, yet high-quality, data? Perspectives Psychological Science, 2011.
Steven Cao, Nikita Kitaev, and Dan Klein. Multilingual alignment of contextual word representa- tions. arXiv preprint arXiv:2002.03518 2020.
Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discriminabil- ity: Batch spectral penalization for adversarial domain adaptation. In ICML, 2019.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.022
2019.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: survey. arXiv preprint arXiv:2209.04744 2022.
Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qingming Huang, and Qi Tian. Towards dis- criminability and diversity: Batch nuclear-norm maximization under label insufficient situations. In CVPR, 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. NeurIPS, 2021.
Benj Edwards. With stable diffusion, you may never believe what you see online again, Sep 2022. URL https::/aasstecmiccaa http://ww....c.ooo.ooommoooooona wwth-stabee stwble-diffuuiioooeee
-etoo--see-onii--aaaaiana/ never be what whw--yo---ee--n wew--eee---nnneeeeeeeee-ee-eeeee
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Man- deep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english-centric multilingual machine translation. JMLR, 2021.
Oran Gafni, Adam Polyak, Oron Ashual Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. arXiv preprint arXiv:2203.13131, 2022.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Paseal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. JMLR, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 12Xvv:660..0844 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurlPS, 2017.
10

---

Under review as a conference paper at ICLR 2023
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NeurlPSW, 2014.
Jonathan Ho and Tim Salimans.
Classifier-free
diffusion
guidance.
arXiv preprint
0rXfv:2207.22596 2022.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. NeurIPS,
2020.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. URL hitps : //d/:///o./////// orr//o//////// /10 5281/ bemodo.514377 If you this software, please cite it below.
Saurabh Kulshreshtha, Jose Luis Redondo Garcia, and Ching- Yun Chang. Cross-lingual alignment methods for multilingual BERT: A comparative study.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201.12086 2022.
Bootstrapping language-
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lec. Enhanced deep resid- ual networks for single image super-resolution. In CVPRW, 2017.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.0977 000.000.9777. 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1900./ 2019.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. domain adaptation. NeurlPS, 2018
Conditional adversarial
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. 2017.
Xu Ma. Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local geometry in point cloud: simple residual mlp framework. In ICLR, 2021.
Yi Ma, Doris Tsao, and Heung-Yer Yeung Shum. On the principles of parsimony and self-consistency for the emergence of intelligence. Frontiers of Information Technology & Electronic Engineering, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arX1v:2112.10741, 2021.
Guy Parsons.
Dall-e 2 prompt book. 2022.
https /ttp://daallrrr wawallery.galllry.....
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. ECCV, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS. 2019.
Can Qin, Haoxuan You, Lichen Wang, C-C Jay Kuo, and Yun Fu. Pointdan: A multi-scale 3d
domain adaption network for point cloud representation. NeurlPS, 2019.

---

Under review as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In JCML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/255 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text-to-image synthesis. In /CML, 2016.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic soxt-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprins arXiv:2/11.02// 14, 2021.
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. Wiki- matrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia. arXiv preprint 197.07990, 2019.
Maximilian Seitzer. pytorch- firrch-idd: FID Score for PyTorch pytorch-fid, August 2020. Version 0.2.1.
Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-decoding mod- els for few-shot conditional generation. Advances in Neural Information Processing Systems, 34: 12533-12548, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurlPS, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13466, 2020.
Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu. Df-gan: A simple and effective baseline for text-to-image synthesis. CVPR, 2022.
Ashish Thapliyal, Jordi Pont-Tuset. Xi Chen, and Radu Soricut. Crossmodal 3600; massively multilingual multimodal evaluation dataset. arXiv preprint arXiv:2205. 2022.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams. Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. In CVPR, 2022.
12

---

Under review as a conference paper at ICLR 2023
Iyy O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. NeurlPS, 2021.
Frederick Tung and Greg Mori. Simiearit knowledge distillation. In CVPR, 2019.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling latent space. NeurlPS, 34:1 1287-113821 2021.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, and Laukasz Kaiser. Attention is all you need. NeurIPS, 2017.
Chenfei Wu, Jian Liang, Xiaowei Hu, Zhe Gan, Jianfeng Wang, Lijuuan Wang, Zicheng Liu, Yuejian Fang, and Nan Duan. Nuwa-infinity: Autoregressive over autoregressive generation for infinite visual synthesis. arXiv preprint wrXi:2207.008114 2022a.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang. Yuejian Fang, Daxin Jiang, and Nan Duan. NÜwa: Visual synthesis pre- pre-training for neural visual world creation. In ECCV, 2022b.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In CVPR, 2018.
Guojun Yin, Bin Liu, Lu Sheng, Nenghai Yu, Xiaogang Wang, and Jing Shao. Semantics disentan- gling for text-to-image generation. CVPR, 2019.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022a.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Baldridge Jason, and Yonghui Wu. Vector-quantized image modeling with im- proved vggan. In ICLR, 2022b.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In ICLR, 2017.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
Yufan Zhou, Ruiyi Zhang. Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Lafite: Towards language-free training for text-to-image generation. arXiv preprint arXiv:21 1.11199177, 2021.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adver- sarial networks for text-to-image synthesis. In CVPR, 2019.
13

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
This paper strictly follows the ICLR ethical research standards and laws. All datasets we employed are publicly available, and all related publications and source codes are cited appropriately.
REPRODUCIBILITY STATEMENT
We adhere to ICLR reproducibility standards and ensure the reproducibility of our work in multiple ways, including:
Besides the codes, we will share the our processed data and model checkpoints to public.
We clearly present the design of MTN and joint finetuning in Section 5.1 and Section 4.2
Detailed framework and more experiments are presented in Supplementary Materials.
14

---

Under review as a conference paper at ICLR 2023
A DETAILS OF MODEL TRANSLATION NETWORK (MTN)
Model Translation Network (MTN) the key contribution of this paper, which needs address both cross-model alignment and feature protection. In this section, we will provide more details of the MTN, including implemmentationcconiiguaiio analysis sizes, and other ablation studies.
A.1 ARCHITECTURE AND CONFIGURATION
We have introduced the basic architecture of MTN in Sec. 4.2 in the main paper. MTN could have variants in different sizes whose configurations can be referred to in Tab. 2. The MTN with three residual modules is the smaller one, with 34M parameters in its encoder. The MTN-5RMs has 51M parameters. In general, larger-size models inevitably result in slower speed and more computation costs. Considering the sizes of the image decoder and text encoder as the basis, the MTN, working as an injected module, should be larger than these major characters. Otherwise, the inference speed will be significantly slower, and it may need huge computations in the finetuning stage. We empirically found that assigning RM as five satisfactory choice. The MTN-3RMs seems too weak in representation learning. Except for the MLP-mixee the self-attention-balt model may also be applied for implementing MTN. Exploring more suitable architectures of MTN is our future work.
Table 2: Model configurations for our MTN. We introduce configurations MTN-3RMs and MTN-5RMs.
Stage
Dimensions
Block
MTN-3RMs
MTN-5RMs
Linear, LN, Linear, LN,
Linear, LN, o Linear, LN, o
Token
Head Net
77
MLP
Linear, LN
Linear, LN
Linear, LN, a Linear, LN,
Linear, LN, a
Sequence MLP
1024
Linear, LN,
Linear, LN
Linear, LN
Linear, LN,, o
Linear, LN, o
Token MLP
Body Net
77
Linear, LN, o
X 3
Linear, LN, o
x
Linear, LN
Linear, LN
Linear, LN, o
Linear, LN, o
Sequence MLP
1024
Linear, LN, o
Linear, LN, a
Linear, LN
Linear, LN
Linear, LN,
Linear, LN,
Token MLP
Tail Net
77
Linear, LN, Linear, LN
Linear, LN, o
Linear, LN
Linear, LN, o
Linear, LN,
Sequence MLP
1280
Linear, LN, o
Linear, LN, o
Linear, LN
Linear, LN
A.2 EFFECTS MTN SIZES
Fig. shows a visual comparison of some example prompts over three different methods. The LDM Ori refers to the original LDM model with the checkpoint shared in its repo. MTN-3RMs and MTN-SRMs are our models with or residual modules inside their body nets. The text encoder replaced as the T5-3B, and the image decoder is the same LDM Ori. Both MTN models are trained by the same text corpus, including 18M sentences. From the figure, we can notice that MTN- 3RMs fails on some complicated prompts such "a virus monster is playing guitar, oil on canvas" and "there a penguin with a dog head standing" where the MTN-5RMs performs well. Therefore, conclude that deep MTN is necessary for precise alignment.
15

---

Under review as a conference paper at ICLR 2023
LDM Ori
MTN-3RMs
MTN-5RMs
a blue car
an astronaut riding a horse a pencil drawing
virus monster is playing guitar, oil on canvas
there is a penguin with a dog head standing
Figure 10: Monolingual generation of example prompts in 256 x 256 with guidance weight 7.5 and DDIM steps 200.
A.3 ABLATION OF LOSSES
To analyze the impacts of different losses, we visualize their t-SNE map Fig. 11. We can notice that just MSE loss cannot well align the feature of the two models. The reconstruction loss is vital keep discrimination and avoid overfitting. The adversarial loss seems not to show much improvement in the figure. According to empirical study, adversarial loss only brings limited gains.
We have conducted the quantitative ablation study on subset of COCO with 5k data randomly selected. We chose T5-3B as the text encoder aligned with the LDM Unet with our MTN. denotes the finetuning of Unet. We have reported the FID and CLIP scores in the following table for comparison of image quality and image-text alignment. The testing data are inferred by DDIM with 200 steps. The image size is 256 256. For comprehensive analysis, the experiments are
16

---

Under review as a conference paper at ICLR 2023
Lmss
Emse Lrec
mmse+ Lree L adv
Figure 11: t-SNE (Van der Maaten & Hinton, 2008) cross-model feature map visualization.
Table 3:
Ablation study of T5++MTN+LDM over the 5K COCO subset
Ablations
8=1.5
8=5
8=7.5
loss-mse
loss-ree
loss-adv
Finetuning
CLIP
FID
CLIP
FID.
CLIP
FIDU
X
10.70
98.
10.75
141.19
11.00
156.70
x
18.98
35.76
22.05
49.14
22.49
53.11
X
X
AXX
20.40
34.41
23.01
48.67
23.40
52.78
X
20.53
33.14
23.23
45.96
23.57
48.67
X
V
20.67
32.80
23.24
45.48
23.74
48.51
V
21.14
30.93
23,88
41.92
24.17
44.58
three classifier-free guidances with s=1.5, 5, and 7.5 according to Eq. (2). The Tab 3 summarizes the results of our ablation study. The first row represents the direct combination of T5 and LDM, whose results do not make sense due to the severe misalignment. TS+LDM trained from scratch can referred to the second row. The finetuning of Unet takes 100 GPU Days, whereas MTN training needs only 5 GPU Days. Even with ten times higher cost, its result is still inferior to those of MTNs'. By comparing the 3rd, 4th, 5th, and 6th rows, we can easily conclude the superiority of the full-version model.
A.4 HYPER-PARAMETTE ANALYSIS
More analysis of sensitive hyper-parameters. In Tab. 4, we explored the different learning rates to try MTN transfer OpenClip-L. to Clip-L for Stable Diffusion. It is benchmarking on the average CLIP score with 500 generative samples in the size of 512 x 512.
is included here. The learning rate is one of the most important
A.5 CLOSED-LOOP CONSISTENCY
The closed-loop consistency is a helpful technique to accelerate MTN training. We have briefly introduced it in Sec 4.1 in the main paper. Here, we provide more details as illustrated in Fig. 12. The Fig. 13 and A.5 visualize the curves alignment and reconstruction losses during training.
B MONOLINGUAL T2I GENERATION
B. LATENT DIFFUSION MODEL
As seen in Fig. 15 and 16, we demonstrate more example results with the challenging prompts from Drawbench (Saharia et al., 2022) and Winoground (Thrush et al., 2022). The baseline methods are the original LDM and the finetuned LDM equipped with T5 the text encoder. We catch that the images of FT+ are better aligned with the text and reasonable the layout of different objects.
17

---

Under review as a conference paper at ICLR 2023
Table 4: Analysis of learning rate of MTN in OpenClip CLIP
Ir=le-3
Ir=le-4
Ir=le-5
Ir=le-6
CLIPT 27.27
29.53
27.17
19.40
Syrc
Arre
Model maniin PO00 INNIIN
OOOO0 000
pOoOo
N
align
L
align
13
20Q 00000
M
OooOO
Figure 12: Illustration of Closed-loop Consistency,
B.2 STUDY OF DIFFERENT TEXT ENCODERS
our proposed framework is generic to a wide range text encoders. In this subsection, we have tried many different language models as the text encoder, including Bert-base (110M #params), Roberta-Large (355M #params), Clip-text (123M #params), T5-large (770M #params) and T5-3B (2.8B #params). All these pre-trained text encoders are aligned with the Latent Diffusion Model (LDM) with our proposed model to replace the original LDM text encoder without any finetuning of the LDM UNet.
These plug-and-pln models are verified on a subset of COCO with 5k samples randomly selected. We have reported the FID and CLIP scores in the following table for comparison of image quality and image-text alignment. All the model translation networks (MTN) are trained by 18M sentences sampled from the captions of Laion-400M. The testing data are inferred by DDIM with 200 steps. The image size is 256x256. For comprehensive analysis, we have conducted the experiments on three classifier-fres guidance with s=1.5 and 5. The results are summarized in Tab. 5.
From the table, we can notice that both FID and CLIP have increased with the rise of text model sizes which is well aligned with the conclusion of Imagen(Sahard) et al., 2022). The CLIP-text does not perform well here due to the larger domain gap with text-only text models such as Bert.
B.3 MODEL. TRANSFER ACROSS VARIANT TOKEN LENGTHS
MTN has a strong ability to overcome different lengths. The token number assigned 77, which is fixed in both LDM and SDM. Our proposed MTN can overcome the variant length text encoders without any finetuning of the Unet model. We have completed the experimental study and reported its result in the Tab. verify this point. In this experiment, we have applied Roberta-L and its tokenizers encoding maximum tokens in 77, 128, and 256, respectively. The guidance 5, and other settings follow the Tab. 3.
B.4 STABLE DIFFUSION MODEL.
To illustrate the generics of our proposed framework, we also apply the T5-L to replace the CLIP text encoder of Stable Diffusion (v1-4). As shown in Fig. 17, our model has revealed precise control- lability and excellent visual quality compared with the standard Stable Diffusion model. Improving Stable Diffusion with efficient finetuning is our following focus.
18

---

Under review as a conference paper at ICLR 2023
104
10
Loss- mse Loss-rec
Loss mse Loss-rec
100
Loss
100
Loss
10
10a1
10
100
200
300
400
500
200
300
400
600
Iterations
Iterations
Figure 13: with Closed-loop Consistency.
Figure 14: without Closed-loop Consistency.
Table 5:
Analysis of Different Text Encoders over the 5K COCO subset.
Bert-B (110M)
Roberta-L (355M)
CLIP-Text (123M)
T5-L (770M)
T5-3B (2.8B)
CLIP
FID
CLIPt
FID
CLIP
FID
CLIP
FID
CLIP
FID
8= 1.5
19.42
37.46
20.03
34.06
19.02
38.93
19.78
35.17
20.67
32.80
5
21.97
55.35
22.85
45.88
21.85
47.32
22.76
47.91
23.24
45.48
7.5.
22.68
55.21
23.25
48.96
22.38
49.66
23.23
50.02
23.74
48.51
We also have applied OpenCLIP-L-14 to replace CLIP-L-14 with the similar architecture Stable Diffusion. The quantitative evaluation is reported in Tab. 7. This tested on COCO-5k with the guidance
B.5 COMPUTATION COST PERFORMANCE
The conditional generation itself is very challenging task where how the text encoder aligns with image Unet still needs to be discovered. We empirically find that replacing the text encoder can achieve the comparable results with the source model, but it takes lot of effort to outperform it. this section, we share a precise cost of computations
The Unet may cause this as the bottleneck. with the different sizes of the training set in Tab. 8. This is benchmarked as the same as previous experiments on COCO-SK with T5-3B to replace the LDM text encoder with MTN. From the table, that the performance has increased with the rise of data as well the computation cost.
Moreover, we also include T5+LDM trained from scratch as baseline. For a fair comparison, we have tried our best to train the T5+LDM with 140 GPU Days which is comparable with the sum of MTNI 16 and
The results are shown in Tab. 9.
19

---

Under review as a conference paper at ICLR 2023
Table 6: Model Transfer with variant token lengths (Sre'TokenL.engt) Targgetookenl.enggle
the 5K COCO subset with the guidance as 5, Roberta-L (Liu et al., 2019) is applied as the new text encoder replace LDM text encoder.
77->77
128-+77
256->77
CLIP
22.85
23.19
23.37
FID
45.88
45.67
45.53
Table 7: Transfer from OpenCLIP-L-14 (Ilharco et al., 2021) to CLIP-L-14 on Stable Diffusion.
CLIPt
FID
OpenCLIP-MIN+SDM CLIP (Ori SDM)
24.66
36.18
25.53
35.34
C MULTILINGUAL T2I GENERATION
To verify the functionalities for multilingual T2I generation, have shown results in Fig. 18 and Fig. 19. There is one MTN for one language. To build an automatic pipeline, the model is expected to choose which MTN use according the detection of incoming languages.
C.1 COMPARISON WITH MULTILINGUAL TRANSLATION MODEL
Our proposed framework is model agnostic which should be able to overcome variants domain gaps between models and languages. The cross-lingual alignment is applied as verification of proof of concept. Compared with the standard translation DM, our proposed MTN needs less training and fewer parameters to train. For quantitative evaluation, we have compared it with a popular multilingual translation model M2M100-418M, (Fan al., 2021) (418M #params) trained by 7.5B parallel sentences. In contrast, our MTN uses far fewer training data (2 sentences for each pair). The results are listed in Tab. 10 where reported the Multilingual-LLPP score calculated over the 100 randomly selected multilingual image-caption pairs from Crossmodal dataset (Thapliyal et al., 2022). All the images are sampled by PLMS (Liu et al., 2022) with 50 steps in the resolution 512 x 512.
The cosine similarity of Multilingual-CLA text and image embeddings is reported here. From the Tab. 10, we can see that the MTN is only slightly weaker than M2M-418M despite the huge efficiency in training data. Such result reflects the great potential of MTN in multilingual T2I by aligning the cross-lingual model with the consideration of data efficiency.
20

---

Under review as a conference paper at ICLR 2023
Table 8: Analysis of MTN's training cost.
Data Size
5M
18M
116M
CLIP
20.92
23.24
23.71
FID
48.63
45.48
43.17
GPU Days
1.67
5.89
41.20
Table 9: Analysis of Performance and Computation Cost
CLIP
FID
GPU Days
LDM T5 + FT T5 + MTN18M T5 MTNISM + FT T5 MINNING 16M T5 + MTN1 + FT
23.79
42.83
23.29
45.62
~40
23.24
45.48
5,89
23.88
41.92
~00
23.71
43.17
41,20
24.14
41.53
~100
Comparison with translation model, i.e. M2M100-418M (Fan et al., 2021), for multilin- gual generation. The Multilingual-CLNN score Crossmodal (Thapliyal al., 2022) benchmark is reported here.
Table 10:
M2M100-418M-SDD
XLM-Roberta-MIIN-DD
Chinese
24.50
22.01
Franch
25.08
23,09
Spanish
23.83
22.91
Korean
22.35
21.46
Japanese
23.73
23,99
Ukrainian
23.99
22.81
Italian
21.08
22.40
21

---

Under review as a conference paper at ICLR 2023
LDM Original
TS+FT
T5+MTN
T5+MTN+FT
A green colored
shitt Matzo
banana.
A pink colored car.
BIGSTOCK
S TOPP
A stop sign on the right of refrigerator.
STOP
STOP
STF
An elephant under the sea.
abe Sock
New York skyline with "hello world" written with fireworks on the sky.
HELLO WORLL
Figure 15: Monolingual generation of Drawbench prompts in 256 x 256 with guidance weight 7.5 and DDIM steps 200.
22

---

Under review as a conference paper at ICLR 2023
LDM Original
TS+FT
TS+MTN
TS+MTN+FT
A blanket on a dog with spots.
A child with fake beard and adult.
chutrarstan arsstack
A human being on a device that intercepts the sun's 's rays.
A parked car is partially inside a tree.
A person is close to the water and in the sand.
Biust
BIGSTOCK
A person stands and dog sits.
Blue pants and green top.
Wood floors with concrete- walls.
BIOBADO
Figure 16: Monolingual generation of Winoground prompts in 256 256 with guidance weight 7.5 and DDIM steps 200.
23

---

Under review as a conference paper at ICLR 2023
Stable Diffusion (v1-4)
T5-L MTN
"An astronaut riding a horse."
"A pink car."
"A shark in the dessert."
Figure 17: Monolingual generation of example prompts in 512 x 512 with guidance weight 7.5 and 50 PLMS (Liu et al., 2022) sampling steps.
24

---

Under review as a conference paper at ICLR 2023
Chinese: 66 20
Korcon:
Autn
senovntias
ausonianjo
French:
Japanosi: 1110
Chinese: 3
Korean: (/2)) 28 W giF:
Germant Ein Husky Wuft sut dem Grün
Uemirian: no sonevt fisars sood
Iallan: Un husky corre green
1E.L
Greek: You French: bunby oount
Experantor Hunky kures
Spanish rlne husky enti
/X 87 1-28 2ot
Figure 18: Multilingual generation results in 512 x 512 of XLM-Roberta MTN SDM decoder (sd-v1-4) with the same input captions, i.e, "a blue car" and "a husky running on green field", in English. It can support different languages including Japanese, Ukraine, German, Italian, Korean, Greek, Esperanto, Chinese, French and Spanish with only different MTNs. The guidance weight is 7.5 and PLMS (Liu al., 2022) sampling steps are 50.
25

---

Under review as a conference paper at ICLR 2023
31:3 10
forcon: 9-5 8891 2 -y
Impressorismaammlirr Nachmittaggarions
Urrinian: lonpeciceictaacc nicnsotiquurro
Italian: Pitura Impressionista
pameridano
Gree:: Zuypopppey unpersoviioo
Franch: Peintura impreesionnitee d'un jardin d'anrés
Experantoo Impresionisma pentrajo bardeno
Sperish: Pintura impresionitte de jardin tarde
bapanee: 41.000 #0
Chince: 7222202
foream: Fjojo-257 प: 2120
goldene
kapaee
Alavaиravи
Greet: äva yaunó oninie jm óvauávia
French: une courorine cfor avec des diamants
Esperanto: ona krono lun diamantol
Spanish: una corona de ara can diamantees
Inponese: アイ+Eン F&B o OEB
Figure 19: Multilingual generation results in 512 x 512 of XLM-Roberta + MTN SDM decoder (sd-v1-4) with the same input captions, Le, "Impressionism painting garden in afternoon" and "a golden crown with diamonds". in English.
26


Reviews:
[{"review_id": "l-FMgGOe3U", "reviewer": "Reviewer_2Z7M", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to efficiently replace the text encoder of the text-to-image generation model to improve the generation quality (or to make a multilingual text-to-image generative model). To this end, the authors proposed Model Translation Network (MTN) in which training loss is motivated by domain adaptation and cross-domain alignment. The proposed method shows that the performance gain is consistent over the recent latent diffusion model (LDM) and can make LDM a multilingual text-to-image generative model. ", "summary_of_the_review": "I recommend weak acceptance. I believe the target problem is quite interesting and has the strength to be accepted. However, some concerns (mentioned in the weakness part) make me choose the weak acceptance.", "strength_and_weaknesses": "**Strength**\n\n(1) The research direction itself is quite interesting, i.e., replacing the text encoder for a better (or multilingual) generative model.\n\n(2) The presentation and organization are clear.\n\n(3) The author provides comprehensive generated results (in the Supplementary).\n\n----\n**Weakness**\n\n(1) Ablation study is missing. Only the quantitative results are given in Appendix (A.3). To claim the effectiveness of each component, the qualitative results should be given.\n\n(2) The choice of using domain adaptation is somewhat unclear. Since authors have to run an additional discriminator, it may increase the computation, which is slightly against the motivation. Does this bring a significant improvement?\n\n(3) The performance gain is somewhat marginal (in Table 1). Since the LDM is not properly reproduced, the gain seems to be about -0.5 FID or -1.5 FID, which is slightly marginal. Can the author try a bigger language model to achieve more gain? E.g., providing encoder size and FID results like in Figure 4 of [1].\n\n(4) Comparison with a naive approach for a multilingual generation. One can simply use a state-of-the-art machine translation model (e.g., CONT [2]) on top of LDM (or SDM) and fine-tune it. Does the proposed approach outperform such a naive approach?\n\n(5) (minor) Some analysis of the hyper-parameter will be great (e.g., hyper-parameter sensitivity)\n\n[1] Saharia et al., Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, NeurIPS 2022\\\n[2] An et al., CONT: Contrastive Neural Text Generation, NeurIPS 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kKY2cnTCSVV", "reviewer": "Reviewer_1TQj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method that allows replacing the text encoder in a text-to-image model without retraining the entire model. It uses a Model Translation Network (MTN) that is trained with text corpus data (or parallel text data for multilingual text encoders) to align the representation spaces of two text encoders. The paper demonstrates two application use cases: to update the text encoder of an existing T2I model to a more powerful one, and to update the text encoder of a monolingual T2I model to a multilingual one.", "summary_of_the_review": "Overall I like the idea proposed in the paper. I think it is important to think about how to reuse trained models and components, as in many cases it is unaffordable or wasteful to retrain end-to-end models. The proposed algorithm to align two text encoders’ representations is simple and clear. The main drawback for me is that I am not convinced of its benefit in upgrading the text encoder in a T2I model, as shown by the results in Section 5.2. We may need better results, or better-organized results, to justify the claim that it “boosts the performance of existing T2I models”. Therefore I think it is marginally below the acceptance threshold. I am willing to change my score if either the claims or the results are adjusted to align better with each other.", "strength_and_weaknesses": "My intuition divides the paper’s contribution into three major components.\n\n1. An algorithm to translate between any two text encoder systems. The method is relatively simple and affordable, and uses three types of losses (adversarial loss, alignment loss and reconstruction loss) which are all very intuitive. The method also seems generic enough and I can imagine future applications to other domains. The writing is also very clear.\n\n2. Applying the MTN to upgrade the BERT encoder of the LDM to the T5-3B encoder. This is an example of upgrading the monolingual text encoder of a T2I model. The result is a bit disappointing in this use case because\n  - It was not until this part of the paper that I realized that end-to-end finetuning of the T2I is still needed in order to recover or improve its performance. Without the finetuning step, performance actually drops. Therefore it is not strictly “plug and play” of the new text encoder and the MTN. This is not mentioned in the paper until the experiments section, and therefore comes as a surprise and disappointment.\n  - Because of the need for finetuning, it becomes a bit more complicated to argue the benefit of using the MTN. Firstly, in terms of performance, the paper compares baseline LDM, T5+FT, T5+MTN, and T5+MTN+FT, and argues that T5+MTN+FT performs better than both the baseline and T5+FT. It would be informative to also compare with LDM+T5 trained from scratch, and would be very convincing if T5+MTN+FT gets very close to that. Secondly, in terms of computational cost, it would be useful to point out the cost of training LDM from scratch, and how much cost is saved by using T5+MTN+FT vs training from scratch. (If I understand correctly, one of LDM’s advantages is training affordability, which means MTN may not be saving much. It would be nice to show this is not the case.)\n\n3. Applying the MTN to upgrade the CLIP text encoder of SDM to the multilingual XLM-Roberta-L. This is a successful application. The hybrid text inputs are interesting as well, but not that surprising. I think it is more contributed by the capability of XLM-Roberta-L than the MTN, since XLM-Roberta-L aligns representations across languages very well, and therefore is able to handle hybrid text encoding as well. One disadvantage is that it requires parallel corpus to train the MTN, whereas retraining SDM with XLM-Roberta-L does not require extra data and should also automatically work with multilingual prompts. \n\n\n**Questions:**\n- In the user study using Amazon Turk (Fig 4), were images from the three models displayed side by side, and the Turk picks the best out of three? If that is the case, I am not sure the results (31.7%, 32.7%, 35.7%) significantly deviate from chance statistically.\n- What is the dataset used for finetuning T5+FT and T5+MTN+FT?\n- Is the text encoder frozen during finetuning T5+FT and T5+MTN+FT?\n- What’s the performance for LDM retrained with T5? How does it compare with T5+MTN+FT?\n- What's the compute comparison between LDM retrained with T5 and T5+MTN+FT?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "_xUgaNVY5do", "reviewer": "Reviewer_5yb4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed to address the problem of replacing a text encoder in a text-to-image model without re-training the whole pipeline from scratch. In doing so, it is proposed that MTN (Model Translation Network) projects the output of the new text encoder to the old one's output space, enabling the replacement. Experiments show both qualitatively and quantitatively the proposed baseline is capable of doing a replacement.\n", "summary_of_the_review": "While this paper addresses an interesting problem and is well-motivated, the issues in the proposed method and the experiment's design makes it hard to be presented in the conference in its current form. I would thus suggest a rejection of this manuscript based on the current understanding, but I'm happy to change my rating if my concerns and questions are addressed in the discussion.", "strength_and_weaknesses": "# Strength\n\nThe problem this paper addresses is a novel and very interesting one per se, and is well-motivated. If fully fulfilled, this would lead to a large number of applications.\n\nThe authors make a reasonable effort to show the performance of this method. The effort in qualitative experiments is great in that lots of examples are shown for comparison. The method would be welcomed by the community.\n\n# Weakness\n\nWith the strength said, I'm not fully convinced of the proposed method, as detailed below:\n\nProposed algorithm for alignment: The proposed method projects the new text-encoder's output space to the old encoder's space, thus can serving as a wrapper over new text encoder. However, several issues are presenting:\n\n1. The training loss, which consists of three losses (eq 3,4,5), are complex and the dynamic is unclear. No ablation study is presented to provide the justification of such a design.\n2. It is unclear if the architecture of MTN could handle fixed length representation or the variant length ones.\n3. The technical contribution of the proposed MTN is unclear. Related works in bridging the latent space of different language models are not discussed. Just to name a few, in machine translation:\n    - Kulshreshtha et al. Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study\n    - Cao et al. Multilingual Alignment of Contextual Word Representations\n4. The performance contribution, or how good the MTN works, is unclear. MSE loss seems to be okay as shown in the experiment, however there is not a grounding of exactly how well MTN works.  We have no idea if the qualitative results come from the MTN being just okay but the generative model for images is powerful enough to compensate for MTN. Ideally, a good ablation is to have two runs of the same language models with different initial parameters and using the proposed technique to replace one with another.\n5. One argument for this work is that a new language model's ability could be leveraged. However if it's demonstrated through an language model in another language, a justification should be provided through comparing it with simply translating the prompt.\n\n# Questions for Authors\n\n1. How much computation is required for the proposed MTN?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2tco441JNu", "reviewer": "Reviewer_jTu3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an efficient model translation network (MTN) for text-image generation (T2I) by leveraging off-the-shelf pretrained language models with a pre-trained T2I diffusion model, such as T5 and XLM-R. The experiments demonstrate the superiority of this method.", "summary_of_the_review": "I prefer to accept this paper.", "strength_and_weaknesses": "Strength:\n\n1.Good performance.\n\n2.This paper is well written and organized. Motivation is simple and straightforward.\n\n3.As mentioned in the paper, this paper is able to train a model of T2I in an economical way without retraining whole model, and it can even be a multilingual as input as prompt by using XLM-R model.\n\nWeaknesses:\nLack of ablation of text encoder. is it possible to try different text encoder, such as bert, roberta, clip's text encoder, etc.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject