UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis
Authors: Zhu Zhang
Jianxin Ma
Chang Zhou
Rui Men
Zhikang Li
Ming Ding
Jie Tang
Jingren Zhou
Hongxia Yang
Venue: NeurIPS 2021 Poster (2021)
Keywords: conditional image synthesis
BERT
multi-modal control

Abstract: Conditional image synthesis aims to create an image according to some multi-modal guidance in the forms of textual descriptions, reference images, and image blocks to preserve, as well as their combinations. In this paper, instead of investigating these control signals separately, we propose a new two-stage architecture, UFC-BERT, to unify any number of multi-modal controls. In UFC-BERT, both the diverse control signals and the synthesized image are uniformly represented as a sequence of discrete tokens to be processed by Transformer. Different from existing two-stage autoregressive approaches such as DALL-E and VQGAN, UFC-BERT adopts non-autoregressive generation (NAR) at the second stage to enhance the holistic consistency of the synthesized image, to support preserving specified image blocks, and to improve the synthesis speed. Further, we design a progressive algorithm that iteratively improves the non-autoregressively generated image, with the help of two estimators developed for evaluating the compliance with the controls and evaluating the fidelity of the synthesized image, respectively. Extensive experiments on a newly collected large-scale clothing dataset M2C-Fashion and a facial dataset Multi-Modal CelebA-HQ verify that UFC-BERT can synthesize high-fidelity images that comply with flexible multi-modal controls.


UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis
Zhu Zhang', Jianxin Mat, Chang Zhou', Rui Men', Zhikang Lit, Ming Ding', Jīe Tang', Jingren Zhou', and Hongxia Yang* iDAMO Academy, Alibaba Group, *Tsinghua University (2hangghus503101@@mmaal com
jason.mjz, ericzhou. zc, yang www.areeaabbbbaiinnecaaaaa
Abstract
Conditional image synthesis aims to create an image according to some multi-modal guidance in the forms of textual descriptions, reference images, and image blocks preserve, as well as their combinations. In this paper, instead of investigating these control signals separately, we propose new two-stage architecture, UFC-BERT, unify any number of multi-modal controls. In UFC-BERT, both the diverse control signals and the synthesized image are uniformly represented a sequence of discrete tokens to be processed by Transformer. Different from existing two-stage autoregressive approaches such as DALL-E and VQGAN, UFC-BERT adopts non-autoregressive generation (NAR) at the second stage to enhance the holistic consistency of the synthesized image, support preserving specified image blocks, and to improve the synthesis speed. Further, we design progressive algorithm that iteratively improves the oon-autoregressively generated image, with the help of two estimators developed for evaluating the compliance with the controls and evaluating the fidelity of the synthesized image, respectively. Extensive experiments on a newly collected large-scale clothing dataset M2C-Fashion and a facial dataset Multi- Modal CelebA-HQ verify that UFC-BERT can synthesize high-fidelity images that comply with flexible multi-modi controls.
Introduction
Conditional image synthesis aims to create an image according to the given control signals. With the increasing demand for flexible conditional image synthesis, various kinds of control signals have been introduced into this field, which can be divided into three main modalities: (i) textual
controls (TC), including the class labels | and natural language descriptions 62 54 (ii) visual
controls (V (VC), such as ppatially-aligned sketch map for reference [17] 60 or another image for style
transfer [15 27); (iii) preservation controls (PC), which require the synthesized image to preserve some given image blocks, c.g., image outpainting and inpainting [63] 69]
However, control signals of various modalities possess different characteristics. Existing works 62 26 61 hence typically design separate methods customized for each control modality. Moreover, most of these approaches only utilize one type of control signal and cannot simultaneously combine multiple types of controls in a concise and versatile model. This begs the question: can we integrate any number of multi-modal control signals into unified framework for flexible conditional image synthesis? There are two inevitable challenges in this setting: (i) how to unify the multi-modal controls and represent them in a unified form, especial when employing multiple control signals from different modalities concurrently; (ii) how to guarantee the fulfillment of the multi-modal controls while ensuring the fidelity of the synthesized image.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

TC
Vintage shoulder pad grange suit coat
with
tile bus been clexned.
(a] Text to Image (TC)
(b) Sketch to Image (VC)
VC
UFC-BERT
PC
Style Transfer (VC)
(d) Inpainting (PC)
(e) Multi-Monne Control to Image (Ours)
Figure 1:
The three main modalities of control signals for conditional image synthesis: Textual Controls (TC), Visual Controls (VC), and Preservation Controls (PC).
13 47 has made great progress. The first stage learns convolutional autoencoder with quantized latent representations for converting image into sequence of discrete tokens, e.g., for compressing 256x256 image into sequence of 32 x 32 tokens where each token correlates mainly with an 8 8 block of the image. Converting a sequence of tokens back into an image is also supported. The second stage then typically adopts an autoregressive model, e.g., PixelCNN [4]] or a unidirectional Transformer decoder [55], capture the distribution over sequences of tokens. Particularly, the Transformer-basse methods [3 13 47 exploit the global expressivity of Transformer to capture long- mangge relationships between local constituents.
Recently, two-stage image synthesis [42]
In this paper, we make two key observations about the two-stage framework. First, the two-stage framework has the advantage that can potentially unify the multi-modal control signals and the generated image into a single sequence of discrete tokens. However, existing works 42 48
47 largely neglect this advantage of the two-stage framework approaches such as those based mainly on the generative adversarial networks (GAN) [8] Second, the autoregressive (AR) approach sequence generation, adopted by the existing two-stage methods such as DALL-E [47] and VQGAN [33, brings undesirable shortcomings: (i) the token-by-token synthesis procedure leads slow generation speed, especially for the heavyweight Transformer [3] (ii) each generated token can only catch sight of the previously generated tokens and cannot
the traditional one-stage
13 47 incorporate bidirectional contexts, which may affect the holistic consistency of image synthesis; (iii) the fixed left-to-right order of autoregressive decoding cannot respond to the preservation control signals unless the image blocks to be preserved are at the beginning of the sequence. Notably, different from AR generation, non-uuloregressive (NAR) sequence generation with bidirectional Transformer, i.e., BERT 9 can naturally avoid the three shortcomings.
Based on the aforementioned observations, we propose UFC-BERT, novel BERT-based two-stage framework to UniFy any number of multi-modal Controls for conditional image synthesis. Concretely, the textual, visual, and preservation control signals, as well as the generated image, are uniformly represented as a sequence of discrete tokens, as shown in Figure 2 word tokens for class labels or natural language descriptions. The visual control(s) and the generated image are both represented as discrete tokens due to the first stage, where each token corresponds to block within the reference image(s) or the generated image. Zero, one, more reference images are supported. To preserve a given image block within the generated image, we encode the given image block into discrete tokens and fix corresponding parts of the generated sequence to the tokens.
The textual control consists of
We train UFC-BERT via the masked sequence modeling task, which predicts masked subset of the target image's 's tokens conditioned on both the multi-modal control signals and the generation target's unmasked tokens. During inference, we adopt Mask-Predict, NAR generation algorithm [16] 21 7). which predicts all target tokens at the first iteration and then iteratively re re-mask and re-predict subset of tokens with low confidence scores. To further improve upon the NAR generation algorithm, we exploit the discriminative capability of the BERT architecture [I] 70 and add two estimators (see Figure where one estimator estimates the relevance between the generated image and the control signals, and the other one estimates the image's 's fidelity. The two estimators help improve the quality of the synthesized image, because at each iteration we can generate multiple samples and keep only the highly-score one before starting the next iteration. The two estimators also help save

---

the number of iterations needed, since the algorithm can dynamically terminate if running for more iterations no longer improves the scores.
The extensive experiments on M2C-Fashion, a newly collected clothing dataset with tens of millions of image-text pairs, well on Multi-Modal CelebA-HQ (28] [1], public facial dataset, demonstrate UFC-BERT can synthesize high-quality images that comply with various multi-modal controls.
2 Related Works
We have discussed the connection between our work and Two-Stage Image Synthesis in the Intro- duction. In this section, further discuss related works from other fields.
Conditional Image Synthesis. A variety of control signals have been introduced into conditional
image synthesis. The conditonnii generation task [
[9 adopts class labels as control signals. 62 72 54 further employs natural language descriptions
The texi-to-image synthesis task [64 65 30
as controls. The image-to- -o-eeagtt-ame translation task generates photo-realism images from visual controls,
such as a sketch map 17 60], semantic label map 26 58), human pose [37 or another image
for style transfer [15 27 Moreover, image outpainting and inpainting [25 63 can be regarded
as image synthesis conditioned on preservation control signals, where some image blocks of the
desired image are already specified and need to be preserved in the generated image. However, these works only utilize one kind of control signal and design their methods customized for each kind of
control. Text-guided image manipulation [10 40 69 B 61] semantically edits an image, where the
text description and the original image serve as control signals. But they still fail to unify multiple modalities in universal form and cannot easily extend more control modalities. To promote versatility and extensibility, we propose UFC-BER to unify any number of multi-modal controls.
Visual-Language Transformer. With great progress in language tasks [55
44
45
2 the transformer and audio [6]
architecture is being rapidly transferred to other fields such as vision
68
Recently, pretraining visual-language transformer 43 24
70 7
35
53
(e.g. multi-modal
BERT) has achieved significant improvements on a variety of downstream tasks, c.g. visual question answering, image captioning [70), and text-to-image generation 77. Among them, the single-stream architecture [52] 32 46, 5 43 24 uses a single transformer to jointly model a pair of text and
image. while the two-stream architecture [35] 36 53 applies two transformers separately learn the representations of the text and the image, respectively. Our UFC-BERT also a variant of the single-stream visual- lsngul-anguage transformer, but focuses on flexible multi-mode image synthesis instead of modii pretraining.
Non- Non-Autoregressive Sequence Generation. Though it is natural to autoregressively predict tokens from left to right when generating a sequence, autoregressive decoding suffers from the slow speed and sequential error accumulation. Thus, the non-autoregressive generation (NAR) paradigm is proposed
to avoid these drawbacks in neural machine translation [19 20 29 [6], image captioning [14 22 70),
and speech synthesis [50] 49]. These approaches often employ the bidirectional Transformer (i.e. BERT) as it is not trained with a specific generation order. Our progressive NAR generation algorithm improves upon the Mask-Predia relevance estimator and the fidelity estimator to facilitate sample selection and dynamic termination.
annaatoregressvve algorithm [57 16 38 33 , by introducing the
3 UFC-BERT For Multi-Modal Image Synthesis
3.1 Background: Two-Stage Image Synthesis
In this section, we review the two-stage architecture
42
48
13
47
for image synthesis.
At the first stage, a codebook Z = { K for vector quantization is learned, where 23 I is the k-th code-word in the codebook and K is the number of code-words. An image X IR xx can be transformed into (or from) a collection of code-words Z E kxxxx. Concretely, convolutional encoder E' first encodes the original image X as Ż E(X) E ghxwuxn Then element-wise quantization step q(-) (-) is applied each element Zu, to obtain the element's closest code-word Zı, i.e., (Zz,) = arg min z Zaz znll. For reconstruction, convolutional decoder D is also learned for recovering image X E R H w x3 from Z such that X is close X. The first stage can be denoted by
3

---

Relevance and Fidelity Estimation Task
Mosked Sequence Modeling Task
The Second Stage
0/1
0/1
UFC-BERT
Prxx
Post
0
Pass
Pros
[FD.]
LEOTT
[Ste]]
V
Estimation Tokens
Textual Control T
Visual Controll 1
Visual Coneral
Generated Image
Visual Control
with optional Preservation Control
Sequential Position Embedding
The First Stage
3
Spatial Pasition Embedding
Rashape
Token Embedding
Encoddr
Discretioo
Recover
Decoder
0/1
Relevance and Fidelity Score
Predicted Code
Figure 2: The framework of UFC-BERT, where the textual control (TC), vsiual control (VC), and preservation control (PC), as well the image generate, collectively form sequence of tokens.
Z = q(E()) X -D(Z).
(1)
Due to the convolutional layers, each of the h x w elements of 2 mainly correlates with H x W block of the image, though receptive field may be larger if multiple convolutions are /t
At the second stage, image X's quantized representation Z can be rewritten as a sequence of codes I E {0, (O...., - 11 composed of N, (= h x w)) indices from the codebook 2. Thus, image synthesis can be formulated as autoregressive sequence generation, i.c. predicting the distribution Pr Pr(4 C) of the next token L; conditioned on the preceding tokens I and the control signals C. The distribution is typically modeled using unidirectional Transformer. The likelihood is then Pr(I[C)
II, Lr,, C). Parameters are learned by minimizing CAR Ei |- log Pr(I[C)]
We focus on improving the second stage. Specifically, the autoregressive paradigm adopted by the existing two-stage works [42 bidirectional contexts, and cannot fully support preservation control signals. We thus propose UFC- BERT, novel NAR approach for stage two, to unify any number of multi-modal controls and tackle the shortcomings of AR. As for stage one, we directly follow VQGAN's design [47), which improves upon VQVAE 42 by incorporating perceptual loss (27) and patch-based adversarial training (26)
48
13 47
suffers from slow generation speed, fails capture
3.2 Problem Formulation
Conditional image synthesis aims generate image that satisfies set of control signals C. We consider three major modalities of control signals. A Textual Control (TC) consists of a sequence of words T .O..... | - Nr, where W is the vocabulary and NT is the number of words in the text. In the two-stage framework, an image can be converted into sequence of code-words (i.c. tokens) based on stage one's encoder E' and codebook Z. Thus, Visual Control (VC) denoted by sequence V E 10....,12 2 1) consisting of rode-words from the codebook Z, where Ny is the sequence length. Similarly, the target (i.e., the image to generate) a sequence of code-words 1).,, We support zero, one, or multiple visual controls for flexibility. As for the
I E {O...., |Z| | Preservation Control (PC), it sequence binary masks E (0, 1 with the same length as where 1 means that the token is known (i.e., 7, is ground-truth if P; = 1) while ( means the token needs to be predicted. aim to design a model the second stage synthesize the target image's sequence I conditioned on C, i.... combination of any number of control signals from {,, V, P))

---

3.3 Model Inputs
As shown in Figure 2 our UFC-BERT modifies the original BERT model 9 to accommodate any number of multi-modal controls. Similar to BERT, the backbone is a multi-layer bidirectional Transformer encoder, enabling the dependency modeling between all input elements. The input sequence of UFC-BERT always starts with two special tokens [REL] and [FDL] for relevance estimation and fidelity estimation, then goes on with the word sequence T of textual controls, code sequence V of visual controls, and ends with the code sequence I of the target image to generate. Two special separation tokens [EOT] and [EOV] are appended to the end of the textual and the visual control sequences, respectively. If there are multiple visual controls, another special token [SEP] is inserted to separate them. The sequence of the target image to generate may be partially or fully masked by special token [MASK]. When the preservation control P is present and P; = 1, token I; in I is always set to the code-word corresponding to the given image block to be preserved.
Each input token's 's representation is the sum of the position and token embeddings:
Position Embedding. Our UFC-BERT learns independent sets of position embeddings for the different kinds of the inputs to achieve better distinguishment between the various modalities. The position embeddings for the word sequence are the same BERT, i.c., we use sequential position embeddings. For a visual control or the target image, the position embedding of each token is decided according to where this token lies on the h x grid, i.e.,
spatial position embeddings.
Token Embedding. For textual controls, we use Byte-Pair Encoding to segment each word into sub-words and then learn sub-word embeddings. Each special token, e.g., [REL] or [MASK], is assigned dedicated embedding. For visual controls and the target image, we learn an embedding for each core-word. We do not directly use the embeddings from stage one's codebook due to the decoupling of the two stages.
3.4 Training: Masked Sequence Modeling with Relevance and Fidelity Estimation
As shown in Figure 2 we train UFC-BERT via masked sequence modeling, i.e., predicting the masked tokens in the target image conditioned on the controls. A relevance estimator and fidelity estimator are also trained in the process, and will be key to our progressive NAR generation algorithm.
Task 1: Masked Sequence Modeling. This task is similar to Masked Language Modeling (MLM) in BERT, but incorporates multi modal control signals when predicting the masked tokens. To construct training samples, we mask parts of the target image I to predict using four strategies: (1) randomly decide the number of tokens to mask, and then randomly mask the desired number of tokens; (2) mask all tokens; (3) mask the tokens within some boxed areas of the image, where the number of boxes and the box sizes are randomly decided; (4) mask the tokens outside some random boxed areas of the image. We use the four strategies with probability 0.70, 0.10, 0.10, and 0.10, respectively. To construct multi-modal control signal C for each training sample, there are four different combinations: <TC, VC>, <TC>, <VC>, <empty>, where <TC, VC> means the textual and visual controls are simultancously employed, <TC> or <VC> means only a textual or visual signal is used, and <empty> means no textual or visual control present. Note that the preservation control is already included in the masked sequence modeling task. Since our dataset does not contain ground-trutt pairs of visual controls and target images, we crop one or multiple regions of target image to construct VC for the target image. Because image synthesis from solely textual controls is more challenging than from other signals, we use the four combinations with probability 0.20, 0.55, 0.20, 0.05, respectively, where textual controls get more attention. We feed UFC-BERT's outputs at each position of into a softmax classifier over the codebook Z, which produces a probability score Y, = (r[,, [y, C) for each position E M, where M is the set of masked positions and U is the unmasked set. Finally, the masked sequence modeling task minimizes the softmax cross-entropy loss LMSM Ei Iu |- log Pr Lu, C)]. where Pr Lg C) I Yt-
Task 2: Relevance Estimation. This task is to learn a binary classifier that judges whether the generated image is relevant or irrelevant to the given multi-modaa control C. Briefly, we add linear layer on the output corresponding to the special token [REL]. The linear layer outputs scalar representing the logit, and a binary cross-entropy loss LREI. is added. During training, the training samples from Task serve as the positive instances (i.e. relevant pairs). We construct negative instances (i.e. irrelevant pairs) by swapping the control signals of two training samples.
5

---

Task 3: Fidelity Estimation. This task aims to distinguish whether the generated image is realistic from the view of human visual cognition. Similar to relevance estimation, we feed the output corresponding to [FDL] into a linear layer for binary classification and add another binary cross- entropy loss LIDL. Since the low-fidelity images (i.e. negative instances) do not exist in the dataset, we run UFC-BERT from previous epochs to synthesize images based solely on textual control signals, and use the synthesized images as negative instances.
We combine the three tasks' losses to train UFC-BERT, i.c.,
CUFC-BERT X1 CMSM + >2CREL + +ALLEDL
(2)
where \1, A and A are set to 1.0, 0.5, and 0.5 to balance the three losses. The masked sequence modeling task ignores the negative instances from the other two tasks, ie.. irrelevant pairs or unrealistic instances. And the fidelity estimation task is added only after a certain number of epochs.
3.5 Inference: Progressive Non-Autoorgressive Generation
We design a Progressive Non-Autoreereessive Generation (PNAG) algorithm for conditional image synthesis after training, which improves upon Mask-Predict [6 71 Mask-Predict predicts all target tokens when given a fully-masked sequence at the first iteration, and then iteratively re-mask and re-predict subset of tokens with low probability scores for a constant number of iterations. However, Mask-Predict cannot ensure the efficacy of multi-modal controls and the fidelity of the synthesized images, and requires determining the number of iterations. Our PNAG tackles its drawbacks via sample selection and dynamic termination, based on the relevance and fidelity estimators.
Each iteration of our PNAG algorithm consists of a Mask step and then a Predict step. Let ((,in))
(4((:mm) ...... 1 Nj (f.,m)) and [(t,out) = (1 (t..ma)) ..... Nr N (t..out) be the state of the target image's 's sequence before and after the t-th iteration, respectively. The tokens in (00.out) for t 0 is all set to [MASK] except for the positions that are controlled by the preservation signals, i.e., except for I r(0..out) that
has P.- 1. If a preservation control is present, i.e. P. = 1, we always set ((5.5m) and 1((.0ut) for all t to be the code-word that corresponds to the provided image block to be preserved.
Mask Step. At the beginning of iteration 1 (1 2 1), we construct the input sequence [((,6)) by (re- -ee--makkknn a subset of tokens in the generated sequence (tt-_, from the last iteration. Similar to beam search, we construct B parallel input sequences (I ...., ....,,( I(f.in)) } at each iteration. Specifically, we re-mask n tokens of ((t-,,)))) to produce each 1 I(f.in). We first sample N,
-n tokens from a multinomial distribution (t.(t.)n) proportional to the probability scores ((-1)) (Y:((-1)) ) (see Equation 3), computed by (t.5m)) Soft ( And other tokens re-masked and re-predicted the next Predict Step. Here N1 (8 T=t (a--)))) where is the initial mask ratio, B is the minimum mask ratio, and T is the maximum possible number of iterations, such that the number of tokens to re- re-mask gradually decreases after every iteration.
Predict Step. Given the control C and an input sequence I/ (((,)m) UFC-BERT estimates distribution Pr( (..mm C) for each masked position UFC-BERT also estimates the relevance score S and fidelity score Sf." regarding the image that it is about to synthesize, and summarizes the scores into a comprehensive score s(()) = S (1 SS, where o is coefficient for adjusting the importance of the two. We perform sample selection based on sf0 i.e., we select the b-th sequence T((.5m) with the highest s()), discard the others, and then generate (t.out)) based on the selected I ((,,in) as follows:
(f.out) ~ (r(((11((55m)) C)2 y(()) + Pr(í, = (er(i,=((t.0m) (f.out) ( (1.5m) C),
(3)
where each token l!t.out) is sampled from the multinomial distribution Pr((i,I((..n) .C) and the corresponding probability is assigned to y:(0. Note that we predict tokens for all masked positions regardless of the predictions' confidence. We also implement dynamic termination based on s/ Specifically, if the current iteration's score S.") is higher than Swar (initialized zero), we set Smaz to 56()) and record the current iteration as tmax- If Smaz does not increase after three consecutive iterations, we select [ (lt) as the final result and terminate generation algorithm.

---

MZC-Fashion
Multi-Modal CelebA-HQ
TC
VC
PC
image
TC
VC
PC
image
A Af
high chockbones, and blond hair. She is attractive, young.
Silk gouze stitching A line
0
Ø
He has the receding hairline and hair.
Grey knitted short sleeve T-shirt
She wears heavy makzup, and lipstick. She has arched eyebrows, big lips, and pointy
Two-pleds plecpleeee summer light
silk dress
3R AR
The has mustache, bushy eyebrows, narrow eyes, high
Lapel pad orange sult coat with pockets
Figure 3: Images generated by UFC-BERT under various combinations of textual controls (TC), visual controls (VC), and preservation controls (PC). Please see the supplemental material for more showcases, where we also include study the diversity of the images generated by UFC-BERT and analyze how the multiple control signals interfere with each other.
Experiments
4.1 Datasets and Hyperparameters
experiments, we focus on two practical fields of image synthesis: fashionable clothing and human faces. We collect very acrge-scale clothing dataset M2C-Fashion with Chinese text descriptions, which contains tens of millions of image-text pairs, much larger than the commonly used text-to-image datasets COCO (34) and CUB (56). Details of the dataset provided in the supplementary material. additionally use another high-resolution facial dataset Multi-Modal CelebA-HQ 28 6T].
Following the model setting of VQGAN [3], we use the 256 x 256 image size on the two datasets and transform each image to a discrete sequence 16 x codes, where the codebook size | is set to 1024. For the BERT model, we set the number of layers, hidden size, and the number of attention heads to 24, 1024, and 16, respectively. Our UFC-BERT has 307M parameters, same as the Transformer used by VQGAN. As for tlpee--paaameee of PNAG, we set the parallel decoding number B to 5 and the balance coefficient to 0.5. We set the initial mask ratio a, the minimum mask ratio 8, and the maximum iteration number T to 0.8, 0.2, and 10, respectively.

---

VC-1
VC-2
image VC-1 VC-2
VC-3
image
Figure 4: Image synthesis with multiple visual controls, where we crop regions from 2 images serve as the visual controls. UFC-BERT synthesizes images that naturally fuse the visual elements.
Table Comparisons with GAN baselines for text-to-image synthesis on Multi-Modal CelebA-HQ-
Method
AttnGAN
62
ControlGAN
DF-GAN
DM-GAN 7
TediGAN 61
UFC-BERT (our)
FID
125.98
116.32
137.60
131.05
106.37
66.72
LPIPS 4
0.512
0.522
0.581
0.544
0.456
0.448
Table Comparisons with the autoregressive two-stage method VQGAN for text-to-image synthesis.
4 means the lower the better, while 1 means the opposite. We evaluate speed on the same V100 GPU.
Automatic Metrics
Human Pairwise Study
Dutasets
Methods
Inference Speed
FID.
LPIPS
PSNR1
SSIM
Relevance
Fidelity
VQGAN (AR)
12.48
0.483
10.80
0.56
38.6%
44.2%
8.73 sec/sample
M2C-Fashion
UFC: BERT (NAR)
11.53
0.461
13.14
0.58
61.4%
55.8%
0.81 soc/sample
MudtiMModa
VQGAN (AR)
52.63
0,503
8.98
0.28
42.7%
46.9%
8.66 sec/sample
CelebA-IIQ
UFC-BERT (NAR)
66.72
0.448
9.56
0.29
57.3%
53.1% %
0.79 sec/sample
4.2 Flexibility of Multi-Modal Controls for Conditional Image Synthesis
In this section, we qualitatively verify the synthesis ability of UFC-BERT with three modalities of control signals, i.e., textual, visual, and preservation controls. The textual controls the texts paired with the images, which are already provided by the two datasets, while the visual controls are code sequences of cropped regions, e.g. regions that represent logos texture of clothes.
In Figure 3 we synthesize images conditioned on combinations of the three types of control signals. The results demonstrate UFC-BERT can unify any number of multi-modal controls synthesize high-quality images. Further, UFC-BERT supports one or multiple visual controls for more flexible synthesis, shown in Figure 4 where generate images given 2~3 visual controls. We observe that UFC-BERT can reasonably fuse multiple visual elements and produce a harmonious image.
4.3 Quantitative Comparison to Existing Methods for Text-to-Image Synthesis
In this section, investigate how our UFC-BERT quantitatively compares to existing models. Considering most existing methods only utilize one control signal, we select the most common and challenging task text-to-image synthesis to compare the synthesis ability.
First, we compare our UFC-BERT with GAN-based text-to-image models AttiGAN 62 Control-
GAN 30, DE-GAN 54. DM-GAN 71 and TediGAN [61] the Multi-Modal CelebA-HQ dataset. For evalution, we adopt two automatic metrics FID 23 and LPIPS [66]. We report the results on T and our UFC-BERT achieves the best performance on the two metrics, even outperforming the TediGAN that uses slow and complex instance-level optimization. This demonstrates the two-stage architecture and non aon-eerrrerressive generation of UFC-BERT are suitable for text-to-image synthesis.

---

Women's striped long-sleeved orange shirt.
*
This woman has wavy hair and is wearing earrings, and lipstick.
t=1
t=2
t=3
t=4
t=5
t=
t=7
t=8
t=9
t=10
Figure 6: The iterative inference process of our PNAG algorithm. The red bounding box means the image has the highest comprehensive score and is selected as the final output result.
Besides, compare UFC-BERT with the autoregressive two-stage method VQGAN from three aspects: (i) the automatic metrics FID for image quality,
SHE 2A
7 Two
fair, pointy nese, mouth sightly open, rosy cheeks
Men's danim style lace (counterfactua))
Roral mid- mid-length
well as LPIPS, PSNR [59] and SSIM (59] for the sim- ilarity between the generated image and the ground truth; (ii) the Rele- vance and Fidelity metrics are eval- uated through user study, where the users are asked to judge which model's 's output is more relevant to the
VOGAN
UFG-BERT
textual descriptions, and more photo- realistic; (iii) the synthesis speed of two approaches. Note that autoregressive inference implementa- tion of VQGAN has been optimized by caching the preceding computation as in Transformer XL [8], and UFC- BERT and VQGAN have the same pa- rameter number (307M) for fair com- parison. For the user study, the two models receive the textual signals, and each generates 50 images. We collect the pairwise comparison results from five volunteers.
Real
Not Exist
Figure 5: Typical examples of UFC-BERT and VQGAN for text-to-image synthesis, including a counterfactual case.
As shown in Table 2 our UFC-BERT achieves better performance for almost all criteria with about 11x speedup. This suggests our non-autoregressive UFC-BERT with progressive NAR generation algorithm can synthesize high-fidelity images relevant to textual descriptions. As for the FID metric, UFC-BERT outperforms VQGAN on M2C-Fashion, but has worse performance on Multi-Modal CelebA-HQ. it may be due to the fact that the autoregressive VQGAN the pattern of a small dataset (only 30,000 facial images). In Figure S we further show typical generated examples intuitively display the difference between the approaches, including of counterfactual generation. We find that UFC-BERT can synthesize high-quality images, even for the counterfactual case.
easily memorize
4.4 The Effectiveness of Our Progressive NAR Generation Algorithm
In this section, we first visualize in Figure 6 the iterative process of our PNAG inference method based on the relevance and fidelity estimators. The images with red bounding boxes are the final outputs that match the textual control signals. We can find that the fidelity and relevance of the images increase after a few iterations, verifying our PNAG algorithm can guide the inference process towards better direction and synthesize more realistic images that match the control signals.

---

Table 3: Ablation studies of our PNAG inference algorithm. PNAG(w/, REF) and PNAG(w/o. FDL) set B the default value 5. MNAG the original Mask-Predict algorithm 16
Dataset
Metries
MNAG [[6]
PNAGtWo, REF)
PNAG(Wo. FDL)
PNAG(B=1)
PNAG(B=5)
PNAG(B=10)
FID 4
14.77
12.17
13.14
12.72
11.53
11.14
M2C-Fashion
LPIPS 4
0.488
0.477
0.469
0.456
Multi-Modal CelebA-HH HQ
FID 1
72.04
68.90
70.32
69.49
66.72
65.30
LPIPS 1
0.514
0.469
0.463
0.475
0.448
0,445
We then conduct ablation studies of PNAG. As shown in Table 3 we develop three ablated inference methods PNAG(w/o. REF), PNAG(w/o. FDL) and MNAG, where PNAG(w/o. REF) and PNAG(w/o. FDL) discard the relevance estimator and the fidelity estimator, respectively, and MNAG is the original Mask-Predict method [16] without any estimator. The results demonstrate that the two estimators effectively utilize the discriminative capability of UFC-BERT and do help improve the synthesis quality. Additionally, we vary the crucial hyper-paramete of PNAG B (i.e. the parallel decoding number during inference) from 10, and the results in Table 3 show that larger B is beneficial to the synthesis quality.
5
Conclusions
We proposed UFC-BERT to unify any number of multi-modal controls in universal form for conditional image synthesis. We utilized non-autoregressive generation to improve inference speed, enhance holistic consistency, and support preservation controls. Further, designed a progressive generation algorithm based on relevance and fidelity estimators to ensure relevance and fidelity.
References
[1] Andrew Brock, Jeff Donahue, Karen Simonyan. Large scale gan training for high fidelity natural image synthesis, arXiv preprint arXiv: 1809. /096, 2018.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamners.
arXiv preprint vrXiv:2005.1466. 2020.
[3] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun. David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703,
PMLR, 2020.
[4] Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 151 1511-1, 2017.
[5] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Cheng, and Jingjing
Liu. Uniter: Learning universal image-text representations. 2019.
[6] Rewon Child, Scout Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse
transformers. arXiv preprint arXiv: 10ti://994.00009 10509, 2019.
Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannanch Hajishirzi, and Aniruddha Kembhavi. X-Ixmert: Paint, caption and answer questions with multi-modal transformers. arXiv preprint arXiv:2009. / 1278, 2020. Zihang Dai, Zhilin Yang. Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdi- nov. Transformer-xt xl: Attentive language models beyond a fixed-length context. arXiv preprint urXiv:1901.02800 2019.
[9] Jacob Devlin. Ming-Wel Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of the Conference on The North American Chapter of the Association for Computational Linguistics, 2019.
[10] Hao Dong, Simiao Yu, Chao Wu, and Yike Guo. Semantic image synthesis via adversarial learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 5706-5714 14, 2017.
[11] Li Dong, Nan Yang. Wenhui Wang, Furu Wei, Xiaodong Liu, Wang, Jianfeng Gao, Ming Zhou, and Mkiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. arXiv preprint (t0i:19955009999 2019. 2019.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth Transformers for image recognition scale. arXiv preprint 2rXiv:2010.11922 2020.
16x16 words:
10

---

[13] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[14] Junlong Gao, Xi Meng, Shiqi Wang, Xia Li, Shanshe Wang. Siwei Ma. and Wen Gao. Masked non- autoregressive image captioning. arXiv preprint 1906::9906000771
2019.
[15] Lcon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2414-2423, 2016.
[16] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding
of conditional masked language models. arXiv preprint SE4i:::904409995 2019.
[17] Arnab Ghosh, Richard Zhang, Puneet K Dokania, Oliver Wang, Alexei A Elros, Philipp HS Torr. and Eli Shechtman. Interactive sketch & fill: Multiclass sketch-to-image translation. In Proceedings of the IEEE
International Conference on Computer Vixion, pages 1171-1180, 2019.
[18] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu. David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv::406..2... 1406. 2601, 2014. [19] Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autorogressive neural machine translation. arXiv preprint XrXiv: 1111..122222 1.02281, 2017.
[20] Junliang Guo, Xu Tan, Di Hc, Tao Qin, Linli Xu, and Yan Liu, Non-autoregressive neural machine translation with enhanced decoder input. In Proceedings of the American Association for Artificial Intelligence, volume 33, pages 3723-3730, 2019.
[21] Junliang Guo, Zhirui Zhang, Linli Xu, Hao-Ran Wei, Boxing Chen, and Enhong Chen. Incorporating bert into parallel sequence decoding with adapters. In Advances in Neural Information Processing Systems. 2020.
[22] Longteng Guo, Jing Liu, Xinxin Zhu, Xingjian He, Jie Jiang, and Hanqing Lu. Non-autoregressive image captioning with counterfactuals crittcca multi-agent learning. arXiv preprint arXiv:2005 04690,, 2020.
[23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by two time-scalc update rule converge a nash equilibrium. 2017.
[24] Zhicheng Huang. Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint 2rXiv:2004,00 00849, 2020.
[25] Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Let there be color! joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Transactions on Graphics (ToG), 1-(4):1-11, 2016.
[26] Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexci A Efros. saagg-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pages 1 1125 1134, 2017.
(27) Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In Proceedings of the European Conference on Computer Vision, pages 694-711. Springer, 2016.
[28] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 10196, 2017.
[29] Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. arXiv preprint
2018.
[30] Bowen Li, Xiaojuan Qi. Thomas Lukasiewiez, and Philip HS Torr. Controllable text-to-image to-image generation.
arXiv preprint arXiv:1909.000 10009,0703,..00 2019.
[31] Bowen Li, Xiaojuan Qi. Thomas Lukasiewiez, and Philip HS Torr. Manigan: Text-guided image ma-
nipulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7880-7889, 2020.
[32] Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vi: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the American Association for Artificial Inselligence, volume 34, pages 11336 11344. 2020.
[33] Yi Liao, Xin Jiang, and Qun Liu. Probabilistically masked language model capable of autoregressive generation in arbitrary word order. arXiv preprint arXiv:2004. 11579, 2020.
[34] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitiick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision, pages 740-755. Springer, 2014.
[35] Jiasen Lıi, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint
(tte::900.00220 2019.
[36] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10437-10446, 2020.
[37] Liqian Ma. Xu Jia. Qianru Sun. Bernt Schiele, Tinne Tuytelaars, and Lue Van Gool. Pose guided person image generation. arXiv preprint arXiv: 0ttp/7055.938... 2017.
[38] Elman Mansimov, Alex Wang, Sean Welleck, and Kyunghyun Cho. A generalized framework of sequence generation with application undirected sequence models. arXiv preprint arXiv:190..12794 2019.
11

---

[39] Takeru Miyato and Masanori Koyama.
cgans with projection discriminator.
arXiv preprint
180iv.8800..80 05630, 2018.
[40] Seonghyeon Nam, Yunji Kim, and Seon Joo Kim. Text-adaptive generative adversarial networks: manipu- lating images with natural language. arXiv preprint arXiv:1810.11 11919, 2018.
[41] Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelenn decoders. arXiv preprint arXiv: 1606.05328, 2016.
[42] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. arXiv preprint arXiv: 1009300077222 2017.
[43] Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. Imagebert: Cross-modal pre-training with large-scale weak-supervvsed image-text data. arXiv preprint arXiv:2001.0990 079666, 2020.
[44] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
[45] Alce Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8),9, 2019.
[46] Wasifur Rahman, Md Kamrul Hasan, Amir Zadeh, Louis-Philippe Moreney, and Mohammed Ehsan Hoque. M-bert: Injecting multimodal information in the bert structure. arXiv preprint arXiv:1908.0577 2019. [47] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint urXiv:2102. 2021.
[48] Ali Razavi. Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. arViv preprint arXiv: /00/ 000600446, 2019.
[49] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tic-Yan Liu. Fastspeoch 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.044 04558, 2020.
[50] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tic-Yan Liu. Fastspeech: Fast, robust and controllable text speech. arXiv preprint wxXv::195.00926 2019.
[51] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv: 07909, 2015.
[52] Weijie Su, Xizhou Zhu, Yue Cao. Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual- -inguuiic representations. arXiv preprint arXiv: /908 08530, 2019.
[53] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint pxxivv/90..000
2019.
[54] Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao-Yuan
Yuan Jing, Fei Wu, and Bingkun Bao. Df-gan; Deep fusion generative adversarial networks for text-lo-image synthesis. arXiv preprint arXiv: 05865, 2020. [55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008, 2017.
[56] Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The caltech-uesd
2irds-200-2011 dataset. 2011.
[57] Alex Wang and Kyunghyun Cho. Bert has a mouth, and it must speak: Bert as a markov random field language model. arXiv preprint 19Xxvv/902.00000 040944 2019,
[58] Ting-Chun Wang. Ming- Yu Liu, Jun- Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8798-8807, 2018
Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. JEEE transactions on image processing, 13(4)x600-6) 612, 2004. [60] Weihao Xia, Yujiu Yang. and Jing-Hap Xuc. Cali-sketch; Stroke calibration and completion for high-quality face image generation from poorly-drawn sketches, arXiv preprint arXiv:1911 0atp//111.002.... 2019.
[59]
[61] Weihao Xia, Yujiu Yang, Jing- Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse image generation and manipulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[62] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang. Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 1316 1324, 2018.
[63] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEF/CVF Internationaal Conference on Computer Vision, pages 4471-4480, 2019.
[64] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 5907-5915, 2017. 2017.
[65] Han Zhang. Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang. Xiaolei Huang, and Dimitris N Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. JEEE transactions on pattern analysis and machine intelligence, -147 2018
12

---

[66] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features a perceptual metric. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 586-595, 2018.
[67] Shengyu Zhang, Tan Jiang. Tan Wang, Kun Kuang, Zhou Zhao, Jianke Zhu, Jin Yu, Hongxia Yang. and Fei Wu. Devlbert: Learning deconfounded visio-linguistic representations. In Proceedings of the 28th ACM International Conference on Multimedia, pages 4373 43322, 2020.
[68] Shengyu Zhang, Ziqi Tan, Zhou Zhao, Jin Yu, Kun Kuang, Tan Jiang, Jingren Zhou, Hongxia Yang, and Fei Wu. Comprehensive information integration modeling framework for vidco titling. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2744 22544, 2020.
[69] Zijian Zhang, Zhou Zhao, Zhu Zhang, Baoxing Huai, and Jing Yuan. Text- Text-guided image inpainting. In Proceedings of ACM International Conference on Multimedia, pages
2020.
[70] Luowei Zhou, Hamid Palangi, Lei Zhang. Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision- language pre-training for image captioning and vqa. In Proceedings of the American Association for Artificial Intelligence, volume 34, pages 3041-130499 2020.
[71] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for -iktee synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5802-5810, 2019.
[72] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region- adaativee normalization. In Proceedings of the IEEE Conference Computer Vision and Pateen Recognition, pages 5104-5113, 2020.
13


Reviews:
[{"review_id": "Rr3L55beSv", "reviewer": "Reviewer_5cvG", "summary": "In this paper, instead of using auto-regressive transformer to generate the discrete tokens like \"taming transformer\", it cleverly employs the well-known BERT model of NLP area, which not only supports the bi-directional conditions for local image editing, meanwhile vastly speeding up the sampling efficiency. The experiments also demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n* The idea of using extra global tokens to judge the current generation status is very interesting.\n* The designed progressive non-autoregressive generation is promising.\n* This paper is well-written and easy to follow.\n\n\nCons:\n\n* What are the details of sampling the re-mask tokens? I am not very clear about the descriptions in Line232-Line233.\n* I think the authors ignore one very important problem of the whole framework. The original pixel contents may not be well preserved while giving **the masks with arbitrary shapes**, especially for the boundary regions, which is caused by the mis-alignment of downsampled masks and discrete representation.  This issue should be discussed in the submission.\n* The papers which also use bi-directional transformer for image generation like [1,2] should be discussed in the related works.\n\n1. High-Fidelity Pluralistic Image Completion with Transformers, arxiv 2021 \n2. Diverse Image Inpainting with Bidirectional and Autoregressive Transformers, arxiv 2021\n\nI will consider further raising the scores if the concerns could be well solved.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the above mentioned comments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "GQk67JsvLb5", "reviewer": "Reviewer_BADA", "summary": "### Summary\nThe paper works on conditional image synthesis using a transformer-based network. The proposed method uses one network to combine multi-modal conditional signals, such as textual information and visual information, for image generation. The proposed non-autoregressive training strategy is validated on two datasets, the collected clothing dataset and Multi-Modal CelebA-HQ, and shows better results than GAN-based methods and a transformer-based method.\n\n### Strengths\n- The proposed method can utilize multi-modal controls for image synthesis and show convincing synthesized images containing input condition information. The image generation results conditioned on multi-modal inputs are interesting and inspiring.\n- While recent autoregressive-based methods demonstrate good text-to-image generation results using transformers, the paper shows that non-autoregressive transformers (e.g., BERT used in the paper) can also be trained for generating a sequence of image tokens.\n- The careful design of the relevance and fidelity estimation tasks is novel and helps the training of the network.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have some questions regarding the methods and experiment details.\n\n### Methods\n- On Ln. 189, the paper mentions that there could no textual or visual control. In this case, how could the network be trained if all the image tokens are masked (case (2) on Ln. 183)?\n- For constructing negative pairs in Task 2 (Relevance Estimation), do authors use images without masked tokens, or could the tokens still be masked? If the tokens still could be masked, do authors perform Task 1 for the generated missing tokens?\n- I'm a little confused about Ln. 232-233. The method seems to mask tokens with high probability scores by sampling from a multinomial distribution. But my understanding is that the method aims to mask tokens with low probability scores. Is the description on Ln. 232-233 accurate?\n\n### Experiments\n- For experiments on M2C-Fashion and Multi-Modal CelebA-HD, do authors train the network from scratch or from pre-trained models? If from pre-trained models, how much performance improvement compared with training from scratch?\n- I wonder whether authors have tried any experiments on the CUB200 bird dataset.\n- For training the VQGAN transformer, could the authors provide more details on how to process the textual information? Is it the same for training UFC-BERT?\n- On Ln.309-310, the authors mention autoregressive VQGAN can more easily remember the small dataset. I wonder would authors explain why autoregressive training can more easily remember the dataset?\n\n### Minor\nWill the code and dataset be released?\n\n===============\n\nThanks authors for submitting the rebuttal. The rebuttal addresses most of my concerns and answers the questions from other reviewers. Although I am concerned about the performance of the proposed framework on different datasets (the paper only shows two datasets), the paper introduces a new way for generating images using multi-modalities, and it is more flexible than auto-regressive transformers. I believe the method can contribute to more potential applications.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It would be better if authors could discuss more about the limitation of the methods.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "BnHgrOdREsQ", "reviewer": "Reviewer_wtTT", "summary": "This paper proposes a two-stage non-autoregressive (NAR) transformer-based approach for multi-modal conditional image generation in the discrete latent space of a pretrained VQGAN, called UFC-BERT. More specifically, the proposed model draws inspiration from BERT [1] and Mask-Predict [2] and uses masked language (token) modeling to (i) incorporate bidirectional context and (ii) significantly speed up generation over autoregressive baselines. Sampling from the masked model is performed via an extension of the progressive mask-predict algorithm, where two newly introduced discriminative heads of the transformer model guide the sampling process to generate samples that are consistent with the given conditioning and have high fidelity.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Strengths:__\n\nThe paper presents high-quality image synthesis results with a BERT-style model, previously unseen with these type of models.\nFurthermore, the paper does a great job at motivating the proposed solution (especially in line 43-56).\nThe proposed architecture and sampling process is simple but effective. Additionally, the two discriminative tasks (relevance and fidelity estimation) can be effectively trained via swapping of the conditioning information (relevance) and replay of samples from previous model checkpoints (fidelity), which is a nice feature. Using the BERT-formulation allows to condition on arbitrary patches of images (in contrast to the \"completion\" property of autoregressive models), where especially Fig. 4 provides impressive results on mixing different patches into a coherent image.\n\nThe model is described in detail in Sec. 3 and it should be easy to re-implement, given the provided information.\n\n__Weaknesses:__\n\n- What's the purpose of Figure 1? Neither sketch-to-image nor artistic style transfer are demonstrated with UFC-BERT. This should be clarified in the corresponding caption.\n- It remains somewhat unclear how the cropping for VC and PC works. Is this performed in the latent space of the VQGAN, and correspondences are assigned via the mentioned $\\frac{H}{h} \\times \\frac{W}{w}$ token-pixel correlations? If so, what is the effect of the non-local block (=attention layer) present in the VQGAN? Furthermore, crops can then only be made with a stride of 16 pixels.\n- The paper repeatedly states that the relevance and fidelity estimators are \"key\" to the generation algorithm, but does not provide experiments to justify this claim.\n- No metric that explicitly measures correspondence between generated images and corresponding textual descriptions is provided, the paper rather takes a detour by measuring similarity of ground-truth images and samples. However, since there is not a one-to-one correspondence between textual description and image, these metrics (LPIPS, SSIM, PSNR) might not be a good fit. On the contrary, LPIPS has been used to measure _diversity_ of generated samples, where higher scores are better [3]. Using R-precision as in [4] or CLIP-scores [5] could be a simple fix.\n\n__Originality:__\nThe proposed method of sampling latent tokens learned via vector-quantized representation learning (VQVAE/VQGAN) with a BERT-style model is new and allows for applications such as image completion or harmonization of different patches. Furthermore, multi-modal control has been underexplored in the context of VQVAE/VQGAN models.\n\n__Clarity/Quality:__\nThe presentation is clear, and, as described above (strengths), the model should be easy to re-implement.\n\n__Significance:__\nGiven the promising results on face and fashion datasets, the approach could have potential to serve as a generic neural image manipulation/creation mechanism when trained in a large-scale setting.\n\n__Mixed Comments/Questions/Suggestions:__\n- How does the diversity of samples generated with IFC-BERT compare to an explicit AR-likelihood model?\n- I would rather call the \"counterfactual case\" (l. 312 and Fig. 5) a \"conflicting prompt\".\n- Will the dataset (M2C) be released? A large-scale dataset of (text, image) pairs would be of great interest to the community.\n- The results depicted in Figure 4 are nice, but more should be added (maybe in the appendix) to demonstrate that the method works robustly.\n- After how many epochs exactly is the $\\mathcal{L}_{FDL}$-loss applied?\n- How robust is the model to different choices of $\\lambda_1$, $\\lambda_2$ and $\\lambda_3$ in Eq. (2)?\n- What is the effect of codebook size $|\\mathcal{Z}|$ on the model?\n- Fig. 3 should include a visualization when gradually adding more and more information of the _same_ ground truth prompt, i.e. TC, TC+VC and finally TC+VC+PC.\n- How big (if at all) is the problem of overfitting?\n\n__References:__\n - [1]: Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec-\ntional transformers for language understanding. In Proceedings of the Conference on The North American\nChapter of the Association for Computational Linguistics, 2019.\n - [2]: Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding\nof conditional masked language models. arXiv preprint arXiv:1904.09324, 2019.\n - [3]: Mahajan, Shweta, Iryna Gurevych, and Stefan Roth. \"Latent normalizing flows for many-to-many cross-domain mappings.\" arXiv preprint arXiv:2002.06661 (2020).\n - [4]: Xu, Tao, et al. \"Attngan: Fine-grained text to image generation with attentional generative adversarial networks.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n - [5]: Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" arXiv preprint arXiv:2103.00020 (2021).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal implications are sufficiently discussed. However, I find it interesting that regarding the facial manipulation possibilities of the proposed approach, the only \"negative\" point raised is the possible deception of facial recognition systems. Depending on one's perspective and culture, one could probably assume the complete opposite here.\n\nThere is no section specifically addressing the limitations of the proposed approach.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a method to unify any number of multi-modal control signal inputs for conditional image synthesis. The key thing was a transformer model that could convert a variable number of multi-modal control inputs to the discrete latent space of a VQGAN. All the reviewers rated the paper above the bar, with one reviewer upgraded the score from 6 to 7 after the rebuttal. The meta-reviewer agreed with the assessment and concluded the paper is above the bar. Please include the reviewer feedback in the updated manuscript in the final version.
Decision: Accept (Poster)