UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis
Year: 2021

Abstract:
Conditional image synthesis aims to create an image according to some multi-modal guidance in the forms of textual descriptions, reference images, and image blocks to preserve, as well as their combinations. In this paper, instead of investigating these control signals separately, we propose a new two-stage architecture, UFC-BERT, to unify any number of multi-modal controls. In UFC-BERT, both the diverse control signals and the synthesized image are uniformly represented as a sequence of discrete tokens to be processed by Transformer. Different from existing two-stage autoregressive approaches such as DALL-E and VQGAN, UFC-BERT adopts non-autoregressive generation (NAR) at the second stage to enhance the holistic consistency of the synthesized image, to support preserving specified image blocks, and to improve the synthesis speed. Further, we design a progressive algorithm that iteratively improves the non-autoregressively generated image, with the help of two estimators developed for evaluating the compliance with the controls and evaluating the fidelity of the synthesized image, respectively. Extensive experiments on a newly collected large-scale clothing dataset M2C-Fashion and a facial dataset MultiModal CelebA-HQ verify that UFC-BERT can synthesize high-fidelity images that comply with flexible multi-modal controls.

heading: 1 Introduction
text: Conditional image synthesis aims to create an image according to the given control signals. With the increasing demand for flexible conditional image synthesis, various kinds of control signals have been introduced into this field, which can be divided into three main modalities: (i) textual controls (TC), including the class labels [1] and natural language descriptions [62, 54]; (ii) visual controls (VC), such as a spatially-aligned sketch map for reference [17, 60] or another image for style transfer [15, 27]; (iii) preservation controls (PC), which require the synthesized image to preserve some given image blocks, e.g., image outpainting and inpainting [63, 69].
However, control signals of various modalities possess different characteristics. Existing works [62, 26, 61] hence typically design separate methods customized for each control modality. Moreover, most of these approaches only utilize one type of control signal and cannot simultaneously combine multiple types of controls in a concise and versatile model. This begs the question: can we integrate any number of multi-modal control signals into a unified framework for flexible conditional image synthesis? There are two inevitable challenges in this setting: (i) how to unify the multi-modal controls and represent them in a unified form, especial when employing multiple control signals from different modalities concurrently; (ii) how to guarantee the fulfillment of the multi-modal controls while ensuring the fidelity of the synthesized image.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Recently, two-stage image synthesis [42, 48, 3, 13, 47] has made great progress. The first stage learns a convolutional autoencoder with quantized latent representations for converting an image into a sequence of discrete tokens, e.g., for compressing a 256×256 image into a sequence of 32 × 32 tokens where each token correlates mainly with an 8×8 block of the image. Converting a sequence of tokens back into an image is also supported. The second stage then typically adopts an autoregressive model, e.g., PixelCNN [41] or a unidirectional Transformer decoder [55], to capture the distribution over sequences of tokens. Particularly, the Transformer-based methods [3, 13, 47] exploit the global expressivity of Transformer to capture long-range relationships between local constituents.
In this paper, we make two key observations about the two-stage framework. First, the two-stage framework has the advantage that it can potentially unify the multi-modal control signals and the generated image into a single sequence of discrete tokens. However, existing works [42, 48, 13, 47] largely neglect this advantage of the two-stage framework over the traditional one-stage approaches such as those based mainly on the generative adversarial networks (GAN) [18]. Second, the autoregressive (AR) approach to sequence generation, adopted by the existing two-stage methods such as DALL-E [47] and VQGAN [13], brings undesirable shortcomings: (i) the token-by-token synthesis procedure leads to slow generation speed, especially for the heavyweight Transformer [3, 13, 47]; (ii) each generated token can only catch sight of the previously generated tokens and cannot incorporate bidirectional contexts, which may affect the holistic consistency of image synthesis; (iii) the fixed left-to-right order of autoregressive decoding cannot respond to the preservation control signals unless the image blocks to be preserved are at the beginning of the sequence. Notably, different from AR generation, non-autoregressive (NAR) sequence generation with bidirectional Transformer, i.e., BERT [9], can naturally avoid the three shortcomings.
Based on the aforementioned observations, we propose UFC-BERT, a novel BERT-based two-stage framework to UniFy any number of multi-modal Controls for conditional image synthesis. Concretely, the textual, visual, and preservation control signals, as well as the generated image, are uniformly represented as a sequence of discrete tokens, as shown in Figure 2. The textual control consists of word tokens for class labels or natural language descriptions. The visual control(s) and the generated image are both represented as discrete tokens due to the first stage, where each token corresponds to a block within the reference image(s) or the generated image. Zero, one, or more reference images are supported. To preserve a given image block within the generated image, we encode the given image block into discrete tokens and fix corresponding parts of the generated sequence to the tokens.
We train UFC-BERT via the masked sequence modeling task, which predicts a masked subset of the target image’s tokens conditioned on both the multi-modal control signals and the generation target’s unmasked tokens. During inference, we adopt Mask-Predict, a NAR generation algorithm [16, 21, 7], which predicts all target tokens at the first iteration and then iteratively re-mask and re-predict a subset of tokens with low confidence scores. To further improve upon the NAR generation algorithm, we exploit the discriminative capability of the BERT architecture [11, 70] and add two estimators (see Figure 2), where one estimator estimates the relevance between the generated image and the control signals, and the other one estimates the image’s fidelity. The two estimators help improve the quality of the synthesized image, because at each iteration we can generate multiple samples and keep only the highly-scored one before starting the next iteration. The two estimators also help save
the number of iterations needed, since the algorithm can dynamically terminate if running for more iterations no longer improves the scores.
The extensive experiments on M2C-Fashion, a newly collected clothing dataset with tens of millions of image-text pairs, as well as on Multi-Modal CelebA-HQ [28, 61], a public facial dataset, demonstrate UFC-BERT can synthesize high-quality images that comply with various multi-modal controls.

heading: 2 Related Works
text: We have discussed the connection between our work and Two-Stage Image Synthesis in the Introduction. In this section, we further discuss related works from other fields.
Conditional Image Synthesis. A variety of control signals have been introduced into conditional image synthesis. The class-conditional generation task [1, 39] adopts class labels as control signals. The text-to-image synthesis task [64, 65, 30, 62, 72, 54] further employs natural language descriptions as controls. The image-to-image translation task generates photo-realism images from visual controls, such as a sketch map [17, 60], semantic label map [26, 4, 58], human pose [37] or another image for style transfer [15, 27]. Moreover, image outpainting and inpainting [25, 63] can be regarded as image synthesis conditioned on preservation control signals, where some image blocks of the desired image are already specified and need to be preserved in the generated image. However, these works only utilize one kind of control signal and design their methods customized for each kind of control. Text-guided image manipulation [10, 40, 69, 31, 61] semantically edits an image, where the text description and the original image serve as control signals. But they still fail to unify multiple modalities in a universal form and cannot easily extend to more control modalities. To promote versatility and extensibility, we propose UFC-BERT to unify any number of multi-modal controls.
Visual-Language Transformer. With great progress in language tasks [55, 44, 45, 2], the transformer architecture is being rapidly transferred to other fields such as vision [3, 68, 12] and audio [6]. Recently, pretraining visual-language transformer [43, 24, 70, 7, 35, 53, 67] (e.g. multi-modal BERT) has achieved significant improvements on a variety of downstream tasks, e.g. visual question answering, image captioning [70], and text-to-image generation [7]. Among them, the single-stream architecture [52, 32, 46, 5, 43, 24] uses a single transformer to jointly model a pair of text and image, while the two-stream architecture [35, 36, 53] applies two transformers to separately learn the representations of the text and the image, respectively. Our UFC-BERT is also a variant of the single-stream visual-language transformer, but focuses on flexible multi-modal image synthesis instead of multi-modal pretraining.
Non-Autoregressive Sequence Generation. Though it is natural to autoregressively predict tokens from left to right when generating a sequence, autoregressive decoding suffers from the slow speed and sequential error accumulation. Thus, the non-autoregressive generation (NAR) paradigm is proposed to avoid these drawbacks in neural machine translation [19, 20, 29, 16], image captioning [14, 22, 70], and speech synthesis [50, 49]. These approaches often employ the bidirectional Transformer (i.e. BERT) as it is not trained with a specific generation order. Our progressive NAR generation algorithm improves upon the Mask-Predict non-autoregressive algorithm [57, 16, 38, 33], by introducing the relevance estimator and the fidelity estimator to facilitate sample selection and dynamic termination.

heading: 3 UFC-BERT For Multi-Modal Image Synthesis

heading: 3.1 Background: Two-Stage Image Synthesis
text: In this section, we review the two-stage architecture [42, 48, 13, 47] for image synthesis.
At the first stage, a codebook Z = {zk}Kk=1 for vector quantization is learned, where zk ∈ Rnz is the k-th code-word in the codebook and K is the number of code-words. An image X ∈ RH×W×3 can be transformed into (or from) a collection of code-words Z ∈ Rh×w×nz . Concretely, a convolutional encoder E first encodes the original image X as Ẑ = E(X) ∈ Rh×w×nz . Then an element-wise quantization step q(·) is applied to each element Ẑij to obtain the element’s closest code-word zk, i.e., q(Ẑij) = argminzk∈Z‖Ẑij − zk‖. For reconstruction, a convolutional decoder D is also learned for recovering image X̂ ∈ RH×W×3 from Z such that X̂ is close to X. The first stage can be denoted by
Z = q(E(X)), X̂ = D(Z). (1)
Due to the convolutional layers, each of the h× w elements of Ẑ mainly correlates with an Hh × W w block of the image, though its receptive field may be larger if multiple convolutions are stacked.
At the second stage, image X’s quantized representation Z can be rewritten as a sequence of codes I ∈ {0, . . . , |Z|−1}NI , composed ofNI (= h×w) indices from the codebook Z . Thus, image synthesis can be formulated as autoregressive sequence generation, i.e. predicting the distribution Pr(Ii|I<i,C) of the next token Ii conditioned on the preceding tokens I<i and the control signals C. The distribution is typically modeled using a unidirectional Transformer. The likelihood is then Pr(I|C) =∏
i Pr(Ii|I<i,C). Parameters are learned by minimizing LAR = EI∼data [− log Pr(I|C)] . We focus on improving the second stage. Specifically, the autoregressive paradigm adopted by the existing two-stage works [42, 48, 13, 47] suffers from slow generation speed, fails to capture bidirectional contexts, and cannot fully support preservation control signals. We thus propose UFCBERT, a novel NAR approach for stage two, to unify any number of multi-modal controls and tackle the shortcomings of AR. As for stage one, we directly follow VQGAN’s design [47], which improves upon VQVAE [42] by incorporating a perceptual loss [27] and patch-based adversarial training [26].

heading: 3.2 Problem Formulation
text: Conditional image synthesis aims to generate an image that satisfies a set of control signals C. We consider three major modalities of control signals. A Textual Control (TC) consists of a sequence of words T ∈ {0, . . . , |W| − 1}NT , whereW is the vocabulary and NT is the number of words in the text. In the two-stage framework, an image can be converted into a sequence of code-words (i.e. tokens) based on stage one’s encoder E and codebook Z . Thus, a Visual Control (VC) is denoted by a sequence V ∈ {0, . . . , |Z| − 1}NV consisting of code-words from the codebook Z , where NV is the sequence length. Similarly, the target (i.e., the image to generate) is a sequence of code-words I ∈ {0, . . . , |Z| − 1}NI . We support zero, one, or multiple visual controls for flexibility. As for the Preservation Control (PC), it is a sequence of binary masks P ∈ {0, 1}NI with the same length as I, where 1 means that the token is known (i.e., Ii is ground-truth if Pi = 1) while 0 means the token needs to be predicted. We aim to design a model at the second stage to synthesize the target image’s sequence I conditioned on C, i.e., a combination of any number of control signals from {T,V,P}.

heading: 3.3 Model Inputs
text: As shown in Figure 2, our UFC-BERT modifies the original BERT model [9] to accommodate any number of multi-modal controls. Similar to BERT, the backbone is a multi-layer bidirectional Transformer encoder, enabling the dependency modeling between all input elements. The input sequence of UFC-BERT always starts with two special tokens [REL] and [FDL] for relevance estimation and fidelity estimation, then goes on with the word sequence T of textual controls, code sequence V of visual controls, and ends with the code sequence I of the target image to generate. Two special separation tokens [EOT] and [EOV] are appended to the end of the textual and the visual control sequences, respectively. If there are multiple visual controls, another special token [SEP] is inserted to separate them. The sequence I of the target image to generate may be partially or fully masked by a special token [MASK]. When the preservation control P is present and Pi = 1, token Ii in I is always set to the code-word corresponding to the given image block to be preserved.
Each input token’s representation is the sum of the position and token embeddings:
Position Embedding. Our UFC-BERT learns independent sets of position embeddings for the different kinds of the inputs to achieve better distinguishment between the various modalities. The position embeddings for the word sequence are the same as BERT, i.e., we use sequential position embeddings. For a visual control or the target image, the position embedding of each token is decided according to where this token lies on the h× w grid, i.e., we use spatial position embeddings. Token Embedding. For textual controls, we use Byte-Pair Encoding [51] to segment each word into sub-words and then learn sub-word embeddings. Each special token, e.g., [REL] or [MASK], is assigned a dedicated embedding. For visual controls and the target image, we learn an embedding for each code-word. We do not directly use the embeddings from stage one’s codebook due to the decoupling of the two stages.

heading: 3.4 Training: Masked Sequence Modeling with Relevance and Fidelity Estimation
text: As shown in Figure 2, we train UFC-BERT via masked sequence modeling, i.e., predicting the masked tokens in the target image conditioned on the controls. A relevance estimator and a fidelity estimator are also trained in the process, and will be key to our progressive NAR generation algorithm.
Task 1: Masked Sequence Modeling. This task is similar to Masked Language Modeling (MLM) in BERT, but incorporates multi-modal control signals when predicting the masked tokens. To construct training samples, we mask parts of the target image I to predict using four strategies: (1) randomly decide the number of tokens to mask, and then randomly mask the desired number of tokens; (2) mask all tokens; (3) mask the tokens within some boxed areas of the image, where the number of boxes and the box sizes are randomly decided; (4) mask the tokens outside some random boxed areas of the image. We use the four strategies with probability 0.70, 0.10, 0.10, and 0.10, respectively. To construct multi-modal control signal C for each training sample, there are four different combinations: <TC, VC>, <TC>, <VC>, <empty>, where <TC, VC> means the textual and visual controls are simultaneously employed, <TC> or <VC> means only a textual or visual signal is used, and <empty> means no textual or visual control is present. Note that the preservation control is already included in the masked sequence modeling task. Since our dataset does not contain ground-truth pairs of visual controls and target images, we crop one or multiple regions of a target image to construct VC for the target image. Because image synthesis from solely textual controls is more challenging than from other signals, we use the four combinations with probability 0.20, 0.55, 0.20, 0.05, respectively, where textual controls get more attention. We feed UFC-BERT’s outputs at each position of I into a softmax classifier over the codebook Z , which produces a probability score Yi = Pr(Ii|IU ,C) for each position i ∈ M , where M is the set of masked positions and U is the unmasked set. Finally, the masked sequence modeling task minimizes the softmax cross-entropy loss LMSM = EIM ,IU [− log Pr(IM |IU ,C)], where Pr(IM |IU ,C) = ∏ i∈M Yi.
Task 2: Relevance Estimation. This task is to learn a binary classifier that judges whether the generated image is relevant or irrelevant to the given multi-modal control C. Briefly, we add a linear layer on the output corresponding to the special token [REL]. The linear layer outputs a scalar representing the logit, and a binary cross-entropy loss LREL is added. During training, the training samples from Task 1 serve as the positive instances (i.e. relevant pairs). We construct negative instances (i.e. irrelevant pairs) by swapping the control signals of two training samples.
Task 3: Fidelity Estimation. This task aims to distinguish whether the generated image is realistic from the view of human visual cognition. Similar to relevance estimation, we feed the output corresponding to [FDL] into a linear layer for binary classification and add another binary crossentropy loss LFDL. Since the low-fidelity images (i.e. negative instances) do not exist in the dataset, we run UFC-BERT from previous epochs to synthesize images based solely on textual control signals, and use the synthesized images as negative instances.
We combine the three tasks’ losses to train UFC-BERT, i.e.,
LUFC-BERT = λ1LMSM + λ2LREL + λ3LFDL, (2) where λ1, λ2 and λ3 are set to 1.0, 0.5, and 0.5 to balance the three losses. The masked sequence modeling task ignores the negative instances from the other two tasks, i.e., irrelevant pairs or unrealistic instances. And the fidelity estimation task is added only after a certain number of epochs.

heading: 3.5 Inference: Progressive Non-Autoregressive Generation
text: We design a Progressive Non-Autoregressive Generation (PNAG) algorithm for conditional image synthesis after training, which improves upon Mask-Predict [16, 21, 7]. Mask-Predict predicts all target tokens when given a fully-masked sequence at the first iteration, and then iteratively re-mask and re-predict a subset of tokens with low probability scores for a constant number of iterations. However, Mask-Predict cannot ensure the efficacy of multi-modal controls and the fidelity of the synthesized images, and requires determining the number of iterations. Our PNAG tackles its drawbacks via sample selection and dynamic termination, based on the relevance and fidelity estimators.
Each iteration of our PNAG algorithm consists of a Mask step and then a Predict step. Let I(t,in) = (I
(t,in) 1 , . . . , I (t,in) NI ) and I(t,out) = (I(t,out)1 , . . . , I (t,out) NI
) be the state of the target image’s sequence before and after the t-th iteration, respectively. The tokens in I(0,out) for t = 0 is all set to [MASK] except for the positions that are controlled by the preservation signals, i.e., except for I(0,out)i that has Pi = 1. If a preservation control is present, i.e. Pi = 1, we always set I (t,in) i and I (t,out) i for all t to be the code-word that corresponds to the provided image block to be preserved.
Mask Step. At the beginning of iteration t (t ≥ 1), we construct the input sequence I(t,in) by (re-)masking a subset of tokens in the generated sequence I(t−1,out) from the last iteration. Similar to beam search, we construct B parallel input sequences {I(t,in)1 , . . . , I (t,in) B } at each iteration. Specifically, we re-mask n tokens of I(t−1,out) to produce each I(t,in)b . We first sampleNI−n tokens from a multinomial distribution Pr(t,in) proportional to the probability scores Y(t−1) = {Y (t−1)i } NI i=1 (see Equation 3), computed by Pr(t,in) = Softmax(Y(t−1)). And other tokens are re-masked and re-predicted at the next Predict Step. Here n = NI · (β + T−tT−1 · (α− β)), where α is the initial mask ratio, β is the minimum mask ratio, and T is the maximum possible number of iterations, such that the number of tokens to re-mask gradually decreases after every iteration.
Predict Step. Given the control C and an input sequence I(t,in)b , UFC-BERT estimates a distribution Pr(Îi|I(t,in)b ,C) for each masked position i. UFC-BERT also estimates the relevance score SRb and fidelity score SFb regarding the image that it is about to synthesize, and summarizes the scores into a comprehensive score S(t)b = σS R b +(1− σ)SFb , where σ is a coefficient for adjusting the importance of the two. We perform sample selection based on S(t)b , i.e., we select the b-th sequence I (t,in) b with the highest S(t)b , discard the others, and then generate I (t,out) based on the selected I(t,in)b as follows:
I (t,out) i ∼ Pr(Îi|I (t,in) b ,C), Y (t) i ← Pr(Îi = I (t,out) i |I (t,in) b ,C), (3)
where each token I(t,out)i is sampled from the multinomial distribution Pr(Îi|I (t,in) b ,C) and the corresponding probability is assigned to Y (t)i . Note that we predict tokens for all masked positions regardless of the predictions’ confidence. We also implement dynamic termination based on S(t)b . Specifically, if the current iteration’s score S(t)b is higher than Smax (initialized as zero), we set Smax to S(t)b and record the current iteration as tmax. If Smax does not increase after three consecutive iterations, we select I(tmax,out) as the final result and terminate our generation algorithm.

heading: 4 Experiments

heading: 4.1 Datasets and Hyperparameters
text: In experiments, we focus on two practical fields of image synthesis: fashionable clothing and human faces. We collect a very large-scale clothing dataset M2C-Fashion with Chinese text descriptions, which contains tens of millions of image-text pairs, much larger than the commonly used text-to-image datasets COCO [34] and CUB [56]. Details of the dataset are provided in the supplementary material. We additionally use another high-resolution facial dataset Multi-Modal CelebA-HQ [28, 61].
Following the model setting of VQGAN [13], we use the 256× 256 image size on the two datasets and transform each image to a discrete sequence of 16 × 16 codes, where the codebook size |Z| is set to 1024. For the BERT model, we set the number of layers, hidden size, and the number of attention heads to 24, 1024, and 16, respectively. Our UFC-BERT has 307M parameters, same as the Transformer used by VQGAN. As for hyper-parameters of PNAG, we set the parallel decoding number B to 5 and the balance coefficient σ to 0.5. We set the initial mask ratio α, the minimum mask ratio β, and the maximum iteration number T to 0.8, 0.2, and 10, respectively.

heading: 4.2 Flexibility of Multi-Modal Controls for Conditional Image Synthesis
text: In this section, we qualitatively verify the synthesis ability of UFC-BERT with three modalities of control signals, i.e., textual, visual, and preservation controls. The textual controls are the texts paired with the images, which are already provided by the two datasets, while the visual controls are code sequences of cropped regions, e.g. regions that represent logos or texture of clothes.
In Figure 3, we synthesize images conditioned on combinations of the three types of control signals. The results demonstrate UFC-BERT can unify any number of multi-modal controls to synthesize high-quality images. Further, UFC-BERT supports one or multiple visual controls for more flexible synthesis, as shown in Figure 4 where we generate images given 2∼3 visual controls. We observe that UFC-BERT can reasonably fuse multiple visual elements and produce a harmonious image.

heading: 4.3 Quantitative Comparison to Existing Methods for Text-to-Image Synthesis
text: In this section, we investigate how our UFC-BERT quantitatively compares to existing models. Considering most existing methods only utilize one control signal, we select the most common and challenging task text-to-image synthesis to compare the synthesis ability.
First, we compare our UFC-BERT with GAN-based text-to-image models AttnGAN [62], ControlGAN [30], DF-GAN [54], DM-GAN [71] and TediGAN [61] on the Multi-Modal CelebA-HQ dataset. For evalution, we adopt two automatic metrics FID [23] and LPIPS [66]. We report the results on Table 1 and our UFC-BERT achieves the best performance on the two metrics, even outperforming the TediGAN that uses slow and complex instance-level optimization. This demonstrates the two-stage architecture and non-autoregressive generation of UFC-BERT are suitable for text-to-image synthesis.
Besides, we compare our UFC-BERT with the autoregressive two-stage method VQGAN from three aspects: (i) the automatic metrics FID for image quality, as well as LPIPS, PSNR [59] and SSIM [59] for the similarity between the generated image and the ground truth; (ii) the Relevance and Fidelity metrics are evaluated through a user study, where the users are asked to judge which model’s output is more relevant to the textual descriptions, and more photorealistic; (iii) the synthesis speed of the two approaches. Note that the autoregressive inference implementation of VQGAN has been optimized by caching the preceding computation as in Transformer-XL [8], and UFCBERT and VQGAN have the same parameter number (307M) for fair com-
parison. For the user study, the two models receive the same textual signals, and each generates 50 images. We collect the pairwise comparison results from five volunteers.
As shown in Table 2, our UFC-BERT achieves better performance for almost all criteria with about 11× speedup. This suggests our non-autoregressive UFC-BERT with progressive NAR generation algorithm can synthesize high-fidelity images relevant to textual descriptions. As for the FID metric, UFC-BERT outperforms VQGAN on M2C-Fashion, but has worse performance on Multi-Modal CelebA-HQ, it may be due to the fact that the autoregressive VQGAN can more easily memorize the pattern of a small dataset (only 30,000 facial images). In Figure 5, we further show typical generated examples to intuitively display the difference between the two approaches, including a case of counterfactual generation. We find that UFC-BERT can synthesize high-quality images, even for the counterfactual case.

heading: 4.4 The Effectiveness of Our Progressive NAR Generation Algorithm
text: In this section, we first visualize in Figure 6 the iterative process of our PNAG inference method based on the relevance and fidelity estimators. The images with red bounding boxes are the final outputs that match the textual control signals. We can find that the fidelity and relevance of the images increase after a few iterations, verifying our PNAG algorithm can guide the inference process towards a better direction and synthesize more realistic images that match the control signals.
We then conduct ablation studies of PNAG. As shown in Table 3, we develop three ablated inference methods PNAG(w/o. REF), PNAG(w/o. FDL) and MNAG, where PNAG(w/o. REF) and PNAG(w/o. FDL) discard the relevance estimator and the fidelity estimator, respectively, and MNAG is the original Mask-Predict method [16] without any estimator. The results demonstrate that the two estimators effectively utilize the discriminative capability of UFC-BERT and do help improve the synthesis quality. Additionally, we vary the crucial hyper-parameter of PNAG B (i.e. the parallel decoding number during inference) from 1 to 10, and the results in Table 3 show that a larger B is beneficial to the synthesis quality.

heading: 5 Conclusions
text: We proposed UFC-BERT to unify any number of multi-modal controls in a universal form for conditional image synthesis. We utilized non-autoregressive generation to improve inference speed, enhance holistic consistency, and support preservation controls. Further, we designed a progressive generation algorithm based on relevance and fidelity estimators to ensure relevance and fidelity.

Review summaries:
### Summary The paper works on conditional image synthesis using a transformer-based network. The proposed method uses one network to combine multi-modal conditional signals, such as textual information and visual information, for image generation. The proposed non-autoregressive training strategy is validated on two datasets, the collected clothing dataset and Multi-Modal CelebA-HQ, and shows better results than GAN-based methods and a transformer-based method.  ### Strengths - The proposed method can utilize multi-modal controls for image synthesis and show convincing synthesized images containing input condition information. The image generation results conditioned on multi-modal inputs are interesting and inspiring. - While recent autoregressive-based methods demonstrate good text-to-image generation results using transformers, the paper shows that non-autoregressive transformers (e.g., BERT used in the paper) can also be trained for generating a sequence of image tokens. - The careful design of the relevance and fidelity estimation tasks is novel and helps the training of the network. 

This paper proposes a two-stage non-autoregressive (NAR) transformer-based approach for multi-modal conditional image generation in the discrete latent space of a pretrained VQGAN, called UFC-BERT. More specifically, the proposed model draws inspiration from BERT [1] and Mask-Predict [2] and uses masked language (token) modeling to (i) incorporate bidirectional context and (ii) significantly speed up generation over autoregressive baselines. Sampling from the masked model is performed via an extension of the progressive mask-predict algorithm, where two newly introduced discriminative heads of the transformer model guide the sampling process to generate samples that are consistent with the given conditioning and have high fidelity.