Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, instead of using auto-regressive transformer to generate the discrete tokens like "taming transformer", it cleverly employs the well-known BERT model of NLP area, which not only supports the bi-directional conditions for local image editing, meanwhile vastly speeding up the sampling efficiency. The experiments also demonstrate the effectiveness of the proposed method.
main_review: Pros:

* The idea of using extra global tokens to judge the current generation status is very interesting.
* The designed progressive non-autoregressive generation is promising.
* This paper is well-written and easy to follow.


Cons:

* What are the details of sampling the re-mask tokens? I am not very clear about the descriptions in Line232-Line233.
* I think the authors ignore one very important problem of the whole framework. The original pixel contents may not be well preserved while giving **the masks with arbitrary shapes**, especially for the boundary regions, which is caused by the mis-alignment of downsampled masks and discrete representation.  This issue should be discussed in the submission.
* The papers which also use bi-directional transformer for image generation like [1,2] should be discussed in the related works.

1. High-Fidelity Pluralistic Image Completion with Transformers, arxiv 2021 
2. Diverse Image Inpainting with Bidirectional and Autoregressive Transformers, arxiv 2021

I will consider further raising the scores if the concerns could be well solved.

limitations_and_societal_impact: Please see the above mentioned comments.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a two-stage non-autoregressive (NAR) transformer-based approach for multi-modal conditional image generation in the discrete latent space of a pretrained VQGAN, called UFC-BERT. More specifically, the proposed model draws inspiration from BERT [1] and Mask-Predict [2] and uses masked language (token) modeling to (i) incorporate bidirectional context and (ii) significantly speed up generation over autoregressive baselines. Sampling from the masked model is performed via an extension of the progressive mask-predict algorithm, where two newly introduced discriminative heads of the transformer model guide the sampling process to generate samples that are consistent with the given conditioning and have high fidelity.
main_review: __Strengths:__

The paper presents high-quality image synthesis results with a BERT-style model, previously unseen with these type of models.
Furthermore, the paper does a great job at motivating the proposed solution (especially in line 43-56).
The proposed architecture and sampling process is simple but effective. Additionally, the two discriminative tasks (relevance and fidelity estimation) can be effectively trained via swapping of the conditioning information (relevance) and replay of samples from previous model checkpoints (fidelity), which is a nice feature. Using the BERT-formulation allows to condition on arbitrary patches of images (in contrast to the "completion" property of autoregressive models), where especially Fig. 4 provides impressive results on mixing different patches into a coherent image.

The model is described in detail in Sec. 3 and it should be easy to re-implement, given the provided information.

__Weaknesses:__

- What's the purpose of Figure 1? Neither sketch-to-image nor artistic style transfer are demonstrated with UFC-BERT. This should be clarified in the corresponding caption.
- It remains somewhat unclear how the cropping for VC and PC works. Is this performed in the latent space of the VQGAN, and correspondences are assigned via the mentioned $\frac{H}{h} \times \frac{W}{w}$ token-pixel correlations? If so, what is the effect of the non-local block (=attention layer) present in the VQGAN? Furthermore, crops can then only be made with a stride of 16 pixels.
- The paper repeatedly states that the relevance and fidelity estimators are "key" to the generation algorithm, but does not provide experiments to justify this claim.
- No metric that explicitly measures correspondence between generated images and corresponding textual descriptions is provided, the paper rather takes a detour by measuring similarity of ground-truth images and samples. However, since there is not a one-to-one correspondence between textual description and image, these metrics (LPIPS, SSIM, PSNR) might not be a good fit. On the contrary, LPIPS has been used to measure _diversity_ of generated samples, where higher scores are better [3]. Using R-precision as in [4] or CLIP-scores [5] could be a simple fix.

__Originality:__
The proposed method of sampling latent tokens learned via vector-quantized representation learning (VQVAE/VQGAN) with a BERT-style model is new and allows for applications such as image completion or harmonization of different patches. Furthermore, multi-modal control has been underexplored in the context of VQVAE/VQGAN models.

__Clarity/Quality:__
The presentation is clear, and, as described above (strengths), the model should be easy to re-implement.

__Significance:__
Given the promising results on face and fashion datasets, the approach could have potential to serve as a generic neural image manipulation/creation mechanism when trained in a large-scale setting.

__Mixed Comments/Questions/Suggestions:__
- How does the diversity of samples generated with IFC-BERT compare to an explicit AR-likelihood model?
- I would rather call the "counterfactual case" (l. 312 and Fig. 5) a "conflicting prompt".
- Will the dataset (M2C) be released? A large-scale dataset of (text, image) pairs would be of great interest to the community.
- The results depicted in Figure 4 are nice, but more should be added (maybe in the appendix) to demonstrate that the method works robustly.
- After how many epochs exactly is the $\mathcal{L}_{FDL}$-loss applied?
- How robust is the model to different choices of $\lambda_1$, $\lambda_2$ and $\lambda_3$ in Eq. (2)?
- What is the effect of codebook size $|\mathcal{Z}|$ on the model?
- Fig. 3 should include a visualization when gradually adding more and more information of the _same_ ground truth prompt, i.e. TC, TC+VC and finally TC+VC+PC.
- How big (if at all) is the problem of overfitting?

__References:__
 - [1]: Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec-
tional transformers for language understanding. In Proceedings of the Conference on The North American
Chapter of the Association for Computational Linguistics, 2019.
 - [2]: Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding
of conditional masked language models. arXiv preprint arXiv:1904.09324, 2019.
 - [3]: Mahajan, Shweta, Iryna Gurevych, and Stefan Roth. "Latent normalizing flows for many-to-many cross-domain mappings." arXiv preprint arXiv:2002.06661 (2020).
 - [4]: Xu, Tao, et al. "Attngan: Fine-grained text to image generation with attentional generative adversarial networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
 - [5]: Radford, Alec, et al. "Learning transferable visual models from natural language supervision." arXiv preprint arXiv:2103.00020 (2021).

limitations_and_societal_impact: Societal implications are sufficiently discussed. However, I find it interesting that regarding the facial manipulation possibilities of the proposed approach, the only "negative" point raised is the possible deception of facial recognition systems. Depending on one's perspective and culture, one could probably assume the complete opposite here.

There is no section specifically addressing the limitations of the proposed approach.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: ### Summary
The paper works on conditional image synthesis using a transformer-based network. The proposed method uses one network to combine multi-modal conditional signals, such as textual information and visual information, for image generation. The proposed non-autoregressive training strategy is validated on two datasets, the collected clothing dataset and Multi-Modal CelebA-HQ, and shows better results than GAN-based methods and a transformer-based method.

### Strengths
- The proposed method can utilize multi-modal controls for image synthesis and show convincing synthesized images containing input condition information. The image generation results conditioned on multi-modal inputs are interesting and inspiring.
- While recent autoregressive-based methods demonstrate good text-to-image generation results using transformers, the paper shows that non-autoregressive transformers (e.g., BERT used in the paper) can also be trained for generating a sequence of image tokens.
- The careful design of the relevance and fidelity estimation tasks is novel and helps the training of the network.

main_review: I have some questions regarding the methods and experiment details.

### Methods
- On Ln. 189, the paper mentions that there could no textual or visual control. In this case, how could the network be trained if all the image tokens are masked (case (2) on Ln. 183)?
- For constructing negative pairs in Task 2 (Relevance Estimation), do authors use images without masked tokens, or could the tokens still be masked? If the tokens still could be masked, do authors perform Task 1 for the generated missing tokens?
- I'm a little confused about Ln. 232-233. The method seems to mask tokens with high probability scores by sampling from a multinomial distribution. But my understanding is that the method aims to mask tokens with low probability scores. Is the description on Ln. 232-233 accurate?

### Experiments
- For experiments on M2C-Fashion and Multi-Modal CelebA-HD, do authors train the network from scratch or from pre-trained models? If from pre-trained models, how much performance improvement compared with training from scratch?
- I wonder whether authors have tried any experiments on the CUB200 bird dataset.
- For training the VQGAN transformer, could the authors provide more details on how to process the textual information? Is it the same for training UFC-BERT?
- On Ln.309-310, the authors mention autoregressive VQGAN can more easily remember the small dataset. I wonder would authors explain why autoregressive training can more easily remember the dataset?

### Minor
Will the code and dataset be released?

===============

Thanks authors for submitting the rebuttal. The rebuttal addresses most of my concerns and answers the questions from other reviewers. Although I am concerned about the performance of the proposed framework on different datasets (the paper only shows two datasets), the paper introduces a new way for generating images using multi-modalities, and it is more flexible than auto-regressive transformers. I believe the method can contribute to more potential applications.
limitations_and_societal_impact: It would be better if authors could discuss more about the limitation of the methods.
review_title: null

Meta-review: The paper proposed a method to unify any number of multi-modal control signal inputs for conditional image synthesis. The key thing was a transformer model that could convert a variable number of multi-modal control inputs to the discrete latent space of a VQGAN. All the reviewers rated the paper above the bar, with one reviewer upgraded the score from 6 to 7 after the rebuttal. The meta-reviewer agreed with the assessment and concluded the paper is above the bar. Please include the reviewer feedback in the updated manuscript in the final version.
Decision: Accept (Poster)