Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes CogView - a cross-modal Transformer with VQ-VAE as the tokenizer. The model is primarily designed for text-to-image generation but also shows promising results on various downstream tasks. 
The paper also provides many insights on the effective training tricks (including PB-Relax and Sandwich-LN) to stabilize the training of text-to-image transformers. The experimental results show that CogView outperforms the recent state-of-the-art approach DALL-E in the zero-shot settings. 
main_review: Strengths: 
1) The paper is generally well-written and easy to follow.
2) The paper introduces several effective methods to stabilize the training for text-to-image generation.
3) The proposed CogView can be further extended to more tasks via fine-tuning: style learning, super-resolution, self-reranking.
4) The code and pre-trained model is open-source, which is the first for large text-to-image transformer, many applications will benefit from it.

Weakness: 
1) The technical novelty of the paper is limited, its model is almost the same as DALL-E. The main contribution lies in the training strategy rather than in the techniques.
2) Training data is one possible reason why CogView beats DALL-E. Although Appendix A provides some descriptions of data collection. However, there are still not enough details to replicate the high-quality text-image pairs from the current descriptions. 
3) It is hard to tell whether the improvement comes from the high-quality training data or the training tricks.
4) Self-reranking after fine-tuning is just another CLIP, it is still post-selection and does not get rid of CLIP.

Questions to the authors:
1) CogView used different data to train the VQ-VAE and text-to-image Transformer. DALL-E also released their own trained VQ-VAE, how will the results be like if using the same VQ-VAE and compare the generation performance between CogView and VQ-VAE?
2) How do you evaluate the ratio of training data and training tricks in the success of CogView?
3) In Table 1, how did you replicate the performance of DALL-E, why CapS didn’t apply to DALL-E? Or do you just copy the number of DALL-E on MSCOCO?

limitations_and_societal_impact: The proposed CogView approach may be used for malicious purposes to generate fake images. That's a potential negative societal impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a new text-image pretrained model based on Transformer with VQVAE tokeniser, which can be implemented on different downstream tasks to achieve an appropriate performance. Also, two techniques called PB-relaxation and Sandwich-LN are proposed to reduce overflow in forwarding and to stabilise the training.
main_review: Originality: The proposed method is built on GPT with VQVAE tokeniser.

Quality: The submission is technically sound, claims are well supported, but authors do not discussed strengths and weaknesses of the proposed method.

Clarity: The paper is well organised and easily followed.

Significance: The results look good, which achieves good performance on different downstream tasks.
limitations_and_societal_impact: Limitation:
1. Are authors going to make the collected dataset public? To some extent, the good performance might be benefited from the large-size training dataset. 

2. The proposed method is similar as Dall-E as both rely on the GPT and VQVAE. It might be better if authors can show more discussion and comparison between proposed method and the Dall-E.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a generative, two-stage text-image model that combines a VQVAE (or VQGAN) and a text-conditional autoregressive transformer model for text-to-image synthesis. Here, the VQ-model is trained on image data only (first stage), while in the second stage the transformer is trained to model the distribution of learned discrete image representations (="image tokens") of the (now fixed) first-stage model, given a corresponding textual description. The generated image tokens can then be decoded into the image space via the pre-trained decoder. Additionally, the fine-tuning of the model for specific tasks such as style learning, super-resolution, and image-to-text is explored. The work also provides insights into stabilizing the training of large transformers for text-image synthesis. 
main_review: __Strengths:__ 
The paper provides (very) good qualitative results. The results depicted in Fig. 1 and Fig. 15 are impressive.
Further, interesting insights on training of the VQVAE are presented. For example, apparently a fixed codebook is not much different from a learned one and codebook collapse does not seem to be an issue. (l.119 and l.128-131).

Moreover, the paper gives useful insights on how to stabilize the transformer training (OBS and Sandwich Layer Norm). Additionally, it is interesting that weighing the text loss and the image-token loss equally seems to perform best.
Excluding the dataset-collection, the proposed approach (or rather the model's architecture)  is reproducible from the description provided in the paper.

__Weaknesses:__

- Empirical justifications of the above statements about VQVAE and Transformer trainining is missing. The proposed methods are simply described as "working well". The same holds for the statements about the weighing of the text loss and the "center-continuous sliding window". The latter needs to be compared to ordering that do not allow for inconsistencies between tokens, i.e token 4 & 7 and 5 & 6 in Fig. 5a (counting from 1 to 16, raster-scan order).
- Evaluation is only done for a fixed re-ranking factor of 60. How does the model perform when re-ranking with less images (similar to Fig. 9 in [1])? How does the model perform when re-ranking is performed with CLIP?
- Very few qualitative examples given for the finetuning, especially "Style Learning".
- The writing should be improved. I would recommend the authors tone down some of their claims (e.g. l. 50: "Our approach is neat and natural", or in the title "mastering") and additionally discuss limitations (e.g. blurriness, sampling speed) of their approach (more) detailed.

__Mixed Comments & Questions:__
- What is the inference time (with and without the superres. model)? How does this compare to other approaches?
- l. 287-288: What does "Samples from CogView itself might be overscored, but it is all right to evaluate other models in the future" mean?
- l. 277-l.281 speculate about the reasons why CogView performs better than DALL-E on blurred MS-COCO. Comparing the first stage models (i.e. the discrete autoencoders) via reconstruction metrics could serve as an objective empirical justification for these claims.
- Section 3.1: Only a single example for style learning is given. Furthermore, only a single realization for each text prompt is depicted. How large is the variance given a single prompt? These question could be easily answered by plotting multiple realizations for each prompt.
- Section 3.3: Could the model be finetuned by reversing the order (i.e. predict text tokens from image tokens from right to left) in the autoregressive NLL training? This would correspond to the proposed "swapping" of text and image tokens, with additional swapping of the (pretrained) positional embeddings.
- Section 3.4: The structure of the paper suggests that this section still belongs to the "Finetuning" chapter, but, at least in my understanding, introduces (i) a new dataset and (ii) another first-stage model (VQGAN), and thus trains a new model from scratch. This should be clarified. To stay with this theme: Why is the VQGAN not used for the "big" data set? Since the resolution is very high with 50x50 tokens, hardly any compression artifacts should be visible according to [2]. Are there other ways to fix the bluriness of the generated samples? How are the massively increased compute requirements (i.e. caused by switching from 32x32 tokens to 50x50 tokens) handled? 
- Why the name "CogView"? I could not find an explanation in the paper.
- How does the model perform in a "zero-shot" setting on other classic text-to-image datasets such as CUB [3]? Can the model be fine-tuned on these datasets?
- What is the stopping criterion when training the transformer? Minimal NLL on test data? Something else?
- l.106: which "chinese corpus"? 
- Which model is used in Fig. 11?

_Minor_:

- l. 54: missing word, maybe "our" or "the"?
- l. 94: $z_i$s -> $z_i$
- l. 149: consine -> cosine
- l. 210: is -> are
- l. 523: alleviates -> alleviate

__Summary/Justification of Rating:__

In summary, I have somewhat mixed feelings about the paper. On the one hand, it (i) shows that two-stage training with VQVAE/VQGAN and Transformers can produce highly complex scenes from text conditioning, and provides convincing results (also demonstrated through a user study), and (ii) provides interesting insights into training the VQVAE and Transformer models (see above). On the other hand, many of the proposed modeling decisions feel somewhat "ad-hoc" without being justified by ablation experiments (see _weaknesses_ and _comments_ above). There are also no results supporting the claims about non-present codebook collapse (l.119) or "fixed/learnable embeddings" (l.128-131), which I think are of great interest to the community. In addition, the writing of the paper needs improvement and the limitations of the work need to be discussed. I am currently rating this paper as "just above threshold" but am willing to increase my score if the authors provide satisfactory answers.

__References:__
 - [1]: Ramesh, Aditya, et al. "Zero-shot text-to-image generation." arXiv preprint arXiv:2102.12092 (2021).
 - [2]: Esser, Patrick, Robin Rombach, and Bjorn Ommer. "Taming transformers for high-resolution image synthesis." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
 - [3]: Welinder, Peter, et al. "Caltech-UCSD birds 200." (2010).
limitations_and_societal_impact: The conclusion mentions that the proposed method could potentially be used to spread misinformation, which is a valid concern. Possible biases of the model or the collected dataset are not discussed. Limitations are rarely discussed, see also my above concerns about the writing.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: - CogView is an text-to-image model similar to DALL-E, except it uses a traditional VQ-VAE instead of dVAE. 
- A Chinese (image, text) dataset is collected for supervised pre-training. (30M pairs)
- A couple of techniques are described to stabilize FP16 training: PB-Relax (identity that is more stable for computing attention), Sandwich-LN (pre + post LayerNorm) .
- They fine-tune the model for style-learning, super-resolution, and image captioning, and fashion design, and show samples but do not provide quantitative results.
- CogView is compared (via FID/IS) to DALL-E on MS-COCO by translating captions to Chinese first.
- Human evaluation is used to compare DM-GAN, DF-GAN, AttnGan, and Ground-truth. 37% of the time, CogView is listed as the best.
- CaptionScore is proposed as an image-text metric, derived from the likelihood of text tokens assigned from CogView. This is also used to select the best txt2img candidate amon 60, unlike DALL-E which relies on a separate CLIP model.

main_review: (Please read summary) While there are some interesting empirical findings, the contribution over DALL-E is limited. The primary difference is dataset collection, which focused primarily on photographs. There is no reason provided for collecting the dataset in Chinese, while much of the dataset is originally in English, but translated. The details of the collection are insufficient for reproducibility and it does not seem like this is a dataset paper either. The main reason given for better FID/IS scores on MS-COCO is that pre-training was conducted mainly on photos, which is more similar to MS-COCO, so the result is not surprising.

One missing baseline from the human evaluation is the retrieval one, e.g. 'closest image in training set' from a text2image retrieval system. It is unclear how much of the generated images is memorized/regurgitated.

I do like the idea of CaptionScore to rank generated images instead of a relying on another model. 

Update: Increased score in light of discussion of potentially useful tricks to stabilize training contributed by this work, which may be useful to the community. The appendix has an expansive analysis on attention in the model that motivates PB-relax. Not having to rely on another model like CLIP is also nice.
limitations_and_societal_impact: No, there is no discussion.
review_title: null

Meta-review: This work presents a text-image generation model using VQ-VAE to discretize images, which is then followed by a LM on text + image tokens. There was a debate whether this work is concurrent to or a follow-up of DALL-E (blog post on Jan 5, 2021 and arXiv on Feb 24, 2021, more than two months before NeuRIPS deadline) given similarities among the two approaches. When viewed as a follow-up work, I found that there is a reasonable amount of contributions (including technical ones) in this paper, e.g., the finetuning ideas for super-resolution (which greatly enhances the quality and is actually a neat idea) and self-ranking (without having to train a full CLIP model). The detailed findings of using nearest neighbors in VQ-VAE, setting equal weight to the LM loss for texts, etc., together with tricks for stabilizing training seem informative and valuable to the community. In addition, the authors promised to open-source this work. Because of these reasons, I recommend Accept.
Decision: Accept (Poster)