Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper presents CogView2, which aims for faster and better text2image generation based on its previous work CogView. The key idea is to adopt the spirit of coarse to fine and generate low-resolution tokens first(20x20) and then do a super-resolution to 60x60 tokens for high-resolution images.
strengths_and_weaknesses: Strengths:
1. This paper tries to solve two critical issues of the current auto-regressive-based text2image model: quality and efficiency. 
2. This paper provides many visualization consequences in the paper.


Weaknesses:
1. This paper claims too many things but does not verify them clearly in experiments. a) The training and inference efficiency of text2image generation are not evaluated or compared with previous methods with wall clock time or FLOPS. 2) As introduced in the introduction section the CogLM is general for many tasks like infilling tasks, and image captioning, but these capabilities are not presented in the paper.

2. The comparison with current works is not convincing. For example, CovViiew2 didn't show significant improvements over previous methods on FID-0. Previous SOTA methods like Make-A-Scene and DALL-E2 did not report the results of FID-1 to FID-8 for comparison. Besides, some latest works like latent space diffusion and VQ-Diffusion are missed in the table for comparison.

3. The quality and fidelity of these generated samples presented in the paper are not that impressive. First, the generated image is still blurry. Second, we can observe clear unreasonable structures for the human hands or faces. 

questions: Please refer to the weaknesses part.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Auto-regressive transformer-based text-to-image model can generate elegant pictures, but also face low-generation speed problems for high-resolution image generation. This is due to the extremely long sequence length and the auto-regressive decoding schema. In this work, the authors propose Cogview2, which is a hierarchical transformer and adopts local parallel auto-regressive generation instead of the global AR. The proposed Cogview2 achieves comparable or even better results than Cogview and speeds up the inference a lot. Overall, Cogview2 is an efficient version of Cogview but more fast benefiting from the local parallel decoding.
strengths_and_weaknesses: ## Strength
-   The proposed Cogview2 consists of several modules. First, they use the CogLM to generate a preliminary image with 20 $\times$ 20 tokens. The following super-resolution is based on this initial image, and I think a useful finding here is that we do not need to predict an image with a high resolution from the scratch, since a small 20 $\times$ 20 tokens may involve the information given the text. This may motivate the following researcher to design more efficient text-to-image models.
-   Local parallel autoregressive decoding is interesting. Traditional AR for text-to-image generation flatten the image into a 1-D sequence and generate the visual tokens from left to right. It is quite time-consuming and also ignores the spatial correlations.
-   Thanks to the flexibility of pretrained general language model, Cogview2 can perform image completion naturally 
-   Achieve comparable results with auto-regressive modeling while speeding up the inference quite a lot.

## Weakness
There is no major concerns for me. Maybe, I would like to see a totally non-autoregressive model which can deliver comparable results with AR ones.
questions: 
-   Have you tried other local parallel generation manners? e.g. within a local patch, left to right or top to bottom generation in a non-autoregressive fashion?
-   The attention mask shown in Figure2 is somehow difficult to understand. Do you mean the token in green box is masked and going to be generated? This slightly confuses me.
-   What about the speed of iterative super-resolution compared with stacked image super-resolution models in Imagen?
limitations: There is no limitation discussed in the current version. The authors have addressed the potential negative societal impact.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a pretraining method, a Cross-Modal General Language Model (CogLM), that masks both image and text tokens in input and learns to predict them in an autoregressive manner, while handling bidirectional context. By fine-tuning a pretrained transformer with this approach, the authors construct a hierarchical model, CogView2, which first maps a generated image into a larger image (direct super-resolution) and refines local patches (local parallel autoregressive), thus improving resolution as well as inference speed.

The experiments in the paper suggest that CogView2 performs comparable to other models despite its smaller model size and training data size. Meanwhile, the approach results in a considerable reduction in model run times (e.g. 10x faster than its predecessor, CogView).

Training and evaluation in this paper considers text in both English and Chinese.

strengths_and_weaknesses: Strengths
* The paper is clearly written and aided with great visualizations. Overall, the problems and proposed solutions are well-motivated and supported with appropriate evidence or justification (e.g. findings from existing literature, their own empirical observations, or limitations due to compute resources).
* The topic is timely; the effort for making the task of text-to-image generation faster and better is of great interest to the community.
* Their approach of generating a low-resolution image and refining it to be a high-resolution image is simple and straightforward. The use of various techniques is adequately justified (e.g. masking strategy and attention mask) and ablated (e.g. clustering sampling and attention upweighting).
* “Faster” text-to-image generation:  One of the main contributions of this paper is to make text-to-image generation faster. Although the experiment section doesn’t directly compare different models’ inference time in an end-to-end fashion, the last paragraph of the introduction section mentions that their model run time for local parallel autoregressive generation is 600x faster and overall 10x faster than their previous model.
* “Better” text-to-image generation: The experiments follow popular benchmarking practice and discuss the gap between automatic metrics and human evaluation. With automatic metrics, CogView2 performs comparable to other methods on MS-COCO based on Frechet Inception Distances. Based on human evaluation, CogView2 performs better on all metrics (image clarity, texture quality, and relevance to the caption) than CogView, Lafite, and DF-GAN.

Weaknesses
* “Better” text-to-image generation: As the authors acknowledged in the paper, the automatic metrics on MS-COCO may not be the best way to evaluate these models, hence the claim of CogView2 being “better” (in the title) or competitive (in the abstract) compared to other models such as DALLE2 can use some scoping/hedges. Outside of this paper, there have been some informal qualitative comparisons between several models, which seem to give the impression that CogView2 is not strictly better than other models. 
  * https://huggingface.co/spaces/THUDM/CogView2
  * https://twitter.com/bhagatsurya2/status/1542824988092530689
  * https://www.reddit.com/r/MachineLearning/comments/vkvq0j/r_cogview2_faster_and_better_texttoimage/
* Bilinguality: Since this paper considers text in both English and Chinese and notes that “Chinese input produces better results than English input” (https://huggingface.co/spaces/THUDM/CogView2), it would be interesting to see more in-depth analysis on this bilingual aspect of the model. The authors state that they used [BOE] to denote the beginning of English text and [BOC] for Chinese text, but do not justify the decision or discuss any findings based on two languages.
questions: Questions
* What were the main challenges/blockers for directly comparing different models’ inference time in an end-to-end fashion?

Suggestions
* Discussion on failure modes: Even from the cherry picked examples in Figure 1, there are multiple failure modes observed (e.g. different numbers or lengths of fingers). What are the main types of failure modes the authors or human annotators observed? Any difference when it’s in English vs. Chinese?

Nitpicking
* In Figure 2: “Supports tokenization of both Image Chinese and English” → “Supports tokenization of both images and texts in Chinese and English”
* In Section 2: “DF-GAN, et al.” → “and DF-GAN.”
* In Section 2 and throughout the paper: use either “VQ-VAE” or “VQVAE” to be consistent
* In Section 3.1: consider moving the paragraph “In NLP, the General Language Model [...]” to Related Work
* In Section 3.1: $l$ and $r$ are not defined
* In Section 3.1: “where [BOE], [BOC] are separators meaning beginning-of-English and beginning-of-Chinese” → “where [BOE] and [BOC] are separators to indicate the beginning of English text and that of Chinese text” 
* In Section 3.1: “Ideally, the two tasks should be separated” is not justified
* In Section 3.2: “Image, Chinese and English” doesn’t really type check; maybe it should be “Image and Text in Chinese and English”?
* In Section 5.2: “Frechet Inception Distances and Inception Scores” → “Frechet Inception Distances (FID) and Inception Scores (IS)”
* In Section 6: clarify “third-level super-resolution)
* In Section 7: “it is possible to train a classifier to distinguish the real and CogView2-generated images according to the texture features” is not supported with any evidence

limitations: I think some scoping about text is necessary. The paper generally assumes that input text can be any text in English and Chinese; their github repository explicitly says “any text” (https://github.com/THUDM/CogView2). For preciseness, it would be helpful to note any potential/practical limitations more clearly. For instance, CogLM can accept up to 111 text tokens (Section 3.2). And presumably, there aren’t that many short text inputs (e.g. one or two words) in the training data – then what would be a reasonable minimum length for the text input for the model to perform well? More generally, based on the types of images and texts in the training data, CogLM may perform better for certain kinds of images and texts. This insight can greatly help the future use of this pretrained model.

This applies to images as well. The fact that this paper only considers square images (based on N^2 notation) is not explicitly addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a faster and better text-to-image generation model called CogView2. Compared to the CogView baseline, main contributions are: 1) hierarchical generation that upsamples the original low-resolution image tokens and then refine them, 2) customized CUDA kernel that speeds up training, 3) a special attention masking strategy used in CogLM during training. Experiments clearly demonstrate that CogView2 generates better images than CogView both in terms of FID scores and also in terms of human evaluation. 

strengths_and_weaknesses: Strength:

The paper is mostly easy to follow. Authors’ proposed idea of using super-resolution and refinement modules to hierarchically generate higher resolution images is intuitive and works well in practice. Designing these two modules is clearly non-trivial work. Authors further demonstrate that clustering sampling is better than simple top-k. I also appreciate that they are willing to write a customized cuda kernel to speed up training. The speed-up seems to be significant in the autoregressive case.

Weakness:

Model improvement is limited compared to CogView. Both use Transformer to jointly learn the likelihood of text tokens and image tokens. The main difference is the new masking strategy in CogLM where tokens inside the mask region are trained to predict the next token based on past masked-tokens and non-masked context tokens. Authors stated that this approach unifies autoregressive generation and bidirectional prediction, but I find this design lacking justification. Experiment section also fails to provide any ablation study to support the choice of this particular masking strategy. 

Even with the new hierarchical design, CogView2 still generates image resolution lower than other works such as DALL-E-2 [1] and Imagen [2]. Stacking another direct/iterative super-resolution module is straightforward and should solve the issue. Resource limitation is indeed a problem but still I need to point out the lower resolution as part of the weakness. 

Only a few hand-picked visualizations are provided in the main paper and in the supplement. This makes it very hard to qualitatively judge the performance. 

[1] Ramesh, Aditya, et al. "Hierarchical text-conditional image generation with clip latents." arXiv preprint (2022).

[2] Saharia, Chitwan, et al. "Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding." arXiv preprint (2022).

questions: (1) It will be great if authors can provide more visualizations, especially those with higher resolution. I can not judge the quality of generated images from just a few hand-picked ones. 

(2) What is FID-k in Table 1? I assume it means radius of the Gaussian Filter but couldn’t find any explanation in the text. If my guess is correct then shouldn’t FID–0 be the most important number? Does this mean that CogView2 is worse than multiple baseline methods in terms of image quality?

(3) As stated before, I have concerns about the masking strategy. I do not see a motivation for joint learning of autoregressive generation and bidirectional mask prediction. Authors are encouraged to provide more ablation studies to support the design of CogLM and explain why it is better than the training in CogView. 

(4) For human quality evaluation, I wonder why authors do not include DALL-E as part of the test? That should be a key baseline.


limitations: Limitations and potential negative societal impact are adequately addressed by the authors.
review_title: null

Meta-review: metareview: This paper describes a less auto-regressive approach for text-to-image generation. Reviewers were somewhat split, with one reject and one strong accept. Overall, the results the authors get are perhaps less compelling given the progress the field has made over the past few months since submission time, but I think the focus on less autoregression for this problem is important and potentially useful for other approaches to this problem, and also the authors had far fewer computational resources and smaller datasets than some of the other recent work, which makes me feel there is a chance the less-auto-regressive generation they employ will be useful for other text-to-image generation projects. I'm slightly less certain about this one because the results don't seem as compelling given works published after the submission deadline, and also the lack of experiments to investigate that this approach can be used with other text-2-image generation works. Still, it seems like an important direction that could use more papers, so I think this should be accepted. The bilingual generation is a bonus.
confidence: Less certain
Decision: Accept