CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers
Authors: Ming Ding
Wendi Zheng
Wenyi Hong
Jie Tang
Venue: NeurIPS 2022 Accept (2022)
Keywords: text-to-image generation
pretraining
transformer

Abstract: Development of transformer-based text-to-image models is impeded by its slow generation and complexity, for high-resolution images. In this work, we put forward a solution based on hierarchical transformers and local parallel autoregressive generation.  
We pretrain a 6B-parameter transformer with a simple and flexible self-supervised task, a cross-modal general language model (CogLM), and fine-tune it for fast super-resolution. 
The new text-to-image system, CogView2, shows very competitive generation compared to concurrent state-of-the-art DALL-E-2, and naturally supports interactive text-guided editing on images.

Cog Voew2: Faster and Better Text-to-lmage Generation via Hierarchical Transformers
Ming Dingi Wendi Zheng' Wenyi Hong' Jie Tangi
'Tsinghua University 'BAAI
{dm180mails, jietangdmail) .attemaiiiisstttttat.aaaa edu.. edu. cn
Abstract
Development of transformer-based text-to-image models impeded by its slow generation and complexity, for high-resolution images. In this work, we put forward a solution based on hierarchical transformers and local parallel autoregressive generation. We pretrain 6B dB-parameter transformer with simple and flexible self-supervised task, cross- modal general language model (CogLM), and fine- tune it for fast super-resolution. The new text-to-image system, Cog View, shows competitive generation performance to the concurrent state-of-the-oe the-art DALL-E-2, and naturally supports interactive text-guided editing images.
besutiful girl hugging a husky.
A lon teacher wearing suit is front a bisckboard.
rabot riding under the blue and cioudy
Seversi youthe taiking bar.
$4
O
ZA
Aln R.
aadarius
Zu
Fr in
A lon man the office. young taking photos.
pirate shull
A gid holding an ol-paper ummbrelle in rairy lavee.
Eye.
magnificent
Moure righ. cherry bicssom Akitn dag ON paining. A figger with anger's 's wings.
Figure 1: Text-to- Text-to-Image samples from VieView2, which supports both Chinese and English. The actual input text in Chinese, translated into English here for better understanding. Codes and demo website will be updated at hithtts::// /tttps//gthh / httpchhwww.c.oo///eeeeecatc com/THUDM/CogVieex
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Introduction
Recently, text-to-image generation has been greatly advanced by large-scale pretrained transformers. e.g. DALL-E 22 and CogView B] These models learn to generate image tokens in an autoregressive way. However, they also suffer from the following disadvantages:
Slow generation. of non-autoregressive models, e.g- GANs IT, with the same FLOOss. Instead of employing a large number of parameters, this shortcoming is mainly attributed to the nature of token-by-token generation used in the autoregressive models cannot exploit the parallel computing ability of GPUs, even after caching hidden states 25 This is significant limitation.
Generation of autoregressive models usually is much slower than generation
Expensive high-resolution training. The current large-scale pretrained models are generally based on Transformers K00, where the attention operation has both time and space complexity of O(n²) for training sequences of length n. Within a limited budget, we face a trade-off between the number of parameters, representing the modeling power, and the resolution of the generated images. For this reason, most current text-to-image models choose a resolution of 32 x 32 tokens (usually 256 256 which is far less dense than the resolution of the real photos.
pixels) 3 26
Unidirectionality. For images, autoregressive models, e.g. GPTs, usually generate tokens in raster- scan order. This order shows the best perplexity during the evaluation 7 However, this order makes the models unaware of the tokens below or on the right side during generation, text-guided infilling is not supported. Moreover, the unidirectionality leads to a gap between the pretrained text-to-image models and vision transformers (ViTs) SS based bidirectional masked prediction, c.g. MAE [2] and SimMIM 44 limiting their application on traditional visual tasks, such as image classification and object detection.
result
Present Work. To overcome these defects, we first propose a simple and versatile pretraining method, Cross-Modal general Language Model (CogLM). Our CogLM masks various types of tokens in the sequence of text and image tokens, and learns to predict them autoregressively. Specifically, (1) if mask all the image tokens, the task becomes the same the original Cog View B in performing a text-to-image generation task; (2) if we mask random patches of image tokens, it works similarly to MAE as an infilling task; (3) if we mask text tokens, the task becomes image captioning.
The versatility of CogLM enables us to fine-tune a pretrained CogLM for different downstream tasks, and constructs hierarchical model, Cog View2. 2here are three steps in the hierarchical generation process as follows:
1. First, we generate a batch of low-resolution images (20 x 20 tokens in Viewi) using the pretrained CogLM, and then (optionally) filter out the bad samples based on the perplexity of CogLM image captioning, which is the post-selection method introduced in Cog View B
2. The generated images are mapped into 60 x 60-token images by direct super-resolution module fine-tuned from the pretrained CogL.M We use local attention implemented by our customized CUDA kernel to reduce the training expense. The high-resolution images from this step usually have inconsistent textures and lack details.
3. These high-resolution images are refined via another iterative super-resolution module fine- tuned from the pretrained CogLM. Most tokens are re-masked and re-generated in local parallel autoregressive (LoPAR) way, which is much faster than the original autoregressive generation.
How does View2 conquer the three defects? First, during pretraining the masked patch prediction task trains CogLM to handle bidirectional context, making it easy to adapt to bidirectional tasks, such as direct and iterative super-resolution. Second, the hierarchical design allows us care only about the local coherence at a high-resolution level. In this way, the local attention can be leveraged to reduce the training expense. Third, the local parallel autoregressive generation can reduce model run times from 3,600 to 6 (1/600 only), significantly accelerating the generation of high-resolution images. Cog View2 about x0x faster than the Cog View (with sliding-winde super-resolution) for generating images of similar resolution and better quality.

---

Input Text:
Input
Image:
Attention Mask
Inage Tokenizer
Cat
VOVAE
370
-01
3/2
Rg
24
366
27
J8
Discretize
Icetk package
Supports tokerization
370
both Chinese
English 000 tokens).
J71
Text tolinizzaaooo
20'20=400 image tokens
J/2
Taknn
[R0]]
Image Token
Image
imge
Sample Mask Regions
.5.4
20
11
22
3
324
5
T6
27
28
...
166
Transformer
Target takens
3
34
28
288
prediction
Figure 2: CM (Left) The sequence consists of both text and image tokens. [BOI] (Begin-Of- Image) is the separator token. Mask regions sampled according to different strategies. Only the second-to-Bed tokens in the mask regions are predicted compute the loss. (Right) The mask regions are only implemented by changing the attention mask matrix, without any modification on the input tokens. the attention mask matrix, rows and columns all masked tokens (the 2,3,4,6,7,8 rows and columns) can be extracted together to form low-triangle attention mask matrix.
Related Work
Text-to-image generation for arbitrary inputs is long-held dream for many cross-modal machine- learning researchers. Most early attempts address this challenge
based Generative
Adversarial Nets [00: these include AttnGAN 35 DM-GAN 40), DF-GAN [28] et al. Although they perform vivid synthesis
domain-specific datasets, such Caltech-UCSD Birds 200,
general-domain datasets, such as MS COCO 17
present great challenges for these methods. DALL- leverage VQ-VAE 29
to compress an image sequence of discrete tokens and pretrain large transformers for autoregressive generation, greatly
26]. Cog View B and similar works
improving results in the general domain. LAFITE [39 learns to invert the pretrained CLIP embeddings in the shared space of text and image for text-free training. Recently, many researchers have turned diffusion models, largely due the slow generation defect of autoregressive models.
One example is Glide [19
Non-autoregressive generation (NAR) is recently popular topic in natural language generation- see Mask-Predict [ and GLAT (11], which explores parallel decoding methods for autoregressive- like models. Generation speed was not an issue in the era when GANs dominated the image generation, but constitutes a considerable challenge for current autoregressive text-to-image models. M6-UFC K first introduces NAR methods into the VQ-VAE framework, and similar ideas adopted by VQ-diffusion [1I and MaskGIT []] A possible drawback of pure NAR methods is that tokens sampled the meantime might lead global inconsistency in later steps during the generation of complex scenes. Our method introduces hierarchical design to combine the consistency merit of autoregressive models and the speed ad advantage of NAR methods.
3
Method
3.1
The Cross-Modal General Language Model
While previous supervised pretext tasks often target at mask prediction in the computer vision A4 12, our approach pursues a unification of autoregressive generation and bidirectional context-aware mask prediction.
In NLP, the General Language Model (GLM) suggests changing the direct mask prediction into blockwise autoregressive generation. However, directly applying images would result in redundancy. For instance, the sizes of the masked image patches are fixed, thus do not need the

---

capacity of filling blocks of indefinite length as in NLP. Moreover, GLM inserts sentinel token for each mask region to predict its first token, which greatly increases the sequence length thus restricts the usage of 2D local attention.
Based on the analysis above, we present simpler and more general language model for both text and image data- Coo-s-Coss--ooee general Language Model (CogLM), As shown in Figure CogLM takes as input concatenation of text and images tokenized by icerk (See 3.2), whose dictionary contains 20,000 image tokens and 130,000 text (both Chinese and English) tokens. Formally, let t= ..., ty] be the text tokens and im fim...... im be the image tokens, where M and N2 are the lengths of text and image tokens respectively.
The crucial step in CogLM is to sample l. mask regions R {1o, ro], .m [e,, rul} according to various strategies. In practice, the following two strategies are used:
(Text-to-Image GPT) The input sequence is ft [DI]] im
We mask all the image
tokens, which is similar to the pretraining task of Cog View
(A Combination of Mask Prediction and Image Captioning) The input sequence
[imo im, imy im [BOE/C] c] where [BDE] [BOC] are separators meaning beginning- of English and beginning cffnneeee used for the corresponding language. we mask random patches and the text tokens. Ideally, the two tasks should be separated; but we
combine them together for training efficiency.
Instead of replacing the tokens in the mask regions as [MASK], we make no change in the input but build attention mask A based on the mask regions. All tokens outside mask regions seen as context and can be attended to by all other tokens. A token in mask regions can only be attended to by the tokens in mask regions and behind it. Specifically,
[ L.
v [,, rul E R,j & [l,, Tul, 1, if < and 3 u, v (indices), i E !uirul E R,j E Por R.
A[i,j]
(1)
0, else.
Figure shows example of the attention mask matrix of two mask regions. In the mask regions, the model learns to predict the next token. The loss function can be written as follows:
log p(1:++ | (conteat),
En Tu s=
where the #contert denotes the tokens outside the mask regions.
A fox is siting on the books.
Infilling. Note that the first token in each mask region is not pre- dicted during training. This feature seems to disable CogLM from image infilling cloze filling in natural language, but this problem actually has simple solution. During inference, we can move the last context token before each mask region into as illustrated in Figure 3 Although these moved tokens becomes blind spots for mask regions before them, they have few negative effects prac- tice. To further avoid this minor influence and fully maintain the context information, we deal with each mask region individually. For each region, we move only the last context token before this re- gion, and keep all the known tokens outside the mask regions. Thus, we cannot use the cached hidden states from the previous region, slightly slowing down the multi-region infilling. See Appendix for samples.
The crder of Gemeration
Figure 3: Image Infilling of CogLM. Tokens (viewed as patches) in light green mean mask regions.
GLM 6 and MAE 12 (GPT) The main ad advantage over GPT is that the modeling of bidirectional contexts is considered in CogLM,
Advantages over GPT 22
http://gittaab.ccm///D/////////

---

which will benefit many tasks relying global information, e.g. super-resolution in the next section and image classification. The importance of bidirectional context has been verified in the comparison of BERT [2] and GPT on GLUE 31. (GLM) The main advantage over GLM is simplicity. To unify the generation and bidirectional understanding, GLM needs to define many new special tokens and new type of position embedding, insert sentinel for each mask region and change the order of input tokens. It destroys the spatial relevance in the image data and excludes the possibility of using 2D local attention or convolution. (MAE) MAE is designed for self-supervised learning pure image data and is not ready for generation. Even without text, CogLM more parameter-efficient, because MAE is an encoder-decode structure. A considerable part of parameters in encoders and decoders are learned for the same function, e.g. extracting basic features from inputs.
3.2 Pretraining
As we have introduced CogLM as a general pretraining framework, in this section, we will describe the details and hyperparameters of our pretrained CogLM.
Tokenization. We have developed unified tokenizer icetk of Image, Chinese and English. As shown in D2 D a large vocabulary (128,000 tokens) offers many benefits. For text, we extract a bilingual vocabulary of 1 130,000 tokens in icetk and explicitly classify them as Chinese, English, Common or Rare Symbols, so that we can specify the generated language via sampling mask.
The image tokenizer is a 20,000-token first-stage VQ-VAE [29], largely following the tokenizer in View R Inspired by Esser et al. f a term of perceptual loss 37 is added to the reconstruction loss, significantly improving reconstruction performance. (See Appendix for details.)
Transformer. The backbone of our pretrained CogLM is a Transformer with Sandwich Layer- Norm 3. The model has billion parameters (48 layers, hidden size 3072, 48 attention heads), trained for 300,000 iterations in FP16 with batch size 4,096. The sequence length is 512, consisting of 400 image tokens, 1 separator and up to text tokens.
Masking Strategy. We randomly select a sampling strategy for each training sample. For the mask prediction strategy, the analysis from SimMIM 3 exhibits the great importance of mask percentage and patch distribution. We follow their results to sample 4 x 4 token patches at random until 75% of the tokens are in the mask regions. For bilingual samples, we randomly choose one of the languages during training.
3.3 Hierarchical Generation
Although the pretrained CogLM can generate images from text, the resolution is only 20 x 20 tokens (160 160 pixels). The short sequence is intentional. for fast generation. The versatility of CogLM allows us to fine-tune it into super-resolution models. The whole hierarchical pipeline makes up our CogView2 system.
Direct super-resolution. In this step, we want a model to map a generated low-resolution image token sequence im E [0, 20000) to a higher-resolution sequence im I E [o, 20000) 60000)60x60, fine-tune the pretrained CogLM into encoder-decoder architecture. The input of the encoder is the 20 x 20 sequence of generated image tokens, and the input of the decoder is just a 60 x 60 sequence of [MASK]. We do not follow the original transformer B0 to add cross-attention layer, instead make the tokens in the decoder attend both local tokens in decoder and encoder. This cross-resolution local attention is implemented via customized CUDA kernel introduced in section 42 Both encoder and decoder are initialized using the pretrained CogLM. In practice, we find it enough to only fine-tune the weights of the attention layers the decoder, so that we can fix and share the other parameters between the encoder and decoder to reduce the memory consumption.
60x60, We
Although direct mapping is traditional practice for super-resolution-ee. SRCNN 4 --it is hardly qualified as generation; it focuses more on texture transformation. The loss function of direct mapping token-based or pixel-based (MAE), meaning that it predicts maximizes the marginal distribution p( (im lim") ") for each token instead of p(im" Tim"). As use cross-entropy loss and multinomial sampling during generation, we get
im' [im]... ... im im oo), im. po (im) iim) èm, and im? are independent if i i
5

---

Input text: A great church.
Local window details.
CogLM (20' 20 takens)
Mask 75%.
Direct Super- Resolution. (60'80 takens)
Iterative super-resolution. All the local windows generate simultaneously.
Figure 4: Super-resolution modules. Low-resolution images are mapped into high- nigh-resolution images via the direct super-resolution module. In each snapshot during the iterative super-resolution, all tokens of the same color are generated at the same time. All the local windows work in parallel.
Therefore, we need refine im using another module.
Iterative super-reson resolutios..... In this step, aim to refine the initial high-resolution sequence im into a better one im ? The working principle of the refinement is to break the independence of the generated tokens, while keeping the parallelism. Thus, we propose a local parallel autoregressive (LoPAR) approach.
The
The motivation of LoPAR is that the hierarchical process frees us from global dependence. As long as we maintain 25% ratio from MAE 12] random tokens as context, it is sufficient to recover the global scene the image. If the re- re-generated tokens are coherent locally with 25% kept tokens, global coherence is also guaranteed. We mask 75% of the tokens of im and assume that there is local window size
p(im 2 iim) = p(i (im) di < a and j is not masked.)),
(4)
p(im im im?) p(im if dist(i, >
(5)
so that local attention is sufficient and tokens from different local windows can be generated in parallel. To further increase the parallelism, we find the local inconsistency usually occurs when directly adjacent (vertically or horizontally) tokens are generated at the same time. We factorize the generation process into different iterations diagonally as Figure and below:
row =o
p(im
iim
II R=0
II
piim im ) {im? row(j) col(j) < k))),
where row(i) = |سنا mod and col(i) = (i- 1) mod are the indices of row and column in the
local window.
To implement the iterative super-resolution module, we tne-tune the pretrained CogLM for 20,000 iterations into BERT-style masked prediction model on 60 x 60-token sequences with local attention. The mask ratio is sampled from {0.2, 0.4, 0.6, 0.8, 0.9} for each sample. During inference, we set the local window size arranging the unmasked tokens and merging the first and final iteration
6 and compress the iterative process from 2n
iterations by
IImplemented by manually designed x 6 matrix. Details are included in released codes.

---

4 Plug-in Improved Techniques for Transformers
4.1 Cluster Sampling
In autoregressive generation, the sampling strategy over the predicted distribution of the tokens is crucial. Top-k or top-p (nucleus) sampling 14 are the most common strategies, but suffer from an incomplete truncation problem.
The vocabulary of the image tokens learned by VQVAE 129] where the embeddings of some tokens are very similar. To represent the frequent patterns at a finer granularity, we use a large vocabulary of 20,000 tokens, three times larger than that of the previous works (2 K further exacerbating the situation. For instance, there are about 42 tokens basically "white" in icetk, which show subtle differences only when connected to some other tokens. Although the sum of the probabilities of these "white" tokens might be large enough, most them could be filtered by top-k sampling. Figure 5 iillustaaes the problem.
Truncation with top-k sampling
P
Tokens sorted by probabilities.
Figure 5: Incomplete truncation. The same color indicates very similar embed- dings of the tokens. The hard trun- cation of top-k sampling twists the proportion between blue, green and red tokens.
(Best viewed in color.)
To solve the incomplete sampling problem, we propose cluster sampling. We group the 20,000 tokens into 500 clusters via K- means (18] based on their vectors VQVAE. During sampling, we first sample a cluster using top-k sampling based on the sum of probabilities of tokens in the clusters, and then sample the cluster. All the tokens within cluster are treated as whole and will be filtered or kept together, alleviating the incomplete truncation problem.
4.2 Local Attention
Locality is one of the most important properties of image data. Local operations, e-g. convolution, dominated the visual computing before ViTs [5]. Even attention in the
Mamory consumattoon diffrentt local attertion impamintation.
6000
CUDA attenson (RF-51
Pytorch local attentinn (RF-11)
CLIAA local attennion (Ri-ll))
ViTs mainly deals with the interactions between local to- kens 24
100
We find it possible to fine-tune the pretrained CogLM using local attention and textual attention, which generally compatible with the global attention weights from pretraining. However, 2D local attention cannot be implemented efficiently using high-level framework, e.g. Pytorch (20] We develop a customized CUDA kernel to support both 2D local attention, 2D autoregressive lo- cal attention and cross-resolution local attention. In the CUDA kernel implementation, we can save half of the computation in the matrix multiplication and do not need a causal attention mask for the autoregressive attention. In the super-resol resolution modules, we use local attention with the receptive field (RF) of mark for single-head attention with hidden size 64 on a A100 GPU. The advantage of our method will be more obvious in autoregressive scenarios, which is up to 40x faster and consumes 1% memory than global attention on 4,096 sequences.
300
200
30
92
300
Image stas, the sequuccc langth
Time consumption for different local attention implementation.
full attentin
potorch local attention (RF-5)
355
CLIDA lecal amhention (ii-is)
20-
25-
20-
1.5-
1.0-
Figure O show the bench-
0.0
Figure 6: Comparison between CUDA kernel-based local attention, full atten- tion, and Pytorch implementation based on the unfold (im2col [T5) operation. The hidden size in the benchmark is 64.
4.3 Upweighting Textual Attention
Most text-image pairs are weakly relevant in the large training data of CogLM. Even the model perfectly fits the data, it should have considerable probability to generate irrelevant images. To strengthen the relevance, we leverage the explainability of the attention operation. We add constant e to the attention scores from any token to the text tokens: (The attention mask is omitted for simplicity)
7

---

QºK +1 c
Attention(Q, K, V, A) softmax
0. DIV.
text part image part
This technique costs ignorable time consumption but largely improves the textual relevance of the generated images. In practice, c < will not influence the quality of the images.
Experiments
5.1
Dataset
Our dataset for pretraining contains about 30 million text-image pairs, mostly overlapping with that of Cog View 3. We filter about million text-image pairs from the CogView dataset with some keywords, e.g. "abstract" and "texture", because they are mostly background images used for design. These images consist of repeating patterns and contribute little text-to-image generation. We then replenish the dataset with 5 million tag-image pairs. About half the text is translated from English, and both Chinese English text are kept to train our bilingual CogLM. Only the images whose resolution is at least 480 x 480 are used to train the super-resolution modules.
5.2 Machine Evaluation
To compare with previous and concurrent works, we follow the most popular benchmark originated from DALL-E (26), Fréchet Inception Distances and Inception Scores evaluated on MS-COCO [17 30,000 captions from validation set are sampled to evaluate the FID. Since each image COCO has up to 5 different captions, carefully select the sampled captions describe different images. generate 16 samples for each caption (translated into Chinese), and select the best one with the lowest caption perplexity (the Caption Score in B). Note that FID not the perfect metric to evaluate Cog View2 because (1) the advantage of Cog View2 is to generate high-resolution images, but we need to resize the images back to 256 x 256 for meaningful comparison. (2) There are mistakes when translating English captions into Chinese. (3) Our training data contain many single-object images, which are quite different from those in the distribution of COCO (common objects in context).
Table 1: Machine Evaluation Results on MS-COCO. (Downsampling Cog View images to 256 x 256.) oight means fine-tuning on MS-COCO. "_ technique" is the ablation study without this technique. Cog View2 achieves the best blurred FIDs over all comparable methods.
Model
FID-O
FID-1
FID-2
FID-4
FID-8
IS
AttnGAN
55
35.2
44.0
72.0
108.0
100.0
23.3
DM-GAN
26.0
39.0
73.0
119.0
112.3
32.2
DF-GAN
26.0
33.8
55.9
91.0
97.0
18.7
DALL-E
20
27.5
28.0
45.5
83.5
85.0
17.9
CigView View
3
27.1
19.4
13.9
19.4
23.6
18.2
XMC-GAN
36
9.3
30.5
NÜWA*
33
12.9
13.8
15.7
19.3
24
27.2
LAFITE
39
26.9
23.0
18.7
15.7
14.8
26.0
VQ-diffusion-FF
13.86
Make-A-Scema
7.55
DALL-E-2 27
10.9
Cog View2
24.0
19.7
16.8
17.2
17.2
22.4
clustering sampling
36.4
32.4
28.9
28.5
30.4
18.8
attention upweighting
24.6
20.4
17.5
17.9
18.9
21.1
Cog *
17.5
13.4
10.9
10.6
10.4
25.2
The results of machine evaluation are demonstrated in Table D find that fine-tuning CogLM on the MS-COCO dataset will largely improve the FID. During our fine-tuning, FID diminishes from 24.0 (0 iteration)-> 19.2 (2,500 iterations) - 17.5 (7,500 iterations). However, we find that the quality (human evaluation) of generation deteriorates. Though the style is similar to COCO, the

---

generation is not as accurate as for the non non-fine-tuned version, which also corresponds to the scores human evaluation Figure
5.3 Human Evaluation
As the most persuasive metric, we conduct large-scale human evaluation following the setting (See Appendix for details). The experiments include a total of 4,600 groups of
in Cog VogView B
comparison on COCO captions between some public available text-to-image works, including DF-
39). View [3) Cog View2 (including its finetuned version on COCO) and the recovered ground truth after VQVAE. Note that the VQVAR in CogView2 is much better than that in Cog View, which makes the recovered ground truth a stronger upper bound. The results are demonstrated in Figure 7 An intriguing finding is that the finetuned Cog View2, although with much better FID, performs worse than the original model. We guess that the model might fit the style of complex scenes in COCO, but the generated samples with isolated subjects could be preferred by the annotators.
GAN 2 LAFITE
Cogstew-12
7.45
Cogy
Laftte
3.0
Gruud Truis
tupper
IN Human Prefeeecee The persentage of the model be choson questions.
Overa foores (1- 10) for the models.
09 Scooee (1-5) for the modes on three important repectss
Figure 7: The results of human evaluation. Cog View2 performs the best in all the aspects.
5.4 Analysis the Speed and FLOPs of LoPAR
As discussed in $ our motivation is to increase the degree of parallelism for inference acceleration, even with more FLOPs. Autoregressive generation with cached hidden states have the same FLOPs with teacher-forcing forward step, but is much slower (858ms vs 225.9s in Cog View2 scale). For LoPAR, it is exactly N (N in our setting) times and FLOPs of forward steps. We compare the inference speed of super-resolution stage with different strategies in Table 2
Table 2: The wall-clock time and FLOPs for a 4,096 sequence A100-40GB GPU with different AR-related methods. The model configs are the same as the pretrained CogLM 6B.
FLOPs
Time
Memory (inference)
Forward (also teacher forcing training)
1.17 1014
858 ms
5,041MB
Autoregressive generation (no eache)
4.81 (407
about 1h
5,041MB
Autoregressive generation (cached)
1.17 1014(1x)
225.9s
4,865MB
LoPAR
7.02 (6x)(6))
4.89s
5,041MB
LoPAR local attention
5.82 1014"
3.41s
352MB
6 Discussion
Autoregressive or Diffusion? Although GPTs achieved great success in text generation, diffusion models are becoming increasingly popular in image generation. Here we compare diffusion models with autoregressive models from the aspect of speed, the largest disadvantage of the autoregressive models discussed the section | With the same architectre,, diffusion models require more FLOPs but have a high degree of parallelism. They can also make trade-off between the quality and time consumption by manually scheduling the stride of sampling. For example, Glide [99 samples 250 diffusion steps for evaluation, and 27 steps for interactive sampling, to reduce the latency to 15s.

---

Autoregressive models must generate the imgge token- by-token, but our LoPAR can upsample the image with a high parallelism degree, so that (potentially) we can reduce the time cost by introducing more hierarchies to design models much faster than diffusion models.
Comparison between DALL--E22 and CogView2. DALL-E-2 27 is recently released concurrent work for text-to-image generation on 1024 x 1024 resolution. Its probabilistic model and architecture are quite different from those in Cog View2. But both models share the same spirit - hicrarchical generation. The difference is that DALL-E-2 adopts an additional third-level super-resolution and a generation prior, which help result in potential quality gain, but also lead to expensive resource- consuming. CogViews is able to synthesize similar scenes according to the limited demos of DALL-E-2, e.g. "lion teacher" (Figure [I vs. "panda scientist" (DALL-E-2), considering CogViz View- 2 is trained using only 5% of the total data (650M text-image pairs) by DALL-E-2. For future, Cog View2 can also adopt the third-level super-resolution and the prior, though it is engineering mostly.
7 Conclusion
The breakthrough in the text-to-image domain is made by autoregressive models. However, the slow generation and high complexity hinder researchers attempts to improve the quality in this direction. In this paper, we put forward an approach based on hierarchical transformers to help autoregressive models remedy these disadvantages, and bridge the gap between text-to-image pretraining and recent visual representation learning methods.
Broader Impact. The advancement of text-to-image generation, especially text-guided image editing, will ease the creative efforts of artists and designers, while also causing a risk of misinformation, leading to permanent damages to the reliability of web photos. However, it is possible to train classifier to distinguish the real and Cog Vieee--ggeeee images according to the texture features.
Acknowledgments and Disclosure of Funding
We would like to thank Zhao Xue and Sha Yuan for the help on collecting the dataset, Hanxiao Qu for maintaining the machines, and Yue Cao and Chang Zhou for their useful discussion, Zhendong Zhang for releasing an initial version of CUDA local attention.
Funding in direct support of this work: GPU hours donated by BAAL, NSFC for Distinguished Young Scholar (61 (61825602).
References
[1] H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman. Maskgit: Masked generative image transformer. arXiv preprint arXiv:2202.04222 04200, 2022.
[2] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1/100008805, 2018
[3] M. Ding, Z. Yang, W. Hong, W. Zheng, C. Zhou, D. Yin, J. Lin, X. Zou, Z. Shao, H. Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34, 2021.
[4] C. Dong, C. C. Loy, K. He, and X. Tang. Learning deep convolutional network for image super-resolution. In European conference on computer vision, pages 184-199. Springer, 2014.
[5] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[6] Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang. All nip tasks are generation tasks: general pretraining framework. arXiv preprint arXiv:2103. 2021.
[7] Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. arXiv preprint arXiv:2012.0098/ 2020.
10

---

[8] O. Gafni, A. Polyak, O. Ashual, S. Sheynin, D. Parikh, and Y. Taigman. Make-a-scenn: Scene-based text-to-image generation with human priors. arXiv preprint arX1v:2203.13131, 2022.
[9] M. Ghazvininejad, O. Levy, Liu, and L.. Zettlemoyer Mask-predict Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 6112-6121, 2019.
[10] J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2000 2014.
[11] S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo. Vector quantized diffusion model for text-to-image synthesis. CoRR, abs/2111.14822 2021.
[12] K. Hc, X. Chen, S. Xic, Y. Li, P. Dollár, and R. B. Girshick. Masked autoencoders are scalable vision learners. CoRR, 0ts/2111106677, 2021.
[13] He, X. Liu, J. Gao, and W. Chen. Deberta: Decoding-enhanced bert with disentangled attention, arXiv preprint arXiv:2006.033330 03654, 2020.
[14] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.0977 2019.
[15] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia, pages 675-678, 2014.
[16] T. Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint arXiv:/804 10959, 2018
[17] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[18] J. MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statisties and probability, volume 1, pages 281 1-77. Oakland, CA, USA, 1967.
[19] A. Nichol, P. Dhariwal, A. Ramesh, Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, arXiv preprint arXiv:2112.1074 2021.
[20] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[21] L. Qian, H. Zhou, Y. Bao, M. Wang, L. Qiu, W. Zhang, Y. Yu, and L. Li. Glancing transformer for non-autoregressive neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2003, 2021.
[22] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[23] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2 103.00020, 2021.
[24] M. Raghu, T. Unterthiner, S. Korbllith, C. Zhang, and A. Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances Neural Information Processing Systems, 34, 2021.
11

---

[25] P. Ramachandran, T. L. Paine, P. Khorrami, M. Babaeizadeh, S. Chang, Y. Zhang, M. A. Hasegawa-Johnson, R. H. Campbell, and T. S. Huang. Fast generation for convolutional autoregressive models. arXiv preprint arXiv: 10X.v:/704.0000 700.06001, 2017.
[26] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102. 1202,, 2021.
[27] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0612 2022.
[28] M. Tao, H. Tang, S. Wu, N. Sebe, F. Wu, and X.-Y. Jing. Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprint arXiv:2008.05865, 2020.
[29] A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6309-6311 6319, 2017.
[30] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. arXiv preprint 0rX.v:/706.03777 2017.
[31] A. Wang. A. Singh, J. Michael, F. Hill. O. Levy, and S. Bowman. Glue: A multi-tass benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, 2018.
[32] Z. Wang. E. P. Simoncelli, and A. C. Bovik. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pages 1402-1402. leee, 2003.
[33] C. Wu, J. Liang, L. Ji, F. Yang, Y. Fang, D. Jiang. and N. Duan. NR" uwa: Visual synthesis pre-training for neural visual world creation. arXiv preprint arXiv:2111.12417 2021.
[34] Z. Xie, Z. Zhang. Y. Cao, Lin, Bao, Z. Yao, Q. Dai, and H. Hu. Simmim: A simple framework for masked image modeling. CoRR, abs/2111 0bs/2111.0998 10.988,, 2021.
[35] Xu, Zhang, Q. Huang, H. Zhang, Gan, X. Huang, and X. He. Attngan: Fine- aine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1316-1324, 2018.
[36] H. Zhang, J. Y. Koh, J. Baldridge, H. Lee, and Y. Yang. Cross-modal contrastive learning for text-to-image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 833-842, 2021.
[37] R. Zhang, P. Isola, A. A. Efros, F. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[38] Z. Zhang, Ma, C. Zhou, Men, Li, M. Ding, Tang. J. Zhou, and H. Yang. M6-ufc: Uni- fying multi-modal controls for conditional image synthesis. arXiv preprint 10Xiv:2105.144 2021.
[39] Y. Zhou, R. Zhang, C. Chen, C. Li, C. Tensmeyer, T. Yu, J. Gu, J. Xu, and T. Sun. Lafite: Towards language-free training for text-to-image generation. arXiv preprint arXiv:2111.13792. 2021.
[40] M. Zhu, P. Pan, W. Chen, and Y. Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5802-5810, 2019.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le roe're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [Yes]
13


Reviews:
[{"review_id": "ag5GOZXc_5P", "reviewer": "Reviewer_swZ7", "summary": "This paper proposes a pretraining method, a Cross-Modal General Language Model (CogLM), that masks both image and text tokens in input and learns to predict them in an autoregressive manner, while handling bidirectional context. By fine-tuning a pretrained transformer with this approach, the authors construct a hierarchical model, CogView2, which first maps a generated image into a larger image (direct super-resolution) and refines local patches (local parallel autoregressive), thus improving resolution as well as inference speed.\n\nThe experiments in the paper suggest that CogView2 performs comparable to other models despite its smaller model size and training data size. Meanwhile, the approach results in a considerable reduction in model run times (e.g. 10x faster than its predecessor, CogView).\n\nTraining and evaluation in this paper considers text in both English and Chinese.\n", "questions": "Questions\n* What were the main challenges/blockers for directly comparing different models’ inference time in an end-to-end fashion?\n\nSuggestions\n* Discussion on failure modes: Even from the cherry picked examples in Figure 1, there are multiple failure modes observed (e.g. different numbers or lengths of fingers). What are the main types of failure modes the authors or human annotators observed? Any difference when it’s in English vs. Chinese?\n\nNitpicking\n* In Figure 2: “Supports tokenization of both Image Chinese and English” → “Supports tokenization of both images and texts in Chinese and English”\n* In Section 2: “DF-GAN, et al.” → “and DF-GAN.”\n* In Section 2 and throughout the paper: use either “VQ-VAE” or “VQVAE” to be consistent\n* In Section 3.1: consider moving the paragraph “In NLP, the General Language Model [...]” to Related Work\n* In Section 3.1: $l$ and $r$ are not defined\n* In Section 3.1: “where [BOE], [BOC] are separators meaning beginning-of-English and beginning-of-Chinese” → “where [BOE] and [BOC] are separators to indicate the beginning of English text and that of Chinese text” \n* In Section 3.1: “Ideally, the two tasks should be separated” is not justified\n* In Section 3.2: “Image, Chinese and English” doesn’t really type check; maybe it should be “Image and Text in Chinese and English”?\n* In Section 5.2: “Frechet Inception Distances and Inception Scores” → “Frechet Inception Distances (FID) and Inception Scores (IS)”\n* In Section 6: clarify “third-level super-resolution)\n* In Section 7: “it is possible to train a classifier to distinguish the real and CogView2-generated images according to the texture features” is not supported with any evidence\n", "limitations": "I think some scoping about text is necessary. The paper generally assumes that input text can be any text in English and Chinese; their github repository explicitly says “any text” (https://github.com/THUDM/CogView2). For preciseness, it would be helpful to note any potential/practical limitations more clearly. For instance, CogLM can accept up to 111 text tokens (Section 3.2). And presumably, there aren’t that many short text inputs (e.g. one or two words) in the training data – then what would be a reasonable minimum length for the text input for the model to perform well? More generally, based on the types of images and texts in the training data, CogLM may perform better for certain kinds of images and texts. This insight can greatly help the future use of this pretrained model.\n\nThis applies to images as well. The fact that this paper only considers square images (based on N^2 notation) is not explicitly addressed.\n", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The paper is clearly written and aided with great visualizations. Overall, the problems and proposed solutions are well-motivated and supported with appropriate evidence or justification (e.g. findings from existing literature, their own empirical observations, or limitations due to compute resources).\n* The topic is timely; the effort for making the task of text-to-image generation faster and better is of great interest to the community.\n* Their approach of generating a low-resolution image and refining it to be a high-resolution image is simple and straightforward. The use of various techniques is adequately justified (e.g. masking strategy and attention mask) and ablated (e.g. clustering sampling and attention upweighting).\n* “Faster” text-to-image generation:  One of the main contributions of this paper is to make text-to-image generation faster. Although the experiment section doesn’t directly compare different models’ inference time in an end-to-end fashion, the last paragraph of the introduction section mentions that their model run time for local parallel autoregressive generation is 600x faster and overall 10x faster than their previous model.\n* “Better” text-to-image generation: The experiments follow popular benchmarking practice and discuss the gap between automatic metrics and human evaluation. With automatic metrics, CogView2 performs comparable to other methods on MS-COCO based on Frechet Inception Distances. Based on human evaluation, CogView2 performs better on all metrics (image clarity, texture quality, and relevance to the caption) than CogView, Lafite, and DF-GAN.\n\nWeaknesses\n* “Better” text-to-image generation: As the authors acknowledged in the paper, the automatic metrics on MS-COCO may not be the best way to evaluate these models, hence the claim of CogView2 being “better” (in the title) or competitive (in the abstract) compared to other models such as DALLE2 can use some scoping/hedges. Outside of this paper, there have been some informal qualitative comparisons between several models, which seem to give the impression that CogView2 is not strictly better than other models. \n  * https://huggingface.co/spaces/THUDM/CogView2\n  * https://twitter.com/bhagatsurya2/status/1542824988092530689\n  * https://www.reddit.com/r/MachineLearning/comments/vkvq0j/r_cogview2_faster_and_better_texttoimage/\n* Bilinguality: Since this paper considers text in both English and Chinese and notes that “Chinese input produces better results than English input” (https://huggingface.co/spaces/THUDM/CogView2), it would be interesting to see more in-depth analysis on this bilingual aspect of the model. The authors state that they used [BOE] to denote the beginning of English text and [BOC] for Chinese text, but do not justify the decision or discuss any findings based on two languages.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SR9o2rOH40", "reviewer": "Reviewer_86fW", "summary": "Auto-regressive transformer-based text-to-image model can generate elegant pictures, but also face low-generation speed problems for high-resolution image generation. This is due to the extremely long sequence length and the auto-regressive decoding schema. In this work, the authors propose Cogview2, which is a hierarchical transformer and adopts local parallel auto-regressive generation instead of the global AR. The proposed Cogview2 achieves comparable or even better results than Cogview and speeds up the inference a lot. Overall, Cogview2 is an efficient version of Cogview but more fast benefiting from the local parallel decoding.", "questions": "\n-   Have you tried other local parallel generation manners? e.g. within a local patch, left to right or top to bottom generation in a non-autoregressive fashion?\n-   The attention mask shown in Figure2 is somehow difficult to understand. Do you mean the token in green box is masked and going to be generated? This slightly confuses me.\n-   What about the speed of iterative super-resolution compared with stacked image super-resolution models in Imagen?", "limitations": "There is no limitation discussed in the current version. The authors have addressed the potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n-   The proposed Cogview2 consists of several modules. First, they use the CogLM to generate a preliminary image with 20 $\\times$ 20 tokens. The following super-resolution is based on this initial image, and I think a useful finding here is that we do not need to predict an image with a high resolution from the scratch, since a small 20 $\\times$ 20 tokens may involve the information given the text. This may motivate the following researcher to design more efficient text-to-image models.\n-   Local parallel autoregressive decoding is interesting. Traditional AR for text-to-image generation flatten the image into a 1-D sequence and generate the visual tokens from left to right. It is quite time-consuming and also ignores the spatial correlations.\n-   Thanks to the flexibility of pretrained general language model, Cogview2 can perform image completion naturally \n-   Achieve comparable results with auto-regressive modeling while speeding up the inference quite a lot.\n\n## Weakness\nThere is no major concerns for me. Maybe, I would like to see a totally non-autoregressive model which can deliver comparable results with AR ones.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "IS-KPcyimMM", "reviewer": "Reviewer_1wZ4", "summary": "This paper presents CogView2, which aims for faster and better text2image generation based on its previous work CogView. The key idea is to adopt the spirit of coarse to fine and generate low-resolution tokens first(20x20) and then do a super-resolution to 60x60 tokens for high-resolution images.", "questions": "Please refer to the weaknesses part.", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper tries to solve two critical issues of the current auto-regressive-based text2image model: quality and efficiency. \n2. This paper provides many visualization consequences in the paper.\n\n\nWeaknesses:\n1. This paper claims too many things but does not verify them clearly in experiments. a) The training and inference efficiency of text2image generation are not evaluated or compared with previous methods with wall clock time or FLOPS. 2) As introduced in the introduction section the CogLM is general for many tasks like infilling tasks, and image captioning, but these capabilities are not presented in the paper.\n\n2. The comparison with current works is not convincing. For example, CovViiew2 didn't show significant improvements over previous methods on FID-0. Previous SOTA methods like Make-A-Scene and DALL-E2 did not report the results of FID-1 to FID-8 for comparison. Besides, some latest works like latent space diffusion and VQ-Diffusion are missed in the table for comparison.\n\n3. The quality and fidelity of these generated samples presented in the paper are not that impressive. First, the generated image is still blurry. Second, we can observe clear unreasonable structures for the human hands or faces. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "A_tdGDBj_0u", "reviewer": "Reviewer_GY7Q", "summary": "This paper proposes a faster and better text-to-image generation model called CogView2. Compared to the CogView baseline, main contributions are: 1) hierarchical generation that upsamples the original low-resolution image tokens and then refine them, 2) customized CUDA kernel that speeds up training, 3) a special attention masking strategy used in CogLM during training. Experiments clearly demonstrate that CogView2 generates better images than CogView both in terms of FID scores and also in terms of human evaluation. \n", "questions": "(1) It will be great if authors can provide more visualizations, especially those with higher resolution. I can not judge the quality of generated images from just a few hand-picked ones. \n\n(2) What is FID-k in Table 1? I assume it means radius of the Gaussian Filter but couldn’t find any explanation in the text. If my guess is correct then shouldn’t FID–0 be the most important number? Does this mean that CogView2 is worse than multiple baseline methods in terms of image quality?\n\n(3) As stated before, I have concerns about the masking strategy. I do not see a motivation for joint learning of autoregressive generation and bidirectional mask prediction. Authors are encouraged to provide more ablation studies to support the design of CogLM and explain why it is better than the training in CogView. \n\n(4) For human quality evaluation, I wonder why authors do not include DALL-E as part of the test? That should be a key baseline.\n\n", "limitations": "Limitations and potential negative societal impact are adequately addressed by the authors.\n", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe paper is mostly easy to follow. Authors’ proposed idea of using super-resolution and refinement modules to hierarchically generate higher resolution images is intuitive and works well in practice. Designing these two modules is clearly non-trivial work. Authors further demonstrate that clustering sampling is better than simple top-k. I also appreciate that they are willing to write a customized cuda kernel to speed up training. The speed-up seems to be significant in the autoregressive case.\n\nWeakness:\n\nModel improvement is limited compared to CogView. Both use Transformer to jointly learn the likelihood of text tokens and image tokens. The main difference is the new masking strategy in CogLM where tokens inside the mask region are trained to predict the next token based on past masked-tokens and non-masked context tokens. Authors stated that this approach unifies autoregressive generation and bidirectional prediction, but I find this design lacking justification. Experiment section also fails to provide any ablation study to support the choice of this particular masking strategy. \n\nEven with the new hierarchical design, CogView2 still generates image resolution lower than other works such as DALL-E-2 [1] and Imagen [2]. Stacking another direct/iterative super-resolution module is straightforward and should solve the issue. Resource limitation is indeed a problem but still I need to point out the lower resolution as part of the weakness. \n\nOnly a few hand-picked visualizations are provided in the main paper and in the supplement. This makes it very hard to qualitatively judge the performance. \n\n[1] Ramesh, Aditya, et al. \"Hierarchical text-conditional image generation with clip latents.\" arXiv preprint (2022).\n\n[2] Saharia, Chitwan, et al. \"Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.\" arXiv preprint (2022).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept