CogView: Mastering Text-to-Image Generation via Transformers
Authors: Ming Ding
Zhuoyi Yang
Wenyi Hong
Wendi Zheng
Chang Zhou
Da Yin
Junyang Lin
Xu Zou
Zhou Shao
Hongxia Yang
Jie Tang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transformer
pretraining
generative model
cross-modality

Abstract: Text-to-Image generation in the general domain has long been an open problem, which requires both a powerful generative model and cross-modal understanding. We propose CogView, a 4-billion-parameter Transformer with VQ-VAE tokenizer to advance this problem. We also demonstrate the finetuning strategies for various downstream tasks, e.g. style learning, super-resolution, text-image ranking and fashion design, and methods to stabilize pretraining, e.g. eliminating NaN losses. CogView achieves the state-of-the-art FID on the blurred MS COCO dataset, outperforming previous GAN-based models and a recent similar work DALL-E.


Cog View: Mastering Text-to-lmage Generation via Transformers
Ming Ding', Zhuoyi Yang', Wenyi Hong', Wendi Zheng', Chang Zhou', Da Yin Junyang Lin', Xu Zou', Zhou Shao Hongxia Yang', Jie Tang Tsinghua University *DAMO Academy, Alibaba Group BAAI {dm1 10m180mals, (eetang@mail).tsiinnn tsinghua. edu. cn
Abstract
Text-to-lmage generation in the general domain has long been an open problem, which requires both a powerful generative model and cross-modal understanding. We propose CogView, View, a 4-billion- 4-billion-paramett parameeer Transformer with VQ-VAE tokenizer to advance this problem. We also demonstrate the finetuning strategics for various downstream tasks, e.g. style learning, super-resolution, text-image ranking and fashion design, and methods to stabilize pretraining. e.g. eliminating NaN losses. CieView achieves the state-of-the art FID the blurred MS COCO dataset. outperforming previous GAN-based models and a recent similar work DALL-E.
women mikinng phore
Sen clock London.
or parb rides motoroycle.
footbost
coffee printed with background
bicycle.
attorstock
Chinese masiiional dram- ing. Statue of Liberty.
CN painting. Lion.
Sketoh, Houses.
Cartoon, A Aiper is playing footbal.
Super-resolation: mid- sikee pawlion
Figure 1: Samples generated by CogView. The text in the first line is either from MS COCO (outside our training set) or user queries on our demo website. The images in the second line are finetuned results for different styles super-resolution. The actual input text is in Chinese, which is translated into English here for better understanding. More samples for captions from MS COCO are included in Appendix
Introduction
"There are two things for a painter, the eye and the mind... eyes, through which view the nature; brain, in which we organize sensations by logic for meaningful expression." " (Paul Cézanne [7])
'Codes and models are at httptttt:///tttubbbbbbb http:///ppspaaacoaoooot/u/ooooat We also have a demo website our latest model at htttps://wdda aminer cn/ www.cc/oooo/einnnnccoco wnwp:.//iixxxiem/ html (without post-selection).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

As contrastive self-supprvised pretraining has revolutionized computer vision (CV) 24
21 8 [2]
visual-language pretraining, which brings high-level semantics to images, becoming the next frontier of visual understanding [38 30 39 Among various pretext tasks, text-to-image generation expects the model to (1) disentangle shape, color, gesture and other features from pixels, (2) under- stand the input text, (2) align objects and features with corresponding words and their synonyms and (4) learn complex distributions to generate the overlapping and composite of different objects and features, which, like painting, is beyond basic visual functions (related to eyes and the VI-V4 in brain (22 requiring a higher-level cognitive ability (more related to the angular gyrus in brain (3).
The attempts to teach machines text-to- -ixt-to-image generation can be traced to the early times of deep gen- erative models, when Mansimov et al. (35] added text information DRAW [20]. Then Generative Adversarial Nets [19] (GANs) began to dominate this task. Reed et al. 42 fed the text embeddings to both generator and discriminator as extra inputs. StackGAN [54] decomposed the generation into sketch-refinement process. AttnGAN [51 used attention on words to focus on the corresponding subregion. ObjectGAN [2] generated images following a text-bboes-llayouts >image process. DM-GAN (55] and DF-GAN 45 introduced new architectures, e.g. dyanmic memory or deep fusion block, for better image refinement. Although these GAN-based models can perform reasonable synthesis in simple and domain-speciffe dataset, e.g. Caltech-UCSD Birds 200 (CUB), the results on complex and domain- -enneralgeee scenes, e.g. MS COCO 31, are far from satisfactory.
Recent years have seen a rise of the auto-regressive generative models, Generative Pre- Pre-Training (GPT) models 37 leveraged Transformers 48 to learn language models in large-scale corpus, greatly promoting the performance of natural language generation and few-shot language understanding 33
Auto-regressive model is not nascent CV. PixelCNN, PixeIRNN 447 and Image Transformer 36
factorized the probability density function on an image over its sub-pixels (color channels in pixel) with different network backbones, showing promising results. However, a real image usually comprises millions of sub-pixels, indicating an unaffordable amount of computation for large models. Even the biggest pixel-level auto-regressive model, ImageGPT 77.. was pretrained on ImageNet at a max resolution of only 96 x 96.
The framework of Vector Quantized Variational AutoEncoders (VQ-VAE) [46] alleviates this problem. VQ-VAE trains an encoder to compress the image into a low-dimensional discrete latent space, and a decoder to recover the image from the hidden variable in the stage 1. Then in the stage 2, an auto-regressive model (such as PixeICNN 47) learns to fit the prior of hidden variables. This discrete compression loses less fidelity than direct downsampling, meanwhile maintains the spatial relevance of pixels. Therefore, VQ-VAD revitalized the auto-regressive models in CV [41] Following this framework, Esser et al. [53 used Transformer to fit the prior and further switches from L2 loss to GAN loss for the decoder training, greatly improving the performance of domain-specific unconditional generation.
The idea of View comes naturally: large-scale generative joint pretraining for both text and image (from VQ- VAE) tokens. We collect 30 million high-quality (Chinese) text-image pairs and pretrain a Transformer with 4 billion parameters. However, large-scale rext-to-image generative pretraining could be very unstable due to the heterogeneity of data. We systematically analyze the reasons and solved this problem by the proposed Precision Bottleneck Relaxation and Sandwich Layernorm. As a result, Cog View greatly advances the quality of text-to-image generation.
A recent work DALL-E (39 independently proposed the same idea, and was released earlier than Cog View. Compared with DALL-E, Cog View steps forward on the following four aspects:
CogView outperforms DALL-E and previous GAN-basel methods at large margin ac- cording to the Fréchet Inception Distance (FID) 25 on blurred MS COCO, and is the first open-source large text-to-image transformer.
Beyond zero-shot generation, we further investigate the potential of finetuning the pretrained CogView. CieView can be adapted for diverse downstream tasks, such style learn- ing (domain-specific text-to-image), super-resolution (image-to-image), image captioning (image-to- te-text), and even text-image reranking.
The finetuned Cog View enables self-reranking for post-selection, and gets rid of an additional CLIP model (8 in DALL-E. It also provides a new metric Caption Loss to measure the quality and accuracy for text-image generation at a finer granularity than FID and Inception Score (IS) [43.
2

---

We proposed PB-relaxation and Sandwich-LN to stabilize the training of large Transformers on complex datasets. These techniques are very simple and can eliminate overfoow in forwarding (characterized as NaN losses), and make View able to be trained with almost FP/6 (022) They can also be generalized the training other transformers.
2 Method
2.1 Theory
CogView optimizes the Evidence Lower BOund (ELBO) of joint likelihood of image and text. The following derivation will turn into a clear re-interpretation of VQ-VAE without text
In this section, will derive the theory of CogView from VAE
[26]:
Suppose the dataset (X, T = {æi, 1 consists of i.i.d. samples image variable x and its description text variable t. We assume the image can be generated by random process involving
a latent variable z: (1) t; is first generated from a prior 0). (2) 2; is then generated from the
conditional distribution p(z]t ft - 44; 0). (3) x; finally generated from p(x|z z;; v)) We will use shorthand form like p(t;) to refer to p(x = .x) in the following part.
Let g(Z)z,; Φ) be the variational distribution, which is the output of the encoder & of VAE. The log-likelihook and the evidence lower bound (ELBO) can be written as:
logp( T;O,0)
logp(1;;0)
logp(x;(4;30,))
2-
log p(:;;;
[- p( + KL(a(z/0; Φ) (1((2)45)))
(2)
NL. loss text
KL. between o and (lext conditional) prior
rcconstruction loss
The framework of VQ-VAV VAE differs with traditional VAE mainly in the KL term. Traditional VAE fixes the prior ptz]t;; 0), usually as ((0 I). and learns the encoder Φ. However, leads to posterior collapse (23) meaning that q ((] sometimes collapses towards the prior. VQ-VAE turns fix and fit the prior p(z) f,; 0) with another model parameterized by O. This technique eliminates posterior collapse, because the encoder is now only updated for the optimization of the reconstruction loss. In exchange, the approximated posterior g(z z4; Φ) could be very different for different 2;, so we need a very powerful model for p(z|t,; 0) minimize the KL term.
Currently, the most powerful generative model, Transformer (GPT), copes with sequences of tokens over a discrete codebook. To use it, make codebook and h x w is the number of dimensions of z. The sequences 2j can be either sampled from q(z) x;; Φ), directly 2; argmax, q(z [1;; Φ). We choose the latter for simplicity, that q [z; o)
{o...., .... [V] 1] J where [V] is the size of
becomes a one-point distribution on 24. The Equation (2) can be rewritten as:
N
E (
-- log
- log p(;;((t;)) log p(2) ty:
(3)
NLL Joss for text
NLL loss for
rcconstruction
The learning process is then divided into two stages: (1) The encoder & and decoder 1 learn to minimize the reconstruction loss. (2) A single GPT optimizes the two negative log-likelihool (NLL) losses by concatenating text t; and 24 input sequence.
As result, the first stage degenerates into pure discrete Auto- Auto-Encoder, serving as an image tokenizer to transform an image to a sequence of tokens; the GPT in the second stage undertakes most of the modeling task. Figure iillsstrrsess the framework of Cog
meeanigg that all computation, including forwarding and backwarding in FP16 without any conversion. but optimizer states and the master weights are FP32.
In this paper, bold font denotes a random variable, and regular font denotes a concrete value. See this comprehensive tutorial 12 for the basics of VAE.

---

2.2 Tokenization
In this section, we will introduce the details about the tokemizers in CogView and comparison about different training strategies about the image tokenizer (VQVAE stage 1).
Tokenization for text is already well-studied, e.g. BPE 16 and SentencePiece 28 In Cog View,
ran SentencePiece on large Chinese corpus to extract 50,000 text tokens.
The image tokenizer is discrete Auto-Encoder, which is similar to the stage 1 of VQ-VAE 46 or d-VAE [39]. More specifically, the Encoder & maps an image a of shape x W 3 into Enc, (x) of shape h x d, and then each d' dimennnsioaal vector is quantized to nearby embedding in learnable codebook {vo,... V Veu E Rd The quantized result can be represented by h x w
indices of embeddings, and then we get the latent variable z E {o..... [ 1) 1) The Decoder v maps the quantized vectors back (blurred) image to reconstruct the input. In our 4B-parameter CogView, |V| = 8192, d 256, H = W 256, h = w = 32.
The training of the image tokenizer is non-trivial due the existence of discrete selection. Here introduce four methods to train an image tokenizer.
The nearest-neighber mupping, straight-thruud estimator 2 which proposed by the original VQVAE. A common concemn of this method [39] is that, when the codebook is large and not initialized carefully, only few of embeddings will be used due to the curse of dimensionality. We did not observe this phenomenon in the experiments.
Gumbel sumpling, struight-througe estimater. If we follow the original VAE to reparam- eterize categorical distribution of latent variable z based on distance between vectors, -Iyg-tegg(t)gllla
i.e. p p(Zix++j
ux|2) D Isk (ag(e)yll unbiased sampling strategy is
2ixutt = argmax, 9s || Enc (z):glaa) 9h ~ Gumbel(O. 1), where the temperature 7 is gradually decreased to O. We can further use the differentiable softmax to approximate the one-hot distribution from argmax. DALL-E adopts this method with many other tricks to stabilize the training.
The nearest-neighhor mapping, moving average, where each embedding in the codebook is updated periodically during training the mean of the vectors recently mapped to it 46
The nearest-neighhor mapping. fixed codebook, where the codebook is fixed after initialized.
Comparison. To compare the methods, we train four image tokenizers with the same architecture on the same dataset and random seed, and demonstrate the loss curves in Figure [ We find that all the methods are basically evenly matched, meaning that the learning of the embeddings in the codebook is not very important, initialized properly. In pretraining, use the tokenizer of moving average method.
Toud
tackprin codddook
guaddic
0.00
0.44
The introduction of data and more details about tok- cnization are in Appendix A
0.23
3000
(180)
3300
20000
25000
2.3 Auto-regressive Transformer
Figure L2 loss curves during training image tokenizers. All the above methods finally converge to a similar loss level.
The backbone of View is a unidirectional Trans- former (GPT). The Transformer has 48 layers, with the hidden size of 2560, 40 attention heads and 4 billion parameters in total. As shown in Figure 3 four seperator tokens, [ROI1] (reference text of image), [BASE], [BOI1] (beginning of image), [E011] (end of image) are added to each sequence to indicate the boundaries of text and image. All the sequences are clipped or padded to a length of 1088.
The pretext task of pretraining is left-to-right token prediction, a.k.a. language modeling. Both image and text tokens are equally treated. DALL-E 39) suggests to lower the loss weight of text tokens; on the contrary, during small-scale experiments we surprisingly find the text modeling is the key for the success of text-to-image pretraining. the loss weight of text tokens is set to zero, the model will fail to find the connections between text and image and generate images totally unrelated to the input text.

---

Input Text:
Input Image:
(The head of lovely cat.)
Image Tokenizer
(Discrete AutoEncoder)
Encoder
Decoder
Discratize
Recovar
Tokenizer (sentence pieces)
A
Fattem
[RDI]]
Text Takan
Text Takan
[BASE]
[Snn]
Image Token
Image Taken
[EOI]]
Text tokens, ranging from 8192 58192.
1024 Image tokens, ranging from 8192.
Transformer (GPT)
Figure 3: The framework of View. [ROI1], [BASE1], etc., are seperator tokens.
We hypothesize that text modeling abstracts knowledge in hidden layers, which can be efficiently exploited during the later image modeling.
We train the model with batch size of 6, 144 sequences (6.7 million tokens per batch) for 144,000 steps on 512 V100 GPUs (32GB). The parameters
updated by Adam with max Ir = 3 x 10 81 0.9, B2 = 0.95, weight decay = 4 x 10-2, The learning rate warms up during the first 2% steps and decays with cosine annealing 34 With hyperparameters in appropriate range, we find that the training loss mainly depends on the total number of trained tokens (tokens per batch X steps), which means that doubling the batch size (and learning rate) results in very similar loss if the same number of tokens are trained. Thus, we use a relatively large batch size to improve the parallelism and reduce the percentage of time for communication. We also design three-region sparse attention to speed up training and save memory without hurting the performance, which is introduced in Appendix B
2.4 Stabilization of training
Currently, pretraining large models (>2B parameters) usually relies on 16-bit precision to save GPU memory and speed up the computation. Many frameworks, e.g. DeepSpeed ZeRO [40], even only support FP16 parameters. However, text-to-image pretraining is very unstable under 16-bit precision. Training 4B ordinary pre-LN Transformer will quickly result in NaN loss within 1,000 iterations. To stabilize the training is the most challenging part of Cog View, which is well-aligned with DALL-E.
We summarize the solution of DALL-E as to tolerate the numerical problem of training. Since the values and gradients vary dramatically scale in different layers, they propose a new mixed-precision framework per-resblock loss scaling and store all gains, biases, embeddings. and unembeddings in 32-bit precision, with 32-bit gradients. This solution complex, consuming extra time and memory and not supported by most current training frameworks.
Cog View instead regularizes the values. We find that there are two kinds of instability; overflow (characterized by NaN losses) and underflow (characterized by diverging loss). The following techniques are proposed solve them.
Precision Bottleneck Relaxation (PB-Relax). After analyzing the dynamics of training, we find that overflow always happens two bottleneck operations, the final LayerNorm attention.
In the deep layers, the values of the outputs could explode to be large 10* 103, making the variation in LayerNorm overflow. Luckily, as LayerNorm(z) = LayerNorm(z/ (ax00omm((mmm(())))))))) we can relax this bottleneck by dividing the maximum firs
The attention scores Q K/ Vd could be significantly larger than input elements, and result in overflow. Changing the computational order into Q² (K/vū) alleviates the problem. (oftmax(QTK/V6
To eliminate the overflow, we notice that softmax Q? K/ /d)
We cannot directly divide by large constant, which will lead to underflow in the early stage of training.
5

---

21+11
65536
Four-LV
Sandiwion L.V
32740
516384
6152
Lagen Murre
4056
2040
1024
NORTS
La Nor
A
Acoibe
Pre-UN (1-00.11
Layer Narr
Sandwich-Lk (l=6..3)
Nettoon
Layer horm
Attertie
Max
16
Pre-UN Orw0.00) 0.3
FB Rellx (1r00.01)
(8)
22
271
(b)
Iteration
Figure 4: (a) Illustration of different LayerNorm structures in Transformers. Post-LN is from the original paper, Pre-LN is the most popular structure currently; Sandwich-LN is our proposed structure to stabilize training. (b) The numerical scales in our toy experiments with 64 layers and a large learning rate. Trainings without Sandwich-LN overflow in main branch; trainings without PB-relax overflow in attention;
Only the training with both can continue.
constant), meaning that we can change the computation of attention into
T K
= softmax (190) avd
Q- (0-K- K max
Q K)) x xxa)
softmax
(4)
Vd
avd
where a is a big number, e.g. a = 32 In this way, the maximum (absolute value) of attention scores are also divided by a to prevent it from overflow. detailed analysis about the attention in CogView is in Appendix C
Sandwich LayerNorm (Sandwich-LLN), The LayerNorms
Transformers are essential for stable training. Pre-IN [50] is proven to converge faster and more stable than the original Post-LN, and becomes the default structure of Transformer layers in recent works. However, is not enough
for text-to-image pretraining. The output of LayerNorm (a-2)va 7 / is basically proportional (E
to the square root of the hidden size of which is vd = 2560 50 in CogView. If input values in some dimensions are obviously larger than the others - which is true for Transformers output values in these dimensions will also be large (10 102). In the residual branch, these large values are magnified and be added back to the main branch, which aggravates this phenomenon in the next layer, and finally causes the value explosion in the deep layers.
This reason behind value explosion inspires us to restrict the layer-by-layer aggravation. We propose Sandwich LayerNorm, which also adds LayerNorm at the end of each residual branch. Sandwich- LN ensures the scale of input values each layer within a reasonable range, and experiments on training 500M model shows that its influence on convergence is negligible. Figure 4 a) illustrates different LayerNorm structures in Transformers.
Toy Experiments. Figure 4b) shows the effectiveness of PB-relax and Sandwich-LN with toy experimental setting, since training many large models for verification not realistic. We find that deep transformers (64 layers, 1024 hidden size), large learning rates (0.1 or 0.01), small batch size (4) can simulate the value explosion in training with reasonable hyperparameters. PB-relax + stabilize the toy experiments.
Sandwich-LN
Shrink embedding gradient. Although we did not observe any sign of underflow after using Sandwich-LN, we find that the gradient of token embeddings is much larger than that of the other parameters, so that simply shrinking its scale by a = 0.1 increases the dynamic loss scale to further prevent underflow, which can be implemented by emb=emb+alpha+emt detach ()+ (1-alpha) Pytorch. seems to slow down the updating of token embeddings, but actually does not hurt performance in our experiments, which also corresponds recent work MoCo v3 [9].
Discussion. The PB-relax and Sandwich-LN successfully stabilize the training of Cog and a ---aarmeterr Cog View-large. They are also general for all Transformer pretraining, and will enable the training of very deep Transformers in the future. As an evidence, we used PB-relax successfully eliminating the overflow in training a 10B -ar-marreteer GLM [1]. However, in general,
'The must be least bise, because the values vary greatly in different heads.

---

the precision problems in language pretraining is not significant as in text-to-imen -imgg pretraining. We hypothesize that the root the heterogeneity of data, because we observed that text and image tokens are distinguished by scale in some hidden states. Another possible reason is hard-to-find underflow, guessed by DALL-E. A thorough investigation is left for future work.
Finetuning
Cog View steps further than DALL-E on finetuning, Especially, we can improve the text-to-image generation via finetuning Cog View for super-resolution and self-reranking. All the finetuning tasks can be completed within one day on single DGX-2.
3.1 Super-resolution
Since the image tokenizer compresses 256 x 256- 256-pixel images into 32 x 32-token sequences before training, the generated images are blurrier than real images due to the lossy compression. However, enlarging the sequence length will consume much more computation and memory due to the (nn²) complex of attention operations. Previous works [13 about super-resolution,
image restoration, usually deal with images already in high resolution, mapping the blurred local textures to clear ones. They cannot be applied low-resolution images. Figure [ (b) is an example of our finetuning method, and illustrates our desired behavior of super-resolution.
case, where we need to add meaningful details to the generated
The motivation of our finetuning solution for super-resolution is belief that CogView is trained on the most complex distribution in general domain, and the objects of different resolution has already been covered Therefore, finetuning View for super-resolution should not be hard.
Specifically, we first finetune VieW into conditional super-resolution model from 16 x 16 image tokens to 32 32 tokens. Then we magnify an image of x 32 tokens to 64 64 tokens (512 x 512 pixels) patch-by-pate via a center-continuous sliding-wwr biddnggwwnnow strategy in Figure [5] (a). This order performs better that the raster-scan order in preserving the completeness the central area.
To prepare data, we crop about million images to 256 X 256 regions and downsample them to 128 128. After tokenization, we get 32 x 32 and 16 16 sequence pairs for different resolution. The pattern of finetuning sequence is " [[ROI]] text tokens [BASE] [BOI1] 16 x 16 image tokens [EOI1] [ROI2] [BASE] [BOI2] 32 32 image tokens [EOI2]" ". longer than the max position embedding index 1087. As solution, we recount the position index from 0 at [ROI2]
(a) Center-continuous sliding window
(b) Different super-resolution results for "a tiger playing football".
Figure 5: (a) A 64 64-token image generated patch-by-patch in the numerical order. The overlapping positions will not be overwritten. The key idea is to make the tokens in the 2nd and 4th regions usually regions of faces or other important parts generated when attending the whole region. (b) The finctuned super-resolution model does not barely transform the textures, but generates new local structures, c.g. the open mouth tail in the example.
"An evidence support the belief that if append "close-up view" at the end of the text, the model will generate details a part of object.
'One might worry about that the reuse of position indices could cause confusions, but in practice, the model distinguish the two images well, probably based on whether they can attend [ROI2] in front.

---

3.2 Image Captioning and Self-reranking
To finetune Voew for image captioning is straightforward: exchanging the order of text and image tokens in the input sequences. Since the model has already learnt the corresponding relationships between text and images, reversing the generation is not hard. We did not evaluate the performance due to that (1) there no authoritative Chinese image captioning benchmark (2) image captioning is not the focus of this work. The main purpose of finetuning such a model is for self-reranking.
We propose the Caption Loss (CapLoss) to evaluate the correspondence between images and text. More specifically, CosL.s 1) [ Σ log p(4; (x, foa-1), where
sequence of text tokens and is the image. CapLoss( ar, t) is the cross-entropy loss for the text tokens, and this method can be seen as an adaptation of inverse prompting [56] for text-to-image generation. Finally, images with the lowest CapLosses are chosen.
Compared to additionally training another constrastive self-supervised model, e.g. CLIP 38 for reranking, our method consumes less computational resource because we only need finetuning. The results in Figure 9 shows the images selected by our methods performs better in FID than those selected by CLIP. o shows an example for reranking.
Figure 6: 60 generated images for "A man in red shirt is playing video games" (selected at random from COCO), displayed in the order of CapLoss Most bad cases are ranked in last places. The diversity also eases the concern that CogView View might be overfitting similar image in the training set.
3.3
Style Learning
Although Cog View is pretrained to cover diverse images as possible, the desire to generate images of a specific style or topic cannot be satisfied well. We finetune models on four styles: Chinese traditional drawing, oil painting, sketch, and cartoon. Images of these styles are automatically extracted from search engine pages including Google, Baidu and Bing, etc., with keyword as "An image of {style} style", where {style} is the name of style. We finetune the model for different styles separately, with 1,000 images each.
During finetuning, the corresponding text for the images are also "An image of {style} style". When generating, the text is "A {object} of (style) style", where {object] is the object to generate. In this way, View can transfer the knowledge of shape of the objects learned from pretraining the style of finetuning. Figure 7 shows examples for the styles
A
Stylo of sketch
Style of cartoon
Style cf Chinose pointing
Style of all painting
the Oriental Pearl
Figure 7: Generated images for "The Oriental Pearl" (a landmark of Shanghai) in different styles.
8

---

3.4 Industrial Fashion Design
When the generation targets at single domain, the complexity of the textures are largely reduced. In these scenarios, we can (1) train a VQGAN [T5] instead of VQVAE for the latent variable for more realistic textures, (2) decrease the number of pa- rameters and increase the length of sequences for higher resolution. Our three-region sparse atten- tion (Appendix B can speed up the generation of high-resolution images in this case.
Sim, open novel
Beggy Cropped partss
slove, women's kritting
We train -a-rammmtter model on about 10 million fashion- cattoon pairs, using 50 x0 VQGAN image tokens and decodes them into 800 x 800 pixels.
Figure g shows samples of Cog View fashion Figure 8: Generated images for fashion design.
design, which has been successfully deployed to Alibaba Rhino fashion production.
4 Experimental Results
4.1 Machine Evaluation
At present, the most authoritative machine evaluation metrics for gemeral-domain text-to-image generation is the FID on MS COCO, which not included in our training set. To compare with DALL-E, follow the from the dataset, after applying a Gaussian filter with varying radius to both the ground-truth and generated images The captions To fairly compare with DALL-E, we do not use super-resolution. Besides, DALL-E generates 512 images for each caption and selects the best one by CLIP, which needs to generate about 15 billion tokens. To save computational resource, we select the best one from 60 generated images according their CapLosses. The evaluation of CapLoss on subset of 5,000 images. We finally enhance the contrast of generated images by 1.5. Table shows the metrics for View and other methods.
setting, evaluating Cog View on subset of 30,000 captions sampled
translated into Chinese for Cog View by machine translation.
Table Metrics for machine evaluation. Statistics about DALL-E and GANs are extracted from their figures. FID-k means that all the images are blurred by Gaussian Filter with radius k.
Model
FID-0
FID-1
FID-2
FID-4
FID-8
IS
CapLoss
AttnGAN
35.2
44.0
72.0
108.0
100.0
23.3
3.01
DM-GAN
26.5
39.0
73.0
119.0
112.3
32.2
2.87
DF-GAN
26.5
33.8
55.9
91.0
97.0
18.7
3.09
DALL-E
27.5
28.0
45.5
83.5
85.0
17.9
CogView
27.1
19.4
13.9
19.4
23.6
18.2
2.43
Caption Loss Metric. FID and IS designed to measure the quality of unconditional generation from relatively simple distributions, usually single objects. However, text-to-image generation should be evaluated pair-by-pair. T that DM-GAN achieves the best unblurred FID and IS, but is ranked last in human preference (Figure 10 a)). Caption Loss is an absolute (instead of relative, like CLIP) score, so that it can be averaged across samples. It should be better metrics for this task and is more consistent with the overall scores of our human evaluation in 4.2 Comparing relf-reranking with CLIP. We
evaluate the FID-O and IS of Cog View-generate, images selected by CLIP and self self-reranking on MS COCO. Figure 9 ferent number of candidates. Self-reranke rerankernn gets better FID, and steadily refines FID as the num- ber of candidates increases. CLIP performs bet- ter in increasing IS, but as discussed above, it is not a suitable metric for this task.
shows the curves with dif-
Canilal an leattig
Figure 9; IS and FID-O for CLIP and self-ranking.
We use the same evaluation codes with DM-GAN and DALL-E, which is available at htttts////gttuu com/Minff hwnm/MannfogZZhu/DXo-ooo

---

Discussion about the differences in performance between CogView and DALL-E. Since DALL- is pretrained with more data and parameters than CogView, why Cog View gets better FID even without super-resolution? It is hard to know the accurate reason, because DALL-EE is not open-source, but we guess that the reasons include: (1) CogView View uses PB-relax and Sandwich-I.N for more stable optimization. (2) DALL-E uses many cartoon and rendered data, making the texture of generated images quite different from that of the photos in MS COCO. (3) Self-reranking selects images better in FID than CLIP. (4) CogView is trained longer (96B trained tokens in Cog View vs. 56B trained tokens in DALL-E).
4.2 Human Evaluation
Human evaluation is much more persuasive than machine evaluation on text-to-image generation. Our human evaluation consists of 2.950 groups of comparison between images generated by AttnGAN, DM-GAN, DF-GAN, CogView, and recovered ground truth, i.c., the ground truth blurred by our image tokenizer. Details and example-based comparison between models in Appendix [
Results Figure T0 show that CieView outperforms GAN-based baselines at a large margin. CogView is chosen as the best one with probability 37.02%, competitive with the performance of recovered ground truth (59.53%) Figure [T) also indicates our super-resolution model consistently improves the quality of images, especially the clarity, which even outperforms the recovered ground truth.
DM
Cogglie
Recoveeed Sound fiuh
and uneee firtt lower beind
Ground Thuth oupper bound)
1:2
Human Profenence. pardertage model
Bi Overall accons (1-10) for the models.
Boores (1: 5) for the mo on three important aspeets.
Figure 10: Human Evaluation results. The recovered ground truth is obtained by first encoding the ground truth image and then decoding it, which is theoretically the upper bound of Cog View.
Conclusion and Discussion
Limitations. A disadvantage of Cog View is the slow generation, which common for auto-regressive model, because each image is generated token-by-token The blurriness brought by VQVAE is also an important limitation. These problems will be solved in the future work.
Ethics Concerns.
Similar to Deepfake, Cog View is vulnerable to malicious use 44 because of its
controllable and strong capacity generate images. The possible methods mitigate this issue discussed in a survey (5). Moreover, there are usually fairness problems in generative models about human In Appendix D analyze the situation about faimess in Cog View and introduce simple "word replacing" method to solve this problem.
We systematically investigate the framework of combining VQVAE and Transformers for text-to- image generation. View demonstrates promising results for scalable cross-modal generative pretraining, and also reveals and solves the precision problems probably originating from data heterogeneity. We also introduce methods to finetune Cog View for diverse downstream tasks. We hope that Cog View could advance both research and application of controllable image generation and cross-modal knowledge understanding, but need to prevent it from being used to create images for misinformation.
https://thogradient: wwtpiiteetttunpppuuuueee lessons
10

---

Acknowledgments and Disclosure of Funding
We would like to thank Zhao Xue, Zhengxiao Du, Hanxiao Qu, Hanyu Zhao, Sha Yuan, Yukuo Cen, Xiao Liu, An Yang. Yiming Ju for their help in data, machine maintaining or discussion. We would also thank Zhilin Yang for presenting this work at the conference of BAAI
Funding in direct support of this work: a fund for GPUs donated by BAAI, a research fund from Alibaba Group, NSFC for Distinguished Young Scholar (61825602), NSFC (61836013).
References
[1] L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv: 1607.06450. 2016.
[2] Y. Bengio, N. Léonard, and A. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 2013.
[3] H. M. Bonnici, R. Richter, Y. Yazar, and J. S. Simons. Multimodal feature integration in the angular gyrus during episodic and semantic retrieval. Journal of Neuroscience, 36(20): 1462-5471,2 2016.
[4] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et Language models are few-shot learners. arXiv preprint arXiv:2055.14100 14165, 2020.
[5] M. Brundage, S. Avin, J. Clark, H. Toner, P. Eckersley, B. Garfinkel, A. Dafoe, P. Scharre Zeitzoff, B. Filar, et al. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv preprint arXiv:/802.007:
2018.
[6] A. Caliskan, J. Bryson, and A. Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 38666334):183-1188 2017.
[7] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703, PMLR, 2020.
[8] Chen, S. Kormblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607, PMLR, 2020.
[9] X. Chen, S. Xie, and K. He. An empirical study of training self-supervised visual transformers. arXiv preprint arXhv:2104.02057, 2021.
[10] K. Clark, U. Khandelwal, O. Levy, and C. D. Manning. What does bert look at? an analysis of 2019.
bert's attention. arXiv preprint arXiv:
[11] J. Deng, W. Dong, R. Socher, L.JJ. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database, In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icee, 2009.
[12] M. Ding. The road from MLE EM to VAE: brief tutorial. URL httph:/ttt///t researchgate. Miinooi-ii-ii---n-------
1ee MeliieeMMiiin MMng
from_MLE_ to
tor_tt____-__ EM_ VAE Brief http:raaiiinaa//ffffff21222..... Tro-Road-from-MLB- EM-E------------- to- VAE A-w-----rifff-- Ao
[13] C. Dong, C. C. Loy, K. He, and X. Tang. Learning deep convolutional network for image super-resolution. European conference on computer vision, pages 184-199. Springer, 2014.
[14] Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang. All nip tasks are generation tasks: general pretraining framework. arXiv preprint a1Xiv:2103.10360, 2021.
[15] Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. arXiv preprint arXiv:2012.0098 2020.

---

[16] P. Gage. A new algorithm for data compression. C Users Journal, 12(2):23-38, 1994.
[17] Gasquet. Cézanne. pages 159-186, 1926.
[18] X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statisticy, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
[19] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. arXiv preprint arXiv:
2014.
[20] K. Gregor, I. Danihelka, A. Graves, D. Rezende, and D. Wierstra. Draw: recurrent neural network for image generation. In International Conference on Machine Learning, pages 1462-1471. PMLR, 2015.
[21] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al. Bootstrap your own latent: A new approach to self- supervvpe learning. arXiv preprint arXiv:2006.07733. 2020.
[22] K. Grill-Spector and R. Malach. The human visual cortex. Annu. Rev. Neurosci. 27:649-677 677, 2004.
[23] He, D. Spokoyny, G. Neubig, and T. Borg-Kirkpatrick, Lagging inference networks and posterior collapse in variational autoencoders. In International Conference on Learning Repre- sentations, 2018.
[24] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[25] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference Neural Information Processing Systems, pages 6629-66400 2017.
[26] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 14, 2013.
[27] Koh, Baldridge, H. Lee, and Y. Yang. Text-to-image generation grounded by fine-grained user attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 237-246, 2021.
[28] T. Kudo and J. Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium, Nov. 2018. Association for Computational Linguistics. doi: 10.18653/v1/ D18-2012. URL httph:tt///// / http://wwwclllll... aclweb. wwtg:/aanhoology/1B--000
[29] W. Li, P. Zhang, L. Zhang, Q. Huang. X. He, S. Lyu, and J. Gao. Object-driven text-to-image synthesis via adversarial training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12174- 2019.
[30] Lin, R. Men, A. Yang, C. Zhou, M. Ding, Y. Zhang, P. Wang, Wang, L. Jiang, X. Jia, et al. 2021.
M6: A chinese multimodal pretrainer. arXiv preprint
[31] T.-Y. Lin, M. Maire, Belongie, J. Hays, P. Perona, D. Ramanan, Dollár, and C. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[32] X. Liu, F. Zhang, Z. Hou, Z. Wang, L.. Mian, J. Zhang and J. Tang. Self-supervised learning: Generative contrastive. arXiv preprint arXiv:2006.082/88 1(2), 2020.
[33] X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang. Gpt understands, too. arXiv preprint arXiv:2103.10338 2021.
12

---

[34] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint wrXiv:1608.0398. 2016.
[35] E. Mansimov, E. Parisotto, J. L. Ba, and R. Salakhutdinov. Generating images from captions with attention. ICLR, 2016.
[36] N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran. Image transformer. In International Conference on Machine Learning, pages 2018.
PMLR,
[37] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[38] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.0002. 2021.
[39] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092, 2021.
[40] J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505-3506, 2020.
[41] A. Razavi, A. v. d. Oord, and O. Vinyals. Generating diverse high-fidelity images with vq- va-vae-2. arXiv preprint arXiv: 1e..119000000 00446, 2019.
[42] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Generative adversarial text to image synthesis. In International Conference on Machine Learning, pages 1060-1069. PMLR, 2016.
[43] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 2234-2242, 2016.
[44] Sharma, N. Ding, Goodman, and R. Sorieut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, 2018
[45] M. Tao, H. Tang, S. Wu, N. Sebe, F. Wu, and X.-Y. Jing. Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprint arXiv:2008.0586. 2020.
[46] A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 6309-6316 18, 2017.
[47] A. Van Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning. pages 1747-1756. PMLR, 2016.
[48] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, arXiv preprint arXiv: vrXXiv1706.000 0707777067722. 2017.
[49] M. Westerlund. The emergence of deepfake technology: A review. Technology Innovation Management Review, 9(11), 2019.
[50] R. Xiong. Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T. Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pages 10524-10533, PMLR, 2020.
[51] T. Xu, P. Zhang, Q. Huang. H. Zhang, Z. Gan, X. Huang, and X. He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1314-1324, 2018.
13

---

[52] S. Yuan, H. Zhao, Z. Du, M. Ding, X. Liu, Y. Cen, X. Zou, and Z. Yang. Wudaocorpora: A super large-scale chinese corpora for pre-training language models. Preprint. 2021.
[53] M. Zaheer, G. Guruganesh, A. Dubey, Ainslie, C. Alberti, Ontamon, P. Pham, A. Ravula, Q Wang, L. Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007. 14062, 2020.
[54] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 5907-5915, 2017.
[55] M. Zhu, P. Pan, W. Chen, and Y. Yang. Dm-gan: Dynamic memory generative adversarial networks for text-lo-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5802-588 28005 2019.
[56) X. Zou, D. Yin, Q Zhong, H. Yang, Z. Yang, and J. Tang. Controllable generation from pre-trained language models via inverse prompting. arXiv preprint 103. 2021.
14


Reviews:
[{"review_id": "yk1mHwxmVJ-", "reviewer": "Reviewer_ccEj", "summary": "The paper proposes CogView - a cross-modal Transformer with VQ-VAE as the tokenizer. The model is primarily designed for text-to-image generation but also shows promising results on various downstream tasks. \nThe paper also provides many insights on the effective training tricks (including PB-Relax and Sandwich-LN) to stabilize the training of text-to-image transformers. The experimental results show that CogView outperforms the recent state-of-the-art approach DALL-E in the zero-shot settings. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1) The paper is generally well-written and easy to follow.\n2) The paper introduces several effective methods to stabilize the training for text-to-image generation.\n3) The proposed CogView can be further extended to more tasks via fine-tuning: style learning, super-resolution, self-reranking.\n4) The code and pre-trained model is open-source, which is the first for large text-to-image transformer, many applications will benefit from it.\n\nWeakness: \n1) The technical novelty of the paper is limited, its model is almost the same as DALL-E. The main contribution lies in the training strategy rather than in the techniques.\n2) Training data is one possible reason why CogView beats DALL-E. Although Appendix A provides some descriptions of data collection. However, there are still not enough details to replicate the high-quality text-image pairs from the current descriptions. \n3) It is hard to tell whether the improvement comes from the high-quality training data or the training tricks.\n4) Self-reranking after fine-tuning is just another CLIP, it is still post-selection and does not get rid of CLIP.\n\nQuestions to the authors:\n1) CogView used different data to train the VQ-VAE and text-to-image Transformer. DALL-E also released their own trained VQ-VAE, how will the results be like if using the same VQ-VAE and compare the generation performance between CogView and VQ-VAE?\n2) How do you evaluate the ratio of training data and training tricks in the success of CogView?\n3) In Table 1, how did you replicate the performance of DALL-E, why CapS didn’t apply to DALL-E? Or do you just copy the number of DALL-E on MSCOCO?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The proposed CogView approach may be used for malicious purposes to generate fake images. That's a potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "b1e19pQb3rC", "reviewer": "Reviewer_TthA", "summary": "- CogView is an text-to-image model similar to DALL-E, except it uses a traditional VQ-VAE instead of dVAE. \n- A Chinese (image, text) dataset is collected for supervised pre-training. (30M pairs)\n- A couple of techniques are described to stabilize FP16 training: PB-Relax (identity that is more stable for computing attention), Sandwich-LN (pre + post LayerNorm) .\n- They fine-tune the model for style-learning, super-resolution, and image captioning, and fashion design, and show samples but do not provide quantitative results.\n- CogView is compared (via FID/IS) to DALL-E on MS-COCO by translating captions to Chinese first.\n- Human evaluation is used to compare DM-GAN, DF-GAN, AttnGan, and Ground-truth. 37% of the time, CogView is listed as the best.\n- CaptionScore is proposed as an image-text metric, derived from the likelihood of text tokens assigned from CogView. This is also used to select the best txt2img candidate amon 60, unlike DALL-E which relies on a separate CLIP model.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(Please read summary) While there are some interesting empirical findings, the contribution over DALL-E is limited. The primary difference is dataset collection, which focused primarily on photographs. There is no reason provided for collecting the dataset in Chinese, while much of the dataset is originally in English, but translated. The details of the collection are insufficient for reproducibility and it does not seem like this is a dataset paper either. The main reason given for better FID/IS scores on MS-COCO is that pre-training was conducted mainly on photos, which is more similar to MS-COCO, so the result is not surprising.\n\nOne missing baseline from the human evaluation is the retrieval one, e.g. 'closest image in training set' from a text2image retrieval system. It is unclear how much of the generated images is memorized/regurgitated.\n\nI do like the idea of CaptionScore to rank generated images instead of a relying on another model. \n\nUpdate: Increased score in light of discussion of potentially useful tricks to stabilize training contributed by this work, which may be useful to the community. The appendix has an expansive analysis on attention in the model that motivates PB-relax. Not having to rely on another model like CLIP is also nice.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, there is no discussion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "YJC5rfRUwe", "reviewer": "Reviewer_barm", "summary": "This paper proposes a generative, two-stage text-image model that combines a VQVAE (or VQGAN) and a text-conditional autoregressive transformer model for text-to-image synthesis. Here, the VQ-model is trained on image data only (first stage), while in the second stage the transformer is trained to model the distribution of learned discrete image representations (=\"image tokens\") of the (now fixed) first-stage model, given a corresponding textual description. The generated image tokens can then be decoded into the image space via the pre-trained decoder. Additionally, the fine-tuning of the model for specific tasks such as style learning, super-resolution, and image-to-text is explored. The work also provides insights into stabilizing the training of large transformers for text-image synthesis. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Strengths:__ \nThe paper provides (very) good qualitative results. The results depicted in Fig. 1 and Fig. 15 are impressive.\nFurther, interesting insights on training of the VQVAE are presented. For example, apparently a fixed codebook is not much different from a learned one and codebook collapse does not seem to be an issue. (l.119 and l.128-131).\n\nMoreover, the paper gives useful insights on how to stabilize the transformer training (OBS and Sandwich Layer Norm). Additionally, it is interesting that weighing the text loss and the image-token loss equally seems to perform best.\nExcluding the dataset-collection, the proposed approach (or rather the model's architecture)  is reproducible from the description provided in the paper.\n\n__Weaknesses:__\n\n- Empirical justifications of the above statements about VQVAE and Transformer trainining is missing. The proposed methods are simply described as \"working well\". The same holds for the statements about the weighing of the text loss and the \"center-continuous sliding window\". The latter needs to be compared to ordering that do not allow for inconsistencies between tokens, i.e token 4 & 7 and 5 & 6 in Fig. 5a (counting from 1 to 16, raster-scan order).\n- Evaluation is only done for a fixed re-ranking factor of 60. How does the model perform when re-ranking with less images (similar to Fig. 9 in [1])? How does the model perform when re-ranking is performed with CLIP?\n- Very few qualitative examples given for the finetuning, especially \"Style Learning\".\n- The writing should be improved. I would recommend the authors tone down some of their claims (e.g. l. 50: \"Our approach is neat and natural\", or in the title \"mastering\") and additionally discuss limitations (e.g. blurriness, sampling speed) of their approach (more) detailed.\n\n__Mixed Comments & Questions:__\n- What is the inference time (with and without the superres. model)? How does this compare to other approaches?\n- l. 287-288: What does \"Samples from CogView itself might be overscored, but it is all right to evaluate other models in the future\" mean?\n- l. 277-l.281 speculate about the reasons why CogView performs better than DALL-E on blurred MS-COCO. Comparing the first stage models (i.e. the discrete autoencoders) via reconstruction metrics could serve as an objective empirical justification for these claims.\n- Section 3.1: Only a single example for style learning is given. Furthermore, only a single realization for each text prompt is depicted. How large is the variance given a single prompt? These question could be easily answered by plotting multiple realizations for each prompt.\n- Section 3.3: Could the model be finetuned by reversing the order (i.e. predict text tokens from image tokens from right to left) in the autoregressive NLL training? This would correspond to the proposed \"swapping\" of text and image tokens, with additional swapping of the (pretrained) positional embeddings.\n- Section 3.4: The structure of the paper suggests that this section still belongs to the \"Finetuning\" chapter, but, at least in my understanding, introduces (i) a new dataset and (ii) another first-stage model (VQGAN), and thus trains a new model from scratch. This should be clarified. To stay with this theme: Why is the VQGAN not used for the \"big\" data set? Since the resolution is very high with 50x50 tokens, hardly any compression artifacts should be visible according to [2]. Are there other ways to fix the bluriness of the generated samples? How are the massively increased compute requirements (i.e. caused by switching from 32x32 tokens to 50x50 tokens) handled? \n- Why the name \"CogView\"? I could not find an explanation in the paper.\n- How does the model perform in a \"zero-shot\" setting on other classic text-to-image datasets such as CUB [3]? Can the model be fine-tuned on these datasets?\n- What is the stopping criterion when training the transformer? Minimal NLL on test data? Something else?\n- l.106: which \"chinese corpus\"? \n- Which model is used in Fig. 11?\n\n_Minor_:\n\n- l. 54: missing word, maybe \"our\" or \"the\"?\n- l. 94: $z_i$s -> $z_i$\n- l. 149: consine -> cosine\n- l. 210: is -> are\n- l. 523: alleviates -> alleviate\n\n__Summary/Justification of Rating:__\n\nIn summary, I have somewhat mixed feelings about the paper. On the one hand, it (i) shows that two-stage training with VQVAE/VQGAN and Transformers can produce highly complex scenes from text conditioning, and provides convincing results (also demonstrated through a user study), and (ii) provides interesting insights into training the VQVAE and Transformer models (see above). On the other hand, many of the proposed modeling decisions feel somewhat \"ad-hoc\" without being justified by ablation experiments (see _weaknesses_ and _comments_ above). There are also no results supporting the claims about non-present codebook collapse (l.119) or \"fixed/learnable embeddings\" (l.128-131), which I think are of great interest to the community. In addition, the writing of the paper needs improvement and the limitations of the work need to be discussed. I am currently rating this paper as \"just above threshold\" but am willing to increase my score if the authors provide satisfactory answers.\n\n__References:__\n - [1]: Ramesh, Aditya, et al. \"Zero-shot text-to-image generation.\" arXiv preprint arXiv:2102.12092 (2021).\n - [2]: Esser, Patrick, Robin Rombach, and Bjorn Ommer. \"Taming transformers for high-resolution image synthesis.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n - [3]: Welinder, Peter, et al. \"Caltech-UCSD birds 200.\" (2010).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The conclusion mentions that the proposed method could potentially be used to spread misinformation, which is a valid concern. Possible biases of the model or the collected dataset are not discussed. Limitations are rarely discussed, see also my above concerns about the writing.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10.5"}, "extra_text": {}}, {"review_id": "R-CBI78R-_n", "reviewer": "Reviewer_RfQ5", "summary": "The paper proposes a new text-image pretrained model based on Transformer with VQVAE tokeniser, which can be implemented on different downstream tasks to achieve an appropriate performance. Also, two techniques called PB-relaxation and Sandwich-LN are proposed to reduce overflow in forwarding and to stabilise the training.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The proposed method is built on GPT with VQVAE tokeniser.\n\nQuality: The submission is technically sound, claims are well supported, but authors do not discussed strengths and weaknesses of the proposed method.\n\nClarity: The paper is well organised and easily followed.\n\nSignificance: The results look good, which achieves good performance on different downstream tasks.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitation:\n1. Are authors going to make the collected dataset public? To some extent, the good performance might be benefited from the large-size training dataset. \n\n2. The proposed method is similar as Dall-E as both rely on the GPT and VQVAE. It might be better if authors can show more discussion and comparison between proposed method and the Dall-E. \n\n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a text-image generation model using VQ-VAE to discretize images, which is then followed by a LM on text + image tokens. There was a debate whether this work is concurrent to or a follow-up of DALL-E (blog post on Jan 5, 2021 and arXiv on Feb 24, 2021, more than two months before NeuRIPS deadline) given similarities among the two approaches. When viewed as a follow-up work, I found that there is a reasonable amount of contributions (including technical ones) in this paper, e.g., the finetuning ideas for super-resolution (which greatly enhances the quality and is actually a neat idea) and self-ranking (without having to train a full CLIP model). The detailed findings of using nearest neighbors in VQ-VAE, setting equal weight to the LM loss for texts, etc., together with tricks for stabilizing training seem informative and valuable to the community. In addition, the authors promised to open-source this work. Because of these reasons, I recommend Accept.
Decision: Accept (Poster)