Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents ImageBART, a generative model of images that uses multinomial diffusion to model the distribution of discrete latent codes extracted from a VQ-GAN encoder. Unlike prior diffusion based models, the reverse diffusion processes are modelled autoregressive, which enables a large reduction in the number of diffusion steps required. The model can condition on contexts such as a caption or semantic segmentation map, and unlike typical autoregressive models, is capable of image completion for arbitrary masked regions.
main_review: ImageBART is a fusion of the VQ-GAN introduced in Taming Transformers (TT, Esser et al, 2020), and the multinomial diffusion model introduced by Hoogeboom et al (2021). It aims to bring together the best of (increasingly effective) diffusion based generative modelling, and lossy discrete representation learning (VQ-VAE, VQ-GAN). ImageBART supports image completion with arbitrary masks unlike the autoregressive prior models used in TT. 

Taming transformers is the most relevant baseline, given that it is similarly a likelihood based model trained in the same discrete latent space. I would argue that the comparison in Table 1 isn’t particularly valuable, given that the proposed models have a large number of parameters( >2B parameters for LSUN). Given neural scaling laws for autoregressive models (Kaplan et al, 2020), we know that performance and sampling quality is heavily influenced by model size and computational resources. These results may reflect these factors rather than the expressiveness / inductive biases of the underlying generative models. A more valuable experiment would compare ImageBART to TT using similar parameter counts / a fixed computational budget.

Also why not report ImageNet FID in the main text and compare it to other models (given that it is reported in Figure 10)? 

It would be of interest to understand how the number of diffusion steps impacts model performance, beyond adding more parameters. One of the claims of the paper is that the hierarchical factorization of the data distribution is a helpful choice, but this isn’t experimentally verified other than through comparison to alternative works, where experimental conditions are different.

I find it strange to characterize the L1 loss as a likelihood, given that the autoencoder is trained adversarially, and the unnormalized density in equation (3) is likely to be degenerate. I think it’s fine to say: we’re doing lossy compression with a VQ-GAN, and learning a distribution of the compressed data.

[l257-259] “While our approach also models each transition autoregressively from the top-left to the bottom-right, the ability to attend to global context from the previous scale enables consistent completions of also the upper half” - I couldn’t figure out if the model can “see” context at the current scale when predicting masked pixels via the masked diffusion process, or if the visible context belongs to more coarse scales. If the former then that’s great, and if not a minor limitation is potentially not being able to match fine-grained detail in the masked / non-masked regions.

Conditional image generation results are qualitatively good but unsurprising: We have seen in a range of works that successful unconditional generative models can typically be conditioned on {image, text, other} context. The combination of masked-inpainting + caption-conditioning is a nice application.

Overall, I think this paper takes a natural step in combining two promising areas: diffusion-based generative models, and discrete representation learning. The proposed model offers benefits relative to pixel-level diffusion models or code-level AR models. The qualitative results are convincing, but due to confounding factors (e.g model sizes), it is hard to compare with baselines in an apples-to-apples way. The paper would benefit from some additional analysis of the benefits of using more / less diffusion steps.

limitations_and_societal_impact: The authors provide an extensive discussion of the limitations and potential negative impacts of the work in the supplementary material.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes ImageBART, an improved approach built on a recent success of an adversarial quantized representation learning model (e.g. VQGAN with autoregressive transformer) by replacing the single-scale autoregressive model with a multinomial diffusion process with a diffusion step-wise autoregressive encoder-decoder architecture. The (diffusion step-wise) encoder combined with a global cross attention mechanism from the adjacent diffusion step enables modeling of bidirectional context, which is not considered from the previous work (Taming transformers, TT).
main_review: Overall, the technical novelty feels somewhat incremental, as the proposed methods and the high visual fidelity results may mostly have stemmed from the high-performance compressed codebook with VQGAN. Otherwise, the proposed approaches (diffusion step-wise autoregressive models) would not have been feasible when operating on the pixel-level data space, rather than the compressed feature space. I'm aware that this is not a bad thing per se, so my assessment of this work is based on how well the writing and experiments are executed to demonstrate the clear advantage of the bidirectional context modeling compared to the original TT. 

The overall writing, motivation and the justification of the methodological choices of each components are presented very well with enough materials for any potential audience on the generative models to catch up. Perhaps emphasizing that the autoregressive diffusion becomes feasible by operating on the compressed space to the "reverse diffusion models" section (in page 5) would make the description more complete & remove potential doubts.

The figures in table 1 convincingly demonstrated the potential failure modes of the current pixel-space diffusion models compared to ImageBART that leverages the powerful compression model. What feels to be desired, however, is more direct comparison of ImageBART against TT itself (to the Table 1 (right) and Figure 2, for example), because placing these comparisons for experiments other than the upper-half image completion (Figure 5) would make the improvements of ImageBART over TT more convincing & remove potential doubts on whether the higher quality samples came from the VQGAN itself or the additional multinomial diffusion of ImageBART. FIDs are already better than TT, and the "hierarchical generative model with bidirectional context" of ImageBART is expected to have more capability than just a "single-scale unidirectional autoregressive transformer" of TT. It would be nice if the authors could add more direct comparisons vs. TT to other qualitative results.

Similar impression goes to text-conditional image synthesis & local editing. While the advantage of the bidirectional context from is clear from Figure 5, the reader might ask whether the text-conditional generation was previously impossible with TT. I expect that the previous model can technically perform the same task, but the quality is expected to be worse, especially more so on the local image editing task, where the TT may suffer from degradation if the most upper-left parts are removed for the editing query. With the same logic, the difference between TT and ImageBART might be smaller for the text-conditional synthesis. If the authors have additional bandwidth, I would like to see whether this will be a correct implication of ImageBART on the demonstrated tasks. I understand that training additional transformer models require (huge) resources, but given the scale of the current experimental results, I expect that the authors may able to provide the material potentially in the revision.

Minor typos & questions

The inference speed of ImageBART is ~2x than TT. I expect that this may be from using fewer decoder layers (per step), but describing the main factor for the faster inference will be beneficial as one may naturally be curious about the cause, because ImageBART theoretically have more computational complexity (O(data_dim * num_difusion_step) compared to just O(data_dim) for TT).

Experiments 4.3 points the non-existent Fig. 4.3 (expected to be 5)?

The line spacing between 101 to 103 feels bit crammed
limitations_and_societal_impact: The limitations of this work is handled adequately in the supplementary material (appendix), focusing on 1. an energy requirement for training large-scale generative model, thereby advocating the use of the pre-trained high-performance compression model for downstream tasks , 2. the challenges of fitting the proposed models to small dataset due to a reduced inductive bias from the transformer architecture. I find that the arguments are valid and handled mostly well. However, perhaps it would be better to add an actual GPU runtime to train (or fine-tune) ImageBART for the main experiments to help the reader understand the scale of the computing requirements better.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses image generation problem and proposes a combination of 3 existing methods: VQ-VAE/GAN, multinomial diffusion and BART. The system works as follows: the images are first tokenized with VQ-VAE/GAN, then 3-5 steps of multinomial diffusion are run on discrete tokens, then those steps are inverted with BART at each step (starting from random noise). Quantitatively, paper evaluates FIDs on the LSUN-{Churches,Beds,Cats} and on the FFHQ datasets, choosing 6 methods as baselines (VDVAE, DDPM, StyleGAN2, BigGAN, DCT, TT). Qualitatively, same datasets + additional ImageNet results are presented. Also, the paper contains qualitative results for inpainting and/or text conditioning. A few ablations are presented: joint/independent training of diffusion steps, number of decoder/encoder layers.
main_review: The scores below are consistent with "Rating" legend.

## Originality = 6/10
The proposed method is new, although a combination of 3 well-known techniques: VQ-VAE/GAN, multinomial diffusion and BART. The main conceptual difference to prior work is usage of multinomial diffusion in VQ-space, BART seems like a particular implementation choice. Overall, the related work is reasonably cited, except for DALL-E paper - which is mentioned in results section but omitted from intro/related work. It might be desirable to add a concurrent work section and mention "Diffusion Models Beat GANs on Image Synthesis" paper (https://arxiv.org/abs/2105.05233) to give more context to the reader - but this is optional.

## Quality = 5/10
The submission is technically sound, however quantitative evaluation is quite limited:
1. While ImageNet samples are shown, FID is not provided.
2. An important ablation is missing: FID wrt number of diffusion steps, does it help to have more than 1? AR models are well-known to ignore learned latents, so what if just the last BART-diffusion step brings value?

## Clarity = 6/10
The submission is reasonably well-written. A few comments though:
1. [abstract, L4-6] "Not only is this unidirectional, sequential bias of attention unnatural for images as it disregards large parts of a scene until synthesis is almost complete." - This critique of AR models sounds strange to me. AR uses correct probability factorization, what is unnatural about it? Conversely, what should be considered natural? Also, what is meant by disregarding?
2. [abstract, L8-10] "we incorporate a coarse-to-fine hierarchy of context by combining the autoregressive formulation with a multinomial diffusion process" - Coarse-to-fine is mentioned multiple times in the paper but it's a somewhat questionable use of terminology. When I hear coarse-to-fine (and hierarchy in addition), I expect multiple spatial resolutions, but here it's always modeled at the same vq resolution, right?
3. [abstract, L14-15] "Experiments demonstrate the gain over current autoregressive models, continuous diffusion probabilistic models, and latent variable models" - please be more specific in what sense there is a gain: FID, speed etc.
4. [L33-34] "Imagine that you only have the lower half of an image and are looking for a completion of the upper half then these models fail at this minor variation of the completion task." - a natural answer would be to learn right-to-left AR. And they don't fail.
5. [L82] "Generative Models on Improved Representations" - please mention DALL-E. It's unpublished, but you do mention it in experimental section, so worth adding to related work.
6. [L141-142] "Note that the optimization of Theta via this objective includes the parameters of the learned codebook in addition to the parameters of the encoder and decoder" - I remember in the original VQ-VAE paper (https://arxiv.org/pdf/1711.00937.pdf) there was a problem with directly learning codebook because straight-through doesn't allow codebook updates, so it was instead learned with an additional VQ objective. But here you say codebook is learned with reconstruction objective. How do you circumvent the problem above? Perhaps to make the paper self-contained, it's worth to write out L_rec term in detail.
7. [Table 1] It would be great to mention number of parameters for all compared methods.
8. [Figure 2] Top row?
9. [L350-353] Duplicated citation.

## Significance = 5/10
Without addressing questions in "Quality", it's hard to judge significance:
1. ImageNet quantitative results are generally considered most trust-worthy, so comparing to representative methods there would boost confidence in this method.
2. How much this paper brings in terms of methods is conditioned on how much profit multiple steps of diffusion bring on top of AR applied to VQ.
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposed a coarse-to-fine image synthesis method which aims to incorporate global context into autoregressive models by a diffusion process. The authors divided the whole task into two subtasks: 1) discrete representation learning task 2) learning a Markov chain to reverse a fixed multinomial diffusion process. Experiment on unconditional image synthesis, class/text-conditional image synthesis and image synthesis with mask were shown.
main_review: ## Originality

This work combines discrete image representation using VQGAN[1] with a multinomial diffusion process to introduce global context. The proposed solution of connecting those two ideas is intuitive and reasonable.

## Quality

I don't work on this very specific area, based on my understanding the paper is of good quality, but I just have the following the concerns and might change my rating if more clarifications are provided:
1. How is the bidirectional context incorporated in your method? The multinomial diffusion process is randomly introducing context switching, but there is no guarantee that which type of switching is being generated, please comment.
2. It seems that most of the experiment uses a diffusion process of T=3~6. How could you make sure that enough global context is being considered here? Also, does larger T always give better results? How do you choose T for a given dataset? I think an analysis on this would be helpful to justify the choice of T and show more insights.
3. For 4.3, I wonder how it would work for for TT [1] and your method if the input image is flopped horizontally and vertically (making the masked out region from the upper half to the lower half)? Would that produce different outputs? 

## Clarity

1. The paper is overall well written with proper breakdown of each components of the method.
2. Line 36: Could you clarify why bidirectional context there does not provide valid factorization of the generative model density function?
3. The claimed contribution seems to mixing the concepts of bidirectional context, global context, hierarchical context all together. Therefore, making the relationships among them unclear. Please comment.

## Significance

Addressing unidirectional bias of AR models is an important problem. The proposed solution produces descent results while improving the sampling speed compared to previous SOTA methods. 

[1] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. *CoRR*, abs/2012.09841, 2020.
limitations_and_societal_impact: The authors provided a discussion on the energy impact and trade-off on sequence length and output quality. As indicated in the main review, the second part might need some experimental results and justifications.
review_title: null

Meta-review: This paper presents several improvements to Taming Transformers through the use of a diffusion instead of autoregressive prior that enable more flexible conditional image generation and improved unconditional image generation and speed. Unlike prior work on diffusion that uses independent distributions for each layer, here autoregressive models for the reverse diffusion process are used which enables far fewer steps but results in slower sampling. The effectiveness of the approach is demonstrated in several conditional image generation tasks (e.g. arbitrary order inpainting, text-to-image generation, panorama generation). Reviewers were somewhat concerned of the novelty over Taming Transformers and the performance of the method on ImageNet. The authors addressed concerns with ImageNet results in the rebuttal, and provided more extensive comparison to TamingTransformers highlighting the utility of the diffusion process with autoregressive conditoinal distributions over the larger one step autoregressive model in that prior work. I'd argue for accepting this paper due to the interesting experimental results and applications, as well as the novelty in the diffusion generative model space of utilizing autoregressive decoders and showing it allows for far fewer steps.
Decision: Accept (Poster)