Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper focused on the problem of improving computational efficiencies of auto-regressive (AR) models on high-resolution images. The authors proposed a hierarchical-quantized VAE which models the images at different scales and a Transformer model for the encoded sequence at each scale. Evaluations on class-conditional and text-conditional tasks between the proposed method and the baselines were shown.
strengths_and_weaknesses: ## Strengths

1. The paper is generally nicely written and well-organized. In addition, the paper contains a good coverage of the related background and training details (architecture selection, hyper-parameter settings).
2. The paper included thorough ablation studies to validate the design decisions.

## Weaknesses

1. The proposed methodology seems incremental: representing images using VQ-VAE in a coarse-to-fine manner was previously explored (VQ-VAE2[28], HAM[10], Hierarchical VQ-VAE[22], RQ-VAE[18]). In particular, RQ-VAE[18] also disentangles the information among different code levels (depths) without specialized modules. In addition, [18] proposed RQ-Transformer which also models codes at different levels in an AR fashion. In terms of computation complexity, RQ-Transformer[18] has the same dominating factor as the proposed method as well.
2. There are some concerns regarding the generalizability of the proposed method:
    1. The empirical validation is insufficient for disentanglement of the latent code as only 2 scales were explored. It is also not clear how the proposed approach could be extended to more than 2 scales. Since the claimed contribution involves the disentanglement of information into different codes, having a general principle that could be extended to an arbitrary number of scales is important and more empirical evidence should be present.
    2. It is not clear whether the conditional independence assumption would still hold when $r>2$ which would be important as the target image resolution increases (e.g. generating images with 512x512 resolution and above).
3. The formulation in equation 6 rewrites the joint probability into the product of two terms. During inference, the proposed method maximizes the second term first (by drawing the best $t_i$) and then maximizes the first term based on the chosen $t_i$. This is a sub-optimal strategy as maximizing the second term and then maximizing the first term does not equal maximizing the joint probability. Since the paper presents this as an inference strategy, the corresponding drawbacks/tradeoffs of this approach should be analyzed and discussed.
4. There are some important details missing:
    1. How does the HQ-Transformer generate diverse samples when the top code is fixed? Does it sample $\bf{b}_0$ only or does it apply stochastic sampling for all $\bf{b}_n$ like RQ-Transformer[18]?
    2. There are some missing throughput results in table 1.
5. Some minor issues
    1. In equation 11: there is a duplicate $u_1$, it should be $u_0$ instead?
    2. What were the batch sizes used to time the throughput in Tables 1 & 2 for different models?
questions: It would be helpful if the authors could address the issues and concerns listed above.
limitations: The authors provide some discussions on the limitation regarding the independence assumption. More discussions on how the proposed method would behave when the target resolution increases would be helpful. 

The authors also provided discussion on potential negative social impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a hierarchical autoregressive transformer model for image generation. Based on a quantizing autoencoder (VQGAN/VQ-VAE) the paper splits the quantized representation into two groups called top and bottom codes. The top codes have a smaller spatial resolution **r** (e.g., 8x8) and encode high-level, abstract information. The bottom codes have a larger spatial resolution **2r** (e.g., 16x16) and encode more detailed texture/high-frequency information. Each bottom code is considered to be only dependent on one top code, i.e., each top code is "responsible" for 4 bottom codes. During generation, the top codes are sampled autoregressively, however, the bottom codes can be sampled in parallel (conditioned on the respective top code) which significantly boosts the sampling speed. Results on class- and text-conditional image generation show similar or improved results compared to recent SOTA baselines while having a significantly higher sampling throughput than other autoregressive models.
strengths_and_weaknesses: The paper tackles an important problem of autoregressive image generation models, i.e., their sampling speed. The proposed approach makes the assumption that the high-level image content can be modeled via a relatively small spatial resolution while detailed texture information can be modeled locally without relying on information from far away (in pixel space). This hierarchical approach allows faster sampling speeds due to parallel generation of the texture codes while the high-level image codes (which still have to be sampled autoregressively) can be represented in a smaller spatial dimension.

Overall the paper is well written and motivated. I believe the methodology section could be slightly improved to make it easier to understand. Maybe Fig 2 can be slightly extended to show more details of the transformer. I believe the encoder/decoder structure is quite clear and easier to understand from the textual description.


questions: I would also be interested in the model's performance on unconditional generation or other conditioning, e.g., label map to image or similar.

Also, do you think the performance or resolution could be further improved by adding even more hierarchical layers of quantized codes?
limitations: One of the main limitations still remains as the top codes still have to be sampled autoregressively. Other approaches, e.g. MaskGIT, sample all codes in parallel and then iteratively resample code entries with low probability which overall also reduces the numbe of model evaluation steps. I wonder if approaches like that could be combined with the current approach?

Also, according to table 3 the size of the latent representation does affect the reconstruction capability so it is not clear how well this scales to larger resolutions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, a hierarchical method for synthesizing natural images based on discrete representations is presented. More precisely, a combination of the previous two-stage hierarchical VQVAE-2 [1] and residual vector quantization (RQ-Transformer [2]) is first used to learn a hierarchical two-stage discrete latent space (called HQ-VAE). In a second phase, as usual, a generative prior is learned over the latent space. For this purpose, the "HQ-Transformer" is introduced, which, given a top code, predicts all corresponding local bottom codes in parallel. Experiments show good results in class-conditional modeling of ImageNet and in text-image synthesis of conceptual captions with improved sampling speed over related two-stage discrete methods.
strengths_and_weaknesses: **Strengths**

- The proposed approach is simple (which is good!).
- In my opinion, the parallel local decoding is interesting and quite promising for further upscaling of the model.
Side question: for the parallel decoding scheme, is it possible to make use of a resampling strategy that uses the most confident codes only akin to [3]?
- The study on avg.-pooling vs channel-resampling is nice and the paragraph on using learnable down/upsampling layers here should be moved to the main paper.

**Weaknesses**

- The main drawback of the paper is it somewhat incremental nature over RQ-Transformer[2] and VQVAE-2 [1]. The main novelty seems to be the parallel decoding scheme,  while the increase in sample quality is limited and still contains lots of perceptual artifacts. I am curious about the other reviewers' opinion on this.
- While the comparisons to other "off-the-shelf" models in Tab.1-3 is interesting, I think the more interesting comparison would be to compare to a VQVAE-2 style hierarchical code (which can be trained with VQGAN reconstruction losses) and the strictly hierarchical autoregressive bottom -> top sampling scheme proposed there **in the same setting** (16x16 + 8x8, 8192+8182). In a related manner, what are the autoregressive decoding parameters used for the comparison in Table 5? 
- For text-to-image: My understanding is that the model is directly conditioned on text embeddings obtained from BPE. Does the model further improve by deploying a fixed or learnable (potentially pretrained) text encoder here?

_Minor:_
- some of the samples in Fig.7 show lots of artifacts which are often associated with an overly reliance on perceptual losses such as LPIPS. Does weighing this term down in the reconstruction loss improve upon this? Related: Does the proposed HQ-VAE structure make training the discriminator more difficult?
- Is it possible to add more levels to the model (e.g. training a top-middle-bottom model)? Can this be used to train models with even higher receptive field in the bottom codes (e.g., for further improving sampling speed)?
- l. 196: And.we --> And we 
- l. 199: casual self-attention --> causal self-attention

_Summary:_

This paper presents a simple and straightforward extension of RQ-Transformer and VQVAE-2 to learn a hierarchical two-stage discrete generative model of images. I like the general setup, the parallel (local) decoding strategy is good and a reasonable assumption, and although it is a straightforward combination of existing approaches, I think this approach has some potential to produce really good samples at a decent sampling speed. Further questions: How well does the model scale to even higher resolutions? Training the model on 32x32 and 16x16 latent codes would be interesting. In my opinion, the main drawback of the paper are both its limited increase in quality of the samples and somewhat limited novelty. This is why I currently view the paper as a borderline submission, but I am happy to be convinced otherwise by the authors and/or other reviewers.


_Literature_:
- [1]: Generating Diverse High-Fidelity Images with VQ-VAE-2, Razavi et al
- [2]: Autoregressive Image Generation using Residual Quantization, Lee et al
- [3]: MaskGIT: Masked Generative Image Transformer, Chang et al
questions: Please see the above section for questions. 
limitations: Yes, limitations and societal implications are discussed, albeit very briefly.
review_title: null

Meta-review: metareview: The work proposes hierarchical generation as an approach to mitigate the increasing costs of modelling each pixel autoregressively as images increase to higher resolutions. The paper is easy to read, has strong empirical results, and has impressive ablation studies to understand the idea.

I think the paper could most be improved by clarifying its relationship to prior work, as well as being more specific about generalizability.

1. There are already quite a few works proposing hierarchical generation based on vector quantization such as VQVAE-2, RQ-Transformer, and RQ-VAE. The authors should include any discussion of differences in the paper. In the rebuttal, the authors argue the major difference is parallel decoding based on conditionally independent generation of bottom features given top. These approaches are conceptually so similar that they're worth comparing to in the experiments in a controlled setup, and not simply comparing paper numbers which use completely different setups like code size and number of parameters.

2. I share Reviewer CKx9's concern about the generalizability of the method as it's difficult to see how it could extend to higher resolutions than what it currently studies (256x256. The computational complexity analyzed in Sec 3.2.3 is incorrect as, say, the Prediction Head Transformer can dominate the complexity significantly more than the main Transformer that is assumed to be the most expensive in the analysis.
confidence: Less certain
Decision: Accept