Image Generation with Channel-wise Quantization

Authors: Tong Yang, Xiangyu Zhang, Wenqiang Zhang
Primary area: generative models
Keywords: generative models, image generation, visual tokenization

Abstract:
We present a novel image generation model with channel-wise quantization. Our method quantizes image feature along channel into discrete codes. Then based on the learned codes, our approach adopts masked-prediction paradigm for image generation. Compared with widely used spatial tokenizers, our channel-wise tokenizer has an efficient modeling for image structure and strong representational capacity. Besides, the codebook usage of our tokenizer can reach 100\% under different codebook size. Using the channel-wise tokenizer, our generation framework achieves competitive performances on various benchmarks of image generation. In particular, on ImageNet 256x256 benchmark, our method significantly improve baseline by improving Frechet inception distance (FID) to 1.87. Furthermore, we also validate the effectiveness of our proposed method on text-to-image generation.

Decision: reject
Reviewer scores: 5, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1729861866931

thread_type:
official_review

<thread_object>
creator: reviewer_6xwh

<content>
summary:
This work presents a novel image generation model that utilizes channel-wise quantization to convert image features into discrete codes along the channel dimension, adopting a masked prediction paradigm for image generation. This approach offers efficient modeling of image structures and strong representational capacity, outperforming or matching state-of-the-art methods on the ImageNet benchmark. The authors also performed the text-to-image generation and demonstrated transferability to text-to-image generation on the COCO dataset. The contributions of this work include a simple yet effective visual tokenizer with 100% codebook usage and a generation framework based on channel-wise quantization for image generation tasks.

strengths:
This paper is easy to follow and includes comprehensive experiments to demonstrate the effectiveness of the proposed method. It proposes novel channel-wise quantization, which offers efficient modeling of image structures and strong representational capacity. It also proposed a simple yet effective tokenizer with 100% codebook usage. These features enable the whole image generation framework to achieve superior or comparable performance to state-of-the-art methods across various image generation tasks.

weaknesses:
1) This paper's biggest contribution is proposing a channel-wise quantization for image generation. However, channel-wise quantization is widely used in various applications, such as classification [a], LLM compression [b], and super-resolution [c].

[a] RDO-Q: Extremely Fine-Grained Channel-Wise Quantization via Rate-Distortion Optimization. ECCV 2022.

[b] OutlierTune: Efficient Channel-Wise Quantization for Large Language Models. Arxiv 2024.

[c] DAQ: Channel-Wise Distribution-Aware Quantization for Deep Image Super-Resolution Networks. WACV 2022.

2) Why not combine SD series for comparison? For example, SD1.5, SDXL and SD2.1.

3) More experimental results about high-resolution image generation should be provided to demonstrate the effectiveness of the proposed method.

4) The authors did not evaluate the text-image alignment between the text prompts and the synthesized images. 

5) It is not reasonable to develop a channel-wise tokenizer, which is limited to one specific image resolution. One general channel-wise tokenizer algorithm that could be suitable for all image resolutions is favored. Such limitation heavily limits the contribution of this work.

6) The authors only conducted experiments on MS-COCO and ImageNet datasets with the image resolution 256*256. 

7) Marginal performance. Compared with VAR, the proposed method only shows very marginal performance improvement (or even a slight performance drop) under the same experimental setting: similar network parameters and inference step from Table 2.

Minor issues:

1) The best results under the same setting in Table 2 should be bold. 

2) Figure 2 is a little blur and seems that some parts are screenshots of other images.

questions:
What is the purpose of the image generation? 

Only evaluating the image quality of the synthesized images based on evaluation metrics like FID, SSIM and PSNR is not enough. Can the synthesized images promote the downstream visual perception performance such as classification?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730359289308

thread_type:
official_review

<thread_object>
creator: reviewer_xtug

<content>
summary:
In this paper, the authors propose a new image tokenization method called channel-wise quantization, which quantizes image feature along channel. Then, based on the learned tokenizer, the authors use masked-prediction paradigm similar to MaskGIT to generate images. Experiments show that the proposed method achieves competitive performances compared to other image tokenizers and generative models. Besides, the proposed method can reaches 100% codebook usage under different codebook size.

strengths:
+ The writing is clear and easy to follow.
+ The idea of channel-wise quantization is novel and interesting.
+ The reconstruction ability (especially rFID and SSIM) is greatly improved compared to spatial tokenizers.

weaknesses:
- The motivation of proposing channel-wise quantization is not very strong to me. While spatial tokenizers often suffer from low codebook usage and reduced code embedding dimension limits expressive ability, it's unclear how these issues lead to the design of channel-wise quantization.

- The paper lacks a detailed analysis of how the channel-wise tokenizer behaves differently from spatial tokenizers. For example, the authors claim that channel-wise tokens capture both global structures and local details, but there are no direct experiments to support this. It would be interesting if the authors could visualize the learned channel-wise tokens and discuss each channel's representation, so that we can have a deeper understanding of the channel-wise tokenizer.

- The authors attribute the 100% codebook usage to the nature of channel-wise quantization. However, I note that entropy regularization, which is known to be helpful for increasing codebook usage, is adopted in codebook learning. Additionally, the compared method LlamaGen did not use entropy regularization. Thus, I'm not sure if channel-wise quantization is the main factor behind high codebook usage.

questions:
+ As mentioned in weaknesses part, will the channel-wise quantization still reach 100% codebook usage without entropy regularization?

+ What does a channel token typically represent? Is it possible to visualize the learned channel tokens?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730609638174

thread_type:
official_review

<thread_object>
creator: reviewer_fntd

<content>
summary:
This paper presents an alternative to standard VQ-VAE, which typically quantizes each spatial position as a token. Instead, this work quantizes image features along the channel dimension into discrete codes.

For comparison, in a standard VQ-VAE with a final feature dimension of C*H*W, there would be H*W tokens, whereas this work produces C tokens.

Advantages:
1. The paper is clearly written and easy to understand
2. The metrics appear reasonable

Disadvantages:
1. Channel-wise quantization lacks theoretical justification and Intuitive rationality.
   Without spatial-based quantization, the model training appears to lose its causal nature
2. As the model scales up, the number of tokens would need to increase, potentially making learning more difficult
3. The resulting tokenizer becomes incompatible when image resolution changes

I believe the motivation behind this idea is fundamentally flawed, which leads to several issues:
- Limited generalizability
- Sequence length problems
- Modeling methodology concerns

Therefore, I lean towards a negative assessment of this work.

strengths:
Advantages:
1. The paper is clearly written and easy to understand
2. The metrics appear reasonable

weaknesses:
Disadvantages:
1. Channel-wise quantization lacks theoretical justification and Intuitive rationality.
   Without spatial-based quantization, the model training appears to lose its causal nature
2. As the model scales up, the number of tokens would need to increase, potentially making learning more difficult
3. The resulting tokenizer becomes incompatible when image resolution changes

I believe the motivation behind this idea is fundamentally flawed, which leads to several issues:
- Limited generalizability
- Sequence length problems
- Modeling methodology concerns

Therefore, I lean towards a negative assessment of this work.

questions:
see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712408701

thread_type:
official_review

<thread_object>
creator: reviewer_xmae

<content>
summary:
This paper provides an interesting perspective on image compression through learning a vector quantized autoencoder. Typically, we tokenize images spatially. That is, every (i,j) for i over h and j over w within an hxwxc latent space, each position is mapped to a token in a learned codebook. In contrast, this paper proposes to tokenize along the channel dimension of the latent space within the AE. This results in tokens that capture the global image along a latent channel dimension. Results are conducted on class conditional and text conditional image synthesis with various ablations and analysis.

strengths:
- The paper tackles an interesting approach to tokenize along the channel dimension within a VQ AE for images.
- The paper is well-structured and written.
- The approach is easy to understand and to follow.
- Comparison against many relevant models (albeit not exactly fairly if I understood correctly; see below for details).

weaknesses:
My main concerns are regarding fair evaluation wrt compression ratios, more analysis on various token dimensions and exploration of usage of channel wise tokenization in downstream diffusion and AR tasks.

- Since each token in channel space captures complete global information, configurations such as [4,64,64] result in only four tokens of size 64x64 each, rather than 64x64 tokens of size 4. This impacts discussions on code embedding size and codebook use, which are misleading because they affect the compression ratio. I suggest comparing against a fixed compression budget with various compression ratios rather than focusing solely on code embedding sizes (which does not give the full picture).
- Adding to the above, the token count in Table 5 is potentially misleading, as the models use different compression ratios. If I understand correctly, the comparison is between VQAN using 4x16x16 tokens (4x256) and the proposed model with 256x16x16 tokens (256x256).
- The ablation study for token dimensions is incomplete. Experiments within the range [8, 256] are needed to understand scaling behavior better.
- Tokenizing *global information* along the channel dimension implies a significant correlation among tokens, meaning each image is learned as a whole rather than in parts. Consequently, *C* tokens per image are kind of memorized, making it difficult to repurpose tokens for other images due to global encoding. This also explains the high overall usage of the codebook, as tokens are not easily reusable.
- Adding codebook size, embedding dimensions, and compression ratios to the tables would improve comparability.
- In Table 5, why is rFID significantly better while PSNR is worse, even though SSIM is better? I would expect a consistent trade-off between perceptual- and pixel-wise metrics.
- Exploring channel-wise quantization in autoregressive (AR) or diffusion tasks could be insightful. I assume it may not perform as well for AR tasks, as the AR function would need to predict the entire global image in one step rather than progressively building it up from parts.
- The claims regarding low similarity between channel tokens, efficient modeling of image structure, and strong representational capacity lack clear definition and verification.
- I have ignored going into details regarding quantitative results mainly because the issue of fair evaluation is not cleared yet. As of now, sometimes the model is better, sometimes worse, and it is unclear why and when one would want to choose this method over the spatial tokenization.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1731566712765

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Rebuttal

comment:
We thanks all the reviewers for their hard work! We are very honored that our work has been recognized for: **1) interesting approch(Reviewer Xmae, xTUG); 2) novel idea (Reviewer xTUG); 3) clearly written (Reviewer Xmae, FnTD, xTUG, 6xWH).**

## Updated Performance Results
We find MaskGiT sampling used by our method is out-of-date and is not compatible with our method. Thus, we adopt the guidance schedule from MDTv2 [1]. This guidance schedule is widely used by MagViT-v2, TiTok[2] and MaskBit[3]. Thus, it is fair to use the updated performance results for comparison.  We have updated performance results in the paper. Here, we list some critical results to show the effectiveness of our method. **Our method outperforms VAR on FID and IS by a large margin, especially on ImageNet 512x512**

**ImagNet 256x256**

+: codebook size is 65536.

|   Models    | FID | IS | #Para| #step|
| ----------- | ----------- |----------- |----------- |----------- |
|VAR-d30 | 1.97 | 334.7 | 2.0B | 10 |
|MagViT-2 | **1.78** | 319.4 | 307M | 64 |
|Ours-B | 2.77 | 305.3 | 305M | 10 |
|Ours-L | 2.46 | 302.5 | 634M | 10 |
|Ours-H |2.39 |338.2 | 1.0B | 10|
|Ours-B* | 2.21 | 301.2 | 305M | 64 |
|Ours-L* | 2.02 | 323.4 | 634M | 64|
|Ours-H* |1.91 | **344.9**| 1.0B | 64 |
|Ours-L+ | 1.87 | 320.4 | 634M | 64 |

**ImagNet 512x512**
|   Models    | FID | IS | #Para| #step|
| ----------- | ----------- |----------- |----------- |----------- |
|VAR-d36-s | 2.63 | 303.2 | >2.0B | 10 |
|MagViT-2 | **1.91** | 324.3 | 307M | 64 |
|Ours-B | 2.68 | 318.5 | 305M | 10 |
|Ours-L  | 2.46 | 336.4 | 634M | 10 |
|Ours-B* | 2.22 | 323.4 | 305M | 64 |
|Ours-L* | 2.01 | **341.5**| 634M | 64|

**MS-COCO 256x256**
|   Models    | FID | #step|
| ----------- | ----------- |----------- |
|U-ViT-S/2 | 5.95 | - |
|Ours-L | 6.40 | 10 |
|Ours-L*  | **5.85** | 64 |

[1] Gao, Shanghua, et al. "Masked diffusion transformer is a strong image synthesizer." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Yu, Qihang, et al. "An Image is Worth 32 Tokens for Reconstruction and Generation." arXiv preprint arXiv:2406.07550 (2024).

[3] Weber, Mark, et al. "Maskbit: Embedding-free image generation via bit tokens." arXiv preprint arXiv:2409.16211 (2024).

## Methodology
As shown in section 3.2 Discussion, the spatial tokenizers pay more attention to local image areas and are easily to collapse into a limited number of visual units. If image tokens concentrate on larger image areas, image tokens can be more diverse, resulting in high usage for codebook. For our method, due to large receptive fields, image tokens are forced to capture large image areas and more diverse than ones generated by spatial tokenizers. Meanwhile, for reconstruct images in high quality, our tokenizers must learn local details. We have add visualization of codebook tokens in Appendix A1. As shown in Figure 4, our channel-wise tokenizers can capture image structure and local details simultaneously. Please refer to A.1 for more details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732512742742

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking forward to your feedback

comment:
Dear reviewers,

Thank you for your valuable comments regarding our submission.

We have posted our responses and hope to address your concerns about the motivation of our method. We also show the superior results of out method with widely used sampling scheme, demonstrating the effectiveness of our proposed method.

Could you please take a look at our responses and possibly re-evaluate our work based on the additional input? Thank you!

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces channel-wise quantization, which discretizes image features along channels as an alternative to standard spatial tokenizers. The paper received negative feedback from all reviewers. While the authors’ rebuttal adequately addressed some concerns, several major issues remain unresolved: (1) the performance improvement is somewhat marginal and insufficient to outweigh the limitation of fixed image resolution, (2) the argument regarding the global-local information captured by the proposed tokenizer is not convincingly presented, and (3) the experiments are limited in demonstrating the claimed improvement in codebook usage over spatial tokenizers. Overall, the AC believes the paper presents an interesting alternative to spatial tokenizers, with some promising results (e.g., Figure 4). However, the AC agrees with the reviewers that the paper requires further improvements in experiments and presentation, and is not ready for publication at this time. Therefore, the AC recommends rejection.