A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generation

## Abstract

Abstract This work tackles the information loss bottleneck of vector-quantization (VQ) autoregressive image generation by introducing a novel model architecture called the 2-Dimensional Autoregression (DnD) Transformer. The DnD-Transformer predicts more codes for an image by introducing a new autoregression direction, model depth , along with the sequence length direction. Compared to traditional 1D autoregression and previous work utilizing similar 2D image decomposition such as RQ-Transformer, the DnD-Transformer is an end-to-end model that can generate higher quality images with the same backbone model size and sequence length, opening a new optimization perspective for autoregressive image generation. Furthermore, our experiments reveal that the DnD-Transformer’s potential extends beyond generating natural images. It can even generate images with rich text and graphical elements in a self-supervised manner, demonstrating an understanding of these combined modalities. This has not been previously demonstrated for popular vision generative models such as diffusion models, showing a spark of vision-language intelligence when trained solely on images. Code, datasets and models are open at https://github.com/chenllliang/DnD-Transformer .

## 1 Introduction

The field of autoregressive (AR) image generation is experiencing a resurgence of interest, largely driven by groundbreaking advancements in large language models (LLMs), exemplified by the release of ChatGPT (OpenAI, ). Because typical AR image generation methods also predict output in a next-token prediction manner, this resemblance has sparked significant efforts in two main areas: 1) transferring advanced, large-scale training techniques and expertise from LLMs to AR image generation models (Bai et al., ; Tian et al., ; Sun et al., ), and 2) developing truly multimodal foundation models capable of both understanding and generating multimodal information within a unified training framework (Lu et al., ; ; Team, ). These developments have the potential to lead to more versatile and powerful multimodal AI systems.

A review of the development history of AR image generation approaches reveals significant efforts focused on finding better sequential decompositions of images and balancing reconstruction fidelity with prediction difficulty. Early models, like PixelCNN (van den Oord et al., ), generated images pixel by pixel. This approach was later enhanced by using vector-quantized variational autoencoders (VQVAEs) to compress images and model the prior distribution of discrete tokens in a compact latent space (Van Den Oord et al., ). Vector quantization (VQ) paved the way for notable models such as VQGAN (Esser et al., ), DALL·E (Ramesh et al., ), and MUSE (Chang et al., ), and it remains a core technique in recent AR image generation models like VAR (Tian et al., ) and LlamaGen (Sun et al., ), and multimodal foundation models like LVM (Bai et al., ), Unified-IO (Lu et al., ; ), and Chameleon (Team, ).

However, despite advancements in AR image generation, VQ-based autoregressive methods face two persistent criticisms, especially juxtaposed with latent diffusion models (Rombach et al., ):

Figure: Figure 2: Illustration of the proposed DnD-Transformer. N denotes the number of depth autoregression. O-i denotes the transformer layer index for the i-th prediction head. Each transformer layer predicts the corresponding depth code, achieving multi-code prediction within one forward pass.
Refer to caption: x2.png

#### 1) Information loss inherent in the quantization process.

Quantization, specifically in VQVAE, introduces significant information loss. With a typical configuration (N=8192, f=16), the Information Compression Ratio ($\text{ICR}=\frac{\log N}{24f^{2}}$, explained in Equation 1) is just 0.21%, drastically lower than the 8.3% of Stable Diffusion’s VAE(^1^11The Stable Diffusion VAE ([https://huggingface.co/stabilityai/sd-vae-ft-mse](https://huggingface.co/stabilityai/sd-vae-ft-mse)) uses a downsampling factor (f) of 8 and 4 channels, with fp32 tensor precision ($\log N=4\times 32$).), hindering fine-grained detail reconstruction. According to Chameleon (Team, ), the authors note that their VQ tokenizer struggles to reconstruct finegrained details like text in images, which we believe is due to the low ICR of their tokenizer.

#### 2) Substantially increased computational requirements for producing higher-quality images.

According to Equation 1,Increasing ICR by expanding the latent space (N) is logarithmically limited and computationally expensive leading to potential codebook collapse and more embedding parameters, while reducing the downscaling factor (f) significantly increases computational overhead due to a longer token sequence of $O(1/f^{2})$ and a higher transformer computation complexity of $O(1/f^{4})$.

We draw inspiration from the Residual Quantization method (Lee et al., ), which provides a new dimension for sequentially decomposing the image for better generation quality. However, the proposed RQ-Transformer employs two separate transformer models. This structure presents difficulties in integrating current LLMs for end-to-end training. In this work, we aim to solve the problem covering the two mentioned concerns: Can we overcome the information loss of VQ-based AR image generation without increasing overall computation budget in an end-to-end manner?

We propose a novel paradigm for AR image generation called 2-Dimensional Autoregression (DnD) and DnD-Transformer, an end-to-end model architecture. DnD Autoregression introduces a new depth dimension along with the original spatial dimension. In the depth dimension, the image patch could be decomposed in any causal coarse-to-fine order, including the residual decomposition (Lee et al., ), Gaussian denoising decomposition (Ho et al., ) and etc. With a depth of $d$ and other configurations unchanged, the ICR of DnD Autoregression becomes $d\times\frac{\log N}{24f^{2}}$, more effectively reducing the information loss comparing to increasing the codebook size $N$.

The remaining problem is how to predict the $d$ times more tokens effectively. We propose the DnD-Transformer. As shown in Figure 2, it inserts multiple prediction heads into the backbone transformer decoder model to predict the depth codes and conduct additional autoregressive predictions in each forward process. Different from RQ-Transformer (Lee et al., ), the DnD-Transformer does not require additional modules or increased sequence length, making it applicable to any language model architecture and efficiently generate more fine-grained images.

Our experiments show several interesting results:

- 1.
Superior reconstruction of fine-grained image details using residual image decomposition in VQVAEs, disproving VQ’s limitations with text-rich images
- 2.
More efficient and lower-entropy decomposition with DnD autoregression compared to 1D methods, evidenced by lower training cross-entropy loss despite predicting more codes
- 3.
Significant outperformance of the AR baseline on ImageNet 256x256 generation, achieving up to 1.54 FID and 82.6 IS improvements (XXL model, cfg=2) without increased model size or sequence length, even surpassing larger LlamaGen model trained with longer sequence length
- 4.
A spark of vision-language intelligence for the first time, enabling unconditional rich-text image generation, outperforming diffusion models like DDPM and Stable Diffusion on dedicated rich-text image datasets, highlighting the distinct advantage of autoregressive models for multimodal modeling.

## 2 2D Visual Tokenizer and 2D Autoregression

### 2.1 Understand VQVAE as Compression

We introduce the basics of AR generation in Section A in the appendix. We can better understand the reconstruction ability of VQVAE from the lens of compression. Let us assume a VQVAE with down-scaling factor $f$, codebook size $N$, input image’s size of $H\times W$, then the shape of the quantized code is $h\times w=(H/f)\times(W/f)$. We assume that the code follows a uniform distribution, so each code has $\log N$ bits information. Its information compression ratio (ICR) is as follows.

$$ $ICR(N,f)=\frac{(H/f)\times(W/f)\times\log N}{H\times W\times 3\times\log 256}= \frac{\log N}{24f^{2}}$ (1) $$

A typical configuration (N=8192, f=16) results in 0.21% ICR. This ICR is significantly lower than JPEG’s 5% ICR (). To increase ICR, the 1D AR method could increase $N$ (might face the codebook collapse problem (Mentzer et al., ) and the improvement is logarithmically bounded) or decrease $f$ (more effective, but increases the token count quadratically).

### 2.2 Images’ 2D Decomposition and Quantization

As pointed out by Equation 1, the information compression ratio of VQVAE is bounded by the size of the codebook and the downscaling ratio. Residual Quantization (Lee et al., ) proposes a new direction to quantize the image feature with multiple residual codes to reduce the quantization error and improve the quality of the reconstruction. For a feature map having $h\times w$ vectors, RQVAE uses $h\times w\times d$ codes to quantize the feature map, where $d$ is the depth dimension of the code. For each feature vector $\mathbf{v}$, RQ finds $d$ codes ($q_{1},q_{2},...,q_{d}$) by sequentially conducting $d$ times residual decomposition and quantization operation $\mathcal{Q}(x)$ as finding the closest entry to $x$ from the codebook:

$$ $\displaystyle q_{d}=\mathcal{Q}(r_{d-1}),\quad r_{d}=r_{d-1}-q_{d},\quad r_{0} =\mathbf{v}$ (2) $$

Consequently, the sum of the residual codes $\sum_{i=1}^{d}q_{i}$ is expected to approximate more closely the feature vector $\mathbf{v}$, thus reducing the quantization error. We generalize this process as two-dimensional autoregression (DnD), which extends beyond Markov residual decomposition and can be applied to any decomposition operation, such as the diffusion process (Ho et al., ), etc.

Figure: Figure 3: Performance of our visual tokenizers of different depths. The reconstruction of complex features (i.e., eyes, mouse and text) gains significant improvement as the depth increases.
Refer to caption: x3.png

DnD Autoregression quantizes a 2D feature map $\mathbf{m}\in\mathcal{R}^{h\cdot w\cdot c}$ by decomposing it in two directions. First, $\mathbf{m}$ is divided into $h\cdot w$ feature vectors. Second, each vector $\mathbf{v}$ is decomposed into $n$ codes $(q_{1},...,q_{n})$ using a function $\mathcal{D}^{n}(\mathbf{v},\mathcal{Q})$ based on a codebook $\mathcal{Q}$. The resulting quantized map $\mathbf{q}$ has shape $h\cdot w\cdot n$ and is predicted in depth-first-spatial-second order. This decomposition could also be non-Markov, unlike RQVAE. The selection of potentially better decomposition functions is left for future exploration. We still use the residual quantization from Equation 2 as $\mathcal{D}^{n}$. DnD decomposition increases the ICR $d$ times (Equation 3), more effectively than increasing codebook size. The remaining challenge of predicting $d$ times more codes is addressed by our DnD-Transformer.

$$ $ICR(N,f,d)=d\times\frac{(H/f)\times(W/f)\times\log N}{H\times W\times 3\times \log 256}=d\times\frac{\log N}{24f^{2}}$ (3) $$

### 2.3 Reconstruction Performance

We evaluate the reconstruction performance of our trained visual tokenizers with varying maximum codebook depths using the standard ImageNet dataset as the benchmark. All images are resized to 256×256 resolution. We train the different visual tokenizers using the same training objectives as in  Lee et al. (), and assess the reconstruction Fréchet Inception Distance (rFID) on the ImageNet validation set using ADM’s evaluation suite  (Dhariwal & Nichol, ). The results are presented in Table 1(a). For comparison, we include the rFID from the VAE of SDXL (Podell et al., ) and Stable-Diffusion 3 (Esser et al., ) . Our findings demonstrate that our trained visual tokenizer achieves an rFID lower than 1 with two or more codebook depths, even surpassing the performance of SD3’s continuous VAE with less theoretical information loss. As shown in the example from Figure 3, by increasing code depth, we could reconstruct more fine-grained details in the image.

**(a) Reconstruction Performance on ImageNet 256$\times$256 Validation Set.**
| Depth | ImageNet 256$\times$256 |  |  |
| --- | --- | --- | --- |
| rFID$\downarrow$ | L2 Loss$\downarrow$ | Code Usage$\uparrow$ |  |
| 1 | 2.98 | 0.11 | 100% |
| 2 | 0.93 | 0.08 | 100% |
| 4 | 0.60 | 0.05 | 100% |
| 8 | 0.42 | 0.04 | 100% |
| SDXL | 0.68 | 0.05 | - |
| SD3 | 0.67 | 0.04 | - |

Figure: (a) Layerwise code usage of visual tokenizers.
Refer to caption: x4.png

#### Code Usage.

We further analyze the code usage in each codebook layer, with results shown in Figure 4(a). The analysis indicates that usage generally decreases as depth increases. This is due to the diminishing diversity of code usage as the residual decomposition progresses deeper, resulting in smaller feature norms and more centralized code usage according to Figure 4(b). Interestingly, we do not observe signs of codebook collapse with the DnD visual tokenizers, even when using a large codebook size (16384), as mentioned in previous work (Mentzer et al., ). While they reported much lower code usage ($<50\%$), our tokenizer achieves $100\%$ usage across all maximum depths.

### 2.4 VQVAEs Can Perfectly Reconstruct Rich-text Images

A prevalent criticism of VQVAE has been its alleged intrinsic information loss problem, particularly its inability to reconstruct images with fine details, such as those containing rich text (Team, ). However, we argue that this claim is unfounded. Our findings suggest that VQVAE can indeed achieve perfect reconstruction of detailed images, when provided sufficient data and an increased number of codes used to represent each image. This demonstrates that the perceived limitations of VQVAE can be overcome through appropriate data-centric adjustments and model scaling-up.

#### rOCR - A New Metric.

We proposes rOCR, a novel metric for evaluating rich-text image reconstruction. Unlike rFID/L2 Loss, rOCR measures textual recognizability using the Qwen2-VL-72B (Wang et al., ) visual language model for OCR. The metric computes the Rouge-L score between recognized and groundtruth text (or original image OCR if groundtruth is unavailable).

#### Experiments and Results.

Two rich-text image datasets, Text-Image and arXiv-Image (details in Section 4.1), were used to train visual tokenizers. Performance (rOCR scores) was evaluated on both datasets’ 1K test sets, compared against ImageNet-trained tokenizers, SDXL’s (Podell et al., ) and Stable-Diffusion-3’s VAE (Esser et al., ). Text-Image was also tested at a reduced 256$\times$256 resolution to assess resolution impacts. Table 1(a) shows the rOCR results, with reconstruction examples in Figures 3 and 11. Results indicate more training data and deeper tokenizers improve text reconstruction. Unlike  Team (), our discrete visual tokenizers excel in rich-text image reconstruction even compared to continuous VAEs.

## 3 The DnD-Transformer

Prior section showed DnD visual tokenizers effectively reconstruct fine details like text. However, efficiently predicting the increased number of depth codes ($d$ times more) remains challenging. Existing methods, like RQ-Transformer, use a separate transformer for depth, hindering integration with LLMs. We propose an efficient end-to-end architecture for multi-code prediction.

### 3.1 DnD-Transformer Design

Figure: Figure 5: Different explored multi-token prediction architectures for DnD-Transformer, which are all designed to generate multiple codes with one forward pass.
Refer to caption: x6.png

Figure 5 shows DnD-Transformer and its variants: Parallel and Vertical Prediction. Parallel Prediction adds multiple prediction heads for simultaneous multi-depth code prediction, similar to accelerated LLM inference (Cai et al., ). However, this ignores the coarse-to-fine nature (Figure 4(b)) of code distributions, where deeper codes have smaller norms and are more centered. Vertical Prediction addresses this by sequentially predicting codes. Adding autoregression further refines this by conditioning deeper code predictions on previous ones, achieving the best multi-layer code prediction without increasing model parameters or sequence length. Ablation on the structure design is shown in Table 3 from Appendix.

### 3.2 Implementation Details

As shown in the left part of Figure 5, the increment of DnD-Transformer compared to vanilla transformer decoder is the additional output head and embedding add operation. Let’s assume the linearized codemap’s length is $L=h\times w$ and code depth is $d$. During generation, DnD-Transformer conducts $L$ forward process and each forward process generate $d$ codes sequentially. After generating codes for all depths in a forward process, the embeddings of all codes are added up as the next input token. In this way, the model could generate $L\times d$ tokens with only $L$ forward passes, improving the generation quality with the same inference cost as standard 1D auto-regression transformer. The only additional hyper-parameter is the layer indexes to predict code of different depths. We adopt the same transformer decoder’s architecture as LLaMA (Touvron et al., ) and , please refer to Appendix E for the training details of our DnD-Transformer.

## 4 Experiments and Findings

### 4.1 Tasks and Datasets

#### Class-Conditional Image Generation.

We conduct standard conditional image generation task with ImageNet-1k benchmark. Images are resized to 256$\times$256 resolution during training and evaluation. We sample 50k images with classes uniformly distributed, and compute the FID, IS, Precision and Recall aganist the training set data using the ADM evaluation tool Dhariwal & Nichol ().

#### Unconditional Rich-Text Image Generation.

We collect two datasets for this task. Dataset examples are shown in Figure 6. Models are trained in a unconditional setting in this task. We aim to explore whether the tested vision generation models could understand and generate the complex logical interrelation among the generated elements such as language.

- 1.
Pure Text Images (Text-Image). The dataset is automatically rendered from a portion of English wikipedia (), consisting of 2.4M images. Each image has a original resolution of 512$\times$512 and a font size of 32pt. We set a maximum of 100 words in each image with a paddling margin of 20pt. We use the [Pillow](https://github.com/python-pillow/Pillow) library to render the image.
- 2.
arXiv Images (arXiv-Image) we first download the papers in PDF format from [arXiv.org](arXiv.org), and render the pages to image of A4 resolution ($1260\times 1782$) with [pdf2image](https://github.com/Belval/pdf2image) tool. We then randomly crop ten 512$\times$512 image from each pages and finally collect 2M images.

We have developed an evaluation pipeline that combines Optical Character Recognition (OCR) and Perplexity Measurement for assessing the quality of generated images, with a focus on the textual information they contain. Initially, we employ the state-of-the-art open-source Vision-Language Model, Qwen2-VL-72B, to extract text from the generated images. Subsequently, we utilize the Qwen2.5-72B model to calculate the perplexity of the generated text, where the LLM is regraded as the evaluator. The resulted score is called $PPL_{ocr}$, we also test the score of groundtruth data from the training images as the performance upper-bound.

### 4.2 Models

#### Visual Tokenizers.

We train our visual tokenizer based on RQVAEs (Lee et al., ). We train tokenizers with code depths of $\{1,2,4,8\}$ and scaling factor $f=16$ across different experiments. We choose the checkpoint with best rFID across 150 epochs. Performance comparison of different visual tokenizers is shown in Table 1. We follow  Lee et al. () to train the visual tokenizers. Details of the training of visual tokenizers are listed in Appendix B. Reconstruction performance of the trained visual tokenizers is shown in Table 1.

#### DnD-Transformer.

We train two size of DnD-Transformers across our experiment, namely DnD-Transformer-XXL (1.4B) and DnD-Transformer-XXXL (2.5B). Basically, DnD-Transformer inherits the LLaMA (Touvron et al., ) architecture. The XXL version strictly align with the LlamaGen-XXL baseline to be fairly compared. Details of the model are shown in Appendix E.

Figure: Figure 6: Data examples in of the collected Text-Image and arXiv-Image image datasets.
Refer to caption: x7.png

#### Implemented Baselines for Class-Conditional Image Generation.

LlamaGen (Sun et al., ) is the major baseline and state-of-the-art model for AR image generation on ImageNet. Our implemented code primarily refers to the same training codebase for fair comparison. LlamaGen could be also viewed as a special version of DnD-Transformer where the decomposition depth equals to 1.

**Table 2: Model comparisons on class-conditional ImageNet 256$\times$256 benchmark. The “Reported” results refer to Sun et al. (). The “Implemented” results are conducted in this work. $\dagger$ indicates that the model is unorthodoxly trained at 384$\times$384 resolution, which requires 2.25 times longer sequence length compared to our implemented models. “cfg” means the scale of classifier-free guidance. The number of depth autoregression is 2 for DnD-Transformers.**
| Type | Model | #Para. | FID$\downarrow$ | IS$\uparrow$ | Precision$\uparrow$ | Recall$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Diffusion-Reported | ADM (Dhariwal & Nichol, ) | 554M | 10.94 | 101.0 | 0.69 | 0.63 |
| CDM (Ho et al., ) | $-$ | 4.88 | 158.7 | $-$ | $-$ |  |
| LDM-4 (Rombach et al., ) | 400M | 3.60 | 247.7 | $-$ | $-$ |  |
| DiT-XL/2 (Peebles & Xie, ) | 675M | 2.27 | 278.2 | 0.83 | 0.57 |  |
| AR-Reported | VQGAN (Esser et al., ) | 1.4B | 5.20 | 280.3 | $-$ | $-$ |
| RQTransformer (Lee et al., ) | 3.8B | 7.55 | 134.0 | $-$ | $-$ |  |
| LlamaGen-XXL (cfg=2)  (Sun et al., ) | 1.4B | 3.64 | 296.5 | 0.86 | 0.51 |  |
| LlamaGen-XXL^† (384$\times$384, cfg=2)  (Sun et al., ) | 1.4B | 2.52 | 295.4 | 0.84 | 0.56 |  |
| LlamaGen-3B (cfg=2)  (Sun et al., ) | 3.1B | 4.21 | 325.2 | 0.87 | 0.49 |  |
| LlamaGen-3B^† (384$\times$384, cfg=2)  (Sun et al., ) | 3.1B | 2.81 | 311.6 | 0.84 | 0.54 |  |
| AR-Implemented | LlamaGen-XXL (cfg=4) | 1.4B | 7.67 | 345.1 | 0.89 | 0.35 |
| LlamaGen-XXL (cfg=2) | 1.4B | 4.12 | 266.9 | 0.83 | 0.49 |  |
| DnD-Transformer-XXL (cfg=4) | 1.4B | 6.55 | 427.7 | 0.89 | 0.42 |  |
| DnD-Transformer-XXL (cfg=2) | 1.4B | 2.58 | 295.6 | 0.83 | 0.56 |  |
| DnD-Transformer-XXL (cfg=1.7) | 1.4B | 2.78 | 239.2 | 0.82 | 0.56 |  |
| DnD-Transformer-XXL (cfg=1.5) | 1.4B | 2.96 | 232.5 | 0.80 | 0.57 |  |
| DnD-Transformer-XXXL (cfg=4) | 2.5B | 6.48 | 413.0 | 0.89 | 0.42 |  |
| DnD-Transformer-XXXL (cfg=2) | 2.5B | 2.77 | 319.1 | 0.85 | 0.54 |  |
| DnD-Transformer-XXXL (cfg=1.7) | 2.5B | 2.21 | 279.3 | 0.83 | 0.58 |  |
| DnD-Transformer-XXXL (cfg=1.5) | 2.5B | 2.52 | 244.2 | 0.80 | 0.59 |  |

#### Implemented Baselines for Rich-Text Image Generation.

We select multiple diffusion models as the baselines, including DDPM (Ho et al., ), Stable Diffusion XL (SDXL) (Podell et al., ) and Stable Diffusion v3.0 (SD3) (Esser et al., ). For DDPM, we train the model on the dataset from scratch. For SDXL and SD3, we finetune the checkpoints from the official website.

### 4.3 Results of Class-Conditional Image Generation

As demonstrated in Table 2, our DnD-Transformer significantly outperforms the 1D autoregressive baseline LlamenGen across various scales and generation evaluation metrics, including FID and IS. This superior performance is achieved while maintaining the same number of parameters in the backbone model, based on our reported and implemented results. It is noteworthy that our 2.5B model, trained with a sequence length of 256, even outperforms the 3.1B LlamaGen model, which was trained with a much longer image sequence length of 576. This result demonstrates that the DnD-Transformer can effectively predict a greater number of tokens within a shorter sequence length, highlighting its significant potential to revolutionize the one-dimensional autoregressive paradigm. We randomly sample some generation results as shown in Figure 1 and compare the generation performance with 1D-AR in Figure 12,13 and 14 from the Appendix. The comparative analysis clearly illustrates the effectiveness of our approach to generate high-quality images.

Figure: (a) Comparison of FIDs along training.
Refer to caption: x8.png

### 4.4 Results of Rich-text Image Generation

#### Generation Results on Text-Image.

A DnD-Transformer (depth 1) and a DDPM model were trained on the same text-image dataset. Comparing 250 randomly sampled images from each, the AR model significantly outperformed the diffusion model in generating coherent text (lower OCR perplexity 7(b); Generation examples 1, 16, 17, 18 and 19 ). This suggests the AR model’s discrete token reconstruction enables effective autoregressive modeling. We also find that with a lower sampling temperature, the model would generate text images with lower PPL just like LLMs. Conversely, the diffusion model’s simultaneous generation hinders text coherence.

Figure: Figure 8: Comparison of Unconditional Rich-Text Image Generation on the more complex arXiv-Image dataset. SD3 is hard to generate valid words, while DnD-Transformer demonstrates an ability to generate semantically appropriate phrases, as marked in blue. More baselines are in Figure 15.
Refer to caption: x10.png

Figure: (a) Training Loss for DnD-Transformer trained with different number of prediction heads.
Refer to caption: extracted/5893877/imgs/Training_Loss_Layer.png

Figure: Figure 10: Some cases of the generated text images. We witness similar error pattern (marked in red) to LLMs such as repetition and hallucination in our trained model during sampling.
Refer to caption: x11.png

#### Generation Results on arXiv-Image.

An 8-layer visual tokenizer and corresponding DnD-Transformer trained on arXiv-Image outperformed diffusion model baselines, generating more valid words and phrases (Figure 8). However, arXiv-Image generation lagged behind Text-Image generation, suggesting joint language and figure modeling is more challenging. More results and baselines are in Figure 15 and  20. While SD3’s VAE reconstructs arXiv images well (Table 1(b)), its generative performance is inferior to DDPM and AR, suggesting its latent space is less suitable for language modeling comparing to pixel or discrete space.

#### A Spark of Vision-Language Intelligence.

Autoregressive (AR) image generation exhibits a marked advantage over diffusion models in producing text-rich images, as demonstrated by our results. The pixel-level language generation inherent to AR models facilitates this capability. Despite limitations imposed by our current training data and model size (preventing direct comparison with large language models), these findings suggest a promising pathway towards vision-language intelligence where language understanding emerges directly from visual perception. Furthermore, our pure image learners display behaviors mirroring language model issues such as repetition and hallucination (Figure 10), implying the potential for integrating pure language modeling into a unified autoregressive framework for joint vision-language image modeling.

### 4.5 Training Becomes Easier when Predicting Multiple Codes, Sampling Not

Deeper DnD-Transformer codes achieve lower cross-entropy loss during training (Figure 9(a)), indicating lower entropy image decompositions. However, despite this, increased depth doesn’t improve ImageNet generation fidelity, possibly due to the larger sampling space. Exploring this multi-depth sampling space for better generation is a promising research direction.

### 4.6 AR Training Loss for Different Domains align with Inner Randomness

Training loss for the same DnD-Transformer varies significantly across datasets (Figure 9(b)), being notably higher for ImageNet than rich-text images. While rich-text image loss nears that of LLMs, ImageNet loss sits between text and natural image datasets. The AR model’s LLM-like training suggests it learns language from visual input alone, implying language’s visual representation has lower entropy than natural images, easing the learning process.

## 5 Related Work

#### Image Generation with VQVAE.

The vector quantization (VQ) method has been pivotal in the development of generative models (Ramesh et al., ; Yu et al., ; Chang et al., ), which achieve image generation through the prediction of discrete image tokens. Efforts in this area focus on two main directions: the optimization of image tokenization techniques (Esser et al., ; Mentzer et al., ; Yu et al., ; ; Weber et al., ), and the strategic planning of effective decompositions of image tokens, such as MaskGit (Chang et al., ) and VAR (Tian et al., ).
Meanwhile, alongside the advancement of large language models, there is growing interest in autoregressive image generation, which predicts image tokens sequentially (Tian et al., ; Sun et al., ).
Recent research has also focused on developing multimodal foundation models (Lu et al., ; Kondratyuk et al., ; Wang et al., ) that integrate both understanding and autoregressive image generation capabilities. They typically convert images or videos into sequences of discretized tokens and train over combined text-image/video token sequences within the AR modeling framework (Lu et al., ; Bai et al., ; Xie et al., ; Team, ).
However, these models often struggle with inherent information loss during the image quantization and the significantly increased computational demands when generating higher-quality images.
The DnD-Transformer that adopts the residual 2D decomposition of image features does not require additional modules or increased sequence length for high-quality and fine-grained image generation.

#### Rich-Text Image Generation.

Despite recent significant progress in image generation, the task of rich-text generation within images remains a persistent challenge (Chen et al., ; Ma et al., ; OpenAI, ).
Most advancements have been witnessed in diffusion models (Betker et al., ; Saharia et al., ; ), these models either leverage large language models to enhance the character spelling capabilities of generative models (Saharia et al., ; Balaji et al., ; Saharia et al., ) or attempt to explicitly control the position and content of the text using additional supervision from different modules (Tuo et al., ; Yang et al., ; Liu et al., ).
However, most diffusion-based methods have primarily focused on text rendering Chen et al. (; ); Balaji et al. (); Saharia et al. () in image generation, often limited to generating short words for logos and posters (Yang et al., ; Ma et al., ; ). The full potential of rich-text image generation remains largely unexplored. Our methods, which build on the foundation of DnD Autoregression, show substantial progress in generating rich-text images in an unconditional manner, highlighting the feasibility of conducting joint vision-language modeling tasks using purely images.

## 6 Conclusion

This paper investigated the limitations of autoregressive (AR) image generation methods, particularly the information loss and computational burden associated with vector quantization (VQ). We introduced 2-Dimensional Autoregression (DnD) and a novel end-to-end architecture, DnD-Transformer, which leverages a depth dimension autoregression alongside the spatial dimension to mitigate these limitations. Our experiments demonstrate that DnD-Transformer achieves significant improvements in image quality, outperforming strong baselines like LlamaGen without increasing model size or sequence length. Notably, DnD-Transformer showcases emergent vision-language intelligence, generating text-rich images unconditionally, a known weakness of diffusion models. These findings highlight the potential of DnD for efficient and high-quality AR image generation and underscore the promise of this approach for advancing multimodal foundation models.

## Appendix A Preliminary: Autoregressive Image Generation

In this section, we introduce the fundamentals of autoregressive image generation. The pipeline is rooted in the Vector Quantized Variational Autoencoder (VQVAE)  (Van Den Oord et al., ) and the autoregressive Transformer  (Vaswani et al., ). This approach has been adopted from the early DALLE  (Ramesh et al., ) to the latest LlamaGen  (Sun et al., ).

### A.1 Step1: Train the Visual Tokenizer and Tokenize the Images

Images initially exist in the pixel-level RGB color space, which consists of little semantic information and makes it challenging to directly model prior knowledge. For example, an image with a resolution of $256\times 256$ comprises $256\times 256\times 3=196,608$ distinct values, representing the individual red, green, and blue intensities for each pixel. The large sequence length makes it difficult to train in autoregressive manner similar to language models’ technique. Van Den Oord et al. () proposed the Vector Quantized Variational Autoencoder (VQVAE), which significantly alleviates the problem. It downscales and tokenizes the image from the original sparse RGB space into a dense and discrete representational space (codebook) $\mathcal{Q}$ by finding the nearest entry. The VQVAE is typically implemented in an encoder-decoder architecture, with its primary training objective being to minimize the image reconstruction loss. You could refer to  Van Den Oord et al. () for details in training a standard VQVAE.

### A.2 Step2: Learn the Prior Distribution of Image Tokens

Having tokenized the source images into discrete tokens and trained a visual decoder to map these tokens back to real images, the next crucial step is to learn the prior distribution of the discrete tokens. This distribution enables the sampling process, which is essential for generating new images. AR Image generation generally first linearizes the $h\times w$ image tokens $q\in\mathcal{Q}$ in a raster scan order and formalize 1D sequence ($q_{1},q_{2},q_{3},...,q_{h\times w}$) for the transformer (Vaswani et al., ) model to learn.

During training, the training objective is the same as GPT’s next token prediction task (Radford et al., ), that the model is required to predict the next image token given the previous tokens and class or text conditional tokens $\prod_{t=1}^{h\times w}p\left(q_{t}\mid q_{<t},c\right)$. After training, we can generate images by autoregressively sampling $h\times w$ tokens from the model. The sampled 1D sequence of image tokens is then reshaped to 2D code map with height $h$ and width $w$. This reshaped token map is subsequently fed into the trained VQVAE decoder, which reconstructs the final image from the code representation.

#### Classifier-Free Guidance

As a technique to enhance the visual quality and text-image alignment, classifier-free guidance (Ho & Salimans, ) has been adopted across the diffusion models (Rombach et al., ; Podell et al., ), VQ models (Chang et al., ) and autoregressive models (Sun et al., ) for image generation.
During the training, the model is exposed to data with and without conditioning: the conditioning is randomly discarded from a fraction of the training samples.
We have implemented this approach in our model as well. Specifically, during training, we randomly replace the conditional embedding with a learnable unconditional embedding in 10% of the cases.
At the inference stage, the logits $\ell_{g}$ are recalculated for each generated token.
We form the $\ell_{g}$ by subtracting the unconditional logits $\ell_{u}$ by conditional logits $\ell_{c}$ with the guidance scale $t$ through the following equation:

$$ $\ell_{g}=\ell_{u}+(\ell_{c}-\ell_{u})\times t$ (4) $$

## Appendix B Training Details of Visual Tokenizers

We follow  (Lee et al., ) to train the 2D tokenizers with residual decomposition a combined objective of l2 loss, GAN loss and perceptual loss. Codes from different depth share the same codebook. We train all tokenizers a fixed learning rate of 4e-5, a total batch-size of 256 for 100 epochs and select the one with lowest validation loss as the final tokenizers. We conduct all training on 8$\times$A100 GPUs.

## Appendix C Reconstruction Results of Texts

Figure 11 shows the reconstruction result on arXiv images of different visual tokenizers.

Figure: Figure 11: Reconstruction Results of Texts. With training data and enough depths of codes, RQ visual tokenizers can well reconstruct the text in the images.
Refer to caption: x12.png

## Appendix D Ablation on DnD-Transformer’s Structure

**Table 3: Ablation of DnD-Transformer Architecture on ImageNet dataset. All models follow the same training setting as in Appendix E.**
| Model | Parameters | FID | IS | Precison | Recall |
| --- | --- | --- | --- | --- | --- |
| 1D | 1.4B | 4.12 | 266.9 | 0.83 | 0.49 |
| 2D Parallel | 1.4B | 6.32 | 232.1 | 0.79 | 0.44 |
| 2D Vertical | 1.4B | 3.18 | 289.7 | 0.83 | 0.57 |
| DnD-Transformer | 1.4B | 2.58 | 295.6 | 0.83 | 0.56 |

## Appendix E Details of hyper-parameters of DnD-Transformer

Table 4 shows the hyper-parameters of our trained models. The XXL model has the same setting as in GPT2 (Radford et al., ) and LlamaGen (Sun et al., ) for fair comparisons. For DnD-Transformer with multiple prediction heads, the prediction layers’ indexes are set to $[39,48]$ when there are two heads, $[39,42,45,48]$ when there are 4 heads in the ImageNet experiments, $[27,30,33,36,39,42,45,48]$ when there are 8 heads in the arXiv-Image experiments.

**Table 4: Model sizes and architecture configurations**
| Model | Parameters | Layers | Hidden Size | Heads |
| --- | --- | --- | --- | --- |
| XXL | 1.4B | 48 | 1536 | 24 |
| XXXL | 2.5B | 48 | 2048 | 32 |

All transformer models were trained using settings similar to LlamaGen (Sun et al., ): a base learning rate of $10^{-4}$ per 256 batch size, the AdamW optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.95$, and a weight decay of 0.05, along with gradient clipping at 1.0. A dropout of 0.1 was consistently applied to the input token embedding, attention module, and feed-forward network (FFN) module. Similarly, a dropout of 0.1 was used for the class condition embedding for classifier-free guidance. Training was performed for 300 epochs, and the final checkpoint was used for performance evaluation.

## Appendix F Generation Results of DnD-Transformers

Figure: Figure 12: Conditional generation comparisons between LlamaGen-XXL and DnD-Transformer-XXL on class “golden retriever” from ImageNet. We random sampled 16 images with cfg=4. DnD-Transformer generates images with higher quality than the 1D AR model.
Refer to caption: extracted/5893877/imgs/golden.png

Figure: Figure 13: Conditional generation comparisons between LlamaGen-XXL and DnD-Transformer-XXL on class “volcano” from ImageNet. We random sampled 16 images with cfg=4. DnD-Transformer generates images with higher quality than the 1D AR model.
Refer to caption: extracted/5893877/imgs/vol.png

Figure: Figure 14: Conditional generation comparisons between LlamaGen-XXL and DnD-Transformer-XXL on class “husky” from ImageNet. We random sampled 16 images with cfg=4. DnD-Transformer generates images with higher quality than the 1D AR model especially for the more complex eyes of husky.
Refer to caption: extracted/5893877/imgs/dog.png

Figure: Figure 15: Comparison of Unconditional Rich-Text Image Generation on the more complex arXiv-Image dataset. All models are trained on the same dataset. The generated images are all in 256x256 resolution. Diffusion-Family models are hard to generate valid words, while DnD-Transformer demonstrates an ability to generate semantically appropriate phrases, as evidenced by the correct clause ”it should be” observed in the second example.
Refer to caption: x13.png

Figure: Figure 16: Unconditional Generation examples of DDPM on Image-Text.
Refer to caption: extracted/5893877/imgs/ddpm-samples_ep0_examples.png

Figure: Figure 17: Unconditional Generation examples of DnD-Transformer on Image-Text with temperature=0.1.
Refer to caption: extracted/5893877/imgs/GPT-XXL-0340000-size-384-size-256-topk-0-topp-1.0-temperature-0.1-cfg-1.0-seed-0.png

Figure: Figure 18: Unconditional Generation examples of DnD-Transformer on Image-Text with temperature=0.5.
Refer to caption: extracted/5893877/imgs/GPT-XXL-0340000-size-384-size-256-topk-0-topp-1.0-temperature-0.5-cfg-1.0-seed-0.png

Figure: Figure 19: Unconditional Generation examples of DnD-Transformer on Image-Text with temperature=1.0.
Refer to caption: extracted/5893877/imgs/GPT-XXL-0340000-size-384-size-256-topk-0-topp-1.0-temperature-1.0-cfg-1.0-seed-0.png

Figure: Figure 20: Unconditional Generation examples of DnD-Transformer on arXiv data with temperature=1.
Refer to caption: extracted/5893877/imgs/GPT-XXL-0040000-size-256-size-256-topk-0-topp-1.0-temperature-0.5-cfg-1.0-seed-0.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: I'm not sure how to interpret the code usage reported in Table 1a vs. Figure 4a. The table shows 100% usage up to depth=8, but the chart shows code usage falling off with depth. Is this a mistake or are they showing something different?

A minor point (that's not actually a question): In Section 2.1 the paper says "each code has log N bits [of] information". It should be "log_2 N bits" or "log N nats".
soundness: 4
strengths: The main strength of the paper is the authors' creative solution to predicting multiple codes per token in an efficient and effective manner.  The approaches used in other works all have significant drawbacks:
 - use more tokens, which is computationally expensive
 - use a second transformer as in RQ-Transformer, which complicates the model, especially if you're building on top of an existing LLM
 - grow the codebook, which is memory-limited for VQ (though not FSQ or binary quantization as in MagVit v2 and MaskBit) and makes prediction more difficult
 - use multiple codes and predict them in parallel, which doesn't work very well (also discussed in this paper).
summary: This paper addresses the problem of image modeling and generation using discrete latent representations and autoregressive (AR) sampling. The authors note that larger codebooks are need to improve visual quality but scaling up the codebook directly is difficult. They suggest a using multiple codes per token and adopt residual vector quantization (RVQ or just RQ) based on an earlier model called the RQ-Transformer.

They then present a creative approach for predicting the residual codes by adding a prediction head between different layers within the normal multi-layer transformer. This approach adds only a trivial amount of extra parameters and computational cost, and requires only minor changes to existing LLMs. Contrast with the RQ-Transformer that adds a new (albeit small) transformer, which is harder to integrate into existing LLM code. Through empirical validation, the authors show that their "DnD-Transformer" yields good generation results compared to RQ-Transformer, LlamaGen, and other methods.

Using RVQ to decompose a large codebook is not new, but adding prediction heads between the layers of the LLM transformer is. The authors show that this approach outperforms parallel prediction and vertical prediction (visualized in Fig. 5), two other approaches for predicting multiple codes from a single forward pass.

In addition to generation on ImageNet-256 evaluated using standard metrics (gFID, IS, etc.), the authors also introduce rOCR that evaluates an autoencoder in terms of how well text in a reconstructed image can be recognized. They show that larger depth values (more residual codes) improves this metric, as expected (see Table 1b). For generation, they measure perplexity using Qwen2.5-72B over text extracted with Qwen2-VL-72B and show that their approach performs better that a diffusion-based baseline (Fig. 6b). They argue that AR prediction is better suited to generating images of text compared to the "simultaneous" generation of diffusion models. This is where the "spark of vision-language intelligence" comes from: the DnD-transformer is trained on images of text and can then generate images of text with a few recognizable words and short phrases (see Fig. 7).
weaknesses: I see two main weaknesses in the paper. First, I'd like to see more evaluation of the DnD-Transformer in terms of sensitivity to layer indexes for the prediction heads both within an otherwise fixed architecture (e.g., stick with DnD-Transformer-XXXL and vary the indexes) and for smaller/larger models (e.g., if #heads == #layers, the smallest option, does the approach still work?). While the layer indexes are "just" a hyperparameter, it would be quite interesting to know if the approach is relatively robust to the layer choices or if there's a pattern. For example, for a fixed number of layers (and thus compute), you can ask how that compute should be used: do you want more layers before the first code is predicted? An even distribution across codes? Maybe the first code is relatively easy and you need successively more compute for later codes.

The second weakness deals with evaluation of generation results as a function of depth. If I'm interpreting the paper correctly, the ImageNet results in Table 2 use depth=2 (see the caption for Table 2), and the text-image generation results use depth=1 (see Section 4.4). Note that depth=1 is the baseline where no residuals are used (i.e., it's just an standard next-token predictor with one code per token). These low depth values seem to undermine the core contribution of the DnD-Transformer.

Note that the "Generation Results on arXiv-Image" sub-section does say that an "8-layer visual tokenizer and corresponding DnD-transformer..." If that means that depth=8, then I'm less concerned. Nonetheless, I'd like to see a chart showing gFID vs. depth (much like other VQVAE-based papers show gFID vs. codebook size). Figure 1b has the right structure, but it shows *reconstruction* metrics, which should always improve as depth increases. This does not always translate to generation quality.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: #1. I'm not trying to pinpoint this assertion, but I don't personally buy this argument, since autoregressive image generation research was already very popular before the release of ChatGPT. Additionally, by 2022, the research community had already shifted to diffusion models. GLIDE (2021), Latent Diffusion (2022), DALL-E 2 (2022), and Imagen (2022) were all published before ChatGPT.

#2. The claim in the phrase “a spark of vision-language intelligence for the first time” appears to be overstated, especially given the limited scope of the experiments conducted. It is notable that AR-based image generation demonstrates strong performance in text rendering, particularly for documents, compared to diffusion models. However, can this really be considered a spark in vision-and-language intelligence?"

#3. How is the ICR of JPEG computed? Additionally, it would be preferable to include a proper reference from a primary source or technical documentation rather than Wikipedia.

#4. In section 2.2, the authors discuss the differences between the proposed approach and RQ-VAE, but I find the explanation somewhat unconvincing. Could you elaborate on the distinctions and explain why the newly proposed component might be expected to outperform RQ-VAE?

#5. In Table 1(a), code usage is reported at 100%. Did you apply any specific techniques to enhance code usage, such as restarting dead codes or similar methods?

#6. In Table 1(b), SDXL and SD3 are trained on ImageNet and should be considered zero-shot tokenizers. However, this isn’t indicated in the table, which leads to misinterpretation of the results.

#7. As I understand it, unlike RQ-Transformer, DnD-Transformer predicts codes along the depth dimension simultaneously. However, HQ-Transformer, a follow-up to RQ-Transformer, also explores predicting codes in this way. If my understanding is correct, it might be beneficial to include HQ-Transformer as a baseline for comparison.

#8. Regarding Figure 3, which dataset was used for the tokenizer in SD3? Did you use the original SD3 tokenizer? If so, I question whether this is a fair comparison, as open-source image generation models generally do not include OCR images in their datasets. If the claim is that DnD-Transformer performs well in generating rich-text images, it would be more appropriate to train a tokenizer specifically on rich-text images. I suggest training an LDM, including a tokenizer (such as continuous VAE f8), on rich-text images and then comparing DnD-Transformer with LDM under these conditions. In that case, I am not certain that DnD-Transformer would outperform LDM.
soundness: 2
strengths: It’s intriguing to see the potential for training a vision-and-language model solely on images. When training such models using textbooks or other materials, preprocessing to associate images with the relevant text is often challenging. Training an LMM purely on images might offer a more principled and human-like learning approach.
summary: This work addresses the information loss bottleneck in vector-quantized (VQ) autoregressive image generation by introducing a novel model architecture, the 2-Dimensional Autoregression (DnD) Transformer. The DnD Transformer is an end-to-end model capable of generating higher-quality images while maintaining the same backbone model size and sequence length, presenting a fresh optimization perspective for autoregressive image generation. Experiments on ImageNet and rich-text images demonstrate that the DnD Transformer successfully generates detailed, high-quality images, highlighting its potential for advancing vision-and-language models with a new approach.
weaknesses: #1. Compared to the RQ-Transformer paired with RQ-VAE, what are the benefits of this work? The proposed method appears largely similar, with no observed improvements over RQ-VAE.

#2. I believe the manuscript requires extensive revision. Some references are incomplete, with some citing Wikipedia, which is not a valid source. Furthermore, the comparison between DnD-Transformer and existing approaches, especially RQ-Transformer, lacks clarity and is not fully convincing.

#3. I believe many experiments in this paper may lead to misinterpretation, as the comparisons are not conducted under fair conditions. I suggest that the authors address my questions in #5 and #8 to ensure a more accurate comparison.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. It's not very clear about the gain from depth-wise autoregression. It would be good to have results with depth 1 or 3 on ImageNet reported as well. Also on rich-text image generation, DnD is trained with depth 1. How would the performance change if more depths are applied?
2. There're some AR-based image generation tasks that are not included in ImageNet256 benchmark. For example, VAR [1] curates a resolution-changing tokenizer for image generation. 
3. For rich-text image generations, how is SD3 implemented? Is it re-trained on tokenizer trained on rich-text datasets or finetuned with its original tokenizer? More details would help understand the performance gap between SD3 and DnD. 
4. Also, are there quantitative evaluations for rich-text image generation, like the rOCR used in evaluation of reconstruction performance?
5. The tokenizers are trained separately for ImageNet and rich-text datasets. Will tokenizers trained on merged datasets hurt the performance on either side?

References: 

[1] Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction, https://arxiv.org/abs/2404.02905
soundness: 2
strengths: 1. The work investigates an interesting problem about improving autoregressive image generation on discrete tokens. 
2. The model achieves competitive performance on standard ImageNet benchmark. 
3. The model demonstrates capability of generating images with rich text and graphical elements.
summary: In this work, the authors introduce DnD Transformer, which predicts discrete image tokens in both sequence and depth direction. In sequence direction, the model is trained to generate tokens one by one as standard AR. In depth direction, the model generate one complete token in a residual-quantized manner. Experiments on ImageNet show competitive performance against baseline AR models. The model also show promising results on text-heavy image generation tasks.
weaknesses: 1. Some implementation details are missing which affect the evaluation of proposed method.
2. Some baselines are missed on standard ImageNet generation tasks. 
3. The model claims it shows a spark of vision-language intelligence by showing results on the rich-text image generation. However, it's a bit unclear to me whether these results lead to actual language understanding. It may beyond the scope of this work but it would be interesting to see how it performances in vision-language understanding tasks. The authors may consider being careful about the claim. 

Please find more details in Questions section below.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - My biggest question is on computational complexity. We need more clarification on the additional compute needed for different depths of DnD-Transformer layers, for both inference and training, and comparisons to other work.
- Is there any reason text-to-image is not explored?
- Can the method generalize to higher resolutions without exploding complexity?

My final score will depend on these clarifications.
soundness: 3
strengths: We know that iterative refinement is currently key in methods to generate images. For example, diffusion and rectified flow models which are SoTA have this iterative refinement quality. Embedding iterative refinement in the AR transformer block is a very interesting idea. Further, the paper presents ablations that show monotonically improving FID and other scores when the depth is higher, which is consistent with this claim. Also, they present results that rival SoTA diffusion models for class-to-image generation, which is very promising. The images they show in the paper are also convincing, with very high quality - and the interesting phenomenon of coherence in text generation even when trained without text-labels. Overall it's an interesting paper with a strong directive idea.
summary: The paper proposes a 2-Dimensional Autoregressive Transformer (DnD-Transformer) which, essentially, adds iterative refinement to visual token generation in the autoregressive regime. This iterative refinement is reminiscent of diffusion model inference (but not the same).
weaknesses: 1. The paper needs objective data on FLOPs, training time, and inference speed. This is very important since adding the DnD-Transformer layers increases all of these. It needs comparisons to SoTA diffusion, rectified flow, and AR models. This is very important for judging the paper since increasing computation has to be balanced out by increased performance - and both need to be presented, not just one.
2. Only explores class-to-image and unconditional image generation.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: see weakness

In summary, I find this paper to be more focused on practical engineering experience. DnD enhances the performance of generative models with relatively low computational cost. However, there are notable issues regarding the theoretical analysis and clarity of the writing. As a result, my rating is a marginal rejection.
soundness: 3
strengths: 1. The motivation behind DnD is well-founded, as addressing the effects introduced by VQ can lead to improvements in generation quality.
2. The author provides ablation studies to demonstrate the effectiveness of the proposed architecture.
3. DnD (almost) does not introduce an increase in the number of parameters compared to conventional architectures.
summary: This paper introduces DnD aimed at addressing information loss in vector quantization decoder for AR transformer. 

The DnD-Transformer incorporates an additional depth dimension and extends sequence length, enabling it to predict a greater number of image encodings, which in turn leads to the generation of higher-quality images.

To my knowledge, it looks like introducing simplified diffusion-like process for each token while keep the AR schedule for the whole token sequence. 

The experimental results demonstrate the effectiveness of the proposed method.
weaknesses: I still have the following concerns regarding this work:

1. The paper’s theoretical analysis is significantly lacking. The author’s explanation of how the depth dimension is used for prediction remains unclear. While the author attempts to explain the work through the lens of RQ, I find this explanation insufficient.

2. The experimental setup for the arxiv-images presented by the author raises some confusion. What exactly can similar experiments demonstrate? I had anticipated that **DnD would, through these experiments, show that the text in the generated images is coherent and carries semantic meaning.** However, based on the examples provided, the content appears to remain **disorganized**, with improvements observed only in terms of fidelity. 

3. Additionally, I find the decision to fine-tune models like SDXL or SD3 for **unconditional** generation tasks unclear. Why didn’t the author employ models like SiT or DiT to validate DnD’s effectiveness?

4. I did not find sufficient evidence in the experiments to convincingly show a reduction in VQ loss. While there is an improvement in generation quality, it appears that this improvement is not solely attributable to VQ.


The following are some comments that do not influence my overall assessment:
1. In comparison to other contemporary works, e.g. [1], the author’s approach seems to represent a particular case.
2. I know it would be tough but could the author provide reconstructed results to better illustrate that the VQ loss has been significantly reduced?

[1] Li, T., Tian, Y., Li, H., Deng, M. and He, K., 2024. Autoregressive Image Generation without Vector Quantization. arXiv preprint arXiv:2406.11838.

Meta-review: The paper introduces a new formulation for vector-quantized autoregressive image generation that overcomes complications of existing 1D approaches (namely information loss and computational complexity) by predicting tokens that are expanded depth-wise. This gives rise to a 2D sequence for modeling image content where the first dimension is spatial and the second dimension encodes visual content at progressive levels of depth/detail. By predicting tokens at multiple depths simultaneously, the method reduces the complexity relative to the standard approach of increasing the sequence length, while increasing the fidelity/compression ratio. The approach is validated by a series of experiments comparing to existing state of the art approaches, both for conditional and unconditional image generation. In terms of conditional image generation, the method improves significantly in FID over existing AR baselines. More interestingly, the method does well at unconditional image generation where the images contain fine grained text, hinting that the model is capable of generating text with a sophistication previously reserved for standard language models.

Strengths:
The approach is well motivated by referring to the compression view of vector quantization and the resulting complications. The 2D solution is elegant and simple, and seems to work well in practice, including rivaling state of the art diffusion approaches. The idea of training a joint image-text model purely with images is a potentially exciting direction motivating early fusion approaches. Analyses and ablations confirm the method and demonstrate efficiency gains.


Weaknesses:
Some concerns that the sparks of vision language intelligence may be a bit overclaimed, though authors softened these claims. Also, further analysis/theory could illuminate why the single-stage parallel prediction of depth might improve over two independent precictions as in the original RQ work. 

Decision reasoning:
This paper is well motivated and backed up by strong empirical results. Reviewers were generally positive about the paper and one noted that it is worthy of highlighting the contribution. I agree that this work deserves visibility particularly due to its practical value in improving fidelity without significant increased computational overhead, and I think the community can build on the work to better undestand why a single stage predictor is so effective and to consider additional variants of the idea.

Decision: Accept (Poster)