A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generation
Authors: Liang Chen
Sinan Tan
Zefan Cai
Weichu Xie
Haozhe Zhao
Yichi Zhang
Junyang Lin
Jinze Bai
Tianyu Liu
Baobao Chang
Venue: ICLR 2025 Poster (2025)
Keywords: Autoregressive Model
Image Generation
Vision-Language Model
Large Language Model
Primary area: foundation or frontier models, including LLMs

Abstract: This work tackles the information loss bottleneck of vector-quantization (VQ) autoregressive image generation by introducing a novel model architecture called the 2-Dimensional Autoregression (DnD) Transformer. The DnD-Transformer predicts more codes for an image by introducing a new direction, **model depth**, along with the sequence length. Compared to 1D autoregression and previous work using similar 2D image decomposition such as RQ-Transformer, the DnD-Transformer is an end-to-end model that can generate higher quality images with the same backbone model size and sequence length, opening a new optimization perspective for autoregressive image generation. Furthermore, our experiments reveal that the DnD-Transformer's potential extends beyond generating natural images. It can even generate images with rich text and graphical elements in a self-supervised manner, demonstrating an understanding of these combined modalities. This has not been previously demonstrated for popular vision generative models such as diffusion models, showing a spark of vision-language intelligence when trained solely on images. Code, datasets and models are open at https://github.com/chenllliang/DnD-Transformer.

Published as a conference paper at ICLR 2025
A SPARK OF VISION-LANGUAGE INTELLIGENCE: 2- D-DIMENSIONAL AUTOREGRESSIVE TRANSFORMER FOR EFFICIENT FINEGRAINED IMAGE GENERATION
Liang Chen', Sinan Tan', Zefan Cai², Weichu Xie*, Haozhe Zhao', Yichi Zhang!
Junyang Lin', Jinze Bai?, Tianyu Liu', Baobao Chang Pekiing University AAlibaba Group
University of Wisconsin-Madison
*Beijing Institute of Technology
leo. eeo.liang.cheree wwwn@outlokk..coo
(a) Generation Examples from DnD Tra-fransformer
Largs Over-iap-Ei,
A.I. Paprooesing Dais
aburat parance of the poth mocssut a cmnoss for lese
There are nf passet mock esaponaste vic in
0.773 dst
0.121
012211
B14
1.K
Sow jian, Domolory Saktnis, aumpei' (hawrt -he conl s) cooi density
ii in log()- -Bi 80 001.9. int E
1.15
0.3780
L1M
01-880
01301111
M11 2.3M
38.9
K.3.2
1:60
5.18
1.76175
15.3
(46.3
8
1.91
022003
1770854
23.3 3.380
711 146.7
(I file, am intreaace e a. 2001. with trmiams, of 'nnn Remaaton and 3b culitration waees var of and about paricks and arisasy fish depett mel. njja, mini oi-o ac"d"
Lintuaz
115500
Jeen
39.04
0000
0.8019
00070
3.3000
3000 1.8
3100
012977
180 3-110)
14011 1.
9.10
(b. Diffusion
(b.2) Autoregressive Image Generation
cofdred. Drrott blent HPRiene 'oi
whenr ots Hens thes lo mostniceict efcr enondlandes
the presionsty underworkrnide service and survoinded during had trde union orginatory law services. The proviously larg untonitt worbers duzing bead composttoon fourder Samantan Baanck Ohamaa was held on 15 Septenberr 1911. Germuny had served us Goverroo Centio Barock Ubasa subjected to new more yniogist than whose countess Enited
Herman Maria David (homn 15 Jnne 19823 is an American owimemen and soins cuchia Andy in canser. Later in Mary, he founded the Miristry diserring services level che and resiined tramp puenni ally with the protested flanform of districts to plan the woeds
hiaSlleoss repcrstient; A
Sunenivain
Alend
chile a Hwbey Enor midhight asc
(nd3--991) 196 Dipcin Seilt the 12h:
Ioorerdditaus olies ocst Ho Iceaee, Hisgas
chsiggl tn Ciy conduerras, merfooumm 1n5gef
Chowicoon boqu
Tloopt 2ich-Uss 5 Ahesstyy pas lyes, wi cit berotl
Eanth by Nia. leotes lelrd lovat
9 ahiitalnno brettanrrwste Ause th (Sbirme
Lleeheoolinn beanburt halposss der opvoessheniass Mirddi pehmeat
Tim
with
deciv crance mutirrityy laads with power and struciural influenees Hi was seniee in Malaysian statitioal
muo Kin Obicis
Dusnndeh. di J. jelown:Srrryy Kink,
Hanos cildden
Ben
falicuso Dupa Geese hrom przrlor.
wowuulppiiinnaaeeeeeee liech Bise. Hhoihe Pans Fipllouns,
proceduces Occam in Senteate Distiet. References Swinsmes
LaubliRee Abcalhouure sushi) Aliuk FiRkee casto
U'alale
Univenstyy
iesalienclls. posalarios Tonler
Tuningne ilred isal"] Dilo Eocadihi
modemist the saving into Weslock. As commonly. hut "ramained and improved welded remote and alterumbyy nnvepuution in Orisrbh. The scrse muliipli and modem repeceive an innovative scem: throghh similar raie, such w perfermance which also mecurs and ware unlikely to compare human reactions and those horm
of
wiasti is related to the not
enghächonäste
atdds Wiffugndon Hagnijm
tilti Horid diedbuinnient
huroon trast
Eaennt
incullolpl
months. Eaglish (English)
4248836, che Gaam denwtise
Bay many 1888 Inzell
Englond the sella mints
words
C1201120, duf
DistoElaiie
land
howw doviiledoo linhet des,
but Micshall loonnbuurgh forms meson-linder
peot were 2int 1913 lith
lengrle fly Kifanu ole ing we tam
m megeiicooonn
and. NoGurdozzali lepor0 rodoom
The secies slower traneition
honidwepijul Kile Rith ieathonest
peneral Antaratic prolection agenda and the sementatiomun (and sumarious Eistern Herilage protection). The sutues estrances the woeld of
Meefiik
Sis fitee lec, moketner
sobunit, Ruatsbbwood bosh Fut
beep Jab irs Nabwot iosan A
oonditioss Paul Marning complotee the instrmenation from some commes ocigins Brazilust
D' D'Mosmoo Ramncbaied thep kaed wodiiile, Eepemess leann to Upiis
driig whaas oerirtuc 19201
s096, and archnne Pl Teai
?
Figure Generations from DnD-TTranformest trained on class-conditional Im6g (a.top) and unconditional arXiv images (a.bottom). Unconditional rich-text image generations by trained diffusion I) and autoregressive model (b.2), where autoregressive model has dominating performance, showing spark of vision-language intelligence after purely training on images.

---

Published as a conference paper at ICLR 2025
ABSTRACT
This work tackles the information loss bottleneck of vector-quantization (VQ) au- toregressive image generation by introducing novel model architecture called the 2-Dimensional Autoregression (DnD) Transformer. The TnD-Transformer predicts more codes for an image by introducing new direction, model depth, along with the sequence length. Compared to ID autoregression and previous work using similar 2D image decomposition such as RQ-Transformer, the DnD- Transformer is an end-to-end model that can generate higher quality images with the same backbone model size and sequence length, opening a optimiza- tion perspective for autoregressive image generation. Furthermore, our exper- potential extends beyond generating
iments reveal that the natural images. It can even generate images with rich text and graphical elements in self-supervised manner, demonstrating an understanding of these combined modalities. This has not been previously demonstrated for popular vision gener- ative models such as diffusion models, showing spark of vision-language intel- ligence when trained solely on images. Code, datasets and models are open at httpss//gihubb.oo//hen//////
Tran-foroorrm
INTRODUCTION
The field of autoregressive (AR) image generation is experiencing a resurgence of interest, largely driven by groundbreaking advancements in large language models (LLMss, exemplified by the re- lease of ChatGPT (OpenAl 2022). Because typical AR image generation methods also predict output in next-token prediction manner, this resemblance has sparked significant efforts in two main areas: 1) transferring advanced, large-scale training techniques and expertise from LLMs to AR image generation models (Bai et al., 2023 Tian et al. 2024 Sun et al. 2024), and 2) develop- ing truly multimodal foundation models capable of both understanding and generating multimodal information within unified training framework (Lu al.
2022
2023 Team 2024). These devel-
opments have the potential to lead to more versatile and powerful multimodal AI systems.
A review of the development history of AR image generation approaches reveals significant efforts focused on finding better sequential decompositions of images and balancing reconstruction fidelity with prediction difficulty. Early models. like PixeICNN (van den Oord al.] (2016). generated images pixel by pixel. This approach was later enhanced by using vector-quantized variational autoencoders (VQVAFs) to compress images and model the prior distribution of discrete tokens in a compact latent space (Van Den Oord et al. (017). Vector quantization (VQ) paved the way for notable models such as VQGAN (Esser et al., 2021). DALL-E (Ramesh et al. (022), and MUSE (Chang et al.) (023), and it remains core technique in recent AR image generation models like 2024). and multimodal foundation models like VAR (Tian et al. 2024) and LlamaGen (Sun et al.
LVM (Bai et al. 2023). Unified-IO (Lu et al. 2022
2023 and Chameleon (Team 2024).
However, despite advancements in AR image generation, VQ-based autoregressive methods face two persistent criticisms, especially juxtaposed with latent diffusion models (Rombach et al. 2022):
1) Information loss inherent in the quantization process. Quantization, specifically in VQVAE, introduces significant information loss. With typical configuration (N=8192, f=16), the Informa- tion Compression Ratio (ICR log, N explained in Equation I is just 0.21%, drastically lower 21f=
than the 8.3% of Stable Diffusion's VAIT hindering fine-grained detail reconstruction. Accord- ing to Chameleon (Team] 2024), the authors note that their VQ tokenizer struggles to reconstruct finegrained details like text in images, which we believe is due to the low ICR of their tokenizer.
2) Substantially increased computational requirements for producing higher-quality images. According to Equation Increasing ICR by expanding the latent space (N) is logarithmically limited
/dd vae- ft ft-mse) uses 'The Stable Diffusion VAE hettpa:/hhusee ng face. cc. bot/stabii 1 1 ittai
downsampling factor (f) of 8 and 4 channels, with fp32 tensor precision (log N 4 32).
2

---

Published as a conference paper at ICLR 2025
Denaising Diffusion Pracess
20 Autoregression (DnD, This Work)
Final Transformer Block
Transformer Block O (N-N1)
Transformer Block D-2
28-1
Transformer Black O-1
Ane
1D Autoregression
MnD-Transformer (This Work)
Figure 2:
Illustraiion of the proposed sion. O-i denotes the transformer layer index for the i-th prediction head. Each transformer layer predicts the corresponding depth code, achieving multi-code prediction within one forward pass.
denotes the number of depth autoregres-
and computationally expensive leading to potential codebook collapse and more embedding parame- ters, while reducing the downscaling factor (I) significantly increases computational overhead due to a longer token sequence of O(1/f²) and a higher transformer computation complexity of O(1/f*),
We draw inspiration from the Residual Quantization method (Lee et al.) 2022b), which provides a new dimension for sequentially decomposing the image for better generation quality. However, the proposed RQ- Traassfffre employs two separate transformer models. This structure presents difficulties in integrating current LLMs for end-to-end training. In this work. we aim to solve the problem covering the two mentioned concerns: Can we overcome the information loss of VQ-based AR image generation without increasing overall computation budget in an end-to-end manner?
We propose a novel paradigm for AR image generation called 2-Dimensional Autoregression (DnD) and DnD-Transformer, an end-to-end model architecture. DnD Autoregression introduces new depth dimension along with the original spatial dimension. In the depth dimension, the image patch could be decomposed in any causal coarse-to- fine order, including the residual decomposition (Lee) ctt al. 2022b), Gaussian denoising decomposition (Ho et al.) (2200) and etc. With a depth of d and other configurations unchanged, the ICR of DnD Autoregression becomes dx log reducing the information loss comparing to increasing the codebook size N.
24 more effectively
The remaining problem is how to predict the d times more tokens effectively. We propose the DnD- Transformer. As shown in Figure 2 it inserts multiple prediction heads into the backbone trans- former decoder model predict the depth codes and conduct additional autoregressive predictions in each forward process. Different from RQ-Transformer (Lee et al. 2022b), the DnD- DnD-Transformer does require additional modules or increased sequence length, making applicable any lan- guage model architecture and efficiently generate more fine- ine-grained images.
Our experiments show several interesting results: superior reconstruction of fine-grained image details using residual image decomposition in VQVAEs. disproving VQ's limitations with text-rich images; 2) more efficient and lower-entropy decomposition with Dnl autoregression compared to 1D methods, evidenced by lower training cross-entropy loss despite predicting more codes; 3) sig- nificant outperformance the AR baseline on ImageNet 256x256 generation, achieving up to 1.54 FID and 82.6 IS improvements (XXL model, cfg=2) without increased model size or sequence length, even surpassing larger LlamaGen model trained with longer sequence length; and 4) DnD- Transformer shows that model outperforming diffusion models like DDPM and Stable Diffusion on dedicated rich-text im- age datasets, highlighting the distinct advantage of autoregressive models for multimodal modeling.
conduct accurate language modeling with pure image generation
3

---

Published as a conference paper at ICLR 2025
Frnund Truth (5172)13))
LcSs 0,0027
L2
Lons c 0000
P558 25,68
PSNR 35.27
PSHR 20.56
D:B (AD) 1.2 Loss 0,0014 PSNR 28.69
D=2 (RO)
LMMT DOO L. Loss 6 0322
Ground Truth (2122112
LcSs 0.0218
PSVR 16.01
Figure 3: Performance of our visual tokenizers of different depths. The reconstruction of complex features (i.c., eyes, mouse and text) gains significant improvement as the depth increases.
2 2D VISUAL TOKENIZER AND 2D AUTOREGRESSION
2.1 UNDERSTAND VQVAE AS COMPRESSION
We introduce the basics of AR generation in Section[Aiin the appendix. We can better understand the reconstruction ability of VQVAE from the lens of compression. Let us assume a VQVAE with down- scaling factor f. codebook size N, input image's 's size of H x W, then the shape of the quantized code is h x (H/f) x (W)f). We assume that the code follows uniform distribution. each code has log2 N bits information. Its information compression ratio (ICR) is as follows.
ICR(N, f) (H/f) (W/f) log, N log2 N
H W x 3 log, 256
24
A typical configuration (N=8192, f=16) results in 0.21% ICR. As comparison, JPEG typically achieves a far larger compression ratio from 5% to 10%, resulting in minimal perceptible loss in image quality (Haines) 1992). To increase ICR, the 1D AR method could increase N (might face the codebook collapse problem (Mentzer et al. (023) and the improvement is logarithmically bounded) or decrease f (more effective, but increases the token count quadratically).
2.2 IMAGES' 2D DRCOMPOSITION AND QUANTIZATION
As pointed out by Equation 1. the information compression ratio of VQVAE is bounded by the size of the codebook and the downscaling ratio. Residual Quantization (Lee et al. 2022b) proposes a new direction to quantize the image feature with multiple residual codes to reduce the quantization error and improve the quality of the reconstruction. For a feature map having h x w vectors, RQVAE uses h x w x d codes to quantize the feature map, where d is the depth dimension of the code. For each feature vector v, RQ finds d codes (41, 92.... ..., qa) by sequentially conducting d times residual decomposition and quantization operation ((x)) as finding the closest entry to x from the codebook:
qa = Q(Ta-1), Va = 9d.
TO =
(2)
Consequently, the sum of the residual codes Σd_ feature vector v. thus reducing the quantization error. We generalize this process two-dimensional autoregression (DnD), which extends beyond Markov residual decomposition and can be applied to any decomposition operation, such the diffusion process (Ho et al. 2020), etc.
1=1 9i is expected to approximate more closely the
DnD Autoregression quantizes a 2D feature map m E Rhowe by decomposing it in two directions. First, m is divided into h feature vectors. Second, each vector is decomposed into n codes (q), .... 9n) using a function D" Q) based on a codebook Q. The resulting quantized map has shape h and is predicted in depth--frst-spaaiil--eoomm order. This decomposition could
4

---

Published as a conference paper at ICLR 2025
ImageNNt 256x256
Tex1256
Tex1512 arXiv512
Depth(RQVAE)
iFID
L2 Loss
Code Usaget
rOCR1
2.98
0.11
100%
0.15
0.73
0.14
2
0.93
0.08
100%
1
0.00
0.00
0.00
4
0.60
0.05
100%
2
0.50
0.81
0.49
0.42
0.04
100%
8
0.80
0.83
0.67
SDXL
0.68
0.05
SDXL SD3
0.72
0.83
0.66
SD3
0.67
0.04
0.82
0.83
0.74
(a) Reconstruction Performance on ImageNet
(b) Reconstruction OCR Performance. indicates zero-shot tokenizer trained on ImageNet.
256 256 Validation Set.
Table I: Ablation studies on the reconstruction performance of visual tokenizers. Our trained tokenizers all have a 16 downscaling factor and N 16384 codebook size.
Layerwise Usage Different Visual Tokenizers
Code Norm Distribution Different Depths
1,0
Max Depth (6)
Depth
Depth 1
Code Usage (%)
0.8
Depth 2
Depth
0.6
Density
Depth
Depth
Depth 6
Depth
6.12
Codebook Depth
Norm Code Embeddings
(a) Layerwise code usage of visual tokenizers.
(b) Code Norm Distribution for Tokeniers
Figure Analysis of visual tokenizers.
also be non-Markov, unlike RQVAE. The selection of potentially better decomposition functions is left for future exploration. We use RQ-VAE (Lee al.) 2022b) the implementation of 2D decomposition method in our experiments. D DnD decomposition increases the ICR d times (Equation 3) more effectively than increasing codebook size. The remaining challenge of predicting times more codes addressed by our MnD-Transformer.
still use the residual quantization from Equation as
ICR(NN = dx (H/f) X (W/f) x log, x
log, N
(3)
H x W 3 x log, 256
24
2.3 RECONSTRUCTION PERFORMANCE
We evaluate the reconstruction performance of our trained visual tokenizers with varying maximum codebook depths using the standard ImageNet dataset the benchmark. All images resized to 256x256 resolution. We train the different visual tokenizers using the same training objectives as in Lee et (2022b), and assess the reconstruction Fréchet Inception Distance (rFID) the ImageNet validation set using ADM's evaluation suite (Dhariwal & Nichol 202T). The results are presented Table For comparison. we include the rFID from the VAE of SDXL (Podell al. 2023) and Gaable-Diffusion (Esser et al. 2024). Our findings demonstrate that our trained visual tokenizer achieves an rFID lower than with two or more codebook depths, even surpassing the performance of SD3's continuous VAE with less theoretical information loss. As shown in the example from Figure by increasing code depth, we could reconstruct more fine-grained details the image.
Code Usage. We further analyze the code usage in each codebook layer, with results shown in Figure 4a The analysis indicates that usage generally decreases as depth increases. This is due to the diminishing diversity of code usage as the residual decomposition progresses deeper, resulting in smaller feature norms and more centralized code usage according to Figure 4b) Interestingly, we do not observe signs of codebook collapse with the DnD visual tokenizers, even when using large
5

---

Published as a conference paper at ICLR 2025
Transtomer
Translomer
Transformer Block
short
Trensformer Bork D-(N-1)
Transforme Block (N
Blakk
Translomer Back
Transffrmme
Transformer
[ [x] [a] [s]
DnB Transformer
Parallel Prediction
Vertical Prediction
Figure 5: Different explored multi-token prediction architectures for OnD-Transformer, which are all designed to generate multiple codes with one forward pass.
codebook size (16384), as mentioned in previous work (Mentzer et al. 2023). While they reported much lower code usage (< 50%), our tokenizer achieves 100% usage across all maximum depths.
2.4 VQVAES CAN PERFECTLY RECONSTRUCT RICH-TEXT IMAGES
A prevalent criticism of VQVAE has been its alleged intrinsic information loss problem, particularly its inability to reconstruct images with fine details, such as those containing rich text (Team) 2024). However, we argue that this claim is unfounded. Our findings suggest that VQVAE can indeed achieve perfect reconstruction of detailed images, when provided sufficient data and an increased number of codes used to represent each image. This demonstrates that the perceived limitations of VQVAE can be overcome through appropriate data-centric adjustments and model scaling-up.
rOCR - A New Metric. We proposes rOCR, a novel metric for evaluating rich-text image re- construction. Unlike rFID/L2 Loss, rOCR measures textual recognizability using the Qwen2-VL- 72B <Wang et al., 2024a) visual language model for OCR. The metric computes the Rouge-L L score between recognized and groundtruth text (or original image OCR groundtruth is unavailable).
Experiments and Results. Two rich-text image datasets, Text-Image and arXiv-Image (details in Section 4.1, both datasets' 1K test sets, compared against ImageNet-trained tokenizers, SDXL's (Podell et al., 2023 and Stable-Diffusion3''s VAE (Esser al. (0024). Text-Image was also tested reduced x56x2566 resolution to assess resolution impacts. Table Ta shows the rOCR results, with reconstruc- tion examples in Figures and Results indicate more training data and deeper tokenizers improve text reconstruction. Unlike Team (2024), our discrete visual tokenizers excel in rich-text image reconstruction even compared to continuous VAEs.
used to train visual tokenizers. Performance (rOCR scores) evaluated
3 THE D
Prior section showed DnD visual tokenizers effectively reconstruct fine details like text. However, efficiently predicting the increased number of depth codes (d times more) remains challenging- Existing methods, like RO.rransform,r, use separate transformer for depth, hindering integration with L.M We propose an efficient end-to-end architecture for multi-code prediction.
3.1
DESIGN
Figure shows DnD- Tranfforsfrrmer and its variants: Parallel and Vertical Prediction. Parallel Predic- tion adds multiple prediction heads for simultaneous multi-depth code prediction, similar to accel- crated LLM inference (Cai et al. 2024). However, this ignores the coarse-to-fine nature (Figure 4b) of code distributions, where deeper codes have smaller norms and are more centered. Vertical Pre- diction addresses this by sequentially predicting codes. Adding autoregression further refines this by conditioning deeper code predictions previous ones, achieving the best multi-layer code pre- diction without increasing model parameters or sequence length. Ablation on the structure design is shown in Table from Appendix.

---

Published as a conference paper at ICLR 2025
3.2 IMPLEMENTATION DETAILS
As shown in Figure 5) the DnD -rD-rranfformer enhances the vanilla transformer decoder by adding output heads and embedding addition operation. Assuming the linearized codemap's length is == h x w and code depth is d, the DnD-Transformer performs I forward passes, generating codes sequentially during each pass. After generating for all depths, the embeddings are summed to form the next input token. This allows the model to produce L x d tokens with just L forward passes, improving generation quality without increasing inference costs. The only additional hyper- parameter is the layer indexes for codes at different depths. We use the same architecture as the LLaMA (Touvron et al. 2023) transformer decoder; training details can be found in Appendix
EXPERIMENTS AND FINDINGS
4.1 TASKS AND DATASETS
Image Generation. We conduct standard conditional image generation task with ImageNet-Ik IK benchmark. Images are resized to 256x256 resolution during training and evalu- ation. We sample 50k images with classes uniformly distributed, and compute the FID, IS, Precision and Recall aganist the training set data using the ADM evaluation tool Dhariwal & Nichol (202T).
Unconditional Rich-Text Image Generation. We collect two datasets for this task. Dataset ex- amples are shown in Figure T0Jin the Appendix. Models are trained in a unconditional setting in this task. We aim explore whether the tested vision generation models could understand and generate the complex logical interrelation among the generated elements such as language.
1. Pure Text Images (Text-Image). The dataset is automatically rendered from a portion of English wikipedia (Foundation), consisting of 2.4M images. Each image has original resolution of 512x5I2 and a font size of 32pt. We set a maximum of 100 words in each image with a paddling margin of 20pt. We use the PIILOW library render the image.
2. arXiv Images (arXiv-Imagel we first download the papers in PDF format from arrii xiv. orrg and render the pages to image of A4 resolution (1260 1782) with PDF2IMAGE tool. We then randomly crop ten 512x512 12 image from each pages and finally collect 2M images.
We developed an evaluation pipeline that combines Optical Character Recognition (OCR) and Per- plexity Measurement to assess the quality of generated images, focusing on their textual content. First, we use the state-of-the-art images. Then, we calculate the text's perplexity using the Qwen2.5-72B model, treating it as the evaluator. The resulting score, PPL..r. is compared to the score of ground truth data from the training images to establish a performance upper bound.
Model Owen2-VL-72B to extract text from the
4.2 MODELS
Visual Tokenizers. We train our visual tokenizer based on RQVAEs (Lee et al. 2022b). We train tokenizers with code depths of (1, 2, 4, 4,8} and scaling factor f= 16 across different experiments. We choose the checkpoint with best rFID across 1.50 epochs. Performance comparison of different visual tokenizers is shown in Table [ We follow [ee et al.] (2022b) to train the visual tokenizers. Details of the training of visual tokenizers are listed in Appendix Reconstruction performance of the trained visual tokenizers is shown in Table
DrD-Transformer. We train two size of DnD- TADD-rrasformers across our experiment, namely DnD- Transformer-XXX (I.4B) and TeotTTTranfrom--XXxe (2.5B). Basically, DnD-TTansformer in- herits the LLAMA (Touvron et al. (023) architecture. The XXL version strictly align with the LlamaGen-XXX baseline to be fairly compared. Details of the model are shown in Appendix E
Implemented Baselines for Class-Conditional Image Generation. LlamaGen (Sun et al. 2024 is the major baseline and state-of-the art model for AR image generation on ImageNet. Our imple- mented code primarily refers to the same training codebase for fair comparison. LlamaGen could be also viewed as special version of DrD-Trrasforeer where the decomposition depth equals to 1.
7

---

Published as a conference paper at ICLR 2025
Type
Mode
FiD
Precesiet
Recalft
AUM
Uhhoood Nacell 2017
554M
10.94
101.0
0.00
0.83
Difllkke DiflokkeRRguun
CDM
Pprtie 2000g
4.88
158-7
VOGGAN
Dour Doergall 2011
1.48
5:20
260-3
3.8
1:55
134.0
X3L
AR-Raserted
Lirratio (clen2) Senddtal p3.3
2024
5.15
4.21
325-2
0.40
Larautee 18 (381 200 parte
M044
3.18
2.81
311.6
0.88
0.50
VAR or 1000 (a2h tadtannie comptigg
2.06
1.75
190.
0.00
agect sangling)
spoduo)
Larestam XXL (olgma)
1.48
1.67
345.1
0.89
1.35
Laraaii--X XXL (c1go))
4.12
Delb (ob-iinniionen---e XXL (clen2)
1.45
1.38
0.56
AR Snpappeeeeted
Dedh trasesmmer XXL (clemii))))
1.48
2.78
1.55
276-2
0.82
0.57
0.56
Dnth Uiasiorme XX3L (cfgm2)
2.58
1.77
0.55
0.54
Delh Trasssomer XXXL (sgg)1.7)
258
1.31
275.3
0.83
0.58
Duth Tranctttmmmeee XXXI (rigal.5)
7.48
3.53
0.80
0.59
Table 2: Model comparisons on class-conditional ImageNet x56x256 benchmark. The "Re- ported" results refer to Sun et al. (2024). The "Implemented" results conducted in this work. indicates that the model is unorthdoxly trained at 384 x x384 resolution, which requires 2.25 times longer sequence length compared to implemented models. "cfg" means the scale of classifier- free guidance. The number of depth autoregression is 2 for DnD--Transformers.
Comparison along Training on ImtgeNet
Comparison of PPL along Training on Texb-irage
LlemaGen XXL
Det Tra rasfomee-000
2000
Tempentredd GoursTehh terpestre1
1600
PorplexiyyOOC
1200
800
400
18
Epochs
Epochs
(a) Comparison of FIDs along training.
(b) Sampling PPT. on Text-Image along training.
Figure Curves during training.
We select multiple diffusion models as the baselines, including DDPM (Ho et al.) 2020), Stable Diffusion XL (SDXL) (Podell et al.) 2023) and Stable Diffusion v3.0 (SD3) (Esser et al. 2024]. For DDPM, we train the model on the dataset from scratch. For SDXL and SD3, we finetune the checkpoints from the official websites.
Implemented Baselines for Rich-Text Image Generation.
4.3 RESULTS OF CLASS- LLASS-CONDITIONAL IMAGE GENERATION
As demonstrated in Table 2 our OnD--Transformn significantly outperforms the 1D autoregressive baseline LlamenGen across various scales and generation evaluation metrics, including FID and IS. This superior performance is achieved while maintaining the same number of parameters in the back- bone model, based on our reported and implemented results. It is noteworthy that our 2.5B model, trained with a sequence length of 256, even outperforms the 3.1B LlanaGen model, which was trained with much longer image sequence length of 576. This result demonstrates that the DnD- Transformer can effectively predict greater number of tokens within a shorter sequence length, highlighting its significant potential to revolutionize the one-dimensional autoregressive paradigm. We randomly sample some generation results as shown Figure I and compare the generation per- formance with ID-AR in Figure illustrates the effectiveness of our approach to generate high-quality images.
[I and 13 from the Appendix. The comparative analysis clearly
4.4 RESULTS OF RICHITEXT IMAGE GENERATION
Generation Results on Text-Image. A DnD-Transfore Transformer (depth and DDPM model were trained on the same text-image dataset. Comparing 250 randomly sampled images from each, the AR model significantly outperformed the diffusion model in generating coherent text (lower OCR
8

---

Published as a conference paper at ICLR 2025
tigh salle's geanded orpha
(ie Star Mass
proggaag tartings, Thes as chaaasoe
inalsated fast Aar beakke
nc payyens pettie bot doed
SD3
Halk G
28"
The aad pkllord eduikdd
colla ArCAk
Kely tha 8 docoodd of 3 vameel
6d1 tepti:
teckwwwrg Coapriic nowerack
kaarkk leanverio angreted Repar
Ltd pobal afiicia
rolkke Inkifs Sociel
DnD
Thassooo
Appasia Doarn Li L
(ee dest.
Figure 7: Comparison of Unconditional Rich-Tex Image Generation on the more complex arxiv- Image dataset. SD3 is hard to generate valid words, while unD-Transformer demonstrates an ability to generate semantically appropriate phrases. as marked in blue. More baselines are in Figure 14
Leaming Curve for Different Number Prediction Layers
Leaming Curee for Deffrrnt Number of Prediction Layars
Traning LOG5
Training Loos
Inagelar:
Step
Step
(a) Training Loss for DrD-Transformer trained with different number of prediction heads.
(b) Training Loss when trained on different domain datasets.
Figure 8: Analysis of code depths and domains during training OnD-Transformers.
17 and 18 This suggests the AR model's dis- crete token reconstruction enables effective autoregressive modeling. We also find that with lower sampling temperature, the model would generate text images with lower PPL just like LLMs. Con- versely, the diffusion model's simultancous generation hinders text coherence.
perplexity 6b] Generation examples 1
15
T6
Generation Results on arXiv-Image. An 8-layer visual tokenizer and corresponding DnD- Transformer trained on arXiv-Image outperformed diffusion model baselines, generating more valid words and phrases (Figure 7 However, arXiv-Image generation lagged behind Text- genera- tion, suggesting joint language and figure modeling is more challenging. More results and basclines are in Figure 14 and 19. While SD3's VAE reconstructs arXiv images well (Table its generative performance is inferior to DDPM and AR, suggesting its latent space is less suitable for language modeling comparing to pixel or discrete space.
A Spark of Vision-Language Intelligence. Autoregressive (AR) image generation exhibits a marked advantage over diffusion models in producing text-rich images, as demonstrated by our results. The pixel-level language generation inherent to AR models facilitates this capability. De- spite limitations imposed by our current training data and model size (preventing direct comparison with large language models), these findings suggest a promising pathway towards vision-language intelligence where language understanding emerges directly from visual perception. Further- more, our pure image learners display behaviors mirroring language model issues such as repetition and hallucination (Figure 20). implying the potential for integrating pure language modeling into a unified autoregressive framework for joint lisionllanguag image modeling.
4.5 TRAINING BECOMES EASTER WHEN PREDICTING MULTIPLE CODES, SAMPLING NOT
Deeper DnD- TrDnnrssffrrmr codes achieve lower cross-entropy loss during training (Figure Sa indi- cating lower entropy image decompositions. However, despite this, increased depth doesn' 'i improve
9

---

Published as a conference paper at ICLR 2025
ImageNet generation fidelity, possibly due to the larger sampling space. Exploring this multi-depth sampling space for better generation is a promising research direction.
4.6 AR TRAINING LOss FOR DIFFERENT DOMAINS ALIGN WITH INNER RANDOMNESS
Training loss for the same DnD-Transforme varies significantly across datasets (Figure Sb being notably higher for ImageNet than rich-text images. While rich-text image loss nears that of LLMs, ImageNet loss sits between text and natural image datasets. The AR model's LLM-like training suggests it learns language from visual input alone, implying language's visual representation has lower entropy than natural images, easing the learning process.
RELATED WORK
Image Generation with VQVAE, The vector quantization (VQ) method has been pivotal in
the development of generative models (Chen et al., 2024, Ramesh et al.] 2021 Yu et al. 2022]
Chang ct al., 2023), which achieve image generation through the prediction of discrete image to-
kens. Efforts in this area focus on two main directions: the optimization of image tokenization
techniques (Esser et al. 2021; Mentzer al. 2023 Yu et al. 2023 2024| Weber et al.] 2024]
You et al. 2022). and the strategic planning of effective decompositions of image tokens, such as MaskGit (Chang et al.. 2022) and VAR (Tian et al.] (024) incorporating a diffusion loss such as MAR (Li al. 2024). Meanwhile, alongside the advancement of large language models, there is growing interest in autoregressive image generation, which predicts image tokens sequentially (Tian) et al. 2024 Sun et al. 2024). Recent research has also focused on developing multimodal founda-
tion models (Lu et al. 2023] Kondratyuk et al., 20244 Wang al. (024b) that integrate both under- standing and autoregressive image generation capabilities. They typically convert images or videos into sequences of discretized tokens and train over combined eex-imaaee/vide token sequences within the AR modeling framework (Lu et al. [022] [ai et al.) [203]] Xie et al., 2024] [eam (0024) However, these models struggle with inherent information loss during the image quantization and the significantly increased computational demands when generating higher-quality images.
Rich-Text Image Generation. Despite recent significant progress image generation, the task of rich-text generation within images remains persistent challenge (Chen et al. 2023b Ma al.
2024
OpenAl, 2024). 2024). Most advancements have been witnessed in diffusion models (Betker et al,
2023
Saharia al. 2022b a), these models either leverage large language models enhance the
character spelanterpeellingg capabilities of generative models (Saharia et al. [022bb Balaji et al.) 2023 Saharia et al. 2022a) attempt to explicitly control the position and content of the text using additional
pervision from different modules (Tuo et al., 2224]] Yang et al. 2023 Liu et al. (024)). However,
most diffusion-based methods have primarily focused on text rendering Chen et al. (2023a b); Balaji
et al. (2023); Saharia et al.] (2022a) in image generation, often limited to generating short words for logos and posters (Yang et al. [2023] Ma et al.) [023] 2024). The full potential of rich-text image generation remains largely unexplored. Our methods, which build on the foundation of DnD Au- toregression, show substantial progress generating rich-text images in an unconditional manner, highlighting the feasibility of conducting joint vision-language modeling tasks using purely images.
6 CONCLUSION
This paper investigated the limitations of autoregressive (AR) image generation methods, partic- ularly the information loss and computational burden associated with vector quantization (VQ). We introduced 2-Dimensional Autoregression (DnD) and novel end-to-end architecture, DnD- Transformer, which leverages depth dimension autoregression alongside the spatial dimension to mitigate these limitations. Our experiments demonstrate that MnD-Transformer achieves significant improvements in image quality, outperforming strong baselines like LlamaGen without increasing model size or sequence length. Notably, DnD- TADD-Tffformrr showcases emergent vision-language intelligence, generating text-rich images unconditionally, a known weakness of diffusion models. These findings highlight the potential of DnD for efficient and high-quality AR image generation and underscore the promise of this approach for advancing multimodal foundation models.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
We thank all reviewers for the valuable advice. This work is supported by the National Science Foundation of China under Grant No.61876004.
REFERENCES
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. arXiv preprint arXiv:2312.0078. 2023.
Yogesh Balaji. Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Mika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers, 2023. URL
https://arxiv/ org/
absp////////////ea/////////aaeea abs abs/aaas..111111 01324
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e33 pdf. 2(3):8, 2023.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple 11m inference acceleration framework with multiple decoding heads, 2024. URL
wttpxxv..rg//////////// prttps://arxinn http://awxxvv../..aa...o abs 2001///////1177
Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-1125, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- eration via masked generative transformers. arXiv preprint arXiv:2301. 000004, 2023.
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser: Diffusion models as text painters, 2023a. URL httpss:/aaxxv..orrtttttt abs 2305 10855
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser-2: Unleashing the power of language models for text rendering, 2023b. URL
arg/abs/ 22311 16465
Liang Chen, Zekun Wang. Shuhuai Ren, Lei Li, Haozhe Zhao, Yunshui Li, Zefan Cai, Hongcheng Guo, Lei Zhang, Yizhe Xiong, Yichi Zhang, Ruoyu Wu, Qingxiu Dong, Ge Zhang, Jian Yang, Lingwei Meng, Shujie Hu, Yulong Chen, Junyang Lin, Shuai Bai, Andreas Vlachos, Xu Tan, Minjia Zhang, Wen Xiao, Aaron Yee, Tianyu Liu, and Baobao Chang. Next token prediction towards multimodal intelligence: A comprehensive survey, 2024. URL https::/
org/abs ha/aaabb22222 2412.42222. 18619
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8748 8794, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recogni- tion, pp. 12873- 12883, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024.
Wikimedia Foundation. Wikimedia downloads. URL https:///umps:: http:////wmnmssiiiiiaaa...o
Richard F Haines.
The effects of video compression acceptability of images for monitoring life sciences experiments. volume 3239. National Aeronautics and Space Administration, Office of Management... ... 1992.
11

---

Published as a conference paper at ICLR 2025
Jonathan Ho and Tim Salimans. arXiv:2207.12588 2022.
Classifier-fre
diffusion
guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-660 6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):224-2228 2022.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hor- nung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, Krishna Somandepalli, Hassan Ak- bari, Yair Alon, Yong Cheng, Josh Dillon, Agrim Gupta, Meera Hahn, Anja Hauth, David Hen- don, Alonso Martinez, David Minnen, Mikhail Sirotenko, Kihyuk Sohn, Xuan Yang. Hartwig Adam, Ming-Hsuan Yang, Irfan Essa, Huisheng Wang. David A. Ross, Bryan Seybold, and Lu Jiang. Videopoet: large language model for zero-shot video generation, 2024. URL https://erxiv.. wtwp:iivvrgg/b////////o/
14125
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523-11533 2022a.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive im- age generation using residual quantization, 2022b. URtttts::///axiit http:/axxxvv...//bbba//e/e orggaa///........ 01941
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image gen- cration without vector quantization, 2024. URL holl:http://// //arxiv wrgg 11838
Zeyu Liu, Weicong Liang, Zhanhao Liang, Chong Luo, Ji Li, Gao Huang, and Yuhui Yuan. Glyph- bytS: customized text encoder for accurate visual text rendering, 2024. URL https://
arxiv. arxiv.org/aae abs
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.089166 2022.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17112. 2023.
Jian Ma, Mingjun Zhao, Chen Chen, Ruichen Wang. Di Niu, Haonan Lu, and Xiaodong Lin. Glyph- draw: Seamlessly rendering text with intricate spatial structures in text-to-image generation, 2023.
URL httpp:://axxi.. org www./b//333333 17870
Jian Ma, Yonglin Deng. Chen Chen, Haonan Lu, and Zhenyu Yang. Glyphdraw2: Automatic gen- eration of complex glyph posters with diffusion models and large language models, 2024. URL
https::/arxiv..om www.///////////// ab////////22 02252
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: Vq-vae made simple. arXiv preprint 255500000500 2023.
OpenAl. Chatgpt. httpps///opent..co////////aaaatt 2022.
2022.
OpenAl. Hello [01-10-2024.
2024. Accessed:
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4208 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint 2rX.w:2307.01952. 2023.
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre- training. article, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, lla Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot -iitae generation. In International Conference Machine Learning, pp. 8821-8831 PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sal- imans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image dif- fusion models with deep language understanding, 2022a. URL https:://aaxii> org/ / 22C5.11487
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-36494 2022b.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation, 2024. URL httpst:
flaxxiv.org/bbaa 2406. 06525
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.
arXiv preprint
vrXiv:2405.09811 2024.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.00990 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and
efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023.
Yuxiang Tuo, Wangmeng Xiang, Jun-Yan He, Yifeng Geng, and Xuansong Xie. Anytext: Multi- lingual visual text generation and editing, 2024. URL 03C54
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelenn decoders, 2016. URL https://
www.iv.orgabb///o.. /1606. 05328
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz. Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024a. URL
12191
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang. Emu3: Next-token prediction is all you need, 2024b. URL
18869
13

---

Published as a conference paper at ICLR 2025
Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang- Chich Chen. Maskbit: Embedding free image generation via bit tokens, 2024. URL hetps:
//axxiv..roo www.vvorg//aaa. httgg//b//2400/ 16211
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou, Show-o: One single transformer to unify multimodal understanding and generation, 2024. URL httth:://axxiiy www. /bs/
2408 12528
Yukang Yang, Dongnan Gui, Yuhui Yuan, Weicong Liang, Haisong Ding, Han Hu, and Kai Chen. Glyphcontrol: Glyph conditional control for visual text generation, 2023. URL https:
///axxiv.orr/aas/2305555 18259
Tackgeun You, Sachoon Kim, Chiheon Kim, Doyup Lee, and Bohyung Han. Locally hierarchical auto-regressive modeling for image generation. In Proceedings of the International Conference on Neural Information Processing Systems, 2022.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, al. Scaling autoregressive models for content- rich -oxte generation. arXiv preprint arXiv:2206. 2(3):5, 2022.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, el. Language model beats diffusion- tokenizer key to visual generation. arXiv preprint arXiv:2310.0573: 2023.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chies Chen. An image is worth 32 tokens for reconstruction and generation, 2024. URL (RRLhttps:: //atxiiv//rxiv. org/abs/ 07550
A PRELIMINARY: AUTOREGRESSIVE IMAGE GENERATION
In this section, introduce the fundamentals of autoregressive image generation. The pipeline is rooted in the Vector Quantized Variational Autoencoder (VQVAE) (Van Den Oord et al. (2017) and the autoregressive Transformer (Vaswani et al. 2017). This approach has been adopted from the early DALLE (Ramesh et al. 2021) to the latest ClamaGen (Sun et al., 2024).
A.1 STEP1: TRAIN THE VISUAL TOKENIZER AND TOKENIZE THE IMAGES
Images initially exist in the pixel-level RGB color space, which consists of little semantic infor- mation and makes it challenging to directly model prior knowledge. For example, an image with resolution of 256 256 comprises 256 x 256 x 3 196, 608 distinct values, representing the individual red, green, and blue intensities for each pixel. The large sequence length makes it diffi cult to train in autoregressive manner similar to language models' technique. Vaan Den Oord et al.) (017)) proposed the Vector Quantized Variational Autoencoder (VQVAE), which significantly alle- viates the problem. It downscales and tokenizes the image from the original sparse RGB space into dense and discrete representational space (codebook) Q by finding the nearest entry. The VQVAF is typically implemented in an encoder-decoder architecture, with primary training objective being to minimize the image reconstruction loss. You could refer to Van Den Oord et al. (2017) for details in training standard VQVAE.
A.2 STEP2: LEARN THE PRIOR DISTRIBUTION OF IMAGE TOKENS
Having tokenized the source images into discrete tokens and trained a visual decoder to map these tokens back real images, the next crucial step is to learn the prior distribution of the discrete tokens. This distribution enables the sampling process, which essential for generating new images. AR Image generation generally first linearizes the h image tokens q E in raster scan order and formalize ID sequence (q), 92, qa ..., 9 for the transformer /Vaswani et al. 2017) model to learn.
14

---

Published as a conference paper at ICLR 2025
During training, the training objective is the same GPT's next token prediction task (Radford et al. 2018), that the model is required to predict the next image token given the previous tokens and class or text conditional tokens I[ P (qt q<t, c). After training, we can generate images w tokens from the model. The sampled 1D sequence of image tokens is then reshaped to 2D code map with height h and width w. This reshaped token map is subsequently fed into the trained VQVAE decoder, which reconstructs the final image from the code representation.
by autoregressively sampling
Classifier-Free Guidance As technique enhance the visual quality and text-image alignment, classifier- free guidance (Ho & Salimans 2022) has been adopted across the diffusion models (Rom- bach et al., 2022, Podell et al.] 2023). VQ models (Chang et al. (023) and autoregressive mod- els (Sun et al.) (024) for image generation. During the training, the model is exposed to data with and without conditioning: the conditioning randomly discarded from fraction of the training samples. We have implemented this approach in our model as well. Specifically, during training, we randomly replace the conditional embedding with a learnable unconditional embedding in 10% of the cases. At the inference stage, the logits lg are recalculated for each generated token. We form the lg by subtracting the unconditional logits &u by conditional logits & with the guidance scale t through the following equation:
Eg = c (le -a) xt
(4)
B TRAINING DETAILS OF VISUAL TOKENIZERS
We follow (Lee et al. 2022b) to train the 2D tokenizers with residual decomposition a combined objective of 12 loss, GAN loss and perceptual loss. Codes from different depth share the same codebook. We train all tokenizers fixed learning rate of 4e-5, total batch-size of 256 for 100 epochs and select the one with lowest validation loss as the final tokenizers. We conduct all training on 8 A100 GPUs.
C RECONSTRUCTION RESULTS OF TEXTS
Figure shows the reconstruction result on arXiv images of different visual tokenizers.
1 Rabiel marks
2. Itelated merbs
2. Related works
Ita tange guandaas Saappemmp (3) cunz- on to det feos coand tsuael ther cnd mp ato casel eomernee da dup tunkrg en .ts 42i? ved. is ds toundia Acmma Naund Gld Et EL. th, U1 eiis onth e gasneu In usge macsies atl dischiaccs ddrggeti bateaa und axi coanses to cSn. peztil ar (1),
Fled bmagge gessueizo Thst kmaas (TVI)
Test-to-inag youenation Test-loiinage (T20 fio modds lave colivall and lesited foooug sorael thor model avthrnettrns in the deep leaming era An early work is Generative Adveourial Newwor (GAN)) Ds, 16, 20], wasch trais genentor for image gescurionn and dectiiitutor to distngashh betwoee red and generate im apo, paralil dabe noc 111, 4), 35, ni] snceg odeest. Anotier caegany generation
fin taable laxn coulnoi and benard frniigh scoord
the stuli sithleedooss le She daos fetadhey fei. Sa ooly
eods ls the Dicorettor 5dittonale Rtasect (CALD TI, 1.S
212 viddi anoda a patsedex Jae Itage getattatis sid a tkonttionns diecingthn bateese wiil getuoured to Ki] vering
oh mssg obee) Austa aree geszala cxeit lpwars vercaandd condes
eptt la jandki frila one (T), 10,
dieast. Anetice dunoeeay at panmmiiee renoels dicsligo INAADe
develops
bech eachelocel
eoodes (VAt5)) 121,
optimize
bis Eifinor teut (MM) (.) 0U/4) lan sesuund eadre 1?' thb tuard passs vuib cas dntea mam. will at camo cam dnavurtas dax dinno and moe ged muppi datt (21)
klece Pvilasion Mdededs 03sinn E3-AA
Mos Difissoo Macels (DMs) 36, enerraaion on the soer-df-bboom (SFIAA) for Bruqg
di to DA53 brus Cate
bate devipeei shile iim (S077A) losqg tlee-
oobien 1111. Dies ux Hahual as puloute hanger plugeer
untion [15]. DMs tainal pescate imagcs prngec- sirely from cardon moise. with the ability capaac moce
stell lever saadesn totes vou fee abdln se diymin lest
duustii thes Dut Se and calbion moot walvale difffly 1711.
diserry that Go and achiree good sample quality II. Latest Exfieaion Model [+1] are Bather ecinement taat performs the dittukoe process in acompact lacer space for more efficency.
Laze INCsuss cusbis E11 as todio vntiaauus nul poitoos duuttiure an paacs aamppe exme upuee oo mure dthoseag
uiser Eacteder Neetth 133) au a tanoss enbtermeti Vins podiets de dit fee process 18 onagee banne space for trere offieneee.
No arXiv Data
Add arXiv Data
Add arXiv Data
D=1 (VQ)
D=1 (VQ)
D=8 (RQ)
Figure 9: Reconstruction Results of Texts. With training data and enough depths of codes, RQ visual tokenizers can well reconstruct the text in the images.
15

---

Published as a conference paper at ICLR 2025
Model
Parameters
FID
Precison
Recall
ID
1.4B
4.12
266.9
0.83
0.49
2D Parallel
1.4B
6.32
232.1
0.79
0.44
2D Vertical
1.4B
3.18
289.7
0.83
0.57
OnD-Transformer
1.4B
2.58
295.6
0.83
0.56
Table 3: Ablation of OnD-Transformer Architecture on ImageNet dataset. All models follow the same training setting in Appendix E
D ABLATION ON DND-TRANSFORMERRS STRUCTURE
E DETAILS OF HYPER-PARIEE PARAMETERS OF DND- TRANSFORMER
Table shows the hyper-parametest of our trained models. The XXL model has the same set- ting as in GPT2 (Radford et al. 2019) and LlamaGen (Sun et al. 2024 for fair comparisons. For unD-Transformer with multiple prediction heads, the prediction layers' indexes are [39, 48) when there are two heads, [39, 42, 45, 48] when there are 4 heads in the ImageNet experiments, (27, 30, 33, 36, 39, 42, 45, 48) when there are heads
amXiv-Image experiments.
Model
Parameters
Layers
Hidden Size
Heads
XXL
1,4B
48
1536
24
XXXL
2.5B
48
2048
Table 4: Model sizes and architecture configurations
All transformer models were trained using settings similar to LlamaGie (Snn et al. 2024): a base
learning of 10-4 per 256 batch size, the AdamW optimizer with 81 0.9, B2 0.95, and weight decay of 0.05, along with gradient clipping at 1.0. A dropout of 0. was consistently applied the input token embedding, attention module, and feed-forward network (FFN) module. Similarly, a dropout of 0.1 was used for the class condition embedding for classifier free guidance. Training was performed for 300 epochs, and the final checkpoint was used for performance evaluation.
16

---

Published as a conference paper at ICLR 2025
F
EXAMPLES OF RICH-TEXT DATASET
Figure show examples from the collected Text-Image dataset and arXiv dataset.
albedo), albodo refers to the entire
Gairi hoaes in the corraction betwen tien. This iouls in the fullowing way of dhifning the canteatt
m). This spectrum includes visible light (0.40.7 my, which explains why surfaces with a low albedo appear dark (e.g., troes abcorb most radiation), wherens surfaces with high albedo appear bright (e.g. snow reflects most rediation).
Aboi pino
S (realttt
suth idea deriaing combisation surounding langung
(Besgio leamingg cnd graph reprsomtiion ng
lecalbeti feedhack positive
feedback climate process where a change in the area iec caps, alters the albedo
Interstingls,
ginciers, and and surfice lemperature of plunet.
Equxioo 9, suggesi exudly same
defining contest.
ssing lunguage modeligg ing. each
Ice is very refleetive, therofore
cuaclyy serving crnteal exprccniaiann
Text
arXiv
Figure 10: Data examples in of the collected Text-Image and amXiv-Image image datasets.
G GENERATION RESULTS OF DND-TRANSFORMERS
LlamaGen-XXL (d=1)
DnD-Transformer-XXL (d=2)
Figure 11: Conditional generation comparisons between LlamaGen-XXI. and OnD-Transformer- XXL class "golden retriever" from ImageNet. We random sampled 16 images with cfg=4. DnD- Transformer generates images with higher quality than the ID AR model.
H EXPERIMENTS WITH DIT
We conduct additional experiments on DiT-XL model (the largest model supported by (Peebles & Xie 2023)) to conduct unconditional rich-text image generation. We use the VAE from SD3 (Esser al. 2024). which has better text reconstruction ability than the RO-Tokenizers used by DnD- Transformer. We use the same training setting as (Peebles & Xie 2023). The results are shown in
Figure 21 and Figure 22
17

---

Published as a conference paper at ICLR 2025
LlamaGen-XXL (d=1)
DnD-Transformer-XXL (d=2)
Conditional generation comparisons between LlamaGen-XXI. and DnD-Traasffrmer- XXL on class "volcano" from ImageNet. We random sampled 16 images with cfg=4. DnD- Transformer generates images with higher quality than the ID AR model.
Figure 12:
LLamaGen-XXL (d=1)
DnD-Transformer-XXL (d=2)
Conditional generation comparisons between LlamaGen-XXI. and DnD-Transformer- XXL class "husky" from ImageNet. We random sampled 16 images with cfg=4. DnD- Transformer generates images with higher quality than the AR model especially for the more complex eyes of husky.
Figure 13:
I TRAINING/IFFERENCE BUDGETS
We compare the training/inference budgets of OnD-Transfomen and different baseline models as shown Table5] DnD- Transformer almost does not introduce an increase the number of param- eters and inference/training budgets compared to the baseline LlamaGien architectures.
18

---

Published as a conference paper at ICLR 2025
DDPM
sal
SDXL
pean bee
peshardos as handii quass
porpoce And di so of
SD3
Horan car
hove 200
Beaa-bh orggated Bejer rolldee D3, HAlls, Social
DnD-Transformer
Agpasta Beran La
Figure 14: Comparison of Unconditional Rich- Text Image Generation on the more complex Xiv- Image dataset. All models are trained on the same dataset. The generated images are all in 256x256 resolution. Diffusion-Family models are hard to generate valid words, while DrD-Traasformer demonstrates an ability to generate semantically appropriate phrases, evidenced by the correct clause "it should be" observed in the second example.
(maaenne((56x2566)
Total Training FLOPs
Num-Parameters
gFID
InferenceTimme(sccene (inesssccTmmsseccn/mmmng))) Training- -Tim (minutew/ppoch)
DnD- TonnTTTannrmeeeeeaeee depth:))
1.0xx (2.57e17)
1.44 B
2.58
1.23s
23emin
MiDD-Tamsommenddppple
1.04x (2.62c17)
1.45 B
2.53
4,48s
25min
Ganaaeeneepph-11 1)
Ix (2.52)77)
1.43 B
4.12
4.05s
22min
(Pebles 2023
2:27
15.45s steps)
SD3 Bsser ef al. 2024
3B
10.78s (28 sleps)
Table 5: The training and inference budget of different models.
J COMPARISON OF DIT AND DND-TRANSFORMER WITH TOKENIZERS BOTH TRAINED ON PIL-TEXT.
We compare the performance of DiT and BnD-Transformer by training their tokenizers the PIL- Text dataset for the same number of epochs and subsequently evaluating their generation capabil- ities. For the VAE tokenizer in DIT, we employ VAE architecture identical to SD3 (Esser al. 2024]. configured with a scaling factor of and 16 channels. The training objective combines MSE loss with KL. divergence loss. Our trained VAE achieves impressive rOCR score of 0.88 on the Text512 2 dataset, surpassing both our best RQVAE and the VAE from the SD3 checkpoint. Follow- ing this, we conduct unconditional generation experiments with DiT, with the results presented in Figure 23 Despite utilizing VAE trained on exclusively rich-text images, the LDM struggles with this task, significantly underperforming compared to our DnD- MoD-Transformer.
19

---

Published as a conference paper at ICLR 2025
saofiind Sari Kaupgii will Elesaa Utubeeede oleea wapplaly keeg was bectheendsstgee renng odor Soc Crreusietiibaale tinusbey Dealle Surmatry'
thintigse (d 100cotylin est
los NE1 destoning (Ras hi finsd
aBoulie Eloo Nigy une, Ajnellis Pepapboridlion Eeguithond oar. liber Stonetrkiian Bacball), Cooildienap ha daruining poold ity end Reid oipasion Geedib oaf
Guodeg bytis sringgiig 1fv dill ljpaeta "Aaallite Sakiky|
Rodiiliuuier as the ascredifiidd au sng belistremey
Hossis citppet un aftoe 11l
Audingo borth coullad de tiall
Re
Docessel besuggemmry It quirppus
Esboratsokos, griltis Bollias's
Zods leausl's alectionn daude" s:1
agginds Aupspery "wooe" Mot
beow Jaciolly goop sincklivvel
Cantrnn coazce Thes ona 59.6
amsolllllanie
Bebas Uershow Clanter "lpball
Dhrale-iechile iil HBWW Jualt,
nagReggy CSX (Tedicos
Lang hillansoon NNtoell last: Dens
Treseraliiee' remvet--
booboote Notal (lll excil
pperecti Cpeoejanng Holjor]
3nd 5- siigha the Caccia bay?
Loffjeeeeny Ii--
basdls so aweosstyy stu eddt. vuigine
NPAc giaal rocomachig
obeatin Dew stal- hakii the Mol'ysetices
(302mmra ter Sanib Hadions laww
Spet
weu in gen the ftammee
arS tili in fordloodeo
cpssnill Dossddn
skk Subejkell lievali
Phurs citer lleee "smor whs
30 1BIS beak (ittel Bill
Sistay lnd (andeck) Secuiiass
fallozadkniy citsaeanna Bilhiso Pecy ytars Seap hilbad.
wordaniza tory bessnbbeees Raa
Jelgeed 1
(Marrall tINa A Whaaghh
feeily
Pops lec Hnaseded boll Bil I
DDompooee pubitji-1 II
(1082 PIET! (pboogeett nee Iodker dodiiallin
Prioctiitedd lkok aredin
Jobed Banccoon Kansiall,
mallaniel plae mowcloo bur" bebedkonnee" quen cod waalaggo, Upurtdaa fass "Dnjmrihhar Nolufyyees frrepethh
poi leshi log (436 bokt 1-
As mostac 191 manial
tupelo 1yMtz 55420ND
Appll
Kepumepitaells:
sorigg-- anoot (- wigg
glox
2112 Pldd 110
"pydtremnatte moder
boo! p-06- 18 sllee Itl neceee
[a] 15800t (illbey Dol g I1
Roenll mnveiin itiiguaal [-7] II
SecogliPer Tistrid, Oge Dale
titures eoss wenttn Ps
Opending (Acaatist Dlai
Jowynaie alound Aunght sighssallio ver ceela the hander and Depaery DNiSgnae bith, dirmon: hot go. Ageese Thonnigg sooouslapy
Loemoca boolleei
(Stce Berammgoo
trevaceer'
atetiie Fikerood's ploversr
Risns H4g
IIl
ABlSSKS bi'e limeellen Afbell
Scaw- till wess veltocriiglou
Il I1B oveel in Jcoweht I EI
1901 PMhagg sl adcoumee
Schadllimess sugiooing..
Socee Labars ant we booti,
offie lenktt Vorth inecab e Sel za es 2 coooooorr courin goor Bittnears,
Piltccoff ercccal Rittrre
Mamiikle Gpee, lroogine Dhr
rineriraboo wibtttes Pore un soidiad werdiia lick plaider
namocetos semicon icdksh
(933 made atsmoderrssd fall.
(oomeg
be
sevariaks Kllazng
I, Wartes fetaraaCa aaareeee hilldies congsaaphnttnng ecspal by filma,
Jouns 906 wolixfis sead
sinceggs..on and buiel athlryyhho
Inkuat Kiee hndoomapaaii Wayer
Wies lands Thookran latheloat
Bllbiecdekk ofd Pulfxtiee)
W crunn teeolbeeeeegge oflepel- nios sial bevecakl Calpenat
lelegaoo). Henalo cours.
"srichen: View tempetitsbbeess
Bel (3,355]
pesiete
Nile yeure Wargey equmirum
Meen baster
spereso
lic VIM
Chateis bays sigl
diiika MDD Lafles i Destomii
PST))
(ree mags trae conthomddiiar
sant Molds haut filuooke Sie
iailss unmimaave wwo opdllers
loddagg
oogginet
Smull Beokbrlanils offeuecca
Xueveli of tempoor laoe the
Tonter Lecney (likkggooo uppler Ciglti Geclard rnfred paay
and quncaatennaee lisierrare
plyfiuRIS of RReaa
Bong un Thase onll cited Rose
(wet do balk thyrydi betuniedd
OrdaniggNi worert
5 hy'dnd absconoun the nomegadls
tistering ra rustinn inreni 1955:
Sabnsse lome ovend babii
Aumapdan Taedan Tiller
Elpeod"" (ff "Xill whi 4|
no lifeer of 1816 (30 Maael
rebesfoott Mateario ese' Us
Sa8894 155 Arbousd Sev: (10)
s"hyaass" bacieaaat uew thtera
Chant 1300 2566420
Willsk:2000 Ber Xolopeds
22 Doggh
teatiis SXC-1 dps (EGENE "oor's
barpseoe stera
Cascli Pats tugi cettoooee
Turchal
Noliytigh, luowis mukkk
Pragi lebeel lonaads Mae 13x
incmpallatal So
Tin exeal Po XIK
velletnimm diists Tillies
and cookais destecdiaeetliee
Blons 1075) Namo If Valise hit
Kolklii Singieno Hatert seare
mmmerahaicccee ano Idno
takieu Prodocll (N) gane Tandrr
who at nogmerr of the RoolN))
tHaa "ae is 13. -WW.-
glardal
thes
lishea (iice Leed
Lostes 18 sadineed Bose
klage
Peas,
(1966h) lSAdedaa schonee. (Phcicko
Chowsss Uygollatteddo iatledoiimoodi Bloamenta tios Ricosseankazee
Kaknonvent cande Sex las Warple]
Poget I K. hadine doer "ive
Cupser.. lse
vetti 1olt watt---oraeellyy nea Milciin
Nasher U, ahsitoal BSS cooddeetees'
Sobe
FiSa Nown, londortth, Nugay
Vii She Bessardf Clus"
15.81 leach ludiene b1 366 Sempertn liguezzo ind Frainett
"AS* nlyoostall NNNi Konslly)
Balle covera Kjllnooss
Figure 15: Unconditional Generation examples of DDPM on Image-Text.
20

---

Published as a conference paper at ICLR 2025
prepenty flat lasos the eocepeay fanoo is the movi well bol upisiances
the book yasysdiddedd 1992. book of ther
the production
The is seared basked ogg by the Kelkinoara Con estry Gromny Fubhishere Australi Duserdland
the bast way de if sas be the imrpoiani on Bee socedd and
un meleetiiin take manngemeent
The oveer wert what was wayed, and fhom 1984 JAM bocame menagament and premriass coutizzed tike rdvistagge with svusa with do depeas lakk, dic muirserance gas and
atrestively refiemits histry, and consengte. and doczaiss
third, in 1540,, and boaill
faze foond
South Andvinn Sara of Amsstccas origtt relaaed priza
other hemsars (inchddnng 1. Blads
demoontariing the value of's uuiiblly muvi vade undataakiz, and the rak
Demossrited o mooraage
demomasd excnoo, rulling proparty
the moder
coe appeard the Hillyyeed fites Beee and New, but sever used in flas
gand cuinnzee. The seouad
repair
arnouaec von anand, and his secrad usi mamorschhggiit
1956, lysts sold
Gemaan region. Asil Holsens la always wasted conecive the comminoom
depicied
Commaniicar ons where modda of
company
Corsoppnoo, hocmee main
Christie onignaiits,
muantic fouadd
technical
the property that was devclpad in condiaammnizatnn by life. Albough abouly the traliion of modless faz dorign was deddenn doanal most sialent a common exampplc the conmerrcaiiiy atfective red modis penio. The most signifieani festure in the canoii keriiggs are courry mune powor ventrre
eumpas was sold to Harvad Theirs than meragement
the propooed mareet. la 2011, the
the book ao part their scioo bacd on the books, weiterr only thining
and compps with mere
moneyaa Del-Nottle wifh thewe
(Hlrrodd Bni Maxagement, and sbe loing succovdal in company aercccoe entopeicos) liechre home also ben
Wie the word begieting unsoad
the Booke condrrtiinnn group was the fevoriic epposci
about a future Korzan food the Roern beg" plased
market 2014, was placod
propenare Pete Hedat's firs sxccoalal codfal NEI. and the firs, and gosord oxttlad. aut was ferensat Diviration
mioogdd
the For
ofhen
Roman Calliiis usst Jan
production was also unssaceefid
dhe Beptis,, who diad sller buttle
the procuution scasen for the loun. with a nevel liseet trvn as Neut
againei the Ciniiian monadery during camrers 1758 May
ownerding and other industries
mising
Files
pryectied
Journation luprasettaiie Home
Highly Intirmatioss surkes where
(755) was on offloer is the Prassio
book part
the property
known
peoposad chinge
the property the known
Slavidan saan Founded
compaay Tor in Cans, but coninuss groisk a poieer for many intractive maragg and
Morgumes warder dhe cinzs
combitatios of the buckpend
Murio it daylie with
stroca while the working gill. On
windows, with single engleokony skory and conceston esore seating ineid
Decerber 30, 1997 The lown
the day the werdin
porerty
untrohaing oibottaraaion.
decceented
coupeticoo corre
none ownal by Roscoming The teenets with the
difiirent
day. The wodding clarty
Levis
fonnotiilee velionoo and rass
sees in womenn and was alront
slary of mony dell, the lest
Satte irluha lind and the towrs cost
"augaenne,
polsomma wornn who
being meuumena lifted in
edited studects sentenced to inemn ed foceetiss lingriine
Dacembeer 2007. Brouffice Depots
million, land
fatrre a sence of dowepard
comtina rand by The vartiorr or
amoccitt edite of Bobotinn and
collections of photoggaphs the
Ausia, Weild aldes the flooe
morket rats 55%, and the towra
laddest
sevend
poems and mnsical bilibee it
egrwers the hwilding to cruat the cocndl doowings unita the
peopection armally hveroadd, quodans
company was vold to Handley Asstralinn proveaior: the hand eventually trasslighing
the beok a book on the peoperty
property was panchased by the and tie Shore Lity was carted "Making Nowtoon with Hidly Preperry "Pogows's Movencal
company. The compasy
the bend on inmmstty receveredd
dominaiio and signdioane aniiated
from
the ssodit
procass toe Cuastan sepuidicas
oxreany. The heed recenor
surches esteraaliad Sydteyy Melscume, Addaade
aumuuiyy acrces time. The fimm
1981s Hounaa emerges
enge lenves the hord using
conoddred the Widdesker etfician and drroper then any of the most Serves segnos
cottinily producs the Coutton
Interattonal
lompoe wily
conuumiity
emplhsis on Americon
and inland IT's fowthell The fins senson Sorth Anstaiia the first seacnn the new Biody
Award Form dnceallva szwr. In the botorn vasion planned invaders, response of
the
unfiersindacss and any
wams lime, segersenve
dadknone,
uspraditahh offect inmurit
ansustenith
syithetisdelyy stalis rogred the
and immuna
being incraiinee fliing laee Civelian with disapeas
was usder wenerr tut it was
between 1902 and (900).
componies
and
ansigrast
fhan 170,800
excoumyy. CIMD STAR Muln
denied armotiaal moragad coeral
atrmarine induary wow the frece
traising over oedeys for
pew
were
Figure 16: Unconditional Generation examples of DrD-Transformer on Image-Text with tempera- 1.re=0.1.
21

---

Published as a conference paper at ICLR 2025
whill had boen deloyed by was perpowe for seceitted
sonrinsed expeett the and setiess also unall bayy recegriiin the segeistliic sfties, inchadins the thh-kqqtntt by
the United State Court
sparsdy aggopating awiet for Os Acanoera scarityy Argus Kant
Septemivm 2 Dxamrrated soeen happen betvenn colerctaanccibe we
baildings and aho shows up as
Kass SRASIKA was memiiuted
werls The Zentogaai snoup
raquiremnt of its was NP
by oppoetior lasder Kadian diad at the ags of and use repuraaly not
Zjtceative
opec pecattood, strectce opertuntts to fhe saugle per of roeen practse. Adiinons (induaing calle. coureene) dascriptine exompiee as aquityhuue black, low book point and lighroeloww For cumple the contect giving
ladd eeigholi died while
fillwwers exited
diases as 1848 and lis dest.. The
fou 1825 ustil 1828 durisg the dvill
publisber was given
was Referanc- Fain-TTiu 1817.3
Beforenees Entercal links Kallisi comorition anry prefered. oxsiaxted, losulesed A Docameni Tral Ammayss
Life footgesholodad Sull""
1953, maken headal
hiovey
priaselly douanii besartuel, and fallociu yeurs, man of
Sapen Miltay counlate
the solored
Sanlaggo Ovaggavda taee
Conditsion Saqun Millary
cmmttios kerner Ken Ersi and Kai
perpeity danivements Usrmmittes
and jar Conpiiaiion on Huraa of Banasshe (becwe Reprosmaai
Sweakea in 1942. Ske was elected to Tooodoons "Fla" (Miced)) west
reizael in hanzay 2007. The wxx
to the nowly formed Alliazee of cormundes invetted and eapable coytine and who anvanee
emys musican Marrit
lad becoms oomnuwelll times
today, Revolstinn
siming
series tines daze of Bomus
Ranare Libggs Amee Morse Morsemes,, beconsing oeal cahinct remmber
offer
compeodl of the gzeeroon end of on
Choodspeeen, whuch was also
with badking moxicone
daly Adticn commesion with the
Stamus
and
eppaseiiee mad orddams at our Gocaner relving sboective
whes was cocared mayer
Refercnes
1802 doalaa centtry Seedias lighadd
Mizerat, yeur, not engli.
Gasur "s was reeaed a basle
nik "Kapahaui Warr" cpeptiar
where Sala
the prograseea
trakk uiled "Ausl" whishh
expi anerraly, Malaraii Skamna
contenl. Be wisic the book.
Cood Tolannt daptes, 'Sly Dortess Houe."
of Party Independent Society
Swaden (Ciy)) Stockholirr Margeett
greap.. the
popalaroei Paztoo Rics
scloss senge
youlh
and woute is suberssoneza Aaaeut "When
"Mas ticlerrr"" Home thee
the tnck
composed ancical Nabander
connameative calendos which
oppresetation erakellstii
Michiga elo jeining
alber, Farneiag
cappocadly
milliess bowever
whole
and conoiioo and Earoes plato.
fanzly cssurated milgaker
wilaut spotils vingges sharce fhr
Reformmce Estenal link (3e) Scather Misn Boad, Tornerly South,
unti be was eighton His Verinia
lids of "Tecx Ou00-2 which
oppeewion. On inc NRCAR recommended "raked" nox name,
Stase continnous megiotce gads
waee the mere eathh booe
1900s
Tralig
Midhigan Childoe's Hull
refemall to beng "Frm Gach
tscladun NEPRUMA wore also underton howe rweining from
aas der Cigenee" weyler 2003 AD-5000, 26 ATPon 11A
Fame. Both eaagaizess puroded
Hlad to Curcaaa though
Murcaviid and tntal in Mayaros
concerning this manoral can hea
wrikering coads Das /ove)) lad
diplly Romsining fanillar,
Arme sinn drivers tinse hil, oner
commenillly and clam Dhe evolanee hoome used by the nevane patem Also NMCM
prece and contomer com 22 O Chillae Finse Clatsttation
Wartington to show Sturayadd
with the foreliption Adrica Rechl tad intaddd that the altum had io
abost leaving Hollad to work. perticpotion
eriku
from the Louivellle Instinatc tlroghoor
the cnd of Cape Cod At the same
Sandee Perman (been Sepkenber
King the Forcs, (1984) Seripe
of River Averac, has vtita
22, 1968) as Americas, snget,
(1999) Coedese Keagitt Gi
bealds,
musirv. old gen Pro-Cemmats
engweteer and eogroams co'se
Harkon (2011) as Jutt Thening
liteee, fhe dovenitaiios legend rederly, miceere cricke Sea Expioess (1900) 1999 asts, ladt thet aniived is 1853 at thei buspiall
Girmani, When the Hrave foond
upple jiee, soegs" (2036)
(1907) Neeway Sade Gris (1987)
miigned, the renamms
ceder
poraaa une tauz. Coin
Hambe and visins (1900 Piellc
plsttees
of pasis,
out Pans is 1981, in make he
CBS Naticcal lnvesigatoo) The
any masc Beardon an
Murder Amecis (1989, Sereece
arpives of the paople who were
Borwwakes, in Lesieiille lea an
basted in Alleri Cooiistheer in
Hanc: Pamarts brodee. waa quitt
same tides 1200, Seouads.
imppouura divralserr Open
ouprives
at crmeet mestien, when
Blustools, Lei Be Rosked
Persoual
have comendeteres secesesity
childrer Scam Londor
cusessited en the sscusriats
enal-alerei words, and samed
Assssect (Coed
Univenaty to aoady 1900 when
(seeningly
the golloff order to manr voaie
Monnte)): Erery day trees onita foa
Figure 17: Unconditional Generation examples of DrD-Transformer on Image-Text with tempera- ture=0.5.
22

---

Published as a conference paper at ICLR 2025
This caed arose well wlks other nussioem wenus (uadd-l-altrra) spkeyediierrin qurtahittosssr in destaay (K Kn 1% 1.9), 22%, and 2001 (Nki divergent foe clepoe
Conmibiio (3.00 (C) of Prosidert: ladaans were lecodd tmaggent was 46) repers of aborrizall The was, froalls, lonad
Portil invess sir bevething nepical land iesil. lomoatial pulmosic gxatrood mellade firmlly Strepianidae reproduction
Iefornters by hightwey This listrr The clevate that inings
n0th
Tustrikn Zem. Joan Dosusdri
clvose
MIBLE..4 and the ClesSe Five,
opecia qatenal
beginsiag the Deginninn August
19, 200 lnk 4''s 19 St. lintng
Teited than they aill koow The
life 68 days (256 (256- 50 250-
iamell Special Equizenia
17)_ cveis oxperealim body soart 7240545 (fxsi)) pertoulte later somse saailentylor Valoes end gifts exientiice werking and those coctrial to
CD Beaas Pacackuadirs AlKcr/st
ClasA A 12, Mousiny 1s.0
axlans sted eads with oecantng
4.6.m stals, laser. Bach-andsoon
far wilh Wien Traabill
KAP VANE 19.1 ls C.51, andinted
W. C pllnt Cuhanelnnderaa mion..
H64SFAC sei- ienilleetal fight "On War two Savmnnars ID Bays "The
ladis Camaanziva Boof
Fitaarer: volcanc ques were
Waserfoot My Hall:
sebeted X
Naturalyy Zir. Aan T3 PTV 1.+.
H Bloon disected
cobocito
Wi Wigsoch Meeii
advantaace hes beuttind and
Smith roconded a year later and Lander would,
careed aleer lim. Early life be was Robens sutadin (dom 9301, Cansgal 'Aala,,
temeike coul de alematio
Bittenooart spelitc of powod) (83, gir, Eyeeboo, A dlas packer play femule
upyer folksur
ampped Meloo
akormnn Whese illmitaaioon
Liz firey's wool
like Book
Dillard zevaloped and nevened Cille joy As Lifeta of leads vol.
otheresa indated in iil to wch la
Supirop. (5% loeral
encemming hendermooall tumpps,
rcher for bicker' warn Andensnn
fillycadies his jud nos unde
lideed was live olaon, She vennala
Villachis Blackoott Calle Diland EOT workers pelient "paces Martralat) A that of just ous od lho.. majer) Cuuhii agal to Selane Vouce Langhale, ons (Friguec) ani fastasias
Lasa
musticule ueee dismbeatioo wilbou
mosoeciue (soric of) ()
(- beodle, an clan it
Oinartio Munness O. Punrdt)).
(mosted off cansiit drrches hedk Haauuaaa
demenning dosss spubo
pas
hand "sgo Sart sz'i. It-came for Macioon night game had to appear is lindly hous criued caile
Conge Tark
Rocaputs Iichsaal Separa Fig (peraa)
Xit Siee also Xletal partiele
Refassso iP Parodimece (kusho)
MS
Hacn
Sest puzlakeed
Handew)) How ham Orvin 1
murafamned Nat quae all
completng toorien, lecullyy
essciation peinsiples
where
coeedara rarting floee miminne
Mime, Vikerd, Alider by their repe eage's Spensor tig. yo
comsojancd Gulley. Bank
the the ingan. nax, and quadat
(BCC) and nems The som.
Fread- (WCSV groneed." isf
deta other
tidec party
opponillicy To ersdeaee
ullos y. Gallery
turbee
AMS
Jovaneral sounee will) adoenzal'
Galko (IMM KS,
chanswnng
Jin
stare menory in, all curaer
teclaooogy ciles Black Decta will
longiky (Oyaat comuae gosestair
(Oyetlc) Thriight' naveill slaht
Parti.. Nf, I Hoppe Vinthi, schib
vingin With beoasalens dalmn edgulatoool fimn fiayy Imn dits may caliing
amess fortydive eniire
(20 gin Lyw gair Swells
Trigaaduu 15 nors
finessirs) sebels regures
Modan-srgppk Bo, and Faz feor
indedlowdd entire
prosddide
Nepen Alerayy zy ge, Veek
ournen 591 brasdMlootooo
couppesed bojobite looks 3ad
DO Porree Biigg My zu The is Mais (The Tae
apparsin requetce
the "Durry dua take owo
"Argiman, "Arpi, Digurmiliap en Sef, Ings doe 1ik4.
Stus, off across individaa XJ, Becen
hown
(Dhha bebheernd//ceeeeeaddd chaducg()))
Boasn, v.E SMMeeilooolii BZ
kya), jearsee (scar))
Argr sebiirr docemenrnn mas
from Halviper Bengsker, held for the diste of Sammanan whone
Makeyimmwrdd online prewige
Sammeeto Klanicppl Airpeei
larpoi toloisims drand lhe
fat of bezners or beharengens nowing sevand adiveral not,
refor to Siste river de Ayus
scods, sinchoon free dickey duing
attira loeragar couenally ralgsd
Mignatta in Salavearcchann, in
from enae holayand Aetr odaly
[wit
warl] wili Maloe
bandkgo MPP
Andrtted
rago Nowsboat
oth tifes menden, Nibbochotn collooognee
getggo/gi
Silve Variaies,
Heneela Gil ball,
fees
Vuttle Bar, proakh goditaa
Interntioeal (aamm obtanee)
mnttille deperd de vilesks min
scpurt y da Cindaas vadia", Sak
vintai (und only vartedai The bikn.
cocming wolan fir piazes to out
Caull: See doo TVB Gread Theate
reaafalle (b medd (1tal Silon
Coesches The prefomat oxiety
trasagulish, S-par tryy laarn
comervaties werking.
deoon TVB Grend Theate Prinnag
He Naw road D whi four Tallc-oad by
conce)). Pargurs Maaaaaz bal."
boys
bey,
arveratioaal wottel sutwere.
Aoskau Augans (luSSkt
"Ajakmi Argans logo tns..
fasiid veribl bawcil from ofler
TVB-Grour Bilenko--6. X-Y
The Booddogk Botentris IVB
Henaar iise Noote Natheng"
sings nugio
Cajpke TVBb) Trontye promos.
Figure 18: Unconditional Generation examples of D on Image-Text with tempera- ture=1.0.
23

---

Published as a conference paper at ICLR 2025
Lsees South Bred lourse chaclaraantie
Seteoeal weitess Wile siiii de
musteo
bueld lewe die (Yooe
bevele
Liortid feeelis J.........
del lenk cevigraman las coun Lntine
senye
with nemuees sebued thet montis
The 9C Ste sears enterrrddy anntttoiee
ia imptorrsss
talle destrr talli alc In Thach
(428 settrrciieed coarinoned sclechet
sás mahrommn moskiet wal tnalnetonree sociste
Lime sne casora par
pira
Augl kook looken banad
this
the dontard onty volmm vake nadze
(ler) alou A. 18 wick lig
cxsegriin lendan (2)
pocnery aeclec
tnmmmnalittin dente hos pike denn scceecc
swotll nathorastott wde
valasi davil
nid ried turrge dun cocivle eslevce estel
Vidian
F17 C.S.
1.S. Capps satied ael (Guella wou
aleen
Mty (-1g-
sull wadettes
dis oproppes Hill dis clisss,
Fale
FOT
IVLL TOUD PUNAUISSER
ahobk roie
Figure 19: Unconditional Generation examples of DnD-Transformer on arXiv data with tempera- ture=1.
Hampshire County Courthouse is cvertlouse the campus University of Pittsbargh It is loested at 142 East Hampion, Raleigh, Mississippi, north of Cheyrougs. The courthouse was demolishod in to make way for the construction of new buildings.
the previously underworidwide sarvice and surreamdedd during had
trade union in originilory low services. The previsusly large unionist workers during head composition founder Samantan Baraek Obama September 1911. Germany had surved as unienist with the South American Govenmess. Gentus Baraek Obama subjected to new unionist than whose counties developed the United States. The unions were delayod and commonly interpreted in order
on 15
also National Register of
Historie Places listings in Pittshurph
References External links Hampton
CouthhCoub Coorthouses in
Peunsel Ivanin Court hoatees in.
Pennspllania Courhhucces
Repetition
Hallucination
Figure 20: Some cases of the generated text images. We witness similar error pattern LLMs such as repetition and hallucination in our trained model during sampling.
trikoo thi choo detammati auposaaaelknennn
redorman teatiis iitt many auelamn Vili dliiun the ON fact to cheyy ii srndurne So ensts
anail ooe slavsantt cusall coorng Sipereas
duatioat' cediulageer oraal adeustah
Cand denttel sulple sasergans
Coatts wnrld ill,, lifcudaa
bhat' coifeoonstthee (Dun. Yalk
Bainel moppanc. Us of') lictudra Boviss
lett lie medaco tise wynee
deviiestiaeetil lane cuacel thillue PSSI the
lincud of Ask care top Eslit ver
Eigike
siedog Warran
arge
slagelavee clice buatt Cloee
Conerer
sundessne tre kill hull.
Masdaasi lok cyceced srone
regg ge wiitb vetll chcc motmie
hete ot keieeir an alter
geiagannt
Nidlialid
sureve whiskns
sogrc.
Rossegaan fier robe manyy
apbbyeten in lty londdlll
Suepaollssopoo
fouretioez
Seleegs Giveryk quo- For ttilag
Conl Earlai Masia eerr fins givts
ieisem are Iigrtollatioo beges
per tfgleeeree rnd Teatin Thetie A Aoors anlo flovet Voreens
materiind fanllc flaid dan
misenng of udkus rrne.
cabpu puetila Vertrrrmetton metiwallaas
Our pleam Canao Xila
vovewer bear 122 un ifoical
wiooaopppzzz oge conategroi
Stanres Eaire pedila alos ponno. billi inary 3n. annie
Selgres lncel St Servace
Bronnamee Buflone! Pros Not
Figure 21: The results of DiT-XL on rich-text (PIL-Text) generation tasks.
24

---

Published as a conference paper at ICLR 2025
beeria ove't
wil eeloogeeetboii NETAO Rilbber
bod
Daps SUNTS
and light do but
Figure 22: The results of DiT-XL on rich-text (arXiv-image) generation tasks.
DiT (with tokenizer trained on PIL-Text)
srurch tina arwzrase Sdear
redernen-mmiid 99081 usxy
Hitiourmate All rensitar nomg Clippzia
Dirriabbentiiiggaa zust
Manil ciossad sayta reaasyy
Setiamad sundnnaskklnddlety
Farinais Mid Zilla Ses US
Counsiinall Lile, Lifenddrr
bialf lecther anoiaad (Ia.. Wils haveel of prounessip pilid wr
Emiiil fluonas elo Hilh ite
lell Elnonerooo floor eryers
foo Je How dhool Che rrists
ulee Misineeiiieee unst like
doaltalte 783 XBF colo.
sedge
Sunpalmnnt edenhcsal the issrr
nade fow Veteal
Lazasts E541 I l'eal) sequeal fedä e seil olcbati taisel mery
Henggs sarkedd a datsdanquaned thal italie battes wall Nelaaass
sadetoni atus dET1E,
Sdufidalii like tyerred erer.
teo spill, leflosi, au tos Stir thooodatt ritiaear elayete lanasiike fest pen decessveree liaber duale V'anetic
bentrelllieeen (CWoilllaiit
pertgroet
Rogingroor thatide
131
Larire anggeaatined hames
Mipaaoo
Jetcken Cas ask djec 31 filit Zenoonevviztal
Eath edei sochis easy durryann
maes use oun thasfoshettiee h rys
ous in
arovesy sladdenmaties
peells, Wefloeemaateeaaae 191 invirotests,
Minat Ebroogga can lin. is wreld lefls
Fity Un-seeeeenoth
saneere. heiry 71 te idvicgle:
ivrtas openton aaoi in citsaothad
Peariaruu Phere sodiica ne Dlg ponsh. ierry hhaaa
delgras aruck Scginnses Blanes ories
Nocemurr bitteed Pru likey
atmmttcratiooaa worke
DnD-Transformer (with tokenizer trained on PIL-Text)
player team five drodt same. Howwreer the entinely buried posoedare is that lie was used is
particuaar
comppay sold
somowhul sofared using machines a explsiler, an
Amnrtcaan madernia, thereitt poomiseer Asiditon and upurbagi
vore olsumaad the and
production roudl Comn sevarnes drop-owring sales aad products lad players given Dee
insagee were nemorrdd
cuolic beauth paokage la this onse
Berret is the outher of marr books and many artaaii, errezs visto and longiumment
Novensberroookk
Casing
not any game my begin
the rosturusi extemab wedding volune (ensiting cameraad, and
throughh Sandes Marning mayy
ans umoeaa amoien asy reland
playing sually par-tine poseddare. The gare also bands only leweeerr has home will use
vappacocse
Bayytt' first beok
Hoveereer the of the
csicaded petiod of time, landly
sereeall varives advennces orgatize
appeared moder wernnn and
(satered resverk are The the
the
rodoral methern rosio-
the ame procadere gain gasts postioned
to the Gruntiin on en emplave
participanel in the portraaa of
parade
moderr
odiieiining the womes, manriiza saa manieelscogple cooplees. Ei was lesling semder. spending serven
1992,
caest peosigionaly itying to show
the efurnatios the seris compase
propare amnet couple is moree commen in the special arcw
single your cxederization
michines (encept for drive-ost mary stuckerss) in one cosoc at
the company's y deyth and winepower
fom
aranded
Timmayy Majorea V'antno prices
fined Karric
of stock market the higher
zo mones of cocppeteal manr
Proskustooee
Figure 23: We compare DiT and TnD-Transformer when their tokenizers are both trained on PIL- Text dataset and evaluate the generation performance. According to the generation results, even with a VAE trained on pure rich-text images, the LDM (DiT) still could not conduct this task well, lagging far behind our
25


Reviews:
[{"review_id": "6CquuaOS99", "reviewer": "Reviewer_Zh5x", "summary": "This paper addresses the problem of image modeling and generation using discrete latent representations and autoregressive (AR) sampling. The authors note that larger codebooks are need to improve visual quality but scaling up the codebook directly is difficult. They suggest a using multiple codes per token and adopt residual vector quantization (RVQ or just RQ) based on an earlier model called the RQ-Transformer.\n\nThey then present a creative approach for predicting the residual codes by adding a prediction head between different layers within the normal multi-layer transformer. This approach adds only a trivial amount of extra parameters and computational cost, and requires only minor changes to existing LLMs. Contrast with the RQ-Transformer that adds a new (albeit small) transformer, which is harder to integrate into existing LLM code. Through empirical validation, the authors show that their \"DnD-Transformer\" yields good generation results compared to RQ-Transformer, LlamaGen, and other methods.\n\nUsing RVQ to decompose a large codebook is not new, but adding prediction heads between the layers of the LLM transformer is. The authors show that this approach outperforms parallel prediction and vertical prediction (visualized in Fig. 5), two other approaches for predicting multiple codes from a single forward pass.\n\nIn addition to generation on ImageNet-256 evaluated using standard metrics (gFID, IS, etc.), the authors also introduce rOCR that evaluates an autoencoder in terms of how well text in a reconstructed image can be recognized. They show that larger depth values (more residual codes) improves this metric, as expected (see Table 1b). For generation, they measure perplexity using Qwen2.5-72B over text extracted with Qwen2-VL-72B and show that their approach performs better that a diffusion-based baseline (Fig. 6b). They argue that AR prediction is better suited to generating images of text compared to the \"simultaneous\" generation of diffusion models. This is where the \"spark of vision-language intelligence\" comes from: the DnD-transformer is trained on images of text and can then generate images of text with a few recognizable words and short phrases (see Fig. 7).", "questions": "I'm not sure how to interpret the code usage reported in Table 1a vs. Figure 4a. The table shows 100% usage up to depth=8, but the chart shows code usage falling off with depth. Is this a mistake or are they showing something different?\n\nA minor point (that's not actually a question): In Section 2.1 the paper says \"each code has log N bits [of] information\". It should be \"log_2 N bits\" or \"log N nats\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The main strength of the paper is the authors' creative solution to predicting multiple codes per token in an efficient and effective manner.  The approaches used in other works all have significant drawbacks:\n - use more tokens, which is computationally expensive\n - use a second transformer as in RQ-Transformer, which complicates the model, especially if you're building on top of an existing LLM\n - grow the codebook, which is memory-limited for VQ (though not FSQ or binary quantization as in MagVit v2 and MaskBit) and makes prediction more difficult\n - use multiple codes and predict them in parallel, which doesn't work very well (also discussed in this paper).", "weaknesses": "I see two main weaknesses in the paper. First, I'd like to see more evaluation of the DnD-Transformer in terms of sensitivity to layer indexes for the prediction heads both within an otherwise fixed architecture (e.g., stick with DnD-Transformer-XXXL and vary the indexes) and for smaller/larger models (e.g., if #heads == #layers, the smallest option, does the approach still work?). While the layer indexes are \"just\" a hyperparameter, it would be quite interesting to know if the approach is relatively robust to the layer choices or if there's a pattern. For example, for a fixed number of layers (and thus compute), you can ask how that compute should be used: do you want more layers before the first code is predicted? An even distribution across codes? Maybe the first code is relatively easy and you need successively more compute for later codes.\n\nThe second weakness deals with evaluation of generation results as a function of depth. If I'm interpreting the paper correctly, the ImageNet results in Table 2 use depth=2 (see the caption for Table 2), and the text-image generation results use depth=1 (see Section 4.4). Note that depth=1 is the baseline where no residuals are used (i.e., it's just an standard next-token predictor with one code per token). These low depth values seem to undermine the core contribution of the DnD-Transformer.\n\nNote that the \"Generation Results on arXiv-Image\" sub-section does say that an \"8-layer visual tokenizer and corresponding DnD-transformer...\" If that means that depth=8, then I'm less concerned. Nonetheless, I'd like to see a chart showing gFID vs. depth (much like other VQVAE-based papers show gFID vs. codebook size). Figure 1b has the right structure, but it shows *reconstruction* metrics, which should always improve as depth increases. This does not always translate to generation quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cl1DysJJFx", "reviewer": "Reviewer_BF3N", "summary": "This work addresses the information loss bottleneck in vector-quantized (VQ) autoregressive image generation by introducing a novel model architecture, the 2-Dimensional Autoregression (DnD) Transformer. The DnD Transformer is an end-to-end model capable of generating higher-quality images while maintaining the same backbone model size and sequence length, presenting a fresh optimization perspective for autoregressive image generation. Experiments on ImageNet and rich-text images demonstrate that the DnD Transformer successfully generates detailed, high-quality images, highlighting its potential for advancing vision-and-language models with a new approach.", "questions": "#1. I'm not trying to pinpoint this assertion, but I don't personally buy this argument, since autoregressive image generation research was already very popular before the release of ChatGPT. Additionally, by 2022, the research community had already shifted to diffusion models. GLIDE (2021), Latent Diffusion (2022), DALL-E 2 (2022), and Imagen (2022) were all published before ChatGPT.\n\n#2. The claim in the phrase “a spark of vision-language intelligence for the first time” appears to be overstated, especially given the limited scope of the experiments conducted. It is notable that AR-based image generation demonstrates strong performance in text rendering, particularly for documents, compared to diffusion models. However, can this really be considered a spark in vision-and-language intelligence?\"\n\n#3. How is the ICR of JPEG computed? Additionally, it would be preferable to include a proper reference from a primary source or technical documentation rather than Wikipedia.\n\n#4. In section 2.2, the authors discuss the differences between the proposed approach and RQ-VAE, but I find the explanation somewhat unconvincing. Could you elaborate on the distinctions and explain why the newly proposed component might be expected to outperform RQ-VAE?\n\n#5. In Table 1(a), code usage is reported at 100%. Did you apply any specific techniques to enhance code usage, such as restarting dead codes or similar methods?\n\n#6. In Table 1(b), SDXL and SD3 are trained on ImageNet and should be considered zero-shot tokenizers. However, this isn’t indicated in the table, which leads to misinterpretation of the results.\n\n#7. As I understand it, unlike RQ-Transformer, DnD-Transformer predicts codes along the depth dimension simultaneously. However, HQ-Transformer, a follow-up to RQ-Transformer, also explores predicting codes in this way. If my understanding is correct, it might be beneficial to include HQ-Transformer as a baseline for comparison.\n\n#8. Regarding Figure 3, which dataset was used for the tokenizer in SD3? Did you use the original SD3 tokenizer? If so, I question whether this is a fair comparison, as open-source image generation models generally do not include OCR images in their datasets. If the claim is that DnD-Transformer performs well in generating rich-text images, it would be more appropriate to train a tokenizer specifically on rich-text images. I suggest training an LDM, including a tokenizer (such as continuous VAE f8), on rich-text images and then comparing DnD-Transformer with LDM under these conditions. In that case, I am not certain that DnD-Transformer would outperform LDM.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "It’s intriguing to see the potential for training a vision-and-language model solely on images. When training such models using textbooks or other materials, preprocessing to associate images with the relevant text is often challenging. Training an LMM purely on images might offer a more principled and human-like learning approach.", "weaknesses": "#1. Compared to the RQ-Transformer paired with RQ-VAE, what are the benefits of this work? The proposed method appears largely similar, with no observed improvements over RQ-VAE.\n\n#2. I believe the manuscript requires extensive revision. Some references are incomplete, with some citing Wikipedia, which is not a valid source. Furthermore, the comparison between DnD-Transformer and existing approaches, especially RQ-Transformer, lacks clarity and is not fully convincing.\n\n#3. I believe many experiments in this paper may lead to misinterpretation, as the comparisons are not conducted under fair conditions. I suggest that the authors address my questions in #5 and #8 to ensure a more accurate comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jD9jKyQpwg", "reviewer": "Reviewer_FQZc", "summary": "In this work, the authors introduce DnD Transformer, which predicts discrete image tokens in both sequence and depth direction. In sequence direction, the model is trained to generate tokens one by one as standard AR. In depth direction, the model generate one complete token in a residual-quantized manner. Experiments on ImageNet show competitive performance against baseline AR models. The model also show promising results on text-heavy image generation tasks.", "questions": "1. It's not very clear about the gain from depth-wise autoregression. It would be good to have results with depth 1 or 3 on ImageNet reported as well. Also on rich-text image generation, DnD is trained with depth 1. How would the performance change if more depths are applied?\n2. There're some AR-based image generation tasks that are not included in ImageNet256 benchmark. For example, VAR [1] curates a resolution-changing tokenizer for image generation. \n3. For rich-text image generations, how is SD3 implemented? Is it re-trained on tokenizer trained on rich-text datasets or finetuned with its original tokenizer? More details would help understand the performance gap between SD3 and DnD. \n4. Also, are there quantitative evaluations for rich-text image generation, like the rOCR used in evaluation of reconstruction performance?\n5. The tokenizers are trained separately for ImageNet and rich-text datasets. Will tokenizers trained on merged datasets hurt the performance on either side?\n\nReferences: \n\n[1] Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction, https://arxiv.org/abs/2404.02905", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The work investigates an interesting problem about improving autoregressive image generation on discrete tokens. \n2. The model achieves competitive performance on standard ImageNet benchmark. \n3. The model demonstrates capability of generating images with rich text and graphical elements.", "weaknesses": "1. Some implementation details are missing which affect the evaluation of proposed method.\n2. Some baselines are missed on standard ImageNet generation tasks. \n3. The model claims it shows a spark of vision-language intelligence by showing results on the rich-text image generation. However, it's a bit unclear to me whether these results lead to actual language understanding. It may beyond the scope of this work but it would be interesting to see how it performances in vision-language understanding tasks. The authors may consider being careful about the claim. \n\nPlease find more details in Questions section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6zcC6k6Ys3", "reviewer": "Reviewer_3Gvw", "summary": "The paper proposes a 2-Dimensional Autoregressive Transformer (DnD-Transformer) which, essentially, adds iterative refinement to visual token generation in the autoregressive regime. This iterative refinement is reminiscent of diffusion model inference (but not the same).", "questions": "- My biggest question is on computational complexity. We need more clarification on the additional compute needed for different depths of DnD-Transformer layers, for both inference and training, and comparisons to other work.\n- Is there any reason text-to-image is not explored?\n- Can the method generalize to higher resolutions without exploding complexity?\n\nMy final score will depend on these clarifications.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "We know that iterative refinement is currently key in methods to generate images. For example, diffusion and rectified flow models which are SoTA have this iterative refinement quality. Embedding iterative refinement in the AR transformer block is a very interesting idea. Further, the paper presents ablations that show monotonically improving FID and other scores when the depth is higher, which is consistent with this claim. Also, they present results that rival SoTA diffusion models for class-to-image generation, which is very promising. The images they show in the paper are also convincing, with very high quality - and the interesting phenomenon of coherence in text generation even when trained without text-labels. Overall it's an interesting paper with a strong directive idea.", "weaknesses": "1. The paper needs objective data on FLOPs, training time, and inference speed. This is very important since adding the DnD-Transformer layers increases all of these. It needs comparisons to SoTA diffusion, rectified flow, and AR models. This is very important for judging the paper since increasing computation has to be balanced out by increased performance - and both need to be presented, not just one.\n2. Only explores class-to-image and unconditional image generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jZMMHPeLYW", "reviewer": "Reviewer_4s2g", "summary": "This paper introduces DnD aimed at addressing information loss in vector quantization decoder for AR transformer. \n\nThe DnD-Transformer incorporates an additional depth dimension and extends sequence length, enabling it to predict a greater number of image encodings, which in turn leads to the generation of higher-quality images.\n\nTo my knowledge, it looks like introducing simplified diffusion-like process for each token while keep the AR schedule for the whole token sequence. \n\nThe experimental results demonstrate the effectiveness of the proposed method.", "questions": "see weakness\n\nIn summary, I find this paper to be more focused on practical engineering experience. DnD enhances the performance of generative models with relatively low computational cost. However, there are notable issues regarding the theoretical analysis and clarity of the writing. As a result, my rating is a marginal rejection.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation behind DnD is well-founded, as addressing the effects introduced by VQ can lead to improvements in generation quality.\n2. The author provides ablation studies to demonstrate the effectiveness of the proposed architecture.\n3. DnD (almost) does not introduce an increase in the number of parameters compared to conventional architectures.", "weaknesses": "I still have the following concerns regarding this work:\n\n1. The paper’s theoretical analysis is significantly lacking. The author’s explanation of how the depth dimension is used for prediction remains unclear. While the author attempts to explain the work through the lens of RQ, I find this explanation insufficient.\n\n2. The experimental setup for the arxiv-images presented by the author raises some confusion. What exactly can similar experiments demonstrate? I had anticipated that **DnD would, through these experiments, show that the text in the generated images is coherent and carries semantic meaning.** However, based on the examples provided, the content appears to remain **disorganized**, with improvements observed only in terms of fidelity. \n\n3. Additionally, I find the decision to fine-tune models like SDXL or SD3 for **unconditional** generation tasks unclear. Why didn’t the author employ models like SiT or DiT to validate DnD’s effectiveness?\n\n4. I did not find sufficient evidence in the experiments to convincingly show a reduction in VQ loss. While there is an improvement in generation quality, it appears that this improvement is not solely attributable to VQ.\n\n\nThe following are some comments that do not influence my overall assessment:\n1. In comparison to other contemporary works, e.g. [1], the author’s approach seems to represent a particular case.\n2. I know it would be tough but could the author provide reconstructed results to better illustrate that the VQ loss has been significantly reduced?\n\n[1] Li, T., Tian, Y., Li, H., Deng, M. and He, K., 2024. Autoregressive Image Generation without Vector Quantization. arXiv preprint arXiv:2406.11838.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)