Elucidating the design space of language models for image generation
Authors: Xuantong LIU
Shaozhe Hao
Xianbiao Qi
Tianyang Hu
Jun Wang
Rong Xiao
Yuan Yao
Venue: ICML 2025 poster (2025)
Keywords: Image generation
Large language model
Generative model
Primary area: applications->computer_vision
TL;DR: This work fully explores the use of language models for image generation, analyzing their optimization behavior, investigating tokenization, sampling strategies,  and model scalability to achieve optimal performance.

Abstract: The success of large language models (LLMs) in text generation has inspired their application to image generation. However, existing methods either rely on specialized designs with inductive biases or adopt LLMs without fully exploring their potential in vision tasks. In this work, we systematically investigate the design space of LLMs for image generation and demonstrate that LLMs can achieve near state-of-the-art performance without domain-specific designs, simply by making proper choices in tokenization methods, modeling approaches, scan patterns, vocabulary design, and sampling strategies. We further analyze autoregressive models' learning and scaling behavior, revealing how larger models effectively capture more useful information than the smaller ones. Additionally, we explore the inherent differences between text and image modalities, highlighting the potential of LLMs across domains. The exploration provides valuable insights to inspire more effective designs when applying LLMs to other domains. With extensive experiments, our proposed model, **ELM** achieves an FID of 1.54 on 256$\times$256 ImageNet and an FID of 3.29 on 512$\times$512 ImageNet, demonstrating the powerful generative potential of LLMs in vision tasks.

Elucidating the design space of language models for image generation
Xuantong Liu
Shaozhe Hao? Xianbiao Qi
Tianyang Hu
Jun Wang' Rong Xiao Yuan Yao
Abstract
generation is commonly facilitated by autoregressive (AR) language models, like LLaMA-3 (Touvron et al., 2023a) and GPT-4 (Achiam et al., 2023), which operate by predicting subsequent tokens based on preceding ones in sequence. On the other hand, image generation predominantly utilizes diffusion models, such as Dall-E 3 (Betker et al., 2023) and Stable Diffusion v3 (Esser et al., 2024), which learn to gradually denoise images for all pixels simultaneously.
The success of large language models (LLMs) in text generation has inspired their application to image generation. However, existing methods ei- ther rely on specialized designs with inductive bi- ases or adopt LLMs without fully exploring their potential in vision tasks. In this work, we system- atically investigate the design space of LLMSs for image generation and demonstrate that LLMs can achieve near state-of-the art performance with- out domain-specific designs, simply by making proper choices in tokenization methods, modeling approaches, scan patterns, vocabulary design, and sampling strategies. We further analyze autore- gressive models' learning and scaling behavior, revealing how larger models effectively capture more useful information than the smaller ones. Additionally, we explore the inherent differences between text and image modalities, highlighting the potential of LLMs across domains. The ex- ploration provides valuable insights to inspire more effectivv designs when applying LLMs other domains. With extensive experiments, proposed model, ELM achieves an FID of 1.54 on 256 ImageNet and an FID of 3.29 on 512x512 ImageNet, demonstrating the powerful generative potential of LLMs in vision tasks.
The recent success of large language models (LLMs) has bolstered the research community's confidence to achieve artificial general intelligence (AGI) and inspired their ap- plication in computer vision tasks, such as image (Esser et al., 2021; Yu et al., 2021; Tian et al., 2024) and video generation (Kondratyuk et al., 2024). significant advan- tage of integrating LLMs into image generation is the ability to transfer established techniques from text-based applica- tions, such as enabling undefined output length (Xiao et al., 2023) and speed-up strategies (Kwon et al., 2023). More- over, the scalability of LLMs makes them the preferred foundation for building unified multi-modal models (Team et al., 2023; Kondratyuk et al., 2024). Nevertheless, L.LMs for image generation remain underexplored. Recent efforts often rely on domain-specins strategies, such as sequenc- ing visual tokens from low to high resolution (Tian et al., 2024), using continuous space modeling with diffusion loss (Li et al., 2024), or randomly permuting image tokens to utilize bidirectional information (Yu et al., 2024a). These approaches introduce inductive biases depending on domain knowledge, which may limit LLMs' true potential. Mean- while, preliminary attempts (Esser al., 2021; Chang et al., 2022; Yu et al., 2022; Sun et al., 2024; Tian et al., 2024; Yu et al., 2024b) take simpler approach by discretizing images into token sequences with vector quaitiiz-uttiziiionn (VQ) autoen- coders and training LLMS on token prediction objectives, leaving much the design space unexplored.
1. Introduction
In the domain of artificial intelligence generated content (AIGC), text and image generation (Brown, 2020; Ho et al.,
2022) represent the principal focal points. Despite their shared goal of content generation, these two modalities pre- dominantly employ distinct methods. On the one hand, text
In this study, we delve into the potential of language models for image generation tasks. Start with image tokenization, we compare VQGAN (Van Den Oord et al., 2017; Esser et al., 2021) and BAE (binary autoencoder) (Wang et al., 2023; Yu et al., 2023). Our comparison involving recon- struction ability, scalability, and generation performance shows that BAE consistently oulperforms VQGAN across all dimensions. Despite this, current language model-based image generation methods largely rely on VQGAN (Yu
Project lead, 'Corresponding authors 'Hnng Kong Uni- versity of Science and Technology "The University of Hong Kong 'Intellifusion "National University of Singapore. Cor- respondence to: Xuantong Liu <xliude @ww.encttott...i........... Xianbiao Qi <qixianbiao @ www.l..o..........mmmmmm..m Tianyang Hu <hutianyang.uup outl Yuan Yao <yuany @ us..
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Elucidating the design space of language models for image generation
Figure 1: Generated samples by ELM-2B with 2-12 tokenizer trained on ImageNet. Top-row: generated 512 512 samples; mid-row: generated 256 256 samples; others: generated unfixed-size size-samples with streaming behavior.
et al., 2022; Chang et al., 2022; Li et al., 2023; Sun et al., 2024). We believe that a more powerful tokenizer for images can lead to significantly better generation performance.
the importance of local information for image generation. However, larger models also capture global information, which more difficult for smaller models to learn, helping to explain the performance improvements observed with increasing model size. Additionally, we observe that image tokens exhibit greater randomness and lack the inherent or- derliness compared text tokens, showing the fundamental diffference between the language and vision domains, which presents challenges for training image generation with the token prediction objective. LLMs achieving high image generation performance highlights their potential across do- mains.
We then investigate the performance of two primary lan- guage modeling approaches for image generation: au- toregressive (AR) models and masked language models (MLMs). Consistent with findings in the language domain (Henighan et al., 2020; Liao et al., 2020; Zhang al., 2024; Chang & Bergen, 2024), AR models demonstrate superior image generation ability and scalability compared to MI.Ms. explore different token scanning patterns for flattening 2D image tokens; serbfrow-major raster scanning achieves the best performance. By leveraging the flexibility of the binary codes produced by BAE, investigate code decom- position strategies. Through our exploration, we refine the attempt in (Yu et al., 2023) and conclude that splitting the original code into two subcodes reduces learning complex- ity, improves performance, and saves computational costs. During inference, we explore the optimal sampling strate- gies for AR models and MLMs, highlighting the importance of randommess in generating realistic images.
Our research deepens the understanding of the LLM's capa- bility and behavior in vision generation. The insights can contribute to the design of more efficient and unified large models. In conclusion, our main contributions include:
We thoroughly examine two prevalent language model- ing methods, AR models and MLMs, within the realm of image generation. Our findings suggest that AR holds greater potential in image generation.
We analyze how AR models learn to generate images by examining attention scores across different layers and model sizes. Our findings indicate that AR models effectively learn
Leveraging BAE as the image tokenizer, our results reveal that appropriate vocabulary decomposition helps improve performance and reduce computational costs.
2

---

Elucidating the design space of language models for image generation
We show that AR models can learn effective image pat- terns without inductive bias, identify distinct patterns across model sizes, and offer concise explanation of the scaling law.
drawn from a vocabulary Q of size K. The AR model as- sumes that the probability of the current token qi depends only on its preceding tokens (gi, 92...., Qt-1), framing the generation task as a 'next-token' prediction, with a uni- directional attention mechanism. Specifically, the network learns the probability p(g) = pp(q: Q1.-. ; with the loss function:
Combining all key ingredients of the design space, we reach a strong Elucidated Language model for iMage generation, termed as ELM, and achieve top-tiered performance on the ImageNet benchmark.
Lar - -Ewwp()) [log((g)
(3)
2. Preliminary
Masked Language Models (MLMs) involves binary mask m [mu]] to replace a subset of tokens in q with [MASK], then predict the masked tokens based the un- masked ones from both directions. Specifically, if m, = 1, qi is replaced by [MASK]; otherwise, it remains unchanged. Denote qM the result after applying mask m to q. Hence, these models optimize the following loss function:
2.1. Image Tokenization
Image tokenization typically involves an encoder ENC, a quantizer QUANT, and a decoder DEC. Given an im- age x E Hx3 ENC encodes it to latent variables = ENC(2) E B # xD_ Each spatial vector 2;j in z is then quantized to discrete code qk- Let the quantized latent be denoted as zg, & = DEC(2,) denotes the recon- structed image (Van Den Oord et al., 2017; Razavi et al., 2019; Esser et al., 2021; Yu al., 2023; Hu et al., 2023). All the codes form codebook Q {9x {q*) C RD,, which can be viewed the "vocabulary" if we regard the im- age as special kind of language. A sequence of tokens is obtained by = (q). Q2, qL), where L = H y x y, reshaping zq.
Liim - Expxp(())
logp(q qm)
(4)
viejo,L), m;=1
3. Elucidating the Design Space of language models for image generation
W
In this section, we comprehensively explore the design space of adopting language models for vision generation, includ- ing the tokenizer choice, modeling choice, scan pattern choice, model scalability analysis, vocabulary decomposi- tion strategy, and sampling strategy, and conduct compre- hensive experiments on the 256x 256 ImageNet (Deng et al., 2009) benchmark, mainly use Fréchet Inception Distance (FID) (Heusel et al., 2017) as the evaluation metric.
VQGAN trains the codebook Q alongside ENC and DEC. In this method, each spatial latent vector 2ij RD "looks up" the nearest code qu by minimizing the Euclidean distance (Razavi et al., 2019; Esser et al., 2021):
4q arg min ||ا - qull] E # x quEQ
BAE discretizes the scalar value at each position of Zij- converting it to a binary value (Fajtl et al., 2020; Wang et al., 2023; Yu al., 2023). Specifically, suppose the latent vector Ziy RD is normalized, then each value 24, d E {I..., at the d-th position of Ziy is obtained by:
3.1. Tokenizer Choice: VQGAN V.S. BAE
Table 1: Comparison of VQGAN and BAE', Tokenizers trained on the ImageNet. Generation models are I. sized.
VQGAN
BAF
{{ 0, if 0.5.
= sign( (2 -
(2)
K
1024
16384
216
22
22
232
L. otherwise.
iFID
10.54
7.41
3.32
2.24
1.77
1.68
In this way, the codebook is structured within a binary latent space, with K = =2_ This method is also named "look-up up free" quantization (LFQ) (Yu et al., 2023) because the code can be directly converted into an index by transforming the binary value into a decimal value. The sign function can be replaced by Bernoulli sampling, then xy = Bernoulli(z) (Wang et al., 2023).
gFID(M))
11.21
7.81
3.96
3.65
3.91
gFID(A.) 9.68
6.71
2.78
2.46
2.68
In VQGAN, "code collapse" is a critical issue where a large portion of the codebook remains unused as the codebook size increases, severely limiting the model's efficiency and scalability (Zhu al.. 2024; Baykal et al., 2024). This problem does not occur in BAE, where discrete codes generated using scalar quantization (Mentzer et al., 2023).
2.2. Modeling Methods
Autoregressive (A.) Models involve a sequence of dis- m., m. qr), where each token qi is
IIn this work, we adopt down-sampling factor of f 16 for image tokenizers.
crete tokens q = (qı, 92,
3

---

Elucidating the design space of language models for image generation
This approach guarantees 100% code utilization (Appendix A.5.2) and achieves better reconstruction capabilities (rFID). We also compare the generation capabilities (gFID) of both tokenizers using different codebook sizes (K) and genera- tion modeling approaches, ie.,, MLM and AR. Among all the results, BAE consistently demonstrates superior perfor- mance (Table 1). Based on the above reasons, we build our generation model on BAE tokenizer instead of VQGAN.
rowwrster
[b) volumm---sss
column sigaag
disgonal-iggagg
Figure 3: Scan pattern choices of converting 2D image tokens into ID sequences.
For BAE, we observe that the introduction of Bernoulli Sam- pling during quantization improves generation performance (Table 10). Incorporating the probabilistic element reduces the model's sensitivity to prediction errors (Englesson & Azizpour, 2021), leading to a more robust generation.
scan pattern. In this study, we explore five possible options: row-raster, column-raster, row-zigzag, column-zigzag, and diagonal-zigzag, as illustrated in Figure 3. The results (Table 2) reveal the various scanning patterns achieve good performance, demonstrating that AR models can effectively learn image generation regardless of the scanning pattern. In our subsequent exploration, for AR models, we adopt the row-major raster scanning method, which is the most natural one and yields the best results among all scanning methods.
3.2. Modeling Method Choice: AR V.s. MLM
In this subsection, we evaluate the performance of AR and MLM in image generation with the same BAE tokenizer with a vocabulary size of 26 and training strategy.
RBb Compuetion
I Compartinn
Table 2: Comparison of different scan pattern choices. Tok- enizer: BAE; AR model size: L..
242
2
pattem (a)
(h)
(d)
(e)
354
gFID
2.47
2.88
2.62
2.79
2.71
Epects
Figure 2: Comparison of AR and MLM image gen- eration with 50,000 generated samples. AR consistently outperforms MLM across various model sizes.
3.4. Learning and Scaling Behavior
To further understand the model's learned patterns, we visu- alize the attention maps of AR models with different sizes.
Figure presents the FID score and Inception Score (IS) (Salimans et al., 2016) over the training epochs for both AR and MLM. The results show that AR consistently out- performs MLM across various model sizes. Additionally, AR exhibits higher training efficiency compared MLM, particularly as the model size increases. Research in the language domain has widely recognized that AR models possess greater generative capabilities than MLMss particu- larly as model scales increase (Radford et al., 2019; Raffel et al., 2020; Henighan et al., 2020). Our findings align with these works. Besides, for ALM-XL and MLM-XXI a clear divergence between FID and IS is observed in the later stages of training, where FID continues to improve, while IS declines. Studies point out that when models overfit to gen- erate highly realistic samples (low FID), they may sacrifice diversity, which negatively impacts IS (Chong & Forsyth, 2020; Benny et al., 2021). This issue does not occur with AR models, further highlighting the superiority of AR over MLM in maintaining both quality and diversity.
Layer 13
Layer
Query
012
544
475
274
254
Laver 24
204
Layer
0.0
0.8
60
Query
312
112
0.5
0.4
0.2
Key
Figure 4: Visualization of average attention score of head 0 in AR models over 100 images. We show the results of L (top row), and XL (bottom row) models with the BAE 2-10 tokenizer, respectively. Both models effectively learned to focus on localized information. However, the XI. model learns capture richer global information.
3.3. Scan Pattern Choice
For AR modeling, converting 2D image tokens into ID sequences involves the challenge of choosing an appropriate
4

---

Elucidating the design space of language models for image generation
200
The results reveal that the attention mechanism is primarily focused on local regions of the image, indicating that the AR transformer models effectively learn the importance of local patterns for image generation (Vaswani et al., 201 7). This finding is notable because the model was trained with- out any inductive biases tailored to image data, highlighting the strong capability of AR transformer models across dif- ferent domains. Besides, for the attention pattern, models of varying sizes showed subtle differences: the L-sized model mainly focused on local information, struggling capture long-term relations. In contrast, larger models (XL) with more layers exhibited longer-range attention in certain lay- ers, suggesting they also learn global features (Figure 4) This capability contributes to improved generation perfor- mance, as evidenced by better visual quality and lower FID scores.
2.45
00.00
2.75
2:50
1.25
3.75
200
48:00
Eakrizer Types
Figure 5: AR model performance with different BAR.
means the original tokenizer, and "2-8" denotes the code is split into two 8-bit subcodes. Several key insights are revealed through our design:
Optimal decomposition. A decomposition into two sub- codes is generally optimal, which also reduces computa- tional costs, leading to more efficient and effective gener- ation (detailed result in Table 12). Dealing with two sub- vocabularies of smaller size leads to a more manageable set of possible outcomes, largely reducing the cognitive load the model (Ali et al., 2023; Yang, 2024). Meanwhile, fur- ther increasing the number of subcodes significantly raises the prediction complexity, and the model struggles to opti- mize across three or more classification tasks simultaneously (Limisiewicz et al., 2023). (An increasing training loss was observed when moving from tokenizers 2-8 to 3-8 and 4-8 in XL and XXL models, as in Figure 14).
Clearly, the scaling law (Henighan al., 2020; Kaplan al., 2020) holds for AR models in image generation tasks, as reflected in lower training loss (Figure 15), improved generation performance, and an enhanced ability to capture global information as the model size increases.
3.5. Vocabulary Design
The vocabulary size K' in the BAE tokenizer is determined by the code dimension D, ie, K = _2__ When the vo- cabulary size exceeds a certain threshold, such as 2 (ie, 65,536), next-token prediction becomes significantly more challenging (Ali et al., 2023) and may even become infea- sible due to memory constraints. Despite these limitations, the tokenizer's is effectiveness largely depends on the code dimension, increased code dimension results in an improved reconstruction ability (Figure 5). Recent research also indi- cates that a stronger tokenizer leads to a better generation performance in AR models (Tao et al., 2024). Therefore, addressing the challenge posed by larger codebook sizes (greater D) is crucial.
Vocabulary complexity and model capacity. Larger code dimensions also introduce more complex vocabularies, making it harder for the model to predict the next token. Clearly, more complex tokenizers require more powerful models for effective learning (Tao et al., 2024). For example, the 2-10 tokenizer is optimal for L and XL models, while the 2- 12 tokenizer performs best with the XXL model.
These findings demonstrate the trade-offs between model scale, vocabulary complexity, and decomposition strategies, highlighting the potential of the AR model's ability to effec- tively handle complex tokenization while maintaining high performance across model scales.
Fortunately, the flexibility of binary-quantized codes allows us to decompose each code into multiple subcodes (Yu et al., 2023), resulting in feasible vocabularies. For instance, an 8- bit code like [1, O, 1, O. O, 0, 1, can be split into two b-bit codes: [1, O, 1, 0] and [0,0, 1]. As result, we convert the embedding matrix from a size of 28 x Dectture into two matrices of size 2t Dfeature. where Dfeature is the fea- ture dimension within the AR model. The final embedding is achieved by concatenating the two indexed embeddings and applying a projection to map the dimension back to D Deature. Separate prediction heads are applied.
3.6. Sampling Strategy
Sampling strategy plays a crucial role in vision generation for both diffusion models (Karras et al., 2022; Ma et al., 2023) and language models (Chang et al., 2022: Sun et al., 2024). In this subsection, we thoroughly explore the sam- pling strategies for both AR and MLM, including classifier- free guidance (CFG) (Ho & Salimans, 2022) scale, the in- troduction of randomness, and the number of generation iterations for the MLMs.
We conduct experiments using AR models with BAE that have varying code dimensions (D- = 16. 20, 24, and 32). Quantizers with and without code decomposition are viewed as distinct tokenizers; for example, for D= 16, "1-16"
First, regarding the CFG scale, a linearly increased CFG scale shows the best performance among various scheduling
5

---

Elucidating the design space of language models for image generation
methods (Table 13). Secondly, high degree of randomness is erucial during the sampling process (Figure 6) for both methods. Regarding the introduction of randomness, for the AR model, randomness primarily derives from the top-k filter used when selecting next-token indices based on their confidence scores; a larger & introduces more randomness. For the MLM, randomness stems from the coefficient т of Gumbel noise added to the [MASK] token predicted confidence; larger results in greater randomness. Visual results (Figure 17, 18 and 19) further reflect the influence of CFG and top-k sampling for AR models and the difference from language domain.
ELM-L (2-10), ELM-XL (2-10), ELM-XXL (2-12), and
ELM-2B (2-12), with parameters ranging from 315M to 1.9B.
4. Experimental Results
4.1. Conditional Image Generation
In this section, we compare our ELM models with other pop- ular image generation models on the 256x256 ImageNet. We compare with Diffusion Models including DiT (Peebles & Xie, 2023) and SiT (Ma et al., 2024); Masked Language Models such as MaskGIT (Chang et al., 2022); Autoregres- sive models, including VQGAN (Esser et al., 2021) and LlamaGen (Sun et al., 2024); Visual Autoregressive Model (VAR) (Tian et al., 2024) and Masked Autoregressive Model (MAR) (Li et al., 2024). The comparison result presented in Table 3.
Moreover, empirical results in (Figure 6) also indicate when model size and vocabulary increase, the need for random- ness diminishes, indicating that larger models are capable of capturing broader range of patterns and making more accurale predictions. This observation aligns with the atten- tion and scalability analysis discussed earlier, where larger models demonstrated enhanced capacity to manage both local and global information, reducing the need for stochas- ticity to generate realistic samples.
Table 3: Comparison results of class-conditional image generation on 256x256 ImageNet. indicates that the model generates samples at a resolution of 284x384, which are then resized to 256x256. -re denotes rejection sampling is used.
Type Model
Para.
FID
ISt
Pre.t
Rec.1
DiT-L/2
458M
5.02
167.2
DIT-XI./2
675M
2.27
278.2
0.83
0.57
Difr.
SiT-XL/2 (ODE)
675M
2.15
258.1
0.81
0.60
SiT-XL/2 (SDE)
675M
2.06
277.5
0.83
0.59
Figure 6: The influence of 'randomness' during sampling. Left: top-k on AR; right: on MLM.
MLM
MaskGIT
227M
6.18
182.1
0.8
0.51
MaskGIT-re
227M
4.02
355.6
VQGAN
227M
18.65
80.4
0.78
0.26
Finally, in the context of MIMM-based image generation, the number of sampling iterations typically ranges from I to the total sequence length. Our experiments indicate that approximately 10 iterations strike an effective balance between generation quality and computational efficiency (Figure 20), highlighting the sampling efficiency advantage of MLMs over their AR counterparts.
VQGAN-re
1.4B
5.20
280.3
LlamaGen-L
343M
3.81
248.3
0.83
0.52
AR
LlamaGen-XL.
775M
3.39
227.08
0.81
0.54
LlamaGen-XXL.
1.4B
3.09
253.61
0.83
0.53
LlamaGen-3B
3.1B
3.05
222.33
0.80
0.58
LlamaGen-3B**
3.1B
2.18
263.33
0.81
0.58
VAR-d16
310M
3.30
274.4
0.84
0.51
VAR-d20
600M
2.57
302.6
0.83
0.56
VAR
VAR-d24
1.0B
2.09
312.9
0.82
0.59
3.7. ELM Model
VAR-d30
2.0B
1.97
334.7
0.81
0.61
VAR-d30-re
2.0B
1.80
356.40
0.83
0.57
After thoroughly exploring the design space of language models for image generation, via combining the better de- sign trick, we reach our final Elucidated Language model for iMage generation (ELM). ELM adopts BAE as the image tokenizer and AR as the modeling method with a row-raster scanning pattern; and splits the quantized image code into two subcodes. The vocabulary should align with the model's capacity. Larger vocabularies require more powerful mod- els to handle the next-token prediction task (2-12 tokenizer for ELM-XXL), while smaller models perform better with simpler vocabularies (2-10 tokenizer for ELM-L and XL). For sampling strategy, we choose a high randomness and a linear CFG scale. In total, we construct four ELM versions:
MAR-B
208M
2.31
281.7
0.82
0.57
MAR
MAR-L
479M
1.78
296.0
0.81
0.60
MAR-H
943M
1.55
303.7
0.81
0.62
ELM-L (2-10)
315M
2.17
288.59
0.82
0.55
AR
ELM-XL (2-10)
757M
1.79
332.99
0.80
0.59
ELM-XXL (2-12)
1.4B
1.58
330.43
0.80
0.60
ELM-2B (2-12)
1.9B
1.54
332.69
0.81
0.60
We generate 50,000 samples to evaluate performance us- ing FID, IS, Precision, and Recall following (Dhariwal & Nichol, 2021). Implementation details can be found in the Appendix A.5. Our method (ELM) exhibits scaling law be- havior, with performance improving as model size increases.
6

---

Elucidating the design space of language models for image generation
Our tokenizers, ie, BAE, are only trained ImageNet, we believe further training on larger datasets, like OpenImages (Kuznetsova et al., 2020), would enhance the tokenizer and further boost the generation capability of our ELMs.
presented in Appendix A.3. These experiments effectively
showcase the robustness and adaptability of our approach.
5. Related Work
Large Language Models Language models are founda- tional tools in natural language processing, designed to pre- dict the likelihood of sequences of words or tokens, using Transformer architectures with self-attention mechanism (Vaswani et al., 2017). There are two primary types: au- toregressive (AR) models, like GPT (Radford et al., 2019; Brown, 2020; Achiam et al., 2023), LLaMA (Touvron et al., 2023a;b; Dubey et al., 2024), etc., which generate text one token at a time in a left-to-right fashion, and masked lan- guage models (MLM), such as BERT (Devlin, 2018), T5 (Raffel al., 2020), etc.. which predict masked tokens within sequence using bidirectional context. AR models are particularly effective for text generation due to their sequential nature, while MLMs are better suited for repre- sentation learning by leveraging global context (Chang & Bergen, 2024). The scaling law (Henighan et al., 2020; Ka- plan et al., 2020), which describes the relationship between the growth of model parameters, dataset sizes, computa- tional resources, and performance improvements, highlights the immense potential of AR models.
4.2. Efficient transfer to higher resolution
We further train our model on 512x512 ImageNet to vali- date the effectiveness of the AR mechanism in seamlessly transferring from low-resolution (short-sequence) to high- resolution (log-ssqquenc)) image generation tasks. Specif- ically, we fine-tune a model pre-trained on 256x256 Im- ageNet directly on 512x512 ImageNet. Remarkably, we found that the model achieved excellent performance with only a few epochs of fine-tuning, effectively transitioning from generating 256 tokens to generating 1024 tokens. This significantly improves the efficiency of training image gen- eration models on high-resolution datasets, as the model can first be trained on smaller resolutions and then fine- tuned with minimal effort for larger resolutions. The results on 51 512x512 ImageNet are presented in Table 4, with 50 epochs of fine-tuning.
Table 4: Comparison results of class-conditional image generation on 512x512 ImtgeNet.
Model
Para. Steps FIDJ
1St
Pre.t
Rec.t
Vision Generation Vision generation is a key focus in the current AIGC field, primarily relying on diffusion proba- bilistic models, which generate images by progressively de- noising a random Gaussian noise (Song et al., 2020; Peebles & Xie, 2023; Chen et al., 2023). Transformer architectures are also the dominant backbone in these tasks (Yu et al., 2021; Peebles Xie, 2023). Language models have also been applied to vision tasks. For instance, (Chang et al., 2022), (Li et al., 2023), and (Chang et al., 2023) use bidi- rectional MLMs for image generation, while (Esser et al., 2021). (Yu et al., 2022), and (Sun et al., 2024) employ AR models. Specifically, (Tian et al., 2024), (Li et al., 2024), and (Yu et al., 2024a) incorporate domain-specifti designs to adapt AR models for image generation tasks, Moreover, AR models offer a path toward developing unified models for general artificial intelligence across different modali- ties, as seen in systems like Gemini (Team et al., 2023) and Chameleon (Team, 2024). Apart from existing works, our study emphasizes exploring the inherent potential of the vanilla AR paradigm in the vision domain with minimal modifications. We also investigate the learning behavior of language models, underscoring the versatility and potential of AR models across different modalities.
DiT-XL/2
675M 3000k 3.04
240.82
0.84
0.54
MaskGIT
227M
1500k 7.32
156.0
0.78
0.50
ELM-L (2-8) 312M
250k
4.82
246.87
0.81
0.59
ELM- XL (2-12) 757M
250k 3.29
321.21
0.81
0.60
4.3. Visualization of the scaling law
According to the scaling law of ELM transformers, both loss and performance improve as training data and model parameter size increase. In our experiments, although the original image data (ImageNet) remains unchanged, the token set effectively scales up with different BAEs. We present generated samples using different sizes of ELM models (L. XL. XXL) and tokenizers (1-16, 2-10, 2-12) to illustrate the scaling behavior of ELM models in image generation. Following (Tian et al., 2024), we maintain the same seed and teacher-forced initial tokens across models. The results in Figure 7 clearly demonstrate performance improvements as both the token set and model size scale up.
4.4. Generalization of ELM
To further demonstrate the generalization and applicability of ELM, we evaluate its ability to generate samples from novel classes and perform image editing tasks, with the results in Appendix A.2. Additionally, we test our method on specific datasets, including a human face dataset and a texture dataset, to highlight its versatility; the results are
6. Discussion and Conclusion
Token Randomness in Image Data While images can be discretized and treated as token sequences, the inherent differences between vision and text still exist, In our exper-
7

---

Elucidating the design space of language models for image generation
Scaling up the tokenizer
Scaling up the model size
1-16
2-10
2-12
1-16
2-10
2-12
XL
XXL
XL
XXL
L
XL
XXL
Figure 7: Scaling up behavior of tokenizer and model size. From left to right and top to bottom, there is a trend of improved image detail and structure. reflects the enhanced generation ability that comes with a more refined tokenizer and a larger model.
iments, we observe that the training loss did not converge well with the token-prediction paradigm, whether AR or MLM, image tokens, similar result is also presented in Henighan et al. (2020). However, the models can still generate high-quality images with a low FID, indicating that they have learned sufficient patterns for image generation, although the training loss remains high.
Second, token distribution close uniform indicates that the generation task has higher tolerance for errors (Zhang et al., 2016; Arpit et al., 2017). Since all tokens are nearly equally probable, the model can afford to make less precise token predictions without significantly impacting the quality of the generated output. This characteristic explains why our model, despite its high training loss, can still gener- ate high-quality images and underscores the importance of incorporating randomness during inference.
Our analysis shows that image tokens exhibit distribu- tion much closer random, uniform distribution when compared to language tokens, and they exhibit a lack of orderliness based on bigram distribution and n-gram mod- els' perplexity (Appendix A.4). These observations lead to several key implications. First, it suggests that image data lacks the inherent structure and sequential order that typically present language data, implying that image gen- eration is less dependent strict sequential patterns and more on local patterns relevant to visual reconstruction (Ulyanov et al., 2018), aligning with our earlier analysis of the learning behavior of AR models in image generation.
Conclusion In this work, we thoroughly investigate the use of language models for image generation. We eluci- date the design space of language models for vision gener- ation, including tokenizer choice, model choice, scan pat- tern choice, model scalability, vocabulary design, and sam- pling strategy through extensive comparative experiments. Through our analysis, we have the following findings: (I) BAE demonstrates superior performance as an image tok- enizer compared to traditional VQGAN approaches; (2) AR models consistently outperform ML.Ms and show strong
8

---

Elucidating the design space of language models for image generation
scaling law, (3) row-major raster scanning performs best for flattening image tokens, (4) larger vocabulary size and decomposition design benefit the image generation, (5) sam- pling strategies should also allow for greater randomness and a linear CFG scale. By combining these designs, we reach our final ELM model with near SOTA performance on ImageNet. hope this work will motivate further usage of the AR model across other domains.
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 11315-11325, 2022.
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama. J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. Muse: Text-to-image genera- tion via masked generative transformers. arXiv preprint wrXiv:231.007004 2023.
Chang, T. A. and Bergen, B. K. Language model behavior: A comprehensive survey, Computational Linguistics, 50 (1):293-350, 2024.
Impact Statement
This research demonstrates the potential of language mod- els in image generation, advancing AIGC and multimodal AI. The work has broad applications creativity, educa- tion, and visualization, but raises ethical concerns, such as misuse in generating harmful content biases in outputs. Future efforts should focus responsible development and safeguards to ensure positive societal impact while fostering innovation in generative AI.
Chen, J., Yu, J.. Ge, C., Yao, L.,, Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. Pixart-a: Fast training diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023.
Chong, M. J. and Forsyth, D. Effectively unbiased fid and inception score and where to find them. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6070-6079, 2020.
References
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Proceed- ings of the IEFE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.00 087744 2023.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 24882555 Icee, 2009.
Ali, M. Fromm, M. Thellmann, K., Rutmann, R., Lübbering, M., Leveling, J.. Klug, K., Ebert, J., Doll, N., Buschhoff, S., et al. Tokenizer choice for I1m training: Negligible or erucial? arXiv preprint 100.0223...08744 2023.
Devlin, Bert: Pre-training of deep bidirectional trans-
formers for language understanding. arXiv preprint arXiv: 1810.04800. 2018
Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., al. closer look memorization in deep networks. In International conference machine learning, pp. 233-242. PMLR, 2017.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Baykal, G., Kandemir, M., and Unal, G. Edvae: Mitigating codebook collapse with evidential discrete variational autoencoders. Pattern Recognition, 156:110792 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint 2rX7:2240..21788 2024.
Benny, Y., Galanti, T., Benaim, S., and Wolf, L. Evaluation metrics for conditional image generation. International Journal of Computer Vision, 129:1712-1731, 2021.
Englesson, E. and Azizpour, H. Consistency regularization improve robustness to label noise. arXiv preprint 10X0::2110001222, 2021.
Betker, J., Goh, G., Jing, I.., Brooks, T., Wang, J., Li, I.., Ouyang, L., Zhuang, J., Lee, J.. Guo, Y. al. Improving image generation with better captions. Computer Science. https://ccdn. openai. com/papers/dallle--. pdf. 2023.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12888 12883, 2021.
Brown, T. B. Language models are few-shot learners, arXiv preprint ar 2005 2020.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution
Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In

---

Elucidating the design space of language models for image generation
image synthesis. In Forty-first International Conference Machine Learning, 2024.
235 of Proceedings of Machine Learning Research, pp. 25105-2512 PMLR, 21-27 Jul 2024.
Fajtl, J., Argyriou, V., Monekosso, D., and Remagnino, Latent bernoulli autoencoder. International Conference on Machine Learning. pp. 2964-2977 PMLR, 2020.
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. The open images dataset v4: Uni- fied image classification, object detection, and visual re- lationship detection scale. International journal of computer vision, (28(():1956-13)) 2020.
Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural story generation. arXiv preprint arXiv:/805.000 044333 2018.
Gale, W. A. and Church, W. What's wrong with adding one. Corpus-based research into language: In honour of 1994.
Kwon, W., Li, Z.,, Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E.,, Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Jan Aarts, pp.
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C.,, Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallaey, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., and Mc- Candlish, S. Scaling laws for autoregressive generative modeling, 2020.
Li, T., Chang, H., Mishra, S., Zhang, H., Katabi, D., and Krishnan, D. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2142-2152, 2023.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregres- sive image generation without vector quantization. arXiv preprint arXiv:2406.11 11838 2024.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12590 2022.
Liao, Y., Jiang, X., and Liu, Q. Probabilistically masked language model capable of autoregressive generation in arbitrary word order. arXiv preprint arXiv:2004.11579 2020.
Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47): 2022.
Limisiewicz, T., Balhar, J., and Mareček, D. Tokenization impacts multilingual language modeling: Assessing vo- cabulary allocation and overlap across languages. arXiv preprint arXiv:2305.17117 2023.
Holtzman, A., Buys, J., Du, L.,, Forbes, M., and Choi, Y. The curious case of neural text degeneration. arXiv preprint arXiv:1904.0973 2019.
Liu, Z., Luo, P., Wang. X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Hu, T., Chen, F., Wang, H., Li. J., Wang, W., Sun, J., and Li, Z. Complexity matters: Rethinking the latent space for generative modeling. Advances in Neural Information Processing Systems, 0-2229555297799 2023.
Ma, J., Hu, T., Wang, W., and Sun, J. Elucidating the design space of classifier-guided diffusion generation. arXiv preprint arXiv:2310. 2023.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.00 283306000...861,, 2020.
Ma, N., Goldstein, M., Albergo, M. S., Boffi, N. M., Vanden- Bijnden, E.,, and Xie, S. Sit: Exploring flow and diffusion- based generative models with scalable interpolant trans- formers. 2024.
Karras, T., Aittala, M., Aila, T., and Laine. S. Elucidating the design space of diffusion-basse generative models. Advances in neural information processing systems, 35: (6565-265777 2022.
Mentzer, F., Minnen, D., Agustsson, E., and Tschannen, M. Finite scalar quantization: Vq-vae made simple. arXiv preprint 2rxXiv:330.155000 2023.
Kondratyuk, D., Yu, I.., Gu, X., Lezama, J., Huang. J., Schindler, G., Hornung, R., Birodkar, V., Yan, J., Chiu, M.-C,, et al. Videopoet: A large language model for zero-shot video generation. In Proceedings of the 41st International Conference on Machine Learning, volume
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195 4205, 2023.
10

---

Elucidating the design space of language models for image generation
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. At- tention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Razavi, A., Van den Oord, A., and Vinyals, O. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Wang. Z., Wang. J., Liu, Z., and Qiu, Q. Binary latent diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22576 22585, 2023.
Salimans, T., Goodfellow, I. Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv, 2023.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Yang. J. Rethinking tokenization: Crafting better tokenizers for large language models, 2024.
Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint MxXiv2446600552. 2024.
Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge. J., and Wu, Y. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv: 10.0067, 2021.
Tao, C., Liu, Q., Dou, L., Muennighoff, N., Wan. Z., Luo, P., Lin, M., and Wong, N. Scaling laws with vocabulary: Larger models deserve larger vocabularies. arXiv preprint urXiv:2407.13025 2024.
Yu, J. Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Va- sudevan, V., Ku, A., Yang, Y., Ayan, B. K., al. Scaling autoregressive models for content-rich text-to-image gen- eration. arXiv preprint arXiv:2206.1 2(3):5, 2022.
Team, C. Chameleon: Mixed modal early-fusion foundation models. arXiv preprint arXiv:2405.095 09818,, 2024.
Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Gupta, A., Gu, X., Haupt- mann, A. G., et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXhv:2310.0573, 2023.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J..B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M.. Hauth, A.. et al. Gemini: a family of highly capable multimodal models. arXiv preprint 23/2./23//////// 2023.
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang. L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905 2024.
Yu, Q. He, J., Deng, X., Shen, X., and Chen, L.-C. Ran- domized autoregressive visual generation, 2024a.
Yu, Q.. Weber, M., Deng, X., Shen, X., Cremers, D., and Chen, L.-C. An image is worth 32 tokens for reconstruc- tion and generation. arXiv preprint arXiv:2406.07530 2024b.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprini arXiv:2302. 13971, 2023a.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking general- ization, 2016.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Zhang, Q., Du, T., Huang, H., Wang. Y., and Wang, Y. Look ahead or look around? theoretical comparison between autoregressive and masked pretraining. arXiv preprint arXiv:2407.00933, 2024.
Ulyanov, D.,, Vedaldi, A., and Lempitsky, V. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446 9454, 2018.
Zhu, L., Wei, F., Lu, Y., and Chen, D. Scaling the codebook size of vggan to 100,000 with a utilization rate of 99%. arXiv preprint arXiv:2406. 1837, 2024.

---

Elucidating the design space of language models for image generation
A. Appendix
A.I. Selected samples
Figure 8: Selected 256x256 samples in different classes with ELM-2B (2-12).
12

---

Elucidating the design space of language models for image generation
Figure Generated 512x512 images.
13

---

Elucidating the design space of language models for image generation
A.2. Generalization Ability of ELM models
We evaluated the model's performance on generating images with interpolated class conditions, specifically, aA + (1-ol a B, where A and B are two distinct class labels, [0, 1). This approach effectively tests how the model learns and adapts to conditions, especially under complex scenarios. The results show that the model effectively learns the conditional information, rather than simply memorizing it. Interestingly, when the interpolated classes share similarities, such as golden retriever and a husky, the model generates images that blend features of both classes when a is around 0.5. In contrast, for unrelated classes like sorrel and a beer bottle, the generated images only reflect the features of the class with the higher weight.
We also evaluate the image editing task by selecting specific regions of the original image and transforming them into other objects based class conditions. The results further emphasize the flexibility of ELM across diverse application scenarios.
0.1
0.5
0.9
A: 207 (golden retriever) B: 248 (Husky)
17 (jay)
8:92 (bee eater)
Class Interpolation
A: 284 (Siamese
cat) 8:388 (panda)
A: 339 (sorrel)
B: 6:440 (beer bottle)
Image Editing
Figure 10: Zero-shot generalization performance of ELM. Class interpolation generate images with interpolated class condition, i.c.. aA + (1- B, a (0.1, 0.5, 0.7, 0.9). Image editing allows the model edit the masked region based specific class condition.
14

---

Elucidating the design space of language models for image generation
A.3. Performance different datasets
We also conduct experiments on specialized datasets distinct from ImageNet assess the robustness and versatility of ELM models. Specifically, select CelebA (Liu et al., 2015), which includes 202.599 human face images across 10,177 identities, and the Describable Texture Dataset (DTD) (Cimpoi et al., 2014) that comprises 5,640 images across 47 different categories. We train an ELM-L model with a 2-8 tokenizer on each dataset for 400 epochs using a batch size of 256. The qualitative results (Figure 11 and 12) from these experiments demonstrate the high performance of our model across diverse types tasks.
C
Figure 11: Generated human face images with x56 256.
braided
bubbly
chequered
crystalline
gauzy
honey-combed
interlaced
pleated
waffled
zigzagged
Figure 12: Generated special texture images with x56 256.
15

---

Elucidating the design space of language models for image generation
A.4. Intrinsic difference between language and images
We choose ImageNet from the image domain;
OpenWebText and Shakespeare" from the language. The information of the tokenized dataset is shown in Table 5 and the KL-divergence between uniform distribution is shown in Table 6.
We can see that from Table 6, compared to text generation, image generation exhibits higher randomness. Note that although VQGAN generates tokens with a lower level of randomness, the major reason is the low code utilization-only less than 10% code from the vocabulary is used, and the generated image quality is not satisfying due to the extremely low token utilization.
Table 5: Vocabulary (Codebook) information of image and text.
ImageNet
OpenWeb
WallStreetJournal"
Tokenizer
VQGAN
BAE
BPE
BPE
Vocab size
16384
65536
47589
19979
Token num of train set
327M
327M
9B
38M
Token num of val set
12M
12M
4M
1.5M
Table 6: KL-divergence between token distribution and Uniform Distribution, along with the perplexity of n-gram models.
ImageNet
OpenWebText
WallStreetJournal
Tokenizer
VQGAN (V=16384)
BAE (V=65536)
BPE (V=47589)
BPE (V=19979)
unigram
bigram
unigram
bigram
unigram
bigram
unigram
bigram
Train
1.00
2.16
0.24
0.17
3.25
3.35
Val
0.90
2.12
0.22
0.03
3.27
1.94
Perplexity
368
210
52,538
596,855
2087
395
962
170
A.5. Additional Experiment Results
Implementation Details For the BAE tokenizer, we followed the configuration in (Wang et al., 2023), utilizing Bernoulli sampling during quantization, and trained it for 400 epochs on the ImageNet dataset. For the transformer model, we adopted the LLaMA-2 (Touvron et al., 2023b) architecture, as referenced in (Sun et al., 2024). The depth and feature dimensions of each model size are detailed in Table 7. All language models were trained on 80GB A800 GPUs with batch size of 256, for 400 epochs, using constant learning rate of le-4, weight decay of 0.05, and the AdamW optimizer with B1 0.9 and 32 0.95. The L and XL-sized models were trained on 8 A800 GPUs, requiring approximately 6.4 and 10 days, respectively, to complete 400 epochs. The XXL-sized model, trained on 16 A800 GPUs (2 nodes with 8 GPUs each), took around 12 days to finish training.
For the AR model, we implement mainly follow (Sun et al., 2024), except for the 2B-sized model. The MLM and AR models use the same model architecture. For the MLMs training strategy, we mainly follow (Chang et al., 2022). Specifically, at each training step, we sample a mask ratio for each sample, mask tokens based on this ratio, and train the model to predict the masked tokens. The mask ratio follows a cosine schedule across the generation iterations, meaning the process transitions from less to more information. Early in training, most tokens are masked; as training progresses, the mask ratio sharply decreases, revealing more tokens for the model to handle in later stages.
During the exploration of the design space and other ablation studies, we calculate the FID score using only 30,000 samples for efficiency, except for key comparisons such as language modeling choices (AR vs. MI.M). Specifically, unless otherwise noted, the FID results in Section 3, where we elucidate the design space, are calculated based on 30,000 generated samples.
OOtained from
'The information is obtained from Standford lecture note: https:/wwe..stanoorddcoo http://wwnndddeo//uuusss//////// jurafskylslp3/3.pdff
'We calculate the perplexity with Laplace smoothing(Gale & Church, 1994). The first 10 percent of the training data select the efficiently calculate the perplexity of Open WehTex..
Although the VQGAN tokenizer exhibits lower perplexity compared to BAE, its extremely low code utilization significantly impacting the tokenizer's 's effectiveness.
16

---

Elucidating the design space of language models for image generation
is worth noting that the FID values obtained this way consistently higher than those calculated with 50k samples. For fair comparison with other methods, generate 50,000 samples in the comparison results presented in the Section 4.
Table 7: Transformer model architecture information with different sizes.
Size
depth
dimension
num of head
ELM-L
24
1024
16
ELM-XI.
36
1280
20
ELM-XXL
48
1536
24
ELM-2B
48
1792
28
A. EFFICIENCY ANALYSIS
While our study primarily focuses on understanding the modeling behavior and generation quality of LLMs for image synthesis, we briefly summarize efficiency-relatte insights for completeness. Given that modern LLM acceleration suchn as KV-cache, FlashAttention, and quantization-are mature and directly transferable to our setting, we do emphasize efficiency optimization in the main body.
Overall, AR models offer superior training efficiency, MLM achieves the fastest inference, and memory usage differences mainly emerge at inference due to AR's K V-ccchhe overhead. Detailed comparisons are reported in Table 8.
Table 8:
Comparison of training and inference efficiency across models.
Model
Params (M)
FLOPs (G)
Training Epochs (Converge/Total)
Inference Time (s/img)
DiT/XL-2
675
118.64
~160 1400
0.39 (50 steps)
MLM-XL
741
189.51
200 400
0.10 (10 steps)
AR-XI
741
189.98
~ 100 400
0.15 (256 steps w/ KV-cache)
A.5.2. RESULTS OF DIFFERENT TOKENIZERS: BAE u.s. VQGAN
We trained BAE on the ImageNet dataset using the same model architecture and loss functions as VQGAN from the taming transformers framework (Esser et al., 2021). For a fair comparison, we evaluated the VQGAN-16384 model that also trained on the ImageNet dataset, and assessed its code utilization (see Figure 13. The results clearly demonstrate that BAE outperforms VQGAN, achieving lower reconstruction FID (rFID) and generation FID (gFID) (Table 1 and 9) and significantly higher code utilization (100% 8%).
40
owes Po
(a) BAE-65536
(b) VQGAN-16384
Figure 13: BAE-16 exhibits a higher code utilization than VQGAN. This figure shows a log count number of the appearance of codes the ImageNet training dataset in sorted order. (a) BAE-16, with code dimension of 16, has 65,536 unique codes and achieves 100% code utilization, with code showing extremely low usage. In contrast, (b) VQGAN, with codebook size of 16,384, only utilizes around 1,000 codes, and many of these codes have extremely low utilization.
'Dwnloadeed from htwp//ghtii.oooomomoo///tamattme
17

---

Elucidating the design space of language models for image generation
Table 9: Generation FID of L models with different image tokenizers. A. denotes AR, M. denots MLM. All models are trained on the ImageNet for 1,000,000 iterations, about 200 epochs. 'efgl-3' denotes classifier free guidance (cfg) scale gradually increased to 3.0 following a linear schedule across inference iteration. 'cfg1.5" denotes the efg remains fixed at 1.5 during inference.
tokenizer
code dim
v. size & top-k
cfgl-3 FID(A.)
cfgl-3 FID(M.)
efg1.5 FID(A.)
cfg1.5 FID(M))
VQGAN
256
16,384
6.71
7.81
8.12
8.98
BAE
16
65,536
2.78
3.96
3.87
5.01
Table 10: The influence of Bernoulli sampling with BAE on FID (30k) of generation. We test on AR-L model with BAE
with D = 16, and the model is trained for 150 epochs.
cfg
constant 2
linearl-3
W. Bernoulli
4.72
2.88
W.o. Bernoulli
5.05
3.13
A.5.3. RESULT OF BAE W. AND W.O BERNOULLI SAMPLING
When using BAE to tokenize image feature codes into discrete tokens, the process can either be deterministic, by directly converting values to 0 or based on threshold, or nondeterministic by incorporating Bernoulli sampling during quantization. We compared both methods to assess their impact on the generation task. As shown in Table 10, the nondeterministic approach clearly performs better. This result aligns with the inherent randomness of image token distribution, as discussed in Section 6, and offers greater tolerance for classification errors during next-token prediction.
A.5.4. RESULTS OF DIFFERENT MODELING METHODS: AR MLM
Table 11 shows the detailed final result of the different-sized AR models and MLMs using the basic BAE on the ImageNet 256x256 dataset. Clearly, AR models always show better performance than MLMs.
Table 11:
Comparison of AR and MLM on ImageNet 256x256. The auto-encoder is BAE with code dimension 16. The FID results are obtained on 50,000 generation images.
Size
Method
FID
sFID
1St
Precision
Recallt
MLM
3.67
5.34
272.23
0,8561
0.4597
L
AR
2.38
4.78
271.54
0.8201
0.5650
MLM
3.13
4.95
261.59
0.8159
0.5355
XL
AR
2.14
4.92
289.33
0,8162
0.5834
MLM
3.12
4.86
281.75
0.8393
0.4947
XXL
AR
2.10
4.89
301.22
0.8284
0.5839
A.5.5. COMPARISON BETWEEN CODE-DECOMPOSITIO STRATEGIES
Table 12 shows the detailed results of AR models with different BAE tokenizers. The code decomposition strategy significantly influences the model parameter size and the generation performance. For the code decomposition strategy, splitting a large vocabulary into two smaller sub-vocabularies yields optimal performance by balancing vocabulary size with the number of classification heads. However, using more than two classification heads increases the model's training complexity and impacts learning effectiveness. Figure 14 shows the training loss curves of AR models with 2-8 and 3-8 tokenizers. Both begin with the same initial loss, but clearly, models with 2-8 tokenizers converge better than those with 3-8 across various model sizes. This demonstrates that making three classifications at each position during next-token prediction significantly increases the learning complexity, affecting convergence efficiency.
In general, larger code dimensions improve generation performance by offering finer granularity, but they also introduce more complex vocabularies, making it increasingly challenging for the model to predict the next token accurately, therefore,
18

---

Elucidating the design space of language models for image generation
the vocabulary size should match the generation model's capacity.
Table 12: Comparisons of AR models on class-conditinaal ImageNet 256x256 benchmark.
Model
Tokenizer
Params.
FID1
SFID
1St
Precisionf
Recallt
1-16
443M
2.38
4.78
271.54
0.8201
0.565
2-8
312M
2.34
4.86
281.29
0.8190
0.5573
L
2-10
316M
2.17
4.83
288.59
0.8168
0.5536
2-12
328M
2.34
5.12
316.08
0.8197
0.5487
1-16
900M
2.14
4.92
289.33
0.8162
0.5834
2-8
737M
2.01
4.50
298.99
0.8069
0.5979
XL
2-10
741M
1.73
4.50
332.38
0.8183
0.5823
2-12
757M
1.79
4.82
328.99
0.8027
0.5903
3-8
740M
1.99
5.29
329.66
0.8070
0.5906
1-16
1.56B
2.10
4.89
301.22
0.8284
0.5839
2-10
1.37B
1.65
4.33
328.08
0.8144
0.5933
XXL
2-12
1.39B
1.58
4.78
330.43
0.8034
0.6091
3-8
1.37B
1.67
4.99
325.06
0.8020
0.6054
4-8
1.37B
2.02
5.66
321.37
0.7913
0.602
2B
2-12
1.90B
1.54
4.81
332.69
0.8093
0.5968
brite
4.4
123
2.5
425
(a) AR-XL with and 3-8 tokenizer
(b) AR-XXL with and 4-8 tokenizer
Figure 14: Different vocabulary decomposition strategies vary lot on the training losses. Clearly, introducing more than two classification heads will increase the model's training complexity and learning effectiveness.
A.5.6. SCALING BEHAVIOR OF AR MODELS
To further illustrate the scaling behavior of AR models, in addition the learning pattern visualization and image generation results present the main pages, also provide the convergence curves during training. Figure 15 shows the loss trends for all AR model sizes (I., XI., XXL., and 2B) with the 2-12 tokenizer. All models successfully converged, and the final loss consistently decreased as the model size increased.
A.5.7. COMPARISON BETWEEN SAMPLING STRATEGIES
For MLMs, we conduct search to find the optimal CFG scale, iteration number, and temperature 7 for Gumbel noise. For AR models, we search for the best CFG scale and top-k threshold.
Classifier-free guidance (CFG) plays crucial role in conditional image generation, but it involves balancing the trade-off between image diversity and individual image quality. We searched for the optimal CFG scale for all models. Additionally, we found that using dynamic CFG schedule significantly improves performance. We tested several CFG scheduling methods (see Figure 16), with the results summarized in Table 13. Figure 17 and 18 further show the qualitative comparison of different CFG scale.
Notably, the effect of top-k varies substantially between vision and language domains. image generation, small (e.g..
19

---

Elucidating the design space of language models for image generation
Initial Loss: 8.32
ARXXL
6.7
6.5
1.00
1.25
1.50
Rerations
Figure 15: AR exhibits good scaling law. Training losses of all AR models are with the BAE 2- 1-12 tokenizer. All models trained for 2,000,000 iterations, equivalent 400 epochs, except the 2B model, which had to be stopped earlier due time constraints.
Table 13: Different CFG strategies vary greatly on FID. All results are based on AR-L with tokenizer 2-10.
CFG-scale
1.5
2
2.5
cosl-4
logl-4
linearl-4
squarel
r-squarel-4
FID (30k)
2.98
3.35
3.58
2.86
2.70
2.48
4.94
3.57
30
100
130
200
230
Ier
Figure 16:
Curves of CGF scale with respect to iteration times under different CFG schedules.
Table 14:
The influence of top-k in sampling process for AR models with decomposed vocabulary.
2-8
2-10
2-12
k 180
210
256
800
900
1024
2600
2800
3000
L
2.97
2.84
2.74
2.55
2.50
2.48
2.68
2.56
2.67
XL
2.46
2.36
2.40
2.13
2.11
2.03
2.11
2.10
2.11
XXL
2.08
2.04
1.95
1.90
1.90
1.95
<100) leads to overly smooth, low-diversity outputs, and optimal values often reach k z3 0.5x vocabulary size to capture texture richness (Figure 19). In contrast, text generation typically requires only k 20-100 to achieve diverse, fluent
20

---

Elucidating the design space of language models for image generation
Constant CFG 1.5
Constant CFG 4.0
Linear CFG 1.0-3.0
Linear CFG 1.0-6.0
Figure 17: Visual comparison of different CFG setting. The top-k is set 500. Clearly, compared to a constant CFG value, gradually increasing CFG value tends enhance image diversity and the alignment with the real data distribution, explains better FID score. Besides, smaller CFG value also brings more diversity and more background details.
Constant CFG 1.5
Constant CFG 4.0
Linear CFG 1.0-3.0
Linear CFG 1.0-6.0
Figure 18: Visual comparison of different CFG settings. The top-k is set to 100. Compared to the CFG scale, top-k setting has more influence on the image visual quality. When is small, a linearly increased CFG scale will bring more diversity and background information.
outputs, even with vocabularies of 50,000-128,000 tokens (Holtzman et al., 2019; Fan et al., 2018). This reflects fundamental differences in the distributional properties visual and linguistic data.
21

---

Elucidating the design space of language models for image generation
Top-k: 10
Top-k: 100
Top-k: 500
Top-k: 1,000
Figure 19: Visual comparison of different top-k settings. The CFG is set to a constant value of 2. Clearly, when the value of k is small, the generated images tend to be overly smooth and sharp, lacking background and fine details. In contrast, larger values lead to images with richer details that better align with the distribution of real-world datasets.
10
MLM-L
MLM-XL
MLM-XXL
30K-FID
Min: 88
3
Min: 3.37
5
10
15
20
30
60
100
256
Number of Iteration
Figure 20: The influence of iteration time (different mask ratio) in sampling process FID scores for MLMs.
Table 15: The best sampling strategy regard to FID score for all models.
Method
Tokenizer
Model
Best Strategy
1-16
L
linear CFG 1-3; 7=9.0, iteration number=10
MLM
1-16
XL XXL
linear CFG 1-3; 7=5.0, iteration number=10
1-16
L& XL & XXL.
linear CFG 1-3; top-k=65536 (all)
2-8
L
linear CFG 1-4; top-k=256 (all)
2-8
XI. & XXL.
linear CFG 1-4; top-1=210
2-10
L
linear CFG 1-4; top-k=1024 (all)
AR
2-10
XL & XXL
linear CFG 1-5; top-k=1024 (all)
2-12
L& XI. & XXI.
linear CFG 1-5; top-1=2800
3-8
XL & XXL
linear CFG 1-5; top-k=180
4-8
XXL
linear CFG 1-5; top-k=180
22

---

Elucidating the design space of language models for image generation
A.6. The flexibility of ELM to generate any-size image
To further explore the capability of AR models in image generation, generate images with more than 16x16 tokens without modifying the model (Figure 21). Although the model's receptive field is limited to 256 tokens, generate 'streaming" images by looking back at few tokens. This demonstrates the greater flexibility of AR models compared to diffusion models, highlighting the potential of AR models for applications in other domains.
easily
Figure 21: AR models are flexible to generate images with any size based previous context.
A.7. Limitation
Our work has limitations. While we propose several improvements for AR models, the fundamental issue of optimizing highly random token distributions remains. Traditional next-token prediction using classification loss may not be the most optimal training objective for such tasks, suggesting that more suitable objectives should be explored in future research. For instance, MAR (Li et al., 2024) has made promising progress by introducing diffusion loss into AR models, while VAR (Tian et al., 2024) presents valuable perspective by altering the image tokenization approach. We hope our analysis will inspire further exploration and innovation in utilizing language models for vision generation, as well as other modalities.
A.8. More generated samples
We present more randomly picked 256 x 256 samples here to straightforwardly show the performance of our model.
23

---

Elucidating the design space of language models for image generation
Figure 22: Randomly sampled images from classes 88 (macaw), 130 (flamingo), and 145 (king penguin).
24

---

Elucidating the design space of language models for image generation
Figure 23: Randomly sampled images from classes 107 (jelly fish), 980 (volcano), and 511 (convertible).
25


Reviews:
[{"review_id": "X0crEC3gJC", "reviewer": "Reviewer_ED5q", "summary": "In this work, authors focus on the research topic of AR image generation, and conduct extensive studies focusing on 1) tokenizer, 2) AR model design (AR or Mask) 3) image scan direction. With large number of stduies, authors proposed ELM, which is able to achieve sota performance in ImageNet256 generation task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments! Below, we will answer your concern in detail.\n\n```W1: Difference performance of VQGAN from LlamaGen```\n\nTo ensure a **controlled comparison between structurally different quantization methods**, we use the **original VQGAN** from Taming Transformers [Esser et al., 2021], downloaded directly with rFID from the official repository （[taming-transformers-github](https://github.com/CompVis/taming-transformers)).\n\nLlamaGen tends to show that **discrete image tokenization** can achieve a reconstruction ability **close to continuous ones**. They **re-train VQGAN** with **L2-normalization on latent codes** refer to [1] to **improve codebook usage and representation smoothness**, which likely accounts for their stronger rFID results.\n\nTo further address the reviewer’s concern, below table shows the **comparison between LlamaGen's VQGAN, the original Taming VQGAN, and BAE** (all with a down-sampling rate of 16 on 256px ImageNet):\n|     | VQ-16384 (taming) | VQ-16384 (LlamaGen) | BAE-2^16 | BAE-2^24 |\n|-------|-------------------|---------------------|-----------|-----------|\n|code usage| 8% |    97%               |  100%      |   100%      | \n| rFID     | 7.41             | 2.19                | 3.32      | 1.77      |\n| gFID(M.)  | 7.81         | 4.51                | 3.96      | 3.91      |\n| gFID(A.)   | 6.71         | 3.45                | 2.78      | 2.68      |\n\nThis comparison shows that **BAE can still outperform both**,  ensuring **100% code utilization**, and the key advantage of **supporting larger codebooks** with better generation performance remains valid.\n\n```W2. Discuss with Infinity paper - bitwise self-correction method. ```\n\nBoth our method and Infinity∞ highlight that **introducing stochasticity or 'error' during training** —via Bernoulli sampling or bitwise self-correction—can improve **autoregressive image generation** by enhancing **robustness to prediction errors**. This suggests that **stochastic modeling** is a valuable direction for AR-based synthesis.\n\nThe key difference lies in the **series modeling formulation**:\n* Infinity∞ builds on domain-specific serialization and **next-scale prediction**, where self-correction mitigates **error accumulation** across scales.\n* Our approach adopts **standard image tokenization** and treats images as language; Bernoulli sampling helps address the **inherent randomness** of image token sequences, improving **tolerance to uncertainty**.\n\nAs Infinity∞ and our work were developed **concurrently**, we were not aware of it at the time of writing and therefore did not include a comparison. We appreciate the reviewer’s suggestion, and we will incorporate a discussion of this work in the revised version of our paper.\n\n``` W3. We will re-design the Figures to improve the readability ```\n\n```W4. Review on Scan Pattern Choice section```\n\nWe study scan patterns to evaluate whether **generation order** impacts performance in the image-as-language setting, where token sequences lack inherent structure. Zigzag, commonly used in tasks like image compression and video coding, serves as a meaningful alternative to raster.\n\nOur results provide **empirical grounding** for choosing a raster scan in AR image generation, and the **minor performance differences** across scan patterns further suggest that large language models are **robust to token order variations**.\n\n```W5. Discuss with the scaling law in LlamaGen and VAR```\n\nWe agree that works like LlamaGen and VAR have demonstrated the scalability of AR models for image generation. Different from ours, VAR provides a **new** image autoregressive modeling mechanism. Our study is in line with LlamaGen, adopting a **language-centric** approach **without** domain-specific design, and offers complementary insights. Beyond generation quality scaling in LlamaGen, we further analyze **training dynamics, attention behavior, AR vs. MLM scaling trends**, and **the interactions between vocabulary and model capacity**, offering a broader perspective on what enables scalable image generation with LLMs. We believe our findings **complement and extend** those of LlamaGen and VAR and help build a more principled foundation for scaling image-as-language models.\n\n```W6. question about the claim in Line 243, about the vocabulary threshold```\n\nWe appologize that the statement in Line 243 is imprecise. Our intention was to highlight that as the vocabulary size approaches or exceeds **2^20**, the memory required for the output projection (softmax layer) in next-token prediction becomes a major bottleneck—especially in **image generation, where larger vocabularies are often needed for higher fidelity and resolution**.\n\nWhile LLMs like LLaMA handle vocab sizes up to ~200k (2^17) without issue, our use of 2^16 was meant as an **illustrative threshold**, not a hard limit. We will revise the wording for clarity in the final version.\n\n [1] Yu et al. 2021, Vector-quantized image modeling with improved vqgan", "claims_and_evidence": "Partly. Most of the claims are presented in related works.", "methods_and_evaluation_criteria": "Make sense.", "theoretical_claims": "Not suitable.", "experimental_designs_or_analyses": "Yes. Please see the weakness and strengthes.", "relation_to_broader_scientific_literature": "See weakness about Infinity paper and LlamaGen paper\n\nInfinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis\n\nAutoregressive Model Beats Diffusion: Llama for Scalable Image Generation", "essential_references_not_discussed": "See weakness about Infinity paper and LlamaGen paper\n\nInfinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis\n\nAutoregressive Model Beats Diffusion: Llama for Scalable Image Generation", "other_strengths_and_weaknesses": "Strengths: \nExtensive experimental results demonstrate the effectiveness of proposed solution.\n\n\nWeakness:\n1. I have question about the results presented in Table 1. As shown in LlamaGen paper, 16k codebook VQGAN is able to achieve 2.19 rFID, trained with classical training receipe.  However, in the submission, only around 7.41. How to explain this significant performance gap?\n\n2. Starts from line 171, authors claimed that \"the introduction of Bernoulli Sampling during quantization improves generation performance\". I think it is very closely related to Infinity paper  bitwise self-correction method. However, no discussions involoved.\n\n3. I would suggest better re-design the Figure 2, the texts inside are too small.\n\n4. For Scan Pattern Choice section, I was wondering if that makes sense since 1) raster is a common-practice choice 2) studies in Mamba related works have studied this choice and results indicate raster is simple and performs good 3) the common-practice raster choice also performs best as shown in Table 2.\n\n5. I would say that  scaling law for AR image generation has already been validated by LlamaGen and VAR. \n\n6. I have a question about the claim in Line 243 \"When the vocabulary size exceeds a certain threshold, such as 2\n16 (ie, 65,536), next-token prediction becomes significantly more challenging and may even become infeasible due to memory constraints.\" If that is true, how can we train LLMs? for example, the vocabulary size in Llama would be around 128k, and we can train Llama smoothly.", "other_comments_or_suggestions": "Please see the weakness.", "questions_for_authors": "Please check the weakness and looking forward to the rebuttal.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fGeBpRy6fo", "reviewer": "Reviewer_e5YQ", "summary": "This paper systematically explores the design space of large language models for image generation, evaluating factors such as tokenization, model architecture, scanning patterns, vocabulary decomposition, and sampling strategies. The proposed ELM achieves state-of-the-art FID scores, demonstrating the potential of LLMs for vision tasks. Key findings include: 1. Autoregressive models outperform masked language models for image generation, benefiting from sequential modeling. 2. Binary autoencoder tokenization is superior to VQGAN, reducing codebook collapse and improving reconstruction quality. 3. Row-major raster scanning yields the best performance among different image tokenization strategies. 4. Scaling laws hold for AR-based image generation, with larger models capturing both local and global features. Despite promising results, the study lacks comparisons with the latest diffusion models, does not evaluate text-to-image generation, and does not analyze computational efficiency, leaving room for further investigation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments! We will address your concerns in detail.\n\n```Limit 1. no results on text-to-image generation```\n\nOur main goal is to assess the effectiveness of LLMs as a unified generation paradigm, especially when applied to images. To isolate modeling factors like tokenizers, AR vs. MLM, and sampling, we use **class-conditional generation** for its **simplicity and control**. While our findings (e.g., on scalable tokenization and AR modeling) are **relevant to text-to-image generation**, we consider T2I a **distinct challenge**—requiring further study on alignment, prompt structure, and retrieval—which requires delicate future work.\n\n```Limit 2 & W1. Efficiency analysis```\n\nOur work focuses on understanding the **modeling behavior** and asses the **effectiveness** of LLMs for image generation. As LLMs acceleration techniques (e.g., KV-cache, FlashAttention, quantization) are well-developed and **directly transferable** to our setting, we do not emphasize efficiency in this paper. To address your concern, we summarize key efficiency insights below:\n* **Training efficiency**: AR is more efficient than MLM and diffusion. During training, each AR step predicts all tokens, while MLM predicts only a subset. As shown in Figure 2 in our main paper, AR converges in ~100 epochs, MLM in ~200, and DiT needs ~160 (refer to [Scalable Diffusion Models with Transformers]).\n* **Inference speed**: Vanilla AR is slower due to sequential decoding (e.g., 256 steps), but KV-cache accelerates it by ~4×. Summary: MLM > AR (with KV-cache) > Diffusion (DiT).\n* **Memory usage**: All models are **similar in training**. During inference, AR’s KV-cache increases memory usage with sequence length; MLM and diffusion require fewer steps and less memory per sample.\n\nWe summarize FLOPs, training, and inference speed across AR, MLM, and DiT with matched model scales:\n|  | params(M) | FLOPs(G) | training epochs (converge/total) | inference time(sec/img) |\n|---|-----------|-----------|----------------------------------|------------------------|\n| DiT/XL-2 | 675 | 118.64 | ~160 / 1400 | 0.39 (50 steps) |\n| MLM-XL | 741 | 189.51 | ~200 / 400 | 0.10 (10 steps) |\n| AR-XL | 741 | 189.98 | ~100 / 400 | 0.15 (256 steps with KV-cache) |\n\n``` Issue1. rigorous proof on AR&MLM in image domain```\n\nWhile we do not provide formal theoretical proof that AR is superior to MLM for image generation, our systematic empirical study offers **meaningful evidence** supporting this claim.\nTo the best of our knowledge, **no existing work** in language or vision has established a rigorous theoretical justification. However, under our image-as-language framework, some NLP arguments for AR over MLM are partially **transferable**—e.g., AR benefits from **training-inference consistency**, while MLM suffers from **exposure bias** [1], and AR models the **full joint distribution**, unlike MLM [2].\n\nMoreover, the lack of standardized image tokenization adds complexity, making theoretical analysis more difficult and **beyond the scope** of this paper. We believe that our findings provide a **strong foundation** and **motivating evidence** for such future theoretical work.\n\n``` Issue 2. The impact of KL-divergence analysis```\n\nWe included the KL-divergence analysis as an **additional diagnostic tool** to investigate **why**, during next-token prediction on tokenized images, the **training loss remains high**—despite the model generating **high-quality images**, unlike in typical language modeling.  Actually, we conduct some analysis of the impact on model optimization in our paper.\n* Training loss curve **plateaus** rather than fully converging, as shown in **Figure 14 & 16 in Appendix**.  The KL-divergence analysis shows that image tokens are more random and less sequentially structured than language tokens, leading to less accurate predictions. \n* Attention Analysis: Attention maps analysis in Sec. 3.4 reveals strong local structure, and larger models capture more global dependencies, indicating **effective learning and scaling** despite the high loss.\n\n```W2. Why choose VQGAN```\n\nWe acknowledge the emergence of stronger tokenizers beyond VQGAN (e.g., hierarchical, residual, multi-scale) with improved utilization and reconstruction. However, we chose VQGAN as our baseline due to its **widespread use**, **reproducibility**, and **alignment with prior works** (e.g., MaskGIT, LDM, LlamaGen), enabling **controlled** comparisons. We include BAE as a **inherently** different tokenizer: unlike VQGAN’s **vector-level** quantization, BAE performs **bitwise scalar** quantization, allowing us to isolate the effect of granularity on the quantization. Its design is also compatible with other advanced quantization methods, suggesting potential for **future integration**.\n\n[1] Song et al., 2019. MASS: Masked Sequence to Sequence Pre-training for Language Generation.\n\n[2] Ghazvininejad et al., 2019. Mask-Predict: Parallel Decoding of Conditional Masked Language Models.", "claims_and_evidence": "Supported Claims:  \n- The paper claims that large language models (LLMs) can achieve near state-of-the-art image generation without specialized inductive biases, by optimizing tokenization, modeling, scanning patterns, vocabulary, and sampling strategies.\n- Experimental results support this claim, as the proposed ELM model achieves an FID of 1.54 (256×256) and 3.29 (512×512) on ImageNet, which is comparable to leading diffusion-based methods.\n- The study asserts that autoregressive (AR) models outperform masked language models (MLMs) in image generation, which is well-supported by quantitative comparisons (lower FID, better scalability).\n\nPotentially Problematic Claims:  \n- The assumption that better tokenization alone can lead to improved performance is somewhat oversimplified—while BAE performs better than VQGAN, the impact of other factors such as model architecture and training strategy is not isolated/detailed in ablations.", "methods_and_evaluation_criteria": "Appropriateness of Methods:  \n- The study explores multiple key design choices in tokenization, modeling, scanning patterns, vocabulary size, and sampling strategies, which are highly relevant for improving LLM-based image generation.\n- Frechet Inception Distance (FID) is used as the main evaluation metric, which is a standard benchmark for generative models, making the comparison meaningful.\n- Experiments are conducted on both 256×256 and 512×512 ImageNet, testing the scalability of the approach.\n\nPotential Limitations:\n- Lack of real-world benchmarks: Evaluations focus on class-conditional image generation on ImageNet, but no results are shown for text-to-image generation, which is a more practical setting. (This might not be a sufficient reason to reject this paper.)\n- Computational efficiency is not discussed: While the study argues that AR models are effective, there is no analysis of training time, memory usage, or inference speed.", "theoretical_claims": "Key Theoretical Contributions:  \n- The paper discusses scaling laws in AR models, showing that larger models capture both local and global information, leading to better image generation performance.\n- It claims that MLMs struggle with image generation due to the lack of strict sequential dependencies in images, which aligns with prior findings in NLP.\n- The study highlights the difference between text and image token distributions, suggesting that image tokens exhibit greater randomness than text tokens, which presents unique challenges for AR models.\n\nPotential Issues:  \n- While the analysis of AR vs. MLM models is insightful, it lacks rigorous mathematical justification—there are no formal proofs to support the claim that AR is inherently better for image generation.\n- The KL-divergence analysis of token distributions is interesting, but its impact on model optimization is not fully explored—how does randomness in tokenization affect convergence and learning stability?", "experimental_designs_or_analyses": "Strengths:  \n- Comprehensive exploration of design choices: The paper systematically evaluates tokenization, modeling, scanning patterns, vocabulary decomposition, and sampling strategies.\n- Comparative performance analysis: The study includes quantitative results on multiple LLM architectures and different vocabularies, providing strong empirical support for its claims.\n- Visualization of scaling laws: The analysis of attention patterns in AR models provides useful insights into how larger models learn hierarchical image representations.\n\nWeaknesses:  \n- No efficiency analysis: The study does not compare training/inference time, FLOPs, or memory consumption between AR models/MLM models/diffusion models.\n- The choice of tokenizers: The study dives deep into two tokenizers, VQGAN and BAE, and demonstrate the latter one’s effectiveness. However, the reason for choosing VQGAN is unclear and there are many other tokenizers that achieve much better performance than VQGAN with significantly better token utilization ratio.", "relation_to_broader_scientific_literature": "- Language Models for Image Generation: The study builds on prior works applying LLMs to vision tasks (e.g., VQGAN, LlamaGen, VAR, MAR) but extends the exploration of architectural choices beyond existing models.\n- Autoregressive vs. Masked Language Models: Consistent with NLP research, this paper finds that AR models scale better than MLMs, reinforcing prior insights from text-based LLMs.\n- Comparison with Diffusion Models: The study follows research on diffusion-based image generation but suggests that AR models can achieve comparable quality under the right design choices.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "I will raise my score if the above questions are appropriately addressed.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A4vOPr8ubq", "reviewer": "Reviewer_xmQD", "summary": "This paper systematically explores how to utilize LLMs for image generation, providing detailed comparisons and analyses across tokenization methods, modeling approaches, scan patterns, vocabulary design, and sampling strategies, offering some interesting conclusions. Based on these integrated experiments and analyses, the authors propose Elucidated Language Models, achieving remarkably good performance on ImageNet class conditional generation, demonstrating the significant potential of using LLMs for image generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comment! Below, we will address your concern in detail.\n\n```W1. methodological innovation```\n\nWe acknowledge that our work does not center around proposing a new model architecture, but rather focuses on **systematically analyzing and understanding** how existing design components interact within the context of **LLM-based image generation**. We believe this perspective is **important and timely**, especially in light of recent developments such as **GPT-4o**, which has been introduced as a **natively AR large multimodal model** capable of cross-modal reasoning and generation, demonstrating impressive performance across tasks. These advances further support the need for a deeper understanding of AR modeling as a unified generative paradigm, and our study lays the **empirical foundation** for such future multimodal research. In particular, our contributions lie in:\n* Providing a fair comparison between AR and MLM under standard image tokenization;\n* Demonstrating how tokenization and dictionary structure affects model performance and interacts with model scaling;\n* Revealing surprising robustness of AR LMs to image token randomness and their ability to generate high-fidelity outputs;\n* Offering design principles that can guide future LLM-based multimodal generation systems.\n\nWe view this paper as an **analytical and foundational contribution**, aiming to guide future methodological innovations. We appreciate the reviewer’s suggestion and will clarify this positioning in the revised version.\n\n```W2. baselines compared on the ImageNet-512```\n\nOn the ImageNet-512 benchmark, we did **not** re-train a model from scratch at 512×512 resolution. Instead, we performed a **lightweight fine-tuning** of our model that was originally trained on 256×256 images, using only **a few** additional training steps. Our primary goal in this experiment is to demonstrate the **resolution scalability and efficiency** of ELM, so we only choose some typical methods to compare with. Training directly on 512-resolution images from scratch would require approximately **4× more compute** compared to 256-resolution training. However, with a token-prediction-based architecture, we find that simple fine-tuning can achieve strong high-resolution performance with significantly reduced cost—a major advantage over diffusion models, which typically require retraining and redesigning the entire pipeline for higher resolutions.\n\nAs a reference baseline, we chose to compare against DiT, a strong class-conditional diffusion model trained natively on 512×512 images. We also included MaskGIT as a discrete-token-based generation model for completeness. We acknowledge that VAR has also reported ImageNet-512 results. Below, we include a comparison with VAR.\n| Model | Params | Steps | FID ↓ | IS ↑ | Precision ↑ | Recall ↑ |\n|-------|---------|--------|--------|--------|-------------|-----------|\n| DiT-XL/2 | 675M | 3000k | 3.04 | 240.82 | 0.84 | 0.54 |\n| MaskGIT | 227M | 1500k | 7.32 | 156.00 | 0.78 | 0.50 |\n| VAR-d36-s | ~2B | 1250k | **2.63** | 303.2 | - | - |\n| ELM-L (2-8) | 312M | 250k | 4.82 | 246.87 | 0.81 | 0.59 |\n| ELM-XL (2-12) | 757M | 250k | 3.29 | **321.21** | 0.81 | 0.60 |\n\nAlthough our results are slightly behind those of VAR, it is important to note that VAR uses more than 2× the number of parameters and was trained for a significantly longer time. \n\n```W3. scaling law for FID```\n\nThank you for the helpful suggestion. While Figure 7 primarily illustrates visual improvements across scales, we would like to clarify that we **do provide quantitative evidence** of a scaling law for FID throughout the paper:\n* In Table 1, we report FID scores across models with increasing capacities, showing a clear performance improvement as model size increases.\n* In Figure 2, we plot FID curves over training epochs for AR and MLM models of different sizes, which further demonstrates the impact of scaling on convergence and generation quality.\n* In the Appendix (Table 11), we present a comprehensive breakdown of FID scores for AR models across multiple scales—from L, XL, XXL to 2B—under different vocabulary designs, further reinforcing the consistency of the observed scaling trend.\n\nTaken together, these results clearly support a scaling law for FID in AR-based image generation under the image-as-language paradigm, similar in spirit to what is shown in the VAR paper. We will make this connection more explicit in the revised version, and thank you again for encouraging us to clarify this aspect.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "Yes. I checked the results of the proposed method on class-conditional generation for ImageNet-256 and ImageNet-512, as well as the ablation studies in the paper.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "## Strengths\n1. This paper thoroughly explores and analyzes the impact of different designs on LLM performance for image generation, providing valuable analysis and experiments for LLM-based Image Generation.\n2. Based on these designs, the authors propose a strong baseline, achieving competitive results on ImageNet-256 and ImageNet-512 class-conditional image generation benchmarks.\n3. The paper is written clearly and is easy to understand.\n\n## Weaknesses\n1. The paper doesn't offer much methodological innovation, but rather focuses on analyzing which existing designs are more suitable for LLM-based image generation.\n2. There are relatively few baselines compared on the ImageNet-512 Benchmarks - what is the reason for this? VAR also has ImageNet-512 class-conditional image generation results, why weren't these compared?\n3. The scaling law in Figure 7 only provides visual effects. Could the authors provide a scaling law for FID similar to the one in the VAR paper?", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J8B0JYsms0", "reviewer": "Reviewer_uJCp", "summary": "The paper investigates the application of large language models (LLMs) to image generation, demonstrating that LLMs can achieve near state-of-the-art performance without relying on domain-specific designs. The study also analyzes the learning and scaling behavior of autoregressive models, showing that larger models capture more useful information.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your valuable comments. Below, we will address your concern in detail.\n\n``` Discuss with LlamaGen and MAR```\n\nWe agree that both [1]LlamaGen(Sun et al.) and [2]MAR(Li et al.) have made important contributions to advancing autoregressive image generation. We would like to clarify that we **do** compare against both works in **Table 1**, and further discuss MAR in the appendix (**A.7 Limitation section**) as a **parallel study** that proposes new modeling approaches and domain-specifc design for AR to effectively work in the **continuous** image domains.\n\nCompared to MAR, our work is more aligned with LlamaGen, as we both follow the same direction of **directly adapting language models for image generation** to explore their modeling potential. However, our work differs in that we conduct a **more comprehensive and systematic analysis** of the image-as-language modeling framework. In particular, we study: \n* The effect of tokenizer choice (VQGAN vs. BAE), \n* Modeling paradigms (AR vs. MLM),\n* Detailed learning behavior reflected by attention pattern and loss dynamics,\n* Vocabulary design through scalable codebooks under BAE,\n* And an in-depth ablation and analysis of sampling strategies.\n\nWe believe our findings provide a **complementary perspective** to LlamaGen and MAR, and offer insights that could be valuable when combined with more advanced objectives or tokenization schemes in future work. We appreciate the suggestion and will incorporate the detailed discussions more clearly in the revised version.\n\n```W1. technical novelty```\n\nWe believe that our work makes a **meaningful and timely** contribution by **systematically analyzing** the design space of applying **language models in the image domain**, providing insights into adopting LLMs as a unified method and guiding future methodological innovations, especially in light of recent developments of GPT-4o, a powerful natively AR large multimodal model. You may find our detailed response to **W1 from Reviewer xmQD**.\n\n```W2. Comparison with MAR```\n\nThank you for pointing out that our conclusion on the choice of generation paradigm differs from MAR. However, given the differences in our methods, we believe this divergence is entirely reasonable.\n* We adopt a standard language modeling view, treating images as **discrete token sequences** with no image-specific modifications. Under this setup, we systematically compare AR and MLM using **standard tokenizers** and sampling, finding AR consistently superior in quality and scalability. Besides, this discrete token framework can integrate well with **LLM-based multimodal extensions**.\n* MAR presents a **new generative modeling approach** tailored for **continuous** image data with **diffusion loss** and **no quantization**. Their finding that MLM outperforms AR is valid within this setting but reflects a **fundamentally different modeling goal and assumptions** from ours.\n\nWe view these works as complementary: MAR explores **domain-specific continuous modeling**, while we investigate how **standard language modeling paradigms transfer to vision**. Based on the different tokenizers and training objectives, we come to different conclusion according to the generation modeling method, which totally makes sense. We have acknowledged this in our appendix and will add the discussion to the main paper.\n\n```W3. qualitative study and empirical discussion on top-k and cfg```\n \nWe agree that understanding top-k sampling and classifier-free guidance (CFG) is key to characterizing image generation behavior, especially in contrast to standard LLM decoding. We have included an extensive analysis of both strategies (Figure 6, Table 12). We further provide **qualitative** comparisons (linked [here](https://anonymous.4open.science/r/ICML2025rebuttal-D02C/qualitive_topk_CFG.pdf)). We find that image generation requires **larger top-k values** and **moderate CFG weights** to achieve a better FID score. However, there is a **trade-off** between FID score and visual quality. A large k (close to the vocab size) introduces more texture details and diversity, making the outputs closer to the real distribution and thus yielding lower FID. In contrast, a moderate k often produces visually more appealing results, with cleaner and smoother images.\n\nWhile CFG is rarely applied in LLM-based text generation, top-k sampling is widely adopted in both domains:\n* In image generation, small k (less than 100) values lead to **overly smooth, low-diversity images**, lacking texture richness. Conversely, image tokens require broader sampling (e.g., **k ≈ 0.5 * vocab size**) to maintain realism.\n* In language generation, a typical k lies in the **20–100** range with a vocab size of **~50,000 or 128,000**, and can produce **diverse** and **grammatical fluency** texts (Holtzman et al., 2020; Fan et al., 2018). \n\nref: Holtzman et al, 2020, The Curious Case of Neural Text Degeneration.\n\nFan et al, 2018, Hierarchical Neural Story Generation.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The evaluation criteria generally make sense.", "theoretical_claims": "The theroretical claims are correct", "experimental_designs_or_analyses": "The experimental designs are valid.", "relation_to_broader_scientific_literature": "This paper shows that LLMs can achieve strong image generation performance without domain-specific designs, through careful choices in tokenization, modeling, and sampling. It contributes to understanding scaling behaviors and modality differences, offering insights for adapting LLMs to vision tasks and encouraging broader cross-domain applications.", "essential_references_not_discussed": "Two highly relevant papers should be discussed and made for fair comparison:\n\n[1] Sun, Peize, et al. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation\n\n[2] Li, Tianhong, et al. Autoregressive image generation without vector quantization", "other_strengths_and_weaknesses": "Weaknesses\n1. The technical novelty is kind of limited, as the results are mostly empirical.\n2. This paper claims AR \"demonstrate superior image generation ability and scalability compared to MLMs\" in L089 and L090, however, works done to improve MLM for image generation in [1] shows much greater results. Authors should make fair comparison with [1] to defend this point.\n3. More qualitative study and empirical discussion on cfg and top-k are important to understand how image generation differ from original decoding strategy in LLM.\n\n[1] Li, Tianhong, et al. Autoregressive image generation without vector quantization", "other_comments_or_suggestions": "No", "questions_for_authors": "I'm leaning towards weak reject before rebuttal. However, this research is in good shape with valid contributions. I would be happy to raise my score to 3 (weak accept) if my concerns in [W2] and [W3] and quantitative comparisons with [1][2] are addressed during rebuttal.\n\n[1] Li, Tianhong, et al. Autoregressive image generation without vector quantization\n\n[2] Sun, Peize, et al. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Even after the rebuttal and discussions, reviewers could not reach a unanimous verdict. The main remaining concern was the novelty of the paper, which most reviewers found weak. Despite this, the majority of the reviewers recommend accepting the paper, citing the thoroughness of the empirical findings. The rebuttal results also played a critical role in swaying the reviewers over the fence. The negative reviewer also acknowledges the impressiveness of these results and is ultimately okay with the paper being accepted. The AC, thus recommends accepting the paper.

The AC further strongly asks the authors to include the new results at the very least as an appendix to the paper, with the main manuscript pointing to the results, as they were quite important during the discussions.
Decision: Accept (poster)