Autoregressive Models in Vision: A Survey
Authors: Jing Xiong
Gongye Liu
Lun Huang
Chengyue Wu
Taiqiang Wu
Yao Mu
Yuan Yao
Hui Shen
Zhongwei Wan
Jinfa Huang
Chaofan Tao
Shen Yan
Huaxiu Yao
Lingpeng Kong
Hongxia Yang
Mi Zhang
Guillermo Sapiro
Jiebo Luo
Ping Luo
Ngai Wong
Venue: Accepted by TMLR ()

Abstract: Autoregressive modeling has been a huge success in the field of natural language processing (NLP). Recently, autoregressive models have emerged as a significant area of focus in computer vision, where they excel in producing high-quality visual content.  Autoregressive models in NLP typically operate on subword tokens. However, the representation strategy in computer vision can vary in different levels, i.e., pixel-level, token-level, or scale-level, reflecting the diverse and hierarchical nature of visual data compared to the sequential structure of language. This survey comprehensively examines the literature on autoregressive models applied to vision.  
To improve readability for researchers from diverse research backgrounds, we start with preliminary sequence representation and modeling in vision. Next, we divide the fundamental frameworks of visual autoregressive models into three general sub-categories, including pixel-based, token-based, and scale-based models based on the representation strategy. We then explore the interconnections between autoregressive models and other generative models. Furthermore, we present a multifaceted categorization of autoregressive models in computer vision, including image generation, video generation, 3D generation, and multimodal generation. We also elaborate on their applications in diverse domains, including emerging domains such as embodied AI and 3D medical AI, with about 250 related references. Finally, we highlight the current challenges to autoregressive models in vision with suggestions about potential research directions. We have also set up a Github repository to organize the papers included in this survey at: https://github.com/ChaofanTao/Autoregressive-Models-in-Vision-Survey.

Published in Transactions on Machine Learning Rescarch (03/2025)
Autoregressive Models in Vision: A Survey
Jing Xionghi
Gongye Liu
Lun Huang 3,10
Chengyue Wu' Taiqiang Wu
Yao Mu'
Yuan Yao*
Hui Shen
Zhongwei Wan Jinfa Huang"
Chaofan Tao Shen Yan Huaxiu Yao"
Lingpeng Kong!
Hongxia Yang
Mi Zhang
Guillermo Sapiro 8.10
Jiebo Luo Ping Luo' Ngai Wong*
Equal Contributors.
Corresponding Author. Email: cftao@connect. whwnnnett..u.kkee
The University of Hong Kong TTinghua University
Duke University
*University of Rochester TThe Ohio State University
"Bytedance
The University of North Carolina at Chapel Hill AApple The Hong Kong Polytechnic University "'rinccton University
Reviewed on OpenReview:
Abstract
Autoregressive modeling has been a huge success in the field of natural language pro- cessing (NLP). Recently, autoregressive models have emerged as significant area of focus in computer vision, where they excel in producing high-quality visual content. Autoregressive models in NLP typically operate on subword tokens. However, the representation stratogy in computer vision can vary in different levels, i.e., pixcl-level, token-level, or scale-level, reflecting the diverse and hierarchical nature of visual data compared to the sequential structure of language. This survey comprehensively exam- ines the literature autoregressive models applied to vision. To improve readability for researchers from diverse research backgrounds, we start with preliminary sequence representation and modeling in vision. Next, we divide the fundamental frameworks of visual autoregressive models into three general sub-catogorics, including pixcl-based, based, and seale-baser models based on the representation strategy. We then explore the interconnections between autoregressive models and other generative mod- cls. Furthermore, we present a multifaccted categorization of autoregressive models in computer vision, including image generation, video generation, 3D generation, and multimodal generation. We also elaborate on their applications in diverse domains, including emerging domains such as embodied Al and 3D medical AI, with about 250 related references. Finally, we highlight the current challenges to autoregres- sive models vision with suggestions about potential research directions. We have also set up a Github repository organize the papers included this survey at: htt.m//haaaaaaaeeeeetttgrrrreeee https://gi
1 Introduction
Autoregressive models, which generate data by predicting each element in a sequence based on the previous elements through conditional probabilities, initially gained prominence in the field of natural language processing (NLP) (Vaswani, 2017; Radford et al., 2019; Brown et al., 2020; Achiam et al., 2023; Wan et al., 2023; Zhou et al., 2023a). This success can be attributed to their inherent advan- tage of capturing long-range dependencies and producing high-quality, contextually relevant outputs. Especially empirical scaling laws (Henighan et al., 2020; Hoffmannn et al., 2022; Muennighoff et al.,

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Show-o
% bii Fluid
Publicly
x
*
*
Chameleon
8
anus
Available/Unnaailable
GPT
GPT-2 GPT-3
GPT-3.5
GPT-4
Llama-G
HART
<3
be
IVVE E Emu3
DART
Lumina-mGi PT
Emul BAAi
Pyramid Flow
VideoPoet
SPA
Bel
VAR SEED-X
Muse
2024
Parti G
RQ- Tra-nsffffrrer
LVM
CogView2
CM3Lee
DALL-E 1 c
Make-A 0
SEED
VQGAN
CogV ideo @
LQAE
CogView
NÜWA
A12
Unified-IO 2
ImageG PT
2023
ImageBART
GODIVA
PixelRNN G
NÜWA-Infinity Visual Prompting A12 Unified-IO
PixeCCNN G
2022
PixeICNN++ C
VQ-VAE G
2020
2018
Pi
2016
G ImageTransformer
Figure We provide a timoline of representative visual autoregressive models, which illustrates the rapid evolution of visual autoregressive models from early pixel-based approaches like PixelRNN in 2016 to various advanced systems recently. We are excitedly witnessing the rapid growth in this field.
2023; Tao et al., 2024; Lyu et al., 2023) reveal that increasing model size and compute budgets con- sistently improves cross-entropy loss across various domains like image generation, video modeling, multimodal tasks, and mathematical problem solving, following universal power-law relationship. Inspired by their achievements in NLP, autoregressive models have recently bogun to demonstrate formidable potential in computer vision.
The timeline in Figure 1 illustrates the key milestones and developments in the evolution of visual autoregressive models, highlighting their transition from NLP to computer vision. To date, autore- gressive models have been applied to a wide array of generative tasks, including image generation (Parmar et al., 2018; Chen et al., 2020), image super-resolution (Guo et al., 2022; Li et al., 2016), image editing (Yao et al., 2022; Crowson et al., 2022), image-to-image translation (Li et al., 2024e;d) and video generation (Tulyakov et al., 2018; Hong et al., 2022), multi-modal tasks (Yu et al., 2023c; Lu et al., 2022) and medical tasks (Ren et al., 2024a; Tudosiu et al., 2024), .etc. This broad applicability underscores the potential for further exploration and application antoregressive models.
With the rapid proliferation of visual autoregressive models, keeping up with the latest advancements has become increasingly challenging. Therefore, comprehensive survey of existing works is both timely and crucial for the research community. This paper endeavors to provide thorough overview of recent developments in visual autoregressive and explores potential directions for future improvements.
We emphasize that there are at least three distinct categories of visual autoregressive models defined by their sequence representation strategies: pixel-based, token-based, and scale-based models. Pixel- RNN (Van Den Oord et. al., 2016), as a representative pixel-wise model in the pioneering of next- pixel prediction by transforming 2D image into a 1D pixel sequence, capturing both local and long-range dependencies but with high computational cost. Next-token prediction, inspired by NLP, compresses images into discrete tokens for efficient high-resolution processing, exemplified by models like VQ-VAE (Van Den Oord et al., 2017). VAR (Tian et al., 2024) introduces next-scale prediction, a hierarchical method that generates content across multiple scales, from coarse to fine autoregressively

---

Published in Transactions on Machine Learning Research (03/2025)
capturing visual information at multiple resolutions. Each category offers unique advantages and challenges, making them promising directions for future research.
We further introduce a multi-perspective categorization of autoregressive models applied to computer vision, which classifies existing models based on criteria such as the sequence representation strategy, the underlying framework, or the larget lask. Our categorization aims to provide a structured overview of how these models are utilized across various vision tasks. We then present both quantitative and qualitative metrics to assess their performance and applicabilityy. Finally, we highlight the current limitations of autoregressive models, such computational complexity and mode collapse, and propose potential directions for future research. In summary, this survey makes several contributions:
Given the recent surge of advances based on visual autoregressive models, we provide a comprehen- sive and timely literature review of these models, aiming to offer readers a quick understanding of the generic autoregressive modeling framework.
We categorize visual autoregressive models based on their sequence representation strategies and systematically compile applications across various domains. This aims to help rescarchers specific fields quickly identify and learn about related work.
We provide a comprchensive review of autoregressive models in vision from about 250 related references and summarize their evaluations compared with GAN/Diftusion/MA Maniiuussio/ME--baeaeee methods in four image generation benchmarks (ImageNet, MS-COCO, MJHQ-30K, and GenEval bench).
2 Autoregressive Models
2.1
Preliminary
Visual autoregressive models are a class of generative models that sequentially predict visual elements, where each prediction is conditioned the previously generated elements. Visual autoregressive models generally consist of two core components:
Sequence Representation. The visual data is first transformed into sequence of discrete elements. These elements may correspond to pixels, image patches, or latent codes derived from visual content, depending on the specific model architecture. This transformation allows the visual data to framed a sequential modeling problem, akin text generation in natural language processing.
Autoregressive Sequence Modeling. Once the visual content is represented ordered se- quence, the model is trained to generate cach element by conditioning all preceding elements. Matho- matically, this is expressed as:
N p(x)= = II o2,,
(1)
where p( #2, 0) represents the probability of the current element x; conditioned on all previous elements in the sequence, with 9 denoting the model parameters. The training objective is minimize the negative (Ngliikelihoo((NLL) loss, which is formulated
C(0) Σ
log logp(z;/21 x2, 2.11;))))

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Pastoo
Fautihees pidtemar 2017 Uneud Marich Oloed 2000)
Piari-atio Groseation 33 al
PHAAD (leed 2015.
Presbratt, (Thmm (011) Tragg Transkerer (Ptreee at, (R1F), Teapdrt
Meeaa (Cur Chedl au, 3317L HQMARR (Rosers ale 2011),
2000)
MA0Ab--4 (0ka 2086, Disache Dras 2000
MA-VA (iom MAN Otinn 2001) DM tiuee 2033
Unconditionll Image Guseeation (11.1
Toko-vie Geoeriia (B:1.1b)
DCT thw.eeeeeee Plaak 5021))
KQUAN Dine 30211 Manadie
DAANSC 2010 BASSS
2013 MAK 2000
ARN Oliss 20311, AAX (1) 30013)
Scale-ston Goneration [$11.4)
VAR 3004)
DoyVerd liken 3022) Mute (Yu
Harrehoee Clada al 2000 MGAK
2004 ThAt 2011
Vig.eBliiee J0 2010 Kolid
Image Goneratkn ($...)
Tolus-niss Genention (B.1
1ASHIRX (Lx 2010 Y2T 201al, MADD 2001)
Image Nystieek 02.1
treebee (Mi
Trage Painting (8:1-3g)
Easgee-oollaage Uraselaioon (5.1
Multi-riww Gezeention ($11.3))
SIE piice 200 Rosadatte Chnstlyne aeai
Visual Costost Leariin ( (1:33x)
HAB-YUUAN (hur du 2001) VIOE Bhi 2011
Test-driven Imoge Editing (511 La)
VQCAN.CUP (Creves 200 Mtdee (Sote 2003)
Iunge-lrven Inage Bliting (8L14
MSM Diis at 2000 CAR ther 300t,, NOMPNe Blorimas at, 2013 HVU this 2014
MACVET (ou di, Sa30o MACUTE CY 20133
Modella in Vislon
2004),
Cacsoditional Vidao Generation (51223)
19th DN Phiddioamm 200h SMAUSAAN Etanuker ale 2010),
poust (ts it al 20300 D. HAII Chast de 2014
InC.CAN (Phng 2011) Gotha (thir du 1401), DANE (Tis 300r)
Cagrise Oloee 2025 sthea (t al 3852)
Autorogrosk
Video Generatam (S.2)
Leea: Oree al, 3.56- PAY Dus dy (0d1)), (aiaour OW 3001
Cunditional Valto Gessestion ($122)
Condetied 14TM (11 al, 2011 PaetBON Wing (317))
KID.Lem Oftiuge 2014) evaP (houdeedd 201t
(51230
Mation Geneettion (B1S))
Point Cloud Generstion (13.3)
3D Generation (3.3)
Irv cole biid), Argesih cgen
Soms Generation ($13.3)
3D Medical Genemtion ($12.)
Cedts clive Khilert and, MESIUOAN (Phe deate)
thadge ('areai als 2017) Amatiay Cllvng Seaes
Hare (lle 2001 BaTT (Preg 2000, MANV (line 2004E)
Undertanding Branevork (8:43)
VL VL-EEIT this 2015 BErTiv Dilneg 20120)
Multinodal Undecetadding and Geurration (g-4)
stto 300 Beed then 202id LAVIZ
Vatao LaVE Glia 20000) XAILA (Os 2000)
United Promwoork (5) 13)
Chreskos Cllers 9900) Troestarie CDlmos an 2004
teokegion
MING-VA Jonn (Wa al, aviel) VIRLA (Wo al, antie Meek coloag deaee)
fee wnlh
Figure 2: Literature taxonomy of autoregressive models in vision.
2.2 Generic Frameworks
In the preceding sections, we have provided a general overview of visual autoregressive models about the essential concept of sequence representation and sequence modeling. Then, we consider the representa- tion strategy to classify the visual autoregressive models further, i.e. pixel-based models, token-based

---

Published in Transactions on Machine Learning Research (03/2025)
E
Compression Discretization
Discrete Visual S Sequence
Flatten
Pixel Sequencee
D
Reconstruction
Nexb Token Pradiction
(a) Sequence Representation
Tokenizer
Token Sequence
X2
Next-Scale Prediction
Autoregressive Models
Mulb-Scale Tokenizer
sulli Sequercce
(b) Autoregressive Sequence Modeling
Figure 3: Illustration of three types of visual autoregres- sive models general frameworks based their sequence representation strategies. Next-Pixel Prediction flat- tens the image into a pizel sequence. Next-Token Pre- diction converts the image into a token sequence via a visual tokenizer. Next-Scale Prediction employs a multi-scale tokenizer generate multi-scale sequence.
Figure 4: Core components in visual toregressive models. (a) Sequence Rep- resentation encodes visual data. into the discrete visual sequence, followed by recon- struction. (b) Autoregressive Sequence Modeling predicts each element sequen- tially.
models, and scale-based models. Since representation strategy directly reflects how models handle the underlying structure and features of visual data. Different representation strategies influence the model's granularity, efficiency, and applicability. Pixel-based models (Sec. 2.2.1) generate images pixel by pixel, capturing fine-grained spatial details. Token-based models (Sec. 2.2.2) represent images as a set of high-level discrete visual tokens, analogous to words in NLP. Scale-based models (Sec. 2.2.3) generate images through multi-scale representation, allowing the model handle visual content from different resolutions. We detail these three types of generic frameworks below.
Discussion about continuous methods Our categorization of autoregressive generation methods is based the representation strategy of the autoregressive elements. Although our discussion primar- ily focuses on discrete autoregressive methods, we note that contimuous autoregressive approaches (Li et al., 2024c; Zhou et al., 2024), which operate directly in the embedding space without discretization, can naturally accommodated within this framework. Continuous autoregressive models share many similarilies with their discrete counterparts, with two notable differences: (a) they perform regres- sion directly on continuous embeddings, eliminating the need for quantization; and (b) while discrete models typically minimize negative log-likelihood using cross-entropy loss, continuous models require alternative loss formulations (e.g., diffusion loss (Li et al., 2024c)) on case-by-case basis. Despile these differences, the underlying principles of autoregressive generation paradigns remain analogous, and our taxonomy is flexible enough to encompass both discrete and continuous methods.
2.2.1 Pixel-based Models
In pixel-based models, visual data is directly represented at the pixel level. PixeIRNN (Van Den Oord et al., 2016) emerged as the pioneering work for AR image generation. It converts 2D image I E #x into a 1D discrete pixel sequence {x1, ... ,NN] by applying a raster sean order.

---

Published in Transactions on Machine Learning Rescarch (03/2025)
PixeIRNN employs LSTM layers to sequentially predict each pixel based previously generated pix- els, as illustrated in Equation 1. Following this development, series of AR generation works centered around pixelCNN (Van den Oord et al., 2016; Reed et al., 2016; Salimans et al., 2017; Chen et al., 2018; Reed et al., 2017) were proposed, achieving remarkable generation quality on CIFAR-10 and ImageNet 32 x 32. Building upon these advancements, several attempts (Kalchbrenner et al., 2017; Weissenborn et al., 2019) are made to extend the pixel-level AR approach to video generation through per-pixel synthesis.
However, despite these milestones, generating high-resolution images remains significant challenge for pixel-level autoregressive models. This is primarily attributed to 1) the quadratically increasing computational cost with sequence length and 2) the redundancy inherent in per-pixel information. Some rescarch (Roed et al., 2017) has explored the adoption of parallel techniques to generate images at 256 256 resolution. However, these methods lend to yield suboptimal and blurry results.
2.2.2
Token-based Models
The token-based models represent a significant evolution in visual autoregressive modeling, drawing inspiration from natural language (Nocsssing(LP)... Unlike pixel-level models, which operate directly on raw visual data, this paradigm compresses and quantifies an image or video into a sequence of discrete tokens, allowing for more efficient processing of high-rcsolution content.
To achieve this, Vector Quantization(VU) technique employed to transform continuous visual fea- tures into sequence of discrete latent codes. Given raw image R#xwx33 encoder E first maps the image to a latent feature map E(x) Raxwxd These continuous features are then quantized into discrete codes using a learned codebook Z = {= C R containing K entries, where each entry 2k represents a prototype vector in the latent space. The quantization operation is defined as finding the nearest codebook entry for each spatial feature vector &ij- E Rd.
2g (x)x)= arg min | RxE2
2x
Rixwxd
This quantization process enables a compact and discrete representation of the latent feature space, which is particularly advantageous for reducing the computational burden for digh-dimensional image generation tasks. A seminal work on discrete representation is VQVAE (Van Den Oord et al., 2017), which introduced a two-stage paradigm that has become the de-facto standard for autoregressive vision generation. In this paradigm, an encoder-decoder architecture initially trained to learn the discrete image representation. The encoder É maps the input image to latent space ze (xr), where the continuous latent features are quantized into discrete codes using VQ. The decoder D then reconstructs the image from these discrete codes. The training objective consists of reconstruction loss and a commitment loss to ensure the latent codes effectively represent the input:
C C= -= (log(aa)) -2,()2(())))))))
- sglz, (x)]1
(4)
where sg denotes the stop-gradient operator, E(x) is the encoded latent vector, and 2x(x) is the quantized vector.
In the second stage, a powerful autoregressive model is trained to predict the next discrete token given the sequence of previously generated tokens. Building upon the VQVAE, VQ-VAE-2 (Razavi et al., 2019) introduced a multi-scale hierarchical architecture that further enhances the quality and diversity of generated images. By incorporating multiple levels of latent representations, VQ-VAE- captures both global and local details. This hierarchical approach improves the model's 's ability to generate high- resolution images, achieving state-of-the-an results various image datasets. VQGAN (Esser et al., 2021b) enhances the image tokenizer by integrating an additional PatchGAN-bassc discriminator loss into the VQVAE framework. This approach facilitates the learning of a perceptually rich codebook.

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Furthermore, VQGAN employs GPT--2-style decoder-only Transformer (Vaswani, 2017) to model the distribution of tokens in raster-scan order, scaling autoregressive image generation with millions of pixels while maintaining computational efficiency.
2.2.3 Scale-based Models
The scale-based models, as proposed in VAR (Tian el al., 2024); introduce a hierarchical method for autoregressive image generation. Unlike traditional next-token prediction models that operate on a single resolution in a raster-scan order, it generates visual content across multiple scales from coarse fine. The autoregressive unit in this approach is entire token map instead of a single token, which enables the model to process visual data in more structured and elficient. manner.
A foundational idea behind VAR is the Residual Quantization(RRQ) technique introduced in RQ- VAE (Lee et al., 2022a). RQ-VAE improves upon standard VQ by recursively quantizing residuals feature maps. Unlike VQ-VAE, which requires larger codebooks to maintain quality as the reso- lution of the quantized feature map decrcases, RQ-VAE uses fixed-size codcbook and quantizes a vector by approximating the residuals step-by-step a coarse-to-fine manner:
RQ(2;,,)) (k1, k,,...,k.)) kp). where ky = arg min |ra-1 - ze|
55)
4EC
where C' is the shared codebook, D is the quantization depth, and rd-1 is the residual vector at depth d 1. This recursive quantization process allows RQ-VAE to represent high-resolution images compactly, reducing spatial resolution while retaining essential information.
Building upon this idea, VAR employs multi-scale quantization autoencoder to discretize an image into token maps at various scales. Given a raw image x E "xxxxx x3 a multi-scale VQ-VAE encodes the image into a set of token maps {R1, R..... R where each token map R2 E R X corresponds to a different scale & of the image and serves as the autoregressive unit. The generative process hierarchical, starting from the coarsest scale and autoregressively generating higher-resolution token maps. During the k-th autoregressive step, all distributions over the h2 wk tokens in т will be generated in parallel.
Compared to token-based models, the basee models offer several advantages: a). It retains spatial locality, which facilitates zero-shot generalization to novel tasks without the need for task-specific training; b). The efficiency token generation is improved by enabling parallel token generation within each token map, reducing the overall computational complexity. This efficiency gain is particularly beneficial for scaling up to larger image resolutions.
2.2.4 Analysis of Computational Costs
We compare the time complexity and efficiency of three autoregressive generation paradigms: next-pixel prediction, next-token prediction, and next-scale prediction. We consider a baseline "vanilla" architecture comprising a standard self-attention Transformer for autorogressive modeling, and an op- tional CNN-based tokenizer for decoding. Our analysis assumes greedy sampling without any addi- tional efficiency techniques.
In next-pixel prediction, an N N image is generated sequentially, with the Transformer's self-attention incurring per-step cost of (i?) for sequence of length I. Consequently, generating all N pixels requires:
N² Σ
2 i=1
(2 (N² + +1(2222 +1)
6
which is equivalent to Or(n') basic computation. Next-token prediction reduces this cost by incorpo- rating an image tokenizer with a compression ratio k, which shortens the latent sequence to (N/k)²

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Require Tokenizer
Compression Ratio
Complexity
Efficiency
Next-Pixel Prediction
Or(No) Or(Ne (k9) + (c(22)
Token Prediction
k
Next-Scale Prediction
1:
Or Oc(N2)
te d 12
Efficiency Comparison of Different Autoregressive Generation Paradigms. We consider the task of generating an x N image using a standard self-attention Transformer and an optional CNN-based tokenizer.
Table 1:
tokens. The autoregressive cost is thereby reduced to Or(N%/k), and decoding the latent represen- tation back to an N x N image via CNN incurs cost of Oc(N2). The total cost is significantly lower than that of next-pixel prediction, particularly for high-resolution image synthesis. Next-scale prediction further enhances efficiency through a block-wise causal masking strategy and a reduced number of iterative steps, resulting in a time complexity of Dr((N/kk) O,(N2) (Tian al., 2024). Table summarizes comparative computational complexities and efficiency ratings of these paradigms.
2.3 Relation to Other Generative Models
Variational Autoencoder. Variational Autoencoders (VAEs) (Kingma, 2013) are class of genera- tive models that learn to map data into continuous, lower-dimensional latent space and subsequently reconstruct it back to the original data space. This process is governed by optimizing a variational lower bound of the data likelihood. In contrast, autoregressive models directly capture the full data distribution by predicting each element sequentially, optimizing negative (N.-)) as the training objective. While VAEs provide efficient sampling, their reliance on variational approxima- tions often results in less sharp outputs (Child, 2021) and may suffer from posterior collapse (Zhao et al., 2019; Chen et al., 2017) when equipped with powerful decoder. Autoregressive models, on the other hand, generate high-quality samples by modcling data dependencies at the original dimen- sionality, but they are slower in inference due to their sequential generalion process. To address these limitations, hybrid approaches (Gulrajani et al., 2017; Van Den Oord et al., 2017; Chen et al., 2017) have been proposed that integrate the autoregressive process into VAËs. One prominent example is VQ-VAE (Van Den Oord al., 2017), which utilizes a VAE framework learn discrete latent spaces and autoregressive models to refine generation, effectively leveraging the strengths of both models for efficient and high-quality image synthesis.
Generative Adversarial Networks. GANs (Goodfellow et al., 2014) are known for generating high-quality images with fast inference, enabled by their one-shot generation process. GANs demon- strate particular strength in domains with structured data, such as facial synthesis, where their low- dimensional latent space can effectively manipulate visual attributes (Xia et al., 2022). However, the adversarial training objective that GANs rely on often results in training instability and mode col- lapse (Liu et al., 2020), which requires a delicate balance between the generator and discrininator. Additionally, the adversarial training paradigm that underpins GANs can hinder their scalability to more diverse datasets and larger model sizes. In contrast, autoregressive models employ likelihood- based training, which ensures a stable training process. Despite their relatively slow sampling speeds, autoregressive models exhibit. favorable scaling laws, where model performance consistently improves with larger datasets and increased model sizes (Kaplan et al., 2020). These properties make autore- gressive models particularly well-suited for building universal generative models and can be adapted to a wide range of downstream tasks through zero-shot generalization of supervised fine-tuning (SFT), enabling ficxibility and robustness across different applications.
Normalizing Flows. Normalizing flows (Dinh et al., 2014; Rezende & Mohamed, 2015) utilize a series of invertible and differentiable transformations to map a simple Gaussian distribution into a

---

Published in Transactions on Machine Learning Rescarch (03/202))
complex data distribution. Each transformation is designed to have an easily computable Jacobian determinant, enabling efficient computation of the model's likelihood. Both normalizing flows and autoregressive models allow direct optimization via maximum likelihood estimation. However, nor- malizing flows achieves this by enforeing a tractable likelihood, which imposes specific architectural constraints to ensure invertibility. In contrast, autoregressive models optimize the likelihood by dis- cretizing the data and sequentially predicting each element, thereby allowing greater flexibility in model design and enhanced scalability with data and model size.
Diffusion Models. Diffusion models (Ho et al., 2020) have emerged as the state-of-the-art genera- tion paradigm for wide range of vision generation tasks. As recent challengers, autoregressive models share several characteristics with diffusion models. Both methods can generate diverse, high-quality samples, yet. both also suffer from inefficient inference due to their iterative or sequential generation processes. Additionally, both approaches are likelihood- baaieood--bsed and optimize objectives such as Nega- tive Log-Likelihood (NLL) or Evidence Lower Bound (ELBO), making them relatively casy to train. Recent advances (Esser el al., 2021b; Rombach el al., 2022) in both fields have focused on compressing visual content into latent spaces to improve the efficiency of high-resolution generation. Despite these similarities, there are fundamental differences in their generative paradigms. Diffusion models employ a predefined forward process that gradually corrupts data and an iterative denoising process to recover samples. This iterative process retains spatial locality at each step, which benefits visual coherence. However, the necessity to corrupt the training data with Gaussian noise may potentially limit their scalability for understanding lasks. Additionally, while diffusion models operale primarily in contin- uous spaces, their discrete variants still lag behind in performance, posing challenges for multimodal applications like text generation. Autorcgressive models, by contrast, inherently introduce unidiroc- tional biases and discretization, which might not be ideal for visual tasks, partially explaining their lag behind diffuusio models in recent benchmarks. However, autoregressive models offer flexibility in han- dling diverse modalities and combining generation with understanding. This adaptability aligns with the emerging trend of integrating autoregressive models with Large Language Models (LLMs) to create a unified framework for multimodal input-outpul lasks, bridging both generation and understanding capabilities. These distinctions and complementarities between the two approaches suggest potential avenues for future research. Recent efforts have aimed to combine the strengths of both approaches, such as integrating bidirectional contexts (Esser al., 2021a; Zhou et al., 2024; Xie et al., 2024b), autoregressive models with continuous representations (Li et al., 2024c; Zhou et al., 2024), discrete diffusion models (Gu et al., 2022), and using autoregressive models to enhance the understanding capabilities of diffusion models (Gu et al., 2024).
Masked Autoencoder. Masked autoencoders (MAEs) (He et al., 2022) are designed to learn ro- bust data representations by randomly masking portions of the input data and training the model to reconstruct the missing content. Both MAFs and autoregressive models share similarities as they com- press images into discrete sequences of visual elements and model these sequences. In fact, MAEs and autoregressive models inherit two leading paradigms from natural language processing (NLP): MAEs adopt the BERT-style (Devlin et al., 2018) encoder-decoder structure, while autoregressive models follow the GPT-Style (Radford et al., 2019) docoder-only approach. Nevertheless, there are essential differences between them: 1) MAEs are trained by randomily masking visual tokens and reconstructing the missing part, while autoregressive models are trained to predict the next element in an ordered sequence; 2) MAEs utilize full attention, enabling each token to consider the entire surrounding con- texts, whereas autoregressive models typically employ causal attention, restricting each token to focus only on previous ones; 3) During decoding, MAFs randomly generate multiple tokens in parallel, while autoregressive models generate tokens sequentially in a predefined order. These differences lead distinct strengths: MAEs generally excel in representation learning and visual understanding tasks, and recent works like MaskGIT (Chang et al., 2022) and MAGE (Li et al., 2023c) also extend the Masked Image Modeling (MIM) approach to image generation. In contrast, autoregressive models are more suited for generating high-quality samples from scratch and demonstrate stronger scaling

---

Published in Transactions on Machine Learning Rescarch (03/2025)
laws they are scaled up, a property that has also been validated in language generation. Recent research (Li et al., 2024c) has proposed viewing masked generative models as generalized form of autoregressive models and has investigated ways to enhance autoregressive models with MAE tricks. However, given the essential differences between them and the comprehensive discussions on MAEs in prior surveys, our review in this paper is mainly focused on standard autoregressive models. In summary, both methods are based on sequence modeling and have their respective strengths. MAEs provide efficient parallel decoding and the bidirectional attention in MAEs is well-suited for visual data, and autoregrossive models excel in scaling up and adapting to multimodal tasks. Rethinking the relationship between these two approaches could polentially advance autoregressive generation. One method that bridges these two algorithms is Show-o (Xie et al., 2024b), which incorporates MAE-like masking strategics and bidirectional attention within an autoregressive framework to enhance image generation capabilitics.
2.4 Comparison between Autoregressive Methods and Non-Autoregressive Methods
In this subsection, we discuss the unique advantages of autoregressive models for vision in comparison to non-autoregressive models, such as the popular diffusion- (Ho et al., 2020) and rectified -low based models (Liu et al., 2022). We summarize these advantages in three key points:
1. Scaling Laws: The success of the next-token prediction paradigm in NLP (Achiam et al., 2023; Lyu et al., 2023; Dubey et al., 2024) can be largely attributed to well-established scaling laws (Henighan et al., 2020; Kaplan et al., 2020). Although the scaling properties of methods like diffusion and GANs remain relatively underexplored, autoregressive visual models offer the potential to transfer success- ful scaling experiences from NLP. This could enable efficient scaling in visual generation frameworks, particularly in terms of model size and performance. Some work like VAR (Tian et al., 2024), Llama- Gen (Sun et al., 2024a) and FLUID (Fan et al., 2024) exemplify how autorcgressive methods benefit from these scaling laws, highlighting their potential for advancing visual generation.
2: Deppoyment Efficiency: Autoregressive generative models can leverage the existing deployment technologies designed for language models (Wan et al., 2023). Frameworks such as VLLM provide acceleration for autoregressive models, significantly improving generation efficiency. This allows au- toregressive visual models to benefit. from infrastructure that has already been optimized for language tasks, facilitating seamless deployment in real-world applications.
3. Bridging Language and Vision: Autoregressive models offer a promising pathway toward unifying multimodal understanding and generation. By aligning closcly with the structure of large language models, autoregressive approaches may provide natural bridge between vision and language tasks. This synergy opens up the potential for more integrated and versatile models that can handle both vision and language processing in a unified manner, contributing to a deeper understanding of multimodal tasks (Team, 2024; Zhou et al., 2024; Xic et al., 2024b; Wu et al., 2024a).
In addition to these primary advantages, autoregressive models share some secondary benefits with cer- tain non-autoregressive models. For example, autoregressive training demonstrates inherent stability. By directly optimizing the negative log-likelihood (NLL) through the minimization of the cross-entropy loss function, as depicted in Eq. 2, autoregressive models eliminate the need for adversarial training, as is the case with GANs (Goodfellow et al., 2014), or for the complex design of invertible networks like normalizing flows (Dinh et al., 2014; Rezende & Mohamed, 2015). Furthermore, autoregressive models often demonstrates well generation diversity, less worrying about the model collapse curse that can affect non-autoregreesive models.
Despite these advantages, autoregressive models do have certain limitations. One significant draw- back, when compared to diffusion models, is their difficulty in generating ultra-high- images or videos. The higher-order time complexity of autoregressive generation makes this task more chal- lenging. As discussed in Table 1, autoregressive models using next-token prediction paradigm have a
10

---

Published in Transactions on Machine Learning Research (03/2025)
VQ-VAE (Van Den Oord al., 2017), VQ-VAR-2 (Rasavi al., 2019),
Foundational Works
VQ-GAN (Easer et al, 20216)
Vistal Polenniser
VET-VQGAN (Yu et al,, 2021a), Efficient VQQGAN (Cao et al., 2023),
Improvements on VQGAN
LlamaGen (Sun et sl., 2021s), VQGAN-LC (Zhu et al,, 2024b), TiTok (Yu al,, 2024c)
RQ-VAE (Lee et al., 2022a), MoVQ (Zheng et al., 2023), DQ-VAE (Huang et al, 2023b), PSQ (Mentzer et al,, 2028)
Variants of Sector-uunntization
Image Compression Algorithms
DCTransformer (Nash et nl,, 2021), Wavelet Transformer (Mattar al,, 2024)
Figure Taxonomy of visual tokenizer in an unconditional generation.
time complexity of O(N%/1)) when generating an N x N image, whereas a Diffusion Transformer (DiT) model (Peebles & Xie, 2023) in a similar setting typically requires only O(TN*) time complexity (T for the denoising steps). Additionally, while autoregressive generation has shown promising progress in prompt-following when generating images from given prompts (Fan et al., 2024; Wang et al., 2024d), demonstrated by benchmarks like GenEval (Ghosh et al., 2024), the visual quality of single images gen- erated by autoregressive models still lags behind state-of-the-art diffusion flow-based models (Esser et al., 2024; Labs, 2024), and significant research is still needed to bridge this gap. We provide a more detailed discussion of potential future directions for this work in Scction 5.
3
Visual Autoregressive Models
3.1
Image Generation
3.1.1
Unconditional Image Generation
Unconditional image generation refers to models producing images without specific input or guiding conditions, relying solely on learned visual patterns from the training data. This process can be cate- gorized into three paradigms: pixel-level, token-level, and scale-level generation. Pixel-level generation creates images pixel by pixel, while token-level treats images as sequences of discrete tokens for more efficient generation. And scale-level generation builds images progressively from low to high resolution. This subsection will focus on the generation process itself, highlighting how models synthesize images by autoregressive generation of each part based on previously generated content, ensuring cohorence and consistency. While other applications of autoregressive image generation will be introduced subsequent sections.
3.1.1.a Pixel-wise Generation
In the pixel-wise generation, the model constructs an image pixel on a pixel-by-pixel basis, uncondi- tionally predicting each pixel's quantified value based solely on the sequence of previously generated pixels. This approach allows for direct optimization of the likelihood function, thus producing highly detailed outputs, albeit a significant computational cost. This unconditional generation marks the inception of AR generation rescarch, laying the foundational paradigm upon which all subsequent work in this domain has been built.
Recurrent and Parallel Pixel Generation Techniques. PixeIRNN (Van Den Oord al., 2016) uses RNNs to predict pixel values sequentially, effectively capturing complex dependencies between pix- els. This work is the first to propose using an autoregressive approach for image generation. Building on its success, PixelCNN (Van den Oord et al., 2016) adopts dilated convolutions to efficiently capture

---

Published in Transactions on Machine Learning Rescarch (03/2025)
long-range pixel dependencies for generation. PixeICNN++ (Salimans et al., 2017) further optimizes PixelCNN using discretized logistic mixture likelihood function and architectural improvements, im- proving image quality and efficiency. Reed et al. (2017) introduces a parallelized version of PixeICNN, which reduces computational complexity and allows faster high-resolution image generation.
Transformer-based Approaches for Pixel Generation. Traditional CNNs and RNNs face scal- ing issues when used for autoregressive image generation on large datasets, mainly due to their limited ability to model long-term dependencies, and their model architectures also struggle to scale up in size. To address this issue, Transformer (Vaswani, 2017) has been adapted for pixel-wise generation, lever- aging its superior capacity to capture long-range dependencies within the data. PixeISNAIL (Chen et al., 2018) is the first work to combinc causal convolutions with self-attention mechanisms, allowing the model to consider both local and broader contexts while preserving the sequential nature of pixel generation. Inspired by it, The Image Transformer (Parmar et al., 2018) employs self-attention to han- dle local neighborhoods in larger images, improving performance on complex tasks. ImageGPT (Chen et al., 2020) utilizes a decoder-only architecture and goes further by treating images as sequences of pixels, employing an autoregressive framework similar to that used in natural language processing. The above approaches demonstrate the power of Transformers in modcling long-range dependencics between pixels, laying groundwork for future advancements in large-scale unsupervised image learning.
3.1.1.b Token-wisc Generation
Token-wise generation utilizes different approach, where the image is divided into sequence of tokens, and each token is generated based the previously generated tokens, similar to how text is processed in natural language models. This paradigm often leverages models like transformers, which can handle larger contexts in one step compared to pixel-by-pixel generation. These tokens typically represent local patches of the image in a more compact form, allowing for faster generation and scalability in terms of image size and complexity. Token-wise generation focuses on the representation of images as sequences of discrete lokens, i.e., the design of the image lokenizer, as well as the modeling of the autoregressive generation process at the token level.
(1) Design of Image Tokenizer
Brief Recap of Foundational Works. In See 2.2.2, we have briefly introduced seminal works in visual tokenization, including VQ-VAE (Van Den Oord et al., 2017), VQ-VAE-2 (Razavi et al., 2019), and VQGAN (Esser et al., 2021b). These groundbreaking studies collectively established a two-stage paradigm for next-token prediction. This paradigm involves initially training a discrete visual tokenizer, followed by training a powerful sequence model to autoregressively predict the next token. VQ-VAF (Van Den Oord et al., 2017) initially introduces autoencoder-style training and VQ techniques, compressing 128 x 128 image into a 32 x 32 discrete latent. Building on this, VQ- VAE-2 (Razavi et al., 2019) proposes a hierarchical structure that quantizes a 256 x 256 image into both 64 64 and 32 x 32 latent maps, corresponding to the bottom and top levels, respectively. VQGAN(Esser et al., 2021b) further advances this approach by integrating adversarial training and perceptual loss, achieving better perceptual quality and improving image compression rates. Due to its powerful representation capabilities, VQGAN is widely adopted in many autoregressive models and MLLMs, serving as a robust visual encoder.
Improvements in Efficiency and Codebook Utilization. Despite the advances brought by VQGAN, it still exhibits certain limitations, such as low codebook utilization and slow sampling speed. Several works have emerged to improve upon the vanilla VQGAN. Yu et al. (2021a) pro- posed VQG-N, which replaces the CNN network in the encoder-decode architecture with a Vision Transformer (Dosovitskiy, 2020), demonstrating superior reconstruction quality. In addition, they introduce a linear projector to map latent codes from the high-dimensional 578ce vector)
12

---

Published in Transactions on Machine Learning Rescarch (03/2025)
to a lower-dimensional space(32-d vector), facilitating efficient code index lookup. They also apply 12-normalizattin to the codebook vector to enhance training stability. ViT-VQGAN expands the code- book size from 1024 to 8196 without compromising the codebook utilization. Efficient- Efficient-QGGAN (Cao et al., 2023) further improves efficiency by incorporating local attention through Swin Transformer blocks (Liu et al., 2021). TiTok (Yu et al., 2024c) propose a Transformer-basedd 1D tokenizer, built on ViT- VIGAN by introducing a set of fixed length latent tokens K imput to the Vision Transformer. Adopting the architectures similar to Q-Former (Li et al., 2023b), this approach converts 2D image patches into 1D latent tokens of length K, enabling more compact latent representation.
Zhu et al. (2024b) presents VQGAN-IC, which maintains static codebook and trains projector to map the entire codebook into the latent space without requiring modifications to the encoder sncoder/decoderr /deodeer. To initialize a static codebook, they utilize a pretrained CLIP image encoder (Radford et al., 2021) to extract patch-level features from the larget dataset. These features are clustered, and class centers are selected to initialize the codebook. VQGAN-LC successfully scaled the codebook size of VQGAN 100000 with a utilization rate of 99%. LlamaGen (Sun et al., 2024a) conducts a detailed ablation study on the codebook size and vector dimension in VQGAN. Their results reveal that the large codebook dimension and small size used in vanilla VQGAN resulted in each vector containing excessive information. Consequently, attempts to scale the codebook size resulted in poor utilization rates. Therefore, LlamaGen employs a much larger codebook size with smaller vector dimensions, which has been shown to significantly enhance both reconstruction quality and codebook utilization.
Variants of Vector Quantization. Beyond improvements on VQGAN, several works have also proposed variants of Vector Quantization. One such approach is Residual Quantization (RQ), in- troduced by Lee et al. (2022a). As previously discussed in Sec. 2.2.3 and Eq. 5, RQ represents an image as D quantizing residuals of feature maps, recursively approximating the final latent code a in a coarse-to-fins manner. Each depth d shares the same codebook. Under constrained codebook sizes, RQ has been shown to outperform VQ in reconstruction performance and generation quality. Similarly, MoVQ (Zheng et al., 2022) adopts a multichannel index map with a shared codcbook for image quantization. They introduce modulation operations during decoding to incorporate spatial variants and enhance reconstruction performance. Huang et al. (2023b) further propose Dynamic Quantization (DQ), employing hierarchical encoder to represent images at multi-level. DQ employ Dynamic Grained Coding (DGC) modules to assign a dynamic granularity to each region, resulting in a multi-grained representation where each region has a variable length. Other works attempt to simplify the nearest-neighhor lookup in VQ by modifying the quantization process. Mentzer et al. (2023) propose Finite Scalar Quantization (FSQ), which assumes that the distribution of the latent code in dive-ddimnsional space follows a simple, fixed grid partition. FSQ bounds each dimension of the latent code z to L values, avoiding complex nearest-neighhor computations. It prevents codebook collapse and demonstrates high codebook utilization when scaling up the codebook size.
Inspiration from Image Compression. Some works draw inspiration from traditional image compression algorithms to design more effective image tokenizers. DCT-Transsormer (Nash et al., 2021), inspired by the JPEG compression algorithm (Wallace, 1991), designs a Discrete Cosine Trans- fDCT(DCT) (Ahmed al., 1974) based transformer to convert images into quantized DCT coefficients. Similarly, Mattar et al. (2024) develops wavelet-based image coding method leveraging wavelet Tok- enizer (Daubechies, 1992). This approach tokenizes visual details in a coarse-lo-line manner, ordering the information starting with the most significant wavelet coefficients. These approaches provide fresh perspectives on the design of image tokenizers.
(2) Autoregressive Modeling
Foundation Work in Autoregressive Modeling. As previously discussed, VQ-VAE (Van Den Oord et al., 2017) and VQ-VAE-2 (Razavi et al., 2019) introduced the concept of tokenization, steering autoregressive visual generation towards a two-stage paradigm. Despite these advancement,
13

---

Published in Transactions on Machine Learning Rescarch (03/2025)
these models still rely CNNS for token-wise autoregressive modeling. The inherent limitation of CNNs, due to their local attention mechanisms, impedes their ability to effectively model long se- quence dependencies. A significant milestone that unlocked the potential of autoregressive visual generation was the advent of VQ-GAN (Esser et al., 2021b), which combines Transformer Vaasaani, 2017) with next-token prediction, employing a GPT-2 style decoder-only Transformer to predict sub- sequent tokens in a raster-scan order. VQGAN enables the scaling of autoregressive image generation high-resolution images comprising millions of pixels, and surpasses all previous methods in both generation quality and resolution. Since VQ-GAN, Transformer have become the de-facto standard for auloregressive modeling.
Scaling Up to Larger Models. Following the establishment of Transformer structures, rescarchers have sought to investigate whether the scaling laws observed in language models are also applicable to autoregressive vision generation. LLamaGen (Sun et al., 2024a), for instance, directly employs the Llama architecture (Touvron et al., 2023a;b) without additional modifications and trains a series of image generation models with parameters ranging from 111M to 3.1B. Results demonstrate perfor- mance improvements with the increase in model parameters, indicating the potential for scalable image generation. In contrast, DeLVM (Guo et al., 2024) concentrated on minimizing data and parameter requirements to develop data-elficient large vision models. DeLVM achieves high adaptability across diverse vision tasks with significantly reduced data.
Efficiency and Long-sequence Modeling. Linear atlentions have recently gained popularity for their efficiency in long-sequence modeling during autoregressive generation. Both ARM (Ren et al., 2024b) and AiM (Li et al., 2024b) leverage the Mamba architecture to enhance image generation. ARM (Ren al., 2024b) has made successful exploration on combining autoregressive models with the Mamba architecture to enhance the visual capabilities of Mamba and accelerate the training process. AiM (Li et al., 2024b) applies Mamba directly for next-token prediction in autoregressive models, achieving superior quality and faster inference on ImageNetIK with a FID of 2.21x.
Exploration Beyond Raster Order. Traditional autoregressive methods typically serialize 2D images into 1D discrete sequences following a raster order. However, recent research suggests that this reliance on raster order and discrete representations may not be essential for autoregressive generation modeling. Studies indicate that exploring alternative methods could lead to improved performance and open up new possibilities for image representation. SAIM (Qi et al., 2023) introduces stochastic process for autoregressive image modeling by predicting image patches in random orders rather than following a fixed raster order of pixel predictions. This stochastic approach challenges the conventional necessity of raster order in autoregressive modeling. MAR (Li et al., 2024c) provides a theoretical analysis suggesting that raster order and discrete representations are not necessary. MAR further introduces a diffusion loss and trains an autoregressive model(in general definition) with continuous representations in a Masked Image Modcling (MIM) style. They achieve comparable performance traditional discrete autoregressive techniques, questioning the conventional reliance on discrete tokenization. SAR (Liu et al., 2024g) further generalizes the problem into unified framework. SAR enables causal learning with any sequence order output intervals, offering a more flexible and potentially powerful approach to autoregressive image modeling.
Intergration with Other Generative Models. The idea of combining autoregressive models with other generative models like GANs (Goodfellow et al., 2014) and Diffussio (Ho et al., 2020) has also garnered researchers' interest. RAL (Ak et al., 2020) innovatively introduces adversarial learning and policy gradient optimization into the training of autoregressive models address the inherent exposure bias problem, which is common issue in autoregressive models. ImageBART (Esser et al., 2021a) employs coarse-to-fine autoregressive method, combining multinomial diffusion with hierarchical structures to iteratively improve image fidelity and details during generation. DisCo-Diff (Xu et al., 2024) aims to enhance diffusion models by augmenting them with learnable discrete latents. They
14

---

Published in Transactions on Machine Learning Rescarch (03/2025)
adopt autoregressive transformer to model the distribution of these discrete latents. This approach achieves state-of-the-art FID scores the ImageNet benchmark.
3.1.1.c Scale-wise Generation
While next-token prediction has achieved considerable success in vision generation and has almost become the de-facto standard paradigm for visual autoregressive models, several significant challenges still warrant discussion. The first is compulational elficiency. Due to the inherent 2D nature of images, the quadratic computational complexity of self-attention transformers, and the autoregressive steps involved in generation. the overall complexity can reach (nz) when generating an n X n token map. Although efforts have been made to reduce the sequence length through tokenization, the growing image resolution exacerbates this issue, rendering the computational demands increasingly unsustainable. The second challenge is the flattening operation. In the context of next-token prediction, visual tokens are flattened into a 1D sequence after tokenization, which limits the model's ability to leverage spatial locality during autoregressive generation. While recent studies (Sun et al., 2024a) demonstrate that LLM-style autoregressive models can produce high-quality images, the necessity of preserving spatial locality remains an open issue.
In contrast to next-token prediction, the next-scale prediction paradigm utilizes token maps at different scales as the base autoregressive units, progressively generating visual content via a coarse-to- toefiee manner. This paradigm further reduces the sequence length required for autoregressive generation. Generating a complete token map at each step, enables leveraging the inductive biases inherent in images to enhance the quality visual generation. In line with previous sections, we introduce the design of visual tokenizers and autoregressive modcling within the next-scale prediction paradigm.
(1) Design of Video Tokenizer
The paradigm of next-scale prediction, akin to next-token prediction, requires a visual tokenizer to compress an image into a discrete token map. foundational technique behind this is the Residual Quantization (RQ) introduced in RQ- VAE (Lee et al., 2022a). RQ-VAE initially developed to address the low utilization issue of VQGAN when scaling to larger codebook sizes, which is achieved by introducing an additional depth dimension d. In RQ-VAB each latent vector is quantized into D codes, with each code representing the quantization of the residual vector at the current depth. All depths d share the same codebook. RQ recursively approximates the final latent vector in a coarse-to-fine fonee manner, providing a more accurate approximation of z while maintaining the codebook size. Note that RQ still quantizes the image into a token-wise sequence based on spatial positions, with each position having multiple residual codes at different scales. Although RQ is not a next-scale prediction method, its hierarchical approach offers valuable insights for next-scale prediction.
Building on the hierarchical and residual-style quantization concepts of RQ-VAE, VAR (Tian et al., 2024) introduces more concise scale-wise quantization method. VAR employs vanilla VQ- GAN (Esser et al., 2021b) to encode the feature map in the latent space. It then interpolates and com- putes residuals to quantize the feature map f E |gxxex into K multi-scale token maps Tk,,,,,kkkk each at increasingly higher resolutions h& X Wk; culminating in т which matches the original feature map's resolution h x w. Similar to RQ-VAE, VAR quantizes the residuals of each scale relative to the previous scale, with residuals sharing the same codebook across different scales. Through this hierar- chical approach, VAR represents an image as coarse-to-fine scale-wise sequence. The quantization process of this sequence is entirely causal, and each autoregressive unit maintains spatial locality.
(2) Autorogressive Modeling
15

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Scale-wise generation involves creating images at multiple scales or resolutions. The model typically starts by generaling coarse, low-resolution version of the image and progressively enhances details through successive stages. This technique allows the model to capture both global structure and finer details without needing pixel-level precision in the initial stages. As each phase refines the image, it adds more detail and adjusts local features to improve overall fidelity and texture. Scale-wise generation is particularly effective for high-resolution image synthesis, where maintaining consistency across scales is crucial.
Despite its advantages, scale-wise generation is less common in autoregressive models due to the significant complexity it introduces in training and computation. Generating images at multiple scales requires careful coordination between different resolutions, which can be challenging and resource- intensive. Each scale builds upon the previous onc, increasing the risk of crror propagation from lower resolutions. Furthermore, this approach demands more computational power and memory as the model processes and refines images on several scales, making it less efficient. than single-resolution methods.
A pioneering work that achieved this breakthrough is VAR (Tian et al., 2024), which brought the concept of "next-scale prediction' to reality. VAR. innovatively shifts the focus from traditional pixel- wise prediction to scale-wise generation paradigm. As previously introduced, VAR draws inspiration from RQ- VAE, quantizing a single image into token maps of various resolutions from coarse to fine. VAR employs a standard decoder-only transformer to model the next-scale prediction, while a block- wise causal mask applied to enable each scale to depend only on its prefix and allow for parallel token generation. VAR outperforms existing autoregressive models and diffusion transformers, achicving superior results on benchmarks such ImageNet. Furthermore, the model exhibits power-law scaling laws similar to those observed in large language models, which may reveal a research direction for vision autoregressive modcling.
Overall, these sea sciee generation techniques mark significant advancements in high-resolution image synthesis. Following VAR, numerous works have emerged based on VAR including text-to to-gg generation, image editing, and controllable image generation. We will discuss these developments the subsequent chapters.
3.1.2 text-to-lmage Synthesis
Toxa-to-Image generation involves generating images basod specific textual conditions provided to the model. Unlike unconditional image generation, which produces images without any input constraints, text-to-image generation leverages additional information to guide the creation process. By incorporating textual conditions, the model can produce more targeted and contextually relevant images, euhancing the quality and applicability of the generated outputs. The illustration of typical text-to-image synthesis frameworks is shown in Figure 6.
3.1.2.a Token-wise Generation
Exploration across the integration of textual conditions and visual modeling. A pioncering effort in this field is is exemplified by DALL E (Ramesh et al., 2021), which employs BPE-encoder and pretrained VQ tokenizer to convert text and images into discrete tokens, respectively. The model then utilizes text tokens as a prefix condition and learns to predict the subsequent image tokens. Building upon this foundation, CogView (Ding et al., 2021) scales up the autoregressive token- wise model by employing a VQ-VAF tokenizer to transform image patches into discrete tokens. This approach enables the deployment of 4-billion parameter Transformer model with higher performance. CogView2 (Ding el al., 2022) further explores hierarchical strategy for text-to-image generation. It first train base T2I model to generate images at low-resolution, followed by fine-tuning stage aimed at super-resolution This hierarchical approach successfully reduces computational demands and enhances large-scale image generation.
16

---

Published in Transactions on Machine Learning Research (03/2025)
VQVAE Tokenizer
Text
Decudder
Image
Transformrr blocks
Image
Output
M
M
Tokenizer
Encoder
self-attention
txt token
...
bxt token
<SEP>
img token
img token
<END>
Auto-regressive Transformer
(a) Base framework for autorgggsssive text-to-image gemeration on
Cross-attention
incoder
Tokenizer
Text
Image
autoregressively denoise
Text
addd noise
Output head
dog
<SEP>
rego-segeessive Transformer
(c) A Autoregressive generation with random order by bi-directional transformer and masking
(b) utorggressivv image generation combined with denoising process
Figure 6: Illustration of different frameworks for auto-rogrossive text-to-image generation. (a) Generally, autoregressive text-to-image generation involves tokenizing lext and images, and model all the tokens with a autoregressive transformer. (b) With the flexibility of autoregressive transformers, the autoregressive generation can be combined with other process, such as the denoising process or multi-scale generation process. (c) With the bi-directional transformer, the autoregressive generation sequence does not need to be a pre-defined order (such as raster scan), but can be any random order.
Actually, these methodologies treat text tokens prefixes for autoregressive modeling, thereby the transformer is responsible for both capturing text semantics and generating image content within the autoregressive framework. In addition to this technical approach, another line of research involves encoding text tokens using a dedicated text encoder and generating image content using a transformer docoder. For instance, Makeea-Scene (Gafni et al., 2022) employs an external text encoder to process the textual input, while Parti (Yu et al., 2022) adopts encoder-decoder architecture. In Parti, transformer encoder is utilized to encode the text information, and transformer decoder is employed to generate the corresponding image. During period when autoregressive modeling had not yet reached its full potential, these approaches demonstrated a more robust semantic understanding compared to earlier methods.
petuuence-t-ssquenee (seq2seq) model with an
Furthermore, LQAE (Liu et al., 2024b) proposes an unsupervised technique for aligning text with images by quantizing image embeddings into text-like tokens, enabling few-shot multimodal learn- ing. Fluid (Fan et al., 2024) introduces random-order autoregrossive model continuous tokens, highlighting importance of token representation and generation order in scaling autoregressive models.
Integration with Diffusion Models. Inspired by the powerful generative capabilities of diffusion models, recent research has investigated the integration diffusion models with autoregressive models for text-to-image generation. VQ-Diffusion (Gu al., 2022) employs Vector Quantized Variational Autoencoder (VQ-VAE) map images into discrete latent space. Subsequently, it utilizes discrete diffusion decoder instead of transformer decoder reconstruct the images. In contrast, Kaleido Diffusion (Gu et al., 2024) maintains diffusion the core generative framework while incorporating an autoregressive model to handle the latent conditions, thereby enhancing both text understanding and image generation capabilities. Another integrating autoregressive model with diffusion is DART (Zhao
17

---

Published in Transactions on Machine Learning Rescarch (03/2025)
et al., 2024), which unifies autoregressive and diffusion within non-Markovian framework, achieving scalable, high-quality image synthesis.
Integration with Large Language Models. A significant advancement, LLM4GEN (Liu et al., 2024f), presented an end-to-end framework combining text encoders like CLIP with diffusion models, enriching loken-wise generation with the semantic depth of large language models (LLMss). This results in improved semantic correspondence between text and image tokens for complex prompts. V2T (Zhu et al., 2024a) translates images into discrete tokens from an LLM's vocabulary, aligning visual and textual data for tasks such as image denoising. MARS (He et al., 2024) features the Semantic Vision-Language Integration Expert (SemVIF)) module, deeply integrating textual and visual tokens, illustrating the flexibility of token-win baken-wise generation in producing detailed images from textual descriptions. Lumina-mGPT (Liu et al.; 2024a) applies Flexible Progressive Supervised Finetuning (FP-SFT) to Chameleon (Team, 2024) with high-quality image-text pairs to fully unlock the potential of the model for high-acsthetic image synthesis while preserving its general multimodal capabilities.
Expansion to Novel Tasks. IconShop (Wu et al., 2023b) further extends token-wise generation to vector graphies enabling scalable vector icon creation from text prompts. Make-a-story (Rahman et al., 2023) focuses on generating visually coherent stories from text, employing a scale-wise generation process to maintain consistency across multiple image frames within complex narratives. SEED- Story (Yang et al., 2024), a multimodal storytelling model, generates images and text in parallel, ensuring that visual details correspond seamlessly with the narrative across different resolution levels. Its hierarchical attention mechanism preserves coherence between text and images at all scales, from coarse descriptions to intricate image details.
3.1.2.b Scale-wise Generation
As previously discussed, VAR pioncered novel paradigm beyond token-wise gencration, termed scale- wise generation, which generates images progressively from coarse to fine scales. Due to its advantages in efficient generation, coupled with its demonstration of scaling laws analogous to those observed in language models. Scale-wise generation has garnered significant research interest. Efforts have been made towards extending this paradigm to the text-to-image generation domain.
STAR (Ma et al., 2024) is a pioneering work in the domain of scale-wise text-to-image generation. Building upon the common paradigm of diffusion models (Rombach et al., 2022) in image generation, STAR employs features from pretrained text encoder, utilizing cross-attention layers provide detailed textual guidance. In addition, STAR introduces novel techniques such as Rotary Position Embedding (ROPE) (Su et al., 2024) to enhance the autoregressive modeling capabilities. These modifications result in improved text alignment and more efficient generation processes, demonstrating the potential of scale-wise toxt-to-image generation to achieve higher fidelity and coherence.
Concurrently, VAR-CLIP (Zhang et al., 2024a) introduces their approach by concatonating text embed- dings from a pretrained CLIP model with visual embeddings from a VAR encoder. this framework, the text embeddings serve as condition to guide the generation of multi-scale tokens and the final image. This method has demonstrated notable effectiveness in text-to-image generation.
In summary, scale-wise generation models provide powerful framework for generating high-resolution images by progressively refining outputs across multiple levels of detail. This hierarchical approach not only ensures global coherence, but also captures intricate details, making it a robust solution for text-to-image gencration tasks.
3.1.3 Image-condition Synthesis
Autoregressive models have played significant role in advancing image generation tasks, including Image-to-Image translation, where the goal is to translate one image domain to another.
18

---

Published in Transactions on Machine Learning Research (03/2025)
input image
positional
Encoder
embedding
output image
Decoder
Auto-rest Auto-regressive mran
(a) Image inpainting
input image
Patchify
Encoder
blocks
patch tokens
query expansion
Decoder blocks
output tokens
queries
output image
(b) Image outpainting
Figure 7: Frameworks for image painting. (a) BAT-Fill (Yu et al., 2021b) inpaints the image by rearranging the input tokens and auto-regressively generate output. (b) Query-OTR. (Yao et al., 2022) does image outpaintngg by expanding input tokens into query tokens.
3.1.3.a Image Painting
parallel, some works focuses on specific editing tasks such as inpainting and outpainting, where missing or extended visual content needs to be generated coherently. Inpainting fills in missing regions by seamlessly blending new content with the surrounding areas. Outpainting, on the other hand, extends image boundaries by generating new content that aligns naturally with the original scene, creating larger images from smaller inputs. Despite these differences, both techniques face challenge of maintaining visual coherence with existing context. Outpainting (Zhang et al., 2024b) is particularly useful for generating panoramic views or expanding image contexts. QueryOTR (Yao et al., 2022) addresses this by using a hybrid transformer-based encoder-decoder architecture, reframing outpainting as a patch-wise sequence- sequence-to-sequenee to-sequence task. For inpainting, Yu et al. (2021b) combines autoregressive modeling with context aggregation, ensuring consistent reconstruccton of missing image regions by using the information surrounding the pixel. The overview of image painting are shown in Figure
3.1.3.b Multi-view Generation
MIS (Shen et al., 2024) introduces an autoregressive framework for generating multiple interrelated images from the same distribution. By leveraging latent diffusion models, MIS enhances diversity while ensuring semantic coherence across scenes. This approach demonstrates the potential of autoregressive processes in multi-view and sequential image generation tasks. SceneScript (Avetisyan et al., 2024) further explores autoregressive modeling in the context of scene reconstruction, using token-based generation to translate complex 3D scenes into structured commands. Both works illustrate how autoregressive models can be extended beyond single-image generation to more complex tasks like multi-view generation and structured scene reconstruction.

---

Published in Transactions on Machine Learning Research (03/2025)
texut
input image
mask
output image
psychedalic painting of American Gothic 1930
segmentation
Text Encoder
VQ-SEG Encoder
VQ-IMG Decoder
Text tokens
Scene tokens
Image tokens
-egresseeee Transformer
Figure 8: For autoregressive image editing, make-a-scene handles text tokens, scene tokens and image tokens with a autoregressive transformer.
3.1.3.c Visual In-Context Learning
Bar et al. (2022); Bai al. (2024b) has emerged a key prompting method for visual autoregressive models. MAE- VQGAN (Bar et al., 2022) firstly proposes visual prompting, where pretrained visual model adapts to new tasks as image inpainting without finetuning or modification. Inspired by it, VICL (Bai et al., 2024b) adapts in-contexi learning to vision tasks by presenting task descriptions or demonstrations through images or natural language text, encouraging models to generate predictions based on provided image queries.
3.1.4 Image Editing
Image editing is a crucial area in computer vision, focusing on modifying, enhancing, or reconstructing images based various inputs. Recent advancements in deep learning have enabled models to perform complex editing tasks such inpainting, outpainting, and style transfornations, allowing users to manipulate visual content with greater case and precision. These tasks often involve generating new content that seamlessly integrates with the existing image, either by filling in missing parts (inpainting) or extending the image boundaries (outpainting), while maintaining visual consistency.
3.1.4.a Text-driven Image Editing
VQGAN-CLIP (Crowson et al., 2022) introduces a method for generating and editing images using natural language input. Its editing capabilities stem from the combination of VQGAN's image synthesis and CLIP's ability to guide image modifications based on text prompts. This allows users to modify existing images or create new ones by altering styles, adding elements, or transforming parts of an image, while maintaining visual coherence. In contrast, Mako-A-Scene (Gafni et al., 2022), illustrated in Figure 8, enhances this by incorporating scene layouts (segmentation maps) alongside text input. This addition enables more precise control over the structure and content of images, making it especially useful for localized edits. Make-A-Scene provides control over both semantics and spatial arrangement, whercas VQGAN-CLIP focuses more on creative, text-driven modifications.
3.1.4.b Image-driven Image Editing
Image-driven image editing encompasses a range of techniques focused modifying visual content while maintaining consistency with the original image's context and style. In the domain of autore- gressive image generation, the introduction of control mechanisms is pivotal for enhancing both the flexibility and precision of the generated images. For example, ControlAR (Li et al., 2024e) integrates
20

---

Published in Transactions on Machine Learning Rescarch (03/2025)
spatial controls like canny edges and depth maps into the decoding process, allowing for detailed and precise manipulation of image tokens. This approach enables more accurate aligument with the original image's features during generation. Similarly, ControlVAR (Li et al., 2024d) advances this concept by modeling image and pixel-level control representations through a scale-level strategy, thus achieving fine-grained control over the image synthesis process while preserving the inherent properties of autoregressive models. CAR (Yao et al., 20244) elaborates on a similar concept, focusing on advanced control mechanisms in autoregressive models to enhance the detail and adaptability of visual outputs.
On the other hand, for complex image generation tasks involving multiple objects, Many-to-Many Diffusion (M2M) (Shen et al., 2024) applies an autoregressive diffusion model to generate multi-image sequences, scaling across both spatial and temporal dimensions while maintaining consistency across frames. MSGNet (Cardenas et al., 2021) utilizes a voctor-quantizer variational autocncoder (VQ-VAE) combined with autoregressive modeling. This framework prioritizes both spatial and semantic coher- ence within generated images, effectively preserving consistency across multiple objects. Extending this concept to medical imaging, MVG (Ren et al., 2024a) proposes a unified framework for diverse 2D medical tasks such as cross-modal synthesis, image segmentation, denoising, and inpainting. This approach leverages autoregressive training conditioned on prompt image-label pairs, treating these tasks as image-to-image generation problems.
Overall, image-driven image editing techniques focus on refining image synthesis and manipulation tasks, employing control mechanisms, context-aware models, and coherent extensions of existing con- tent to achieve precise and visually consistent outputs.
3.2 Video Generation
Building upon the advancements of autoregressive models in image generation, video generation (Yuan et al., 2024a;b) has similarly seen significant progress by extending these models to capture temporal dynamics in addition to spatial patterns. While autoregressive image models focus on generating static frames, video generation introduces the challenge of producing coherent sequences over time. Like image generation, video generation is categorized into unconditional and conditional approaches with different condition inputs. By leveraging the success of autoregressive models in both domains, researchers continue to push the boundaries of generating realistic and temporally consistent video sequences. Additionally, the integration of video generation techniques into embodied AI systems presents a new frontier, where generated videos are not just an end in themselves but are used to enhance the capabilities of intelligent agents in complex, real-world environments. The basic pipclinc of autoregressive video generation can be found in Figure 9.
3.2.1 Unconditional Video Generation
(1) Design of Vidco Tokenizer
Recent works have expanded visual tokenization to videos by incorporating temporal compression. MAGVIT (Yu et al., 2023a) extends the 2D VQGAN into 3D tokenizer to quantize video data into spatial-temporal visual lokens, employing an inflation technique for initialization using image pre- training. However, it struggles to tokenize images effectively. Building on MAGVTT, MAGVTT-v2 (Yu et al., 2023b) introduces lookup-free quantization (LFQ) method, replacing the traditional codebook with binary latents derived from an integer set of size K, similar to word tokenizers in natural lan- guage processing (Sennrich, 2015). This LFQ approach allows the vocabulary size to grow in a way that benefits the generation quality of autoregressive models. Additionally, MAGVIT-22 integrates C-ViVIT (Villegas et al., 2022) and MAGVIT within causal 3D CNN, enabling unified tokenization of both images and videos using shared codebook. Open-MAGVIT2 (Luo al., 2024) has success- fully implemented and open-sourced MAGVTT-v2 for image tokenization, explored its application in auto-regressive models, and validated its scalability properties. Omni Tonenzoree (ang et al., 2024b)

---

Published in Transactions on Machine Learning Rescarch (03/2025)
employs a transformer-based architecture with decoupled spatial and temporal blocks perform image and video tokenization within a unified framework, utilizing progressive training strategy to enable general-purpose visual encoding across varions modalities.
(2) Autoregressive Modeling
Unconditional video generation focuses creating video sequences from scratch, without any specific imput conditions. One of the foundational approaches in this area is the Video Pixel Networks (VPNs) (Kalchbrenner et al., 2017), which extend the PixeICNN model to video data. VPNs model each pixel in video frame based on the preceding pixels and frames, allowing for the capture of intricate spatial details. However, VPNs face challenges in maintaining temporal coherence over longer sequences, resulting less realistic motion when generating exlended videos.
To address some of these limitations, MoCoGAN (Tulyakov et al., 2018) introduces an approach by decomposing the vidco generation process into two distinct components: motion and content. The model employs Generative Adversarial Networks (GANs) to separately generate static content frame and sequence of motion vectors, which are then combined to form the final video. This decomposition allows MoCoGAN to produce videos with more realistic and temporally coherent motion, although the strict separation of motion and content can sometimes limit the model's ability to generate highly complex and expressive content.
Video Transformer (Weissenborn al., 2019) focuses on leveraging the transformer architecture to enhance the capacity and performance of these models to handle larger datasets and generate higher- resolution, longer videos. These advancements push the boundaries of what is possible with autoregres- sive models, enabling the generation of more detailed and extended video sequences. This approach offers a more efficient. alternative to VPNs and MoCoGAN, particularly in handling the temporal dynamics of video generation.
LVT (Rakhimov et al., 2020) and VideoGPT (Yan et al., 2021) represent significant step forward by combining the power of VQ-VAE with transformers. By operating in a discrete latent space, they achieve a balance between computational efficiency and high-quality video generation, improving both the diversity and fidelity of the generated videos.
To reflect the latest developments, including dongdiuraion video synthesis, models such as TATS (Ge et al., 2022) have been introduced. This model leverages the combination of VQGAN and transformer archilectures produce temporally consistent long videos. Additionally, the PVDM (Yu et al., 2023d) demonstrates advanced diffusion- approaches for video generation, offering improved probabilis- tic modcling in latent spaces. Furthermore, MAGVIT-v2 (Yu et al., 2024b) emphasizes the role of tokenizers in refining autoregressive models for video generation. LARP (Wang et al., 2024a) uses a learned AR generative prior, capturing global semantic content and enhancing compatibility with AR models.
3.2.2 Conditional Video Generation
Conditional video generation involves generating videos based on specific inputs, such as text descrip- tions, images, or existing video frames. This task is more complex than unconditional generation, as it requires the model to ensure that the generated content aligns with the given conditions.
Text-to- Video Synthesis. Text-to-video synthesis challenging because involves translating textual input into coherent and contextually appropriate video sequences. One of the earlier models addressing this challenge IRC-GAN (Deng et al., 2019), which employs a recurrent architecture com- bined with GANs to iteratively refine video generation from text. By using introspective modules that encourage the model to correct its own mistakes and recurrent convolutional layers to handle temporal dependencies, IRC-GAN improves the quality and coherence of generated videos, setting a foundation
22

---

Published in Transactions on Machine Learning Research (03/2025)
input image /frames/ depth optical ffow
outputt video
output audio
Input audio
input text
Viärant parrot with straking red mankings looking oround.
Visual encoder
Sound encoder
Audio decoder
Iasktokens
encoder
tokenss control tokens
decoder
Auto-regressive video model Pretrained LLM
prefix
autoregressiveely generated output
The pipeline of the auto-regressive video generation. The multi-modal input tokens are tokenized by different encoders, and set prefix conditions for autoregressive models. The output tokens are autoregressively generated.
Figure 9:
for future models. Godiva (Wu et al., 2021) utilizes transformers align textual descriptions with video content. GODIVA is capable of producing diverse and realistic video outputs from natural lan- guage descriptions, demonstrating the potential of transformers in bridging the gap between textual and visual modalities.
Building on this foundation, VogVideo (Hong et al., 2022) enhances the text-to-videe video synthesis process through large-scale pretraining. By leveraging vast amounts of data during pretraining, Cog Video improves the modcl's ability generate coherent vidcos from wide range of text inputs, making more robust and versatile in handling different types of textual descriptions.
NÜWA (Wu al., 2022) introduces visual synthesis pretraining approach to further enrich the model's understanding of visual information before fine-tuning on text-to-video tasks. This pretraining phase allows the model to develop a more nuanced understanding of visual data, resulting in more accurate and realistic video generation based on textual input. NUWA-Infinity (Liang et al., 2022) extends the capabilities its predecessor by enabling the generation of videos of arbitrary length. By stacking multiple autoregressive models, NUWA- overcomes the limitations of fixed-length video generation, providing greater flexibility and adaptability.
Phenaki (Villegas et al., 2022) takes different approach by introducing variable-length video gener- ation, allowing the model to produce videos that match the length and detail specified by the input text. This flexibility makes Phenaki particularly suitable for generating videos that need to accommo- date varying levels of detail and duration. ART-V (Weng et al., 2024) introduces a masked diffusion model (MDM) reduce drifting by determining which parts of the frame should rely on reference images versus network predictions. Additionally, anchored conditioning maintains consistency across long vidcos, allowing ART-V to produce detailed and aesthetically pleasing vidcos even with limited training resources. ViD-GPT (Gao et al., 2024) applies GPT-style auloregressive framework within video diffusion models to generate high-fidelity video frames one at time, conditioned on previously generated frames. By leveraging a latent space diffusion process and sophisticated temporal condition- ing, ViD-GPT effectively reduces temporal inconsistencies and visual drift, resulting in videos that are both visually appealing and temporally coherent.. Loong (Wang al., 2024e) showcases capabilities in generating lengthy videos using short-to-long video training strategy. PAV (Xie et al., 2024a) propose progressive diffuusin approach, improving both speed and fidelity in text-conditioned video generation. ARLON (Li et al., 2024f) integrates autoregressive models with diffusion transformers to provide long-range temporal guidance. LWM (Liu et al., 2024c) introduces an architecture capable of handling very long video sequences by leveraging Blockwise RingAttention. iVideoGPT (Wu et al.,
23

---

Published in Transactions on Machine Learning Rescarch (03/2025)
2024c) incorporates interactive mechanisms in video generation, making it more adaptable and scal- able. Pandora (Xiang et al., 2024) uses hybrid autoregressive-diftsiooo model to allow real time text controlled video generation.
Visual Conditional Video Generation. Visual conditional vidco generation involves generating future video frames based on an initial sequence of frames or images. This task requires models to accurately capture and predict the temporal dynamics of video sequences. It relates to unconditional video generation in that the methods developed for unconditional video generation can also be applied video prediction, providing a foundation for predicting future frames based on learned temporal patterns. The Convolutional LSTM Network (Shi et. al., 2015) is one of the foundational models in this area, combining CNNs with LSTM units. This combination allows the model to effectively capture both spatial and temporal dependencies, making it particularly effective for tasks like precipitation nowcasting, where predicting the movement and evolution of visual patterns is crucial. Building upon this, PredRNN (Wang et al., 2017) proposes to use a spatiotemporal memory, and E3D-LSTM (Wang et al., 2019) integrates 3D convolutions into LSTM.
SV2P (Babaeizadeh et al., 2018) introduces stochastic elements into the video prediction process, allowing the model to account for multiple possible futures rather than predicting single deterministic outcome. By learning a distribution over potential future frames, this model improves the realism and diversity of the generated videos, making it more adaptable to real-world scenarios where uncertainty and variability are inherent. PVV (Walker et al., 2021) ) and HARP (Seo et al., 2022) take advantage of latent space prediction to improve the accuracy and efficiency of video prediction. VQ-VAE compresses video data into a discrete latent space, where autoregressive models can more easily predict future frames. HARP builds on this approach by combining latent space prediction with a high-fidelity image generator, resulting in vidco sequences that are not only accurate but also visually detailed and temporally coherent. MaskViT enhances video prediction by using masked visual pretraining, where portions of video frames are masked during training, forcing the model to learn to predict the missing information. This approach improves the model's ability to handle occlusions and other challenging scenarios in vidco prediction tasks.
TVM (Bai et al., 2024a) introduces an approach that enhances the scalability of large vision models, making them more adaptable for multimodal tasks like image-to-video generation. ST-LLM (Liu et al., 2025) demonstrates that large language models are highly effective temporal learners, significantly advancing video generation tasks that rely on visual cues. Furthermore, Pyramid Flow (Jin et al., 2024a) introduces an efficient approach to flow matching, improving the computational efficiency of video generation models without compromising the temporal or spatial consistency of the generated videos. Building on the power of masked auto-regression, MarDini (Liu et al., 2024e) combines Masked AR with diffusion for scalable vidco generation.
Multimodal Conditional Video Generation. Multimodal conditional video generation combines visual and textual inputs to create video sequences. MAGE (Hu et al., 2022) presentes a key example of this approach, allowing users to control the motion and content of the generated video through text descriptions. This model provides precise control over the generated video, making it particularly useful for applications where detailed and specific video content is required, such as animation or visual storytelling.
While most video generation models focus on either unconditonal or conditional generation, there are models that aim to generalize across different types of video generation tasks, allowing single model generate videos from a wide range of inputs without requiring task-specific training. VideoPoet (Kondratyuk et al., 2024) is significant step in this direction, leveraging a large language model (LLM) for zero-shot video generation. VideoPoet can generate videos from diverse inputs, including text, images, and existing videos, demonstrating the potential of LLMs in creating versatile and adaptable video generation models. This approach shows promise for developing truly universal models that can
24

---

Published in Transactions on Machine Learning Rescarch (03/2025)
handle the complex and varied demands of real-world video generation applications. The scalability of multimodal video generation has also been improved by sequential modeling techniques.
Autoregressive models have significantly advanced video generation, offering new capabilities in both unconditional and conditional vidco tasks. These approaches continuc to evolve, improving scalability, flexibility, and generalization, bringing us closer to universal models capable of addressing the diverse demands of real-world video generation applications.
3.2.3 Embodied AI
While the previous subsections have focused on the generation of video sequences as an end goal, the emerging field of embodied AI introduces an important application for video generation. In this context, video generation is not merely about creating visually appealing or realistic sequences; instead, it serves as a critical component in training and augmenting intelligent agents that interact with and navigate their environments. By providing synthetic yet realistic visual data, video generation techniques can significantly enhance the learning process for embodied Al systems, enabling them to better understand and respond to dynamic, real-world scenarios. The following part explores how video generation is being leveraged to empower embodied AI, bridging the gap between visual synthesis and intelligent interaction.
Learning general world models visual domains remains a significant challenge for policy learning in embodied AL. One approach to learn continncond video prediction models (Oh al., 2015; Kaiser et al., 2020), which predict future frames based on current observations and actions. Advanced model-based reinforcement learning (RL) algorithms (Hafner et al., 2020; 2021; 2023; Schrittwieser et al., 2020; Hansen et al., 2022) enhance efficiency and accuracy by leveraging latent imagination. For example, IRIS (Micheli et al., 2023) uses a discrete autoencoder and an autoregressive Transformer to model dynamics as a sequence learning problem, improving sample efficiency with minimal tuning, particularly the Atari 100k benchmark (Kaiser el al., 2020). However, these methods often complicate the process by tightly coupling model learning with policy learning.
To address this issue, recent efforts have focused on building world models that accumulate generaliz- able knowledge beyond specific lasks. Leveraging scalable architectures such Transformers (Micheli et al., 2023) and pre-training on large-scale datasots (Wu et al., 2023a; Mendonca et al., 2023) have shown promise. GAIA-1 (Hu et al., 2023) integrates an autoregressive world model with video dif- fusion decoder to predict future driving scenarios. By discretizing video frames into tokens, GAIA-1 trains a self-supervised autoregressive Transformer to predict the next frame at the representation level. These representations are subsequently decoded via video diffusion process, yielding realis- lic frames with fine-grained details. iVideoGPT (Wu et al., 2024d) adopts a generic autoregressive Transformer framework to enhance the flexibility of scalable world models. Genie (Bruce et al., 2024) pre-trains world models by learning latent actions from videos without ground-truth actions, enabling virtual worlds. GR-1 (Wu et al., 2024b) and GR-2 (Cheang et al., 2024) intro- duce GPT-style models for multi-task, Lunguage-condtioons trained on video datasets and fine-tuned on robot data for end-to end-to-end prediction of actions and future keyframes. GR-2, pretrained on 38 million vidco clips, generalizes across diverse robotic tasks and environments. Their architectures ures incorporate large-scale video generative pre-training into robotic manipulation tasks, thereby enhancing few-shot learning and generalization capabilities.
visual robot manipulation. GR-1 is pre-
3.3 3D Generation
With increasing demand for realistic and controllable 3D content across various fields from gaming and film to medical imaging and autonomous driving researchers have been striving to advance 3D generation methods. Recent approaches leverage autoregressive models to improve control and detail in generated 3D structures. This section provides a comprehensive overview of key 3D generation advancements, focusing on the areas of motion, point clouds, scenes, and medical imaging. By exploring
25

---

Published in Transactions on Machine Learning Rescarch (03/2025)
these advancements, we observe the transformative potential of autoregressive methods in achieving more accurate and versatile 3D models.
3.3.1 Motion Generation
T2M-GPT (Zhang et al., 2023a) leverages pretrained text embeddings to generate detailed human motion sequences from textual descriptions. HiT-DVAE focuses (Bie et al., 2022) on generating 3D human motion sequences, utilizing a dynamical variational autoencoder (DVAE) to encode temporal dependencies in pose sequences into time-dependent latent variables, which are decoded across mul- tiple scales to produce realistic motion. HuMoR. (Rempe et al., 2021) models motion as a series of hierarchical latent variables, ensuring that both global movement patterns and finer details are accu- rately captured produce realistic 3D motion sequences. AMD (Han al., 2024) proposes novel autoregressive model that iteratively generates complex 3D human motions from long text descrip- tions, leveraging the previous time-step text and motion sequences for improved temporal coherence and diversity.
3.3.2 Point Cloud Generation
Recent advances in autoregressive models have greatly enhanced 3D generation tasks. Canonical- VAE (Cheng et al., 2022) propose a transformer-baser model for point cloud generation, decomposing point clouds into semantically aligned shape compositions, improving both reconstruction and un- conditional generation. Octree Transformer (Tbing et al., 2023) introduces an octree-based mothod with adaptive compression, linearizing complex 3D data for efficient autoregressive generation. (Qian al., 2024) present the Improved Auto-regressive Model (ImAM) for 3D shape generation, leverag- ing discrete representation learning for efficiency and supporting conditional generation. Additionally, Argus3D (Qian et al., 2024) scales these autoregressive models with 3.6 billion parameters, achieving impressive results on large-scale datasels.
3.3.3 Scene Generation
Inspired by the success of multi-scale alignment of visual and language tokens, more tasks adopting similar Smproacches.ScenSSrrii (Avetisyan et al., 2024) proposes novel autoregressive image gener- ation approach by using structured language commands to represent scenes. This method provides compact and interpretable scene representations, excelling in architectural layout estimation and 3D object detection, while being casily extendable to new tasks.
3.3.4 3D Medical Generation
3D images are challenging to acquire due to cost, quality, and accessibility constraints. Aiming to produce high-resolution 3D volumctric imaging data with the correct anatomical morphology, Syn- thAnatomy (Tudosiu et al., 2022) and BrainSynth (Tudosiu et al., 2024) scale and optimize VQ-VAE and Transformer models for high-resolution volumetric data. Similarly, ConGe (Zhou & Khalvati, 2024) and 3D- VQGAN (Zhou al., 2023b) introduce the class-conditional generation framework for synthesizing 3D brain tumor MRI ROIs Another way to generate 3D representations is image syn- thesis based on 2D images. To achieve this goal, Unalign (Corona-Figuerdd et al., 2023) reformulate it as a toxel-to-voxel prediction problem and achieve it by conditioning an unconstrained transformer on 2D input views. Specifically, they model such mapping with a conditional likelihood- -aaed generative model, allowing sampled 3D data to sit at arbitrary positions/rotan /rotatios relative to the 2D data. More- over, AutoSeq (Wang et al., 2024c) introduces an autoregressive pre-training framework to represent 3D medical images, which treats them as interconnected visual tokens, which performs well on several downstream tasks in public datasets, demonstrating the strong ability of autoregressive training.
In summary, autoregressive modeling techniques have significantly advanced the generation of complex and diverse 3D structures. By harnessing temporal dependencies, shape compositions, and conditional
26

---

Published in Transactions on Machine Learning Rescarch (03/2025)
representations, these methods enhance fidelity and offer refined control over generated outputs. The diverse applications across motion, point clouds, scene creation, and medical imaging underscore the versatility of autoregressive 3D generation. Future research is poised to further refine these models, pushing the boundaries of realism, scalability, and usability across various domains.
3.4 Multimodal Understanding and Generation
Previous sections discussed visual generation, which transforms language or visual input into visual output. In this section, we shift our focus to multimodal-tor cestinoda--to-mllle to-mllimmoaal tasks. This section is di- vided into two parts: the framework for multimodal understanding and the framework for unifying multimodal understanding and generation.
3.4.1 The Framework for Multimodal Understanding
The evolution of sophisticated models in multimodal understanding has significantly advanced the integration of visual and textual data. A key approach in this domain is the discrete image token masked image modeling (MI) method, which has gained traction as a powerful pretraining strategy. This method instrumental in learning robust. visual representations by predicting missing parts of images, thus allowing models to gain a deeper understanding of visual content in context.
One pivotal model built this groundwork is the BETT (Bao et al., 2021). BETT leverages MIM by treating images analogously to how BERT treats text, masking parts of the image and predicting the discrete tokens for these masked sections. This enables the model to learn high-level visual features efficiently, akin to how BERT captures linguistic nuances.
The lineage of BETT has seen multiple significant enhancements. BEIT-v2 (Peng et al., 2022) intro- ducod the concept of visual quantization knowledge distillation (VQKD), which integrates semantic information into the image tokenization process. By leveraging pretrained models like CLIP, BEiT-v2 incorporates semantic knowledge into the token representations, thereby improving the model's ability to capture both visual and semantic features more effectively.
VI--BEiT (Bao el al., 2022) represents a further advancement in the series, focusing on generative vision-language pretraining. It employs a unified transformer-based architecture to scamlessly process and integrate both image and text data. By using a generative approach for pretraining, VL-BEIT is capable of learning joint visual and textual representations, enhancing its ability to perform tasks such image captioning and visual question answering with high accuracy and coherence. BFiT-v3 (Wang et al., 2022b) continued to refine these ideas, integrating more sophisticated multimodal techniques and achieving state-of-the-arl. results in variely of vision-language benchunark.
Additionally, Flamingo (Alayrac et al., 2022) demonstrated that in-context learning can be effectively scaled to large-scale Aision-language models. Specifically, they enhanced downstream tasks like image captioning by requiring only a few examples for significant improvements.
As the field progressed, new frameworks like LLaVA (Liu et al., 2024d) emerged, representing the latest wave in multimodal understanding. LLaVA is built upon the extensive capabilities of large language models, integrating them with powerful visual representation modules, thereby setting new paradigms in the seamless fusion of language and vision tasks.
Several works like the AIM series (El-Nouby et al., 2024; Fini et al., 2024) have also explored a compelling framework for learning visual representations through generative pre-training. Tn ATM (EI- Nouby al., 2024), images are decomposed into non-overlapping patches, and causal Vision Trans- former (VTT) (Dosovitskiy, 2020) is trained autoregressively predict the next patch's representation. These predicted representations are then mapped to pixel space via a heavyweight MLP for pixel-level reconstruction. This approach enables ATM to learn robust visual representations using only gener- ative objective. AIMV2 Fini et al. (2024) further enhances this by aligning visual representations with
27

---

Published in Transactions on Machine Learning Rescarch (03/2025)
textual features, expanding its applicability to multimodal understanding tasks. Both models rely solely on generative pretraining and exhibit strong scaling properties, mirroring trends observed in LLMs Achiam et al. (2023); Dubey et al. (2024), and demonstrating that large autoregressive models can serve as a versatile paradigm for unified representation learning across modalities.
These advancements mark significant strides in bridging visual and textual data, facilitating richer, more context-aves aware Al systems that can process and interpret the world in more human- like manner.
3.4.2 The Framework for Unifying Multimodal Understanding and Generation
The unified framework extends traditional multimodal understanding (Chen et al., 2022: Li et al., 2023a; Zhang et al., 2023b; Shi et al., 2023; Xie et al., 2024b; Wan et al., 2024) by enabling both visual and textual output generation. Unlike carlier models that focused solely on interpretation, this framework leverages the integration of Large Language Models (LLMs) to enhance multimodal content generation. The overview of regressgres multimodal LLM's application is shown in Figure 10.
From pre-LLM era's separate processing of modalitics, the evolution to LLMs has allowed for seamless convergence of language and visual data. This progress supports more coherent and conlextually rich outputs, using native multimodal architectures to handle integrated inputs and outputs efficiently. As result, the Unified Framework facilitates applications that require simultaneous comprehension and generation across modalities, such as interactive storytelling and dynamic content creation.
Autoregressive Vision-Language Fusion Methods. Before the advent of LLMs, several autore- gressive models focused on integrating vision and language. Notable among these is OFA (Wang et al., 2022a), which unified tasks across modalities using shared transformer backbone, demonstrat- ing versatility in handling tasks such as image captioning and visual question answering. Similarly, CogView (Ding et al., 2021) and M6 (Lin et al., 2021), extended the capabilities to Chinese text inputs, offering robust image generation from textual descriptions. TRNIE-ViLG (Zhang al., 2021) further advanced this by employing multi-modal pretraining strategy that enhanced the understanding of image-text pairs. Unified-IO (Lu et al., 2022) explored input-output transformations across modalities, providing a unified framework for diverse tasks.
Integration with LLMs and Diffusion Models. With the rise of LLMs, autoregressive mod- els have increasingly interfaced with diffusion models to enhance multi-modal capabilities. NEXT- GPT (Wu et al., 2023c) exemplifies this trend by combining the autoregressive approach with diffusion processes to improve image synthesis from textual inputs. The SEED (Ge et al., 2024) and EMU-series (Sun et al., 2024b; 2023; Wang et al., 2024d) further explore this synergy, utilizing diffusion models to refine and stabilize the output quality of generated visual content. LaViT (Jin et al.) employs a token merging strategy, which reduces the number of visual tokens by merging tokens that are spatially semantically similar, thereby lowering computational costs while maintaining high-quality visual generation. Video-LaViT (Jin et al., 2024c), an extension of VaV,, applies a keyframe-basedd token reduction technique to video data, selecting keyframes that capture the most critical information and sparsely sampling intermediate frames, significantly reducing token count while preserving temporal coherence in video understanding and generation. Additionally, models like X-ViLA (Ye et al., 2024) (Cross-modality Vision, Language, Audio) push the boundaries of multi-modal learning by serving as a foundation model for cross- mods-toy understanding, reasoning, and generation across video, image, language, and audio domains. X- X-VILA demonstrates the potential of unified multi-modal architectures to handle diverse lasks across multiple data lypes, offering holistic approach to multi-modal content generation. This integration has proven beneficial in addressing the limitations of purely autoregressive methods, particularly in generating high-fidelity and diverse images.
Autorogressive Models for Seamless Multi-Modal Integration. More recent developments have seen the emergence of native multi-modal autoregressive models designed from the ground up to
28

---

Published in Transactions on Machine Learning Rescarch (03/2025)
VOA
Text
Text
Zecodar
Tirh
Malindi Lhal Talk Png
cuto-Regrossive Multimodal L.MSs
Image
Inegr Faoddr
Linor
Iange Deeala
kaige
(a) Diffusion Based Image Decoder
Time
(b) VQ Based Image Decoder
Embedding
Image Representation
(Discrete)
K
K
V
X
V
K
Decoder
Image Representation
(Continuous)
Figure 10: The overview of auto-regressive multi-modal LLMs' application. The diagram highlights high-level architecture combining both text and image modalities within auto-regressive LLMs. The image and text encoders process input data, which is subsequently integrated by the multimodal model to generate text and image outputs. Two image decoding strategies are shown: (a) Diffusion- based image decoder, where continuous image representations are progressively refined using transformer- based diffusion process, and (b) VQ-based image decoder, where discrete image tokens are transformed into visual outputs. The framework capable of handling a range of tasks, including pure text-based tasks (e.g., translation). vision-language tasks (e.g., VQA), text-to-image generation, image editing, and multi-modal conversational applications.
handle multiple data types. Models such Chameleon (Team, 2024) and Transfusion (Zhou et al., 2024) exemplify this new class of architectures, providing smooth integration of visual and textual data. These models are built to natively support multi-modal tasks, offering enhanced flexibilityy and scalability. Chameleon, for instance, is designed to handle multi-modal imputs with minimal architectural adjustments, making a versatile model for tasks ranging from image captioning to multi-modal reasoning.
Aghajanyan al. (2023) is an important precursor these developments, exploring how scaling laws apply to generative models that handle multiple modalities simultancously. This work provides critical insights into how model size, data quantily, and computation scale influence performance in mixed-modal settings, guiding the design of larger, more capable multi-modal models like Chameleon and Transfusion. Another notable model is RA-CM3 (Yasunaga et al., 2022), which incorporates retrieral-angmented techniques to enhance multi-modal language modeling. By integrating a retrioval mechanism, RA-CM3 improves the model's ability to access relevant external knowledge, thereby boosting performance on tasks requiring a deeper understanding of both visual and textual inputs. This retrieval-augmenin approach demonstrates how external knowledge sources can be leveraged enhance the capabilities of multi-modal models, particularly in tasks that require complex reasoning contextual understanding. SHOW-o (Xie et al., 2024b), for instance, introduces an innovative modeling scheme that aligns image and text representations, facilitating tasks such as image generation

---

Published in Transactions on Machine Learning Rescarch (03/2025)
and visual storytelling, with the use of MAGVIT-v2 (Yu et al., 2023b) tokenizer. VILA-U (Wu et al., 2024e) injects high-level visual information into VAE tokenizer (Lee et al., 2022a) from pretrained CLIP (Radford et al., 2021) to unify understanding and generation. Janus (Wu et al., 2024a) explicitly decouples the understanding and generation vision encoder, which avoids the conflict between vision understanding and generation.
To further enhance the efficacy of visual in-context learning, CoTVL (Ge et al., 2023) successfully applies visual chain-of-thought prompt tuning for vision-language modeling, and performs better in tasks that require more reasoning abilities. These models highlight the potential for autoregressive frameworks to serve as foundational architectures in multi-modal applications, promol more holistic approach to understanding and generating content across modalities.
The previously mentioned multimodal tokenization involves encoding images into tokens, allowing Large Language Models (LLMs) to process both visual and language signals in a unified space. Vision- to-Language (V2T) Tokenizer (Zhu et al., 2024a) maps image patches discrete tokens that correspond to LLM vocabularics, enabling tasks like inpainting and deblurring. Similarly, the Multimodal Cross- Quantization VAE (MXQ- VAE) (Lee et al., 2022b) encodes both image and text inputs as tokens, generating coherent multimodal outputs. These approaches improve image-text generation, though they typically predict pixels in a fixed order without considering random generation strategies.
In summary, the evolution of visual autoregressive models in the realm of multi modaltyy underscores their growing importance and versatility. From early integrations to sophisticated native architoctures, these models continue to push the boundaries of what's possible in generating and understanding multi- modal content. Future research directions may focus improving model efficiency, scalability, and the ability to handle even broader array of modalities.
4 Evaluation Metrics
This section presents commonly used quantitative and qualitative metrics to evaluate the visual au- toregressive models, summarized in Table 2.
4.1 Evaluation of Visual Tokenizer Reconstruction
The Visual Tokenizer serves a crucial role in compressing visual content into discrete token sequences and accurately reconstructing it. Evaluation metrics for these tokenizers primarily focus on reconstruc- tion fidelity, with widely adopted metrics including PSNR, SSIM (Wang et al., 2004), LPIPS (Zhang al., 2018), and rFID (Heusel et al., 2017). PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index Measure) are pixel-level metrics that quantify the degree of pixel-wise alignment be- tween the reconstructed image and its reference. While PSNR captures the average signal-to-noise ratio across image pixels, SSIM accounts for perceptual differences in structure, luminance, and contrast. LPIPS, in contrast, measures perceptual similarity by evaluating features from a pretrained VGG network (Simonyan & Zisserman, 2014). rFID (Reconstruction Fréchet Inception Distance) extends the traditional FID to the reconstruction setting, mcasuring the distributional discrepancy between original and reconstructed image sets in feature space.
4.2 Evaluation of Visual Autoregressive Generation
Beyond the evaluation of tokenizers, comprehensive assessments of visual autoregressive generation models are essential to measure their performance across multiple dimensions. Key metrics include the following five aspects:
1. Visual Quality assesses the realism of generated content relative real data. One fundamental metric is Negative Log- Licehiood (NLL), which directly quantifies the likelihood of generated data under the model's learned distribution. Additional metrics, such as Inception Score (IS) (Salimans

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Table 2: Evaluation metrics used to assess Visual autoregressive models performance. FR: Full- Reference, NR: No-Reference. indicates that the higher the metric the better the model performance and vice versa.
Metric
Reference
Source
Dependency
Reconstruction Fidelity
Peak Signal-to-Noise Ratio (PSNR) f
FR
Structural Similarity Index Measure (SSIM) f
FR
Wang al. (2004)
Learned Perceptual Image Patch Similarity (LPIPS) +
FR
Zhang et al. (2018)
Reconstruction Fréchet Inception Distance (rFID) 4
FR
Heusel al. (2017)
Visual Quality
Negative Likeliihldoood (NIL)) +
NR
Fisher (1922)
Inception Score (IS) f
NR
Salimans al. (2016)
Fréchet Inception Distance (FID) 4
NR
Heusel et al. (2017)
Kernel Inception Distance (KID) +
NR
Binkowski et al. (2018)
Fréchet Video Distance (FVD) 4
NR
Unterthiner et al. (2018)
Aesthetic Score f
NR
Schuhmann (2022)
Diversity
Precision and Recall f
Kynkäänniemi et (2019)
MODE Score f
NR
Che et al. (2022)
Semantic Consistency
CLIP Similarity (CLIPSIM) t
NR
Radford et al. (2021)
R-precision t
NR
Craswell (2009)
Temporal Coherence
Warpping Errors 1
NR
Lai et al. (2018)
CLIP Similarity Among Frames (CLIPSIM-Temp)
NR
Radford al. (2021)
Human- CemaanCentered Assessment
Human Preference Score (HPS) t Quality ELO Score 1
NR
Wu et al. (2023d)
NR
Elo (1978)
Human Study Metrics (Ratings, Preference, etc.)
NR
et al., 2016), Fréchet Inception Distance (FID) (Heusel et al., 2017), and Kernel Inception Distance (KID) (Binkowski et al., 2018), measure the distributional closeness between generated samples and real data. which evaluates different aspects of the generated content's realism and closeness to real data distributions. Aesthetic Score (Schuhmann, 2022), a CLIP-based method, also contributes to this ovaluation by mcasuring the aesthetic appeal of gencrated images through embeddings aligned with human aesthetic preferences. For video generation lasks, the Fréchet Video Distance (FVD) (Un- terthiner et al., 2018) extends FID by taking temporal modeling into consideration, evaluating the distance between distributions of generated and real vidcos.
2. Diversity evaluates the variely and richness of generated outputs within the model's distribution. Precision and Recall (Kynkäänniemi et al., 2019) metrics here assess both fidelity and diversity: Pre- cision measures the proportion of generated samples that lic within the real data distribution, while Recall evaluates the extent to which the real data distribution is represented within the generated outputs. While the MODE Score (Che al., 2022) quantifies both the quality and diversity of the generated samples.
3. Semantic Consistency examines the alignment between generated content and given textual descriptions other modalities of input. Metries such CLIP Score and R-precision (Craswell, 2009) are commonly used to assess this alignment. The CLIP Score quantifies the cosine similarity

---

Published in Transactions on Machine Learning Research (03/2025)
Type
Model
# Params
Resolution
FID
IS
Precision
BigGan (Brock, 2018)
112M
6.95
224.50
GAN
GigaGan (Kang al., 2023)
569M
3.45
225.50
StyleGan-XL (Sauer al., 2022)
166M
256
256
2,30
265.10
0.78
0.53
ADM (Dhariwal & Nichol, 2021)
551M
256 x 256
10.94
101.00
0.69
0.63
CDM (Daras et sl., 2024)
256 x 256
1.88
138.70 70
LDM-4 (Rombach et al., 2022)
400M
256
256
3.60
247.70
Diffusion
DIT-XI//2-G (Peebles & Xie, 2023)
675M
256
256
2.27
278.20
0.83
0.57
VDM++ (Kingua Gao, 2024)
2B
256
256
2.40
225.30
DIT-XL/2-G (Peebles & Xie, 2023)
675M
512
512
3.04
240.82
0.84
0.54
MaskGIT (Chang al., 2022)
227M
182.10
Mask
MaskGIT -re (Chang et al., 2022)
227M
256
2.66
4.02
335.60
VQCAN -re (Esser et al., 2021h)
1.4B
256
256
5.20
280.30
VQGAN (Esser et al., 2021b)
227M
256 x 256
18.65
80.40
0.78
0.26
VQGAN (Esser et. al,, 2021b)
1.4B
256
255
15.78
74.30
RQTran. -re (Lee al., 2022a)
1.7B
256
256
3.80
323.70
RQTran. (Lee al., 2022a)
1.7B
256
256
7.55
134.00
VIT-VQGAN -re (Yu al., 2021a)
1.7B
256
256
3.04
227.40
VIT-VQGAN (Yu
2021a)
3.8B
256
256
4.17
175.10
LlamaGeu-B (Sun al., 2021a)
111M
0.45
LlamaGen-I. (Sum et al., 2024a)
343M
3.07
256.06
0.52
AR
LlamaGen-Xi (Sun et al., 2024a)
775M
2.62
244.08
0.57
LlamaGea-XSa (Sun et al., 2024a)
1.4B
256
256
2.34
253.90
0.80
0.59
LlamaGen-3B (Sun et al., 2024a)
3B
256
256
2.18
263.33
0.81
0.58
Opea-MAGVIT2-B (Luo et sl., 2024)
313M
256
256
3.08
258. 26
0.83
0.52
Open-MAGVIT2.L (Luo et al., 2024)
804M
256
256
2.51
271.70
0.84
0.54
Opm-MAAGVTT2-XI. (Lui et al., 2024)
1.5B
256
256
2.33
271.77
0.84
0.54
VAR-d16 (Tian et al., 2024)
310M
256
256
3.30
274.40
0.84
0.51
VAR-d20 (Tian et al., 2024)
600M
256
256
2.57
302.60
0.83
0.56
VAR-d24 (Tian al., 2024)
1.0B
256
256
2.09
312.90
0.82
0.59
VAR-d30 (Tian al., 2024)
2.0B
1.92
323.10
0.82
SPAE (Yu al., 2024a)
4.41
SPAE (Yu et al. 2024s]
256
256
3.
168.50
MAR-B (Li et al., 2024c)
208M
256
236
3.
192.40
0.78
0.58
MAR-T. (IA al., 2024c)
479M
256
256
2.60
221.40
0.79
0.60
MAR-H (Li et al., 2024c)
943M
256 256
2.35
227.80
0.79
0.62
MAR-H w/ CFG (Li et al., 2024c)
943M
256
256
1.55
303.70
0.81
0.62
DART w/ CFG (Zhao et al., 2024)
812M
256
256
3.98
Table 3: Comparison of Model Parameters, Resolution, FID, IS, Precision, and Recall across various
Types of Generative Models on ImageNet dataset (Deng et al., 2009). -re the generative model with rejection sampling. w/ CFG is the model with classifier-free diffusion guidance (Ho & Salimans, 2022).
between visual outputs and textual input using a pretrained CLIP (Radford et al., 2021) model, while R-precision measures the ranking of relevant generated samples with respect to their alignment with the intended context.
4. Temporal Coherence focuses on the temporal consistency across generated frames. Warping Errors (Lai et al., 2018), based on optical flow (Teed & Deng, 2020), mcasure the consistency of motion and object placement over time. The additional commonly applied metric is CLIPSIM-Temp. which calculates the CLIP similarity between successive frame embeddings.
5. Centereed Assessment is crucial for evaluating the subjective quality of generated visual content, while quantitative metrics provide essential objective assessments. The Human Preference Score (HPS) (Wu et al., 2023d) leverages classifier trained on extensive human preference data, estimating how closely generated outputs align with human aesthetic judgments. Recently, the Quality ELO Score (Elo, 1978) has gained prominence: in this method, generative models compete an online platform (Analysis, 2023) where users express preferences between paired model outputs. These preferences are then used to compute ELO scores, effectively capturing human rankings of models.

---

Published in Transactions on Machine Learning Research (03/2025)
MJHQ-SOK FID 4
MJHQ-30K
MS-COCO
Type
Model
Para.
CLIP-Score
FID-30K
LDM (Rombach et al., 2022)
1.4B
12.64
12.61
DALL-B-2 (Ramesh al., 2022)
6.5B
Imagen (Sahazia et sl., 2022)
3B
7.27
7.27
SD2.1 (Rombach et al., 2022)
860M
26.96
25.90
Diffusion
SD3-Medium (Esser al., 2024)
2B
11.92
28.83
SDXL (Esser et al., 2024)
2.6B
8.76
28.60
PixArt-E (Chen al., 2021)
630M
6.31
27.62
PixArt-a (Chen et al., 2023)
630M
6.14
27.55
Playground
et al., 2024a)
RAPHAEL [Xue al. 2024)
3B
6.61
DALLE (Ramesh et al., 2021)
12B
27.50
Fluid (Fan al., 2024]
10.5B
6.16
6.16
Fluid (Fan et al., 2024]
3.1B
6.41
6.41
(Fan al., 2024)
Fluid (Fan al., 2024)
665M
6.84
6.84
AR (Gen)
Fluid (Fan ct al., 2024)
369M
7.23
7.23
Muse (Chang al., 2023)
3B
7.88
7.88
Parti (Yu et al., 2022)
20B
7.23
7.23
Emaü-GGe (Wang al., 2024d)
8B
LlamaGen (Sun et al., 2024a)
775M
25.59
23.03
HART (Tang al., 2024]
DART (Zhao et al., 2024)
812M
11.12
Seed-X (Ge et al., 2023)
17B
10.82
11.99
LVM (Bai al., 2024b)
17.77
12.68
AR. (Unifled)
Chameleon (Team, 2024)
34B
10.82
Show-o (Xie et al., 2024b)
1.3B
14.99
27.02
VILA-U (Wu et sl., 20216)
7B
12.81
Jamus (Wu al., 2021a)
1.3B
10.10
8.53
Transfusion (Zhon et al., 2024)
7.3B
6.61
6.61
Table Comparison of Model Parameters, FID, CL.P--core across various Types of Generative Models on MJHQQ30K (Li al., 2024a) and MS-COCO (Lin al., 2014) datasets.
Type
Model
Pars..
GenEval Countiug Color
Single Obj.
Two Obj.
Position Color Attri. Overall
LDM (Rombach et al., 2022)
1:4B
0.92
0.29
0.23
0.70
0.02
0.05
0.37
DALL-B-2 (Ramesh al., 2022)
0.77
DALL-E-3 (Betker et al., 2023)
0.96
0.87
0.47
0.83
0.43
0.45
0.67
Diffusion
SD1.5 (Rombach et al., 2022)
860M
0.97
0.38
0.35
0.76
0.04
0.06
0.43
SD2.1 (Rombach et al., 2022)
860M
0.98
0.51
0.44
0.85
0.07
0.17
0.55
SD3-Large (Esser al., 2024)
8B
0.98
0.84
0.66
0.74
0.40
0.43
0.68
SDXL (Esser al., 2024)
2.6B
0.74
0.39
0.85
0.
0.23
0.58
PixArt-a (Chen et al., 2023)
630M
0.98
0.50
0.44
0.80
0.08
0.07
0.48
Pluid (Fan et al., 2024)
0.63
0.69
Fluid (Pan et al., 2024)
3.1B
0.83
0.83
0.60
0.82
0.41
0.53
0.70
Pluid (Fan et al., 2024]
1.1B
0.96
0.77
0.61
0.78
0.34
0.53
0.67
Fluid (Fan al., 2024)
665M
0.96
0.73
0.51
0.77
0.42
0.65
AR (Gen)
Fluid (Fan et al., 2021)
369M
0.96
0.64
0.53
0.78
0.33
0.46
0.62
Emu3-Gen (Wang et al., 2024d)
0.34
LlamaGon (Sun et al., 2024a)
775M
0.71
0.34
0.21
0.58
0-07
0.04
0.32
Seed-X (Ge et al., 2024)
17B
0.97
0.58
0.26
0.80
0.19
0.14
0.49
LVM (Bai sl., 2024bb
0.93
0.46
0.79
0.48
AR (Unified)
Chameleon (Team, 2024)
34B
0.39
Show-o (Xie al., 2024b)
Jamus (Wu ot al., 2024a)
1.3B
0.97
0.68
0.30
0.84
0.46
0.42
0.61
Transfusion (Zhou al., 2024)
0.63
Table 5: Comparison of various types of generative models on GenEval bench (Ghosh et al., 2024).
Additionally, there various User Study Metries, such ratings, preferences, and other subjective measures collected from user studios. These metrics are often case-specific and provide valuable insights into evaluation.

---

Published in Transactions on Machine Learning Rescarch (03/2025)
4.3 Task-Specific Evaluation Metrics
In the previous sections, we categorized and introduced several common metrics based various evaluation dimensions. In this subsection, we reorganize these metrics according to specific tasks. The aim is to provide task-specific classifications, allowing readers to quickly identify the most relevant metrics for each task. Specifically, metrics for visual autoregressive models are categorized as follows:
Reconstruction
PSNR, SSIM (Wang et al., 2004), LPIPS (Zhang et al., 2018), rFID (Heusel et al., 2017) are pixel-level accuracy metrics for restoration tasks, which commonly adopted for evaluating the performance of tokenizers in token-based autoregressive models. Additionally, codcbook utilization is often employed to assess how effectively vectors in VQ codebooks are utilized.
Image Generation
General Metrics. FID (Heusel et al., 2017), IS (Salimans al., 2016), and KID (Binkowssi et al., 2018) are widely adopted for evaluating the quality of generated images. Precision and Recall (Kynkäänniemi et al., 2019) are typically used to assess the fidelity and diversity of generated content.
Text-to-Image Generation. CLIPSim (Radford al., 2021) and R-Precision (Craswell, 2009) are key metrics for evaluating the relevance between text and images in lext-to-image generation tasks.
Inpainting In addition to the general quality metrics, pixel-level metrics are employed to assess whether the inpainting results align with the ground truth.
Video Generation
General Metrics. FVD (Unterthiner et al., 2018) is commonly used to evaluate the video quality. CLIPSIM-Temp (Radford et al., 2021) and Wrapping Errors (Lai et al., 2018) used to assess the temporal coherence across video generation tasks.
Text-to-Vides Generation. Similar to T2I Generation, T2V generation uses semantic con- sistency metrics like CLIPSTM to evaluate alignment with given text descriptions. Benchmarks such as VBench (Huang et al., 2024) and EvalCrafter (Liu et al., 2024h) are also widely used to evaluate the overall performance of T2V generation.
Task-Specific Metrics
In addition, there are various lask-specific metrics and benchmarks tailored to particular tasks. These are designed to assess specified attributes according to task requirements. For example, T21-CompBench (Huang et al., 2023a) is used for evaluating compositional generation tasks, while CLIPSIM-Ing is applied to example-based image editing tasks.
4.4 Comparison of Existing Models
Based on the collected results presented in Table 3, Table 4, and Table 5, conduct comprehen- sive analysis of different model types and their performance characteristics. First, comparing AR and diffusson models reveals interesting patterns across metrics. While both approaches demonstrate com- parable performance in Precision and Recall metrics, as well as FID scores, AR models show a notable advantage IS metrics. For instance, Table 3, AR models consistently achieve higher IS scores, with many models exceeding 250, while diffusion models typically show lower IS values. Second, the comparison between specialized generation models (AR Gen) and unified models (AR Unified) reveals clear performance differences across both Table 4 and Table 5. In Table 4, specialized generation models achieve better FID scores compared to unified models. This pattern continues in Table 5, w where specialized generation models like Fluid demonstrate superior GenEval scores (0.70) compared
34

---

Published in Transactions on Machine Learning Rescarch (03/2025)
to unifidd models. These consistent results across different metrics and datasets suggest that current approaches to unifying understanding and generation might actually limit generation performance. While the initial hypothesis suggested that unified understanding and generation would enhance gen- eration quality, our analysis indicates otherwise. This performance gap might be attributed to the current limitations in vision tokenization methods, which have not yet achieved a seamless integra- tion of understanding and generation capabilities. The distinct requirements for visual understanding versus generation tasks may create competing objectives that current unified architectures struggle optimize simultancously. This observation suggests that further research is needed to develop more effective approaches for unifying visual underslanding and generation. The consistent performance gap indicates that fundamental advances in vision tokenization and architectural design may be required to realize the thcoretical benefits of unified models.
5 Challenges and Future Work
5.1 Technical challenges
How to design a powerful tokenizer? This paper categorizes existing autoregressive models into three primary types: next-pixel prediction, next-token prediction and next-scale prediction. Among these, next-token prodiction and next-scale prediction have gained prominence. Both approaches hinge critically on the availability of a powerful tokenizer that can effectively compress images or videos into discrete visual tokens. However, designing such tokenizer is non-trivial challenge. One of the most widely adopted methods is VQGAN (Esser et al., 2021b), which employs VQ techniques and adversarial training to develop a perceptually rich tokenizer. Vanilla VQGAN adopts relatively large vector dimensionn256)), allowing each vector to represent rich semantic information. However, as the codebook size scales, this capacity becomes a bottleneck, significantly limiting the utilization rate of larger codebooks, and presenting challenges for applying VQGAN in large-scale autoregressive models. A rising trend in codebook design (Sun et al., 2024a) involves using smaller vector dimensions with larger codebook sizes to enhance lookup efficiency and codebook utilization. Some studies (Yu et al., 2023b; Luo et al., 2024) have even pushed the boundaries by reducing the vector dimension to zero, developing a binary lookup-free quantization(LFQ) with codebook size of 218, Additionally, various advanced training strategies (Razavi et al., 2019; Yu et al., 2021a; Zhu et al., 2024b) have been explored to further improve VQGAN's codebook utilization. Another promising avenue is leveraging hierarchical multi-scale properties (Lee et al., 2022a; Tian el al., 2024) to improve the compression of visual data. In summary, designing powerful tokenizer for open-domain visual data is crucial for advancing autoregressive visual generation.
Autoregressive models have traditionally been associated with discrete representations. However, given that visual data are inherently continuous, most approaches require an additional discrotization step, which, as discussed carlier, is far from straightforwar. Recent studies (Li et al., 2024c) argue that the essence of auloregressive models lies in "next element prediction', regardless of whether the elements are discrete or continuous. This perspective revives the debate over the merits of "discrete continuous" representations in autoregrossive models. Continuous representations offer advantages, particularly in simplifyyng the training of visual data compressors. However, they also pose new challenges for designing autoregressive architectures. Firstly, the cross-entropy loss is no longer applicable. Although L2 loss is a simple alternative, it often compromises the quality and diversity of generated outputs (Li et al., 2024c). Thus, developing alternative loss functions tailored for continnous settings is still under exploration. Secondly, the challenge of multimodal adaptability arises. Large language models (LLMs) based on discrete representations have already shown remarkable capabilities, but integrating visual continuous and discrete language representations within a unified autoregressive framework remains complex task. TransFusion (Zhou et al., 2024) has made pioneering efforts in this area, yet significant work remains to build upon these initial efforts. Currently, continuous visual representations have yet to demonstrate a decisive advantage over discrete ones in autoregressive
Discrete or continuous?

---

Published in Transactions on Machine Learning Rescarch (03/2025)
modeling. The full potential of autoregressive model-based continuous representations remains to be realized and represents a promising direction for future research.
Inductive bias in autoregressive model architecture. A key consideration in visual autoregres- sive modeling is whether a vanilla autoregressive model, without any inductive bias, truly represents the most optimal approach. Recent works (Sun et al., 2024a; Team, 2024) have shown that even standard language models like Llama (TTouvron et al., 2023a;b) can generate high-quality images and achieve effective multimodal without requiring specific inductive bias. Nonetheless, there is still merit in exploring architectures that incorporate inductive biases tailored to visual signals. VAR (Tian et al,, 2024) utilizes hierarchical multi-scale tokenization to capture visual structures better, while other ap- proaches (Chang et al., 2022; Xie et al., 2024b) leverage masked image modcling strategies to enhance performance. Additionally, recent works (Zhou et al., 2024; Xie et al., 2024b) highlight the advantages of employing bidirectional attention over unidirectional ones for processing visual signals. Exploring how inductive biases can be integrated into autoregressive models, and their potential impacts on scalability and multimodal fusion, presents a challenging yet promising area for future research.
Downstrcam task. Current rescarch on downstrcam tasks for visual autoregressive models largely focuses on zero-shol capabilities in a limited range of tasks or designing models specifically tailored for individual downstream applications. In this respect, visual autoregressive models lag behind dif- fusion and language models. A critical open question remains whether visual autorogressive models can follow the trajectory of language models, which quickly adapt to new tasks via techniques such as prompt tuning and instruction tuning, or attain the versatility of diffusion models, which excel across various visual tasks including structure control, style transfer, and image editing. As research on autoregressive models advances, developing unified autoregressive model that could adapt ream tasks is primary focus in future work.
5.2 Application Roadmaps
Long Video Generation The generation of long videos remains an unsolved problem for various generative models. Due to the inherent limitations of current generation algorithms, the most advanced flow-based video generation models (OpenAI, 2024; Kuaishou, 2024) typically produce videos with durations similar to the training data, usually around 5 to 10 seconds. However, autoregressive models, particularly in the context of large language models (LLMs), have demonstrated strong extrapolation abilities, generating text sequences that far exceed the length of the sequences seen during training. This capability suggests that visual autoregressive models hold significant, potential for generat ing long video sequences. Recent work (Jin et al., 2024b) has begun to explore the use of autoregressive models for generating longer video sequences, opening up new avenues for this challenging task.
World Simulator for Embodicd AI A promising direction for the future of visual autoregressive generation is the development of world simulators that can facilitate the training of models in dynamic, interactive environments. These simulators would enable models to generate realistic visual content by interacting with simulated worlds in context-aware manner. By integrating physical dynamics, environmental factors, and agent interactions, such simulators could allow visual autoregressive models to generate long and temporally consistent video sequences, building on the model's general ability to extrapolate from sufficient training data. Existing works (Hu et al., 2023; Bruce et al., 2024; Wu et al., 2024b; Cheang et al., 2024) have already made significant progress in leveraging autoregressive models world models. The potential to scale training data and model size to enable the learning of general intelligence, allowing models to interact with the world and provide multimodal feedback, presents an exciting avenue for further exploration.
Unified Multimodal Generation. The future of multimodal autoregressive generation lies in ad- vancing models capable of seamlessly combining and generating content across different modalities,
36

---

Published in Transactions on Machine Learning Rescarch (03/2025)
such as text, images, audio, and video. Autoregressive models naturally lend themselves to cross- modal generation, as they treat different modalities equally in the token space. While existing work has explored the potential of autoregressive models for multimodal generation (Lu et al., 2022; Wang et al., 2024d) and unified models for both understanding and visual generation (Team, 2024; Zhou et al., 2024; Xie et al., 2024b; Wu et al., 2024a), these models still fall short of the state-of-the-art models in each individual modality. The key challenge likely lies in the development of large-scale multmmodl datasets and unified cross-modal representations. We hope an aha moment that, with suf- ficient training, autoregressive models will emerge with powerful cross-modal generation capabilities, where the strengl hs of one modality can boost the generation of others.
6
Conclusion
This survey provides a comprehensive overview of the autoregressive modeling landscape in computer vision, drawing parallels to its foundational success in NLP while highlighting the unique represen- tational strategies inherent to visual data. By categorizing autoregressive models into pixel-based, token-based, and scale-based approaches, we offer a structured understanding of their underlying mechanisms. Furthermore, we illuminate the diverse applications these models across domains such image and vidco generation, multi-modal tasks, and medical applications. Finally, this survey addresses current challenges in visual autoregressive models, offering valuable insights into potential avenues for advancement in this rapidly evolving field.
7
Acknowledgement
This work primarily supported by the Theee-based Research Scheme (TRS) project T45-701/22 R of the Research Grants Council (RGC), Hong Kong SAR, and partially supported by ONR, NSF, the Simons Foundation, as well as gifts and awards from Google, Amazon, and Apple.
37

---

Published in Transactions on Machine Learning Rescarch (03/2025)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. Xiw preprint arXio:2303.08771 08774, 2023.
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer Scaling laws for generative mixed- modal language models. In International Conference Machine Learning, pp. 265 279, PMLR, 2023.
Nasir Ahmed, T Natarajan, and Kamisetty R. Rao. Discrete cosine transform. IEEE transactions on Computers, 100(1):90-2)) 1974.
Kenan E Ak, Ning Xu, Zhe Lin, and Yilin Wang. Incorporating reinforced adversarial learning in autoregressive image generation. In European Conference on Computer Vision, pp. 18 38-34. Springer, 2020.
Jean- Cean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural informalion processing syslems, 35:23716-23777 2022.
Artificial Analysis. Text to image, 2023. URL http://p/arrrffcccaaaassssa///ee
Armen Avetisyan, Christopher Xie, Henry Howard-Jenkins, Tsun-Yi Yang. Samir Aroudj, Suvam Pa- tra, Fuyang Zhang, Duncan Frost, Luke Holland, Campbell Orme, et al. Scenescript: Reconstructing scenes with an antoregressive structured language model. arXiv preprint wxXiv:240..10066 2024.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. In International Conference Learning Representations, 2018.
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L. Yuille, Trevor Darrell, Jitendra Malik, and Alexci A. Efros. Sequential modeling enables scalable learning for large vision models. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 22861 22872, 2024a. URL https:://penacceess... thecff www _Modeling Enables Baaaaaaaaaaaaaeeeeeennnnnn wwwpa//fooo__aaa____eeeee Wwsion MMdell_Moeee___eeeeeeeeeeeeeeeee
http://oneee////o2/////aa/ /Baii S httpooddeesss_d____ee__e__ CVPR_ 2024 20024_ppperr._ html.
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 22861-2283 2024b.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei, Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106. 21222200600824, 2021.
Hangbo Bao, Wenhui Wang, Li Dong, and Furu Wei. VI-beit: Generative vision-language pretraining.
arXiv preprint ar arXiv:2206.01122 011277, 2022.
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 35:25005 25017, 2022.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufci Guo, et al. Improving image generation with better captions. Computer Science. https://odn. openai. com http//apers/deee dall-e--. pdf, 2(3):8, 2023.

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Xiaoyu Bie, Wen Guo, Simon Leglaive, Lauren Girin, Francesc Moreno-Noguer, and Xavier Alameda- Pineda. Hit-dvae: Human motion generation via hierarchical transformer dynamical vae. arXiv preprint 0rXiu:2204.0150 2022.
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXi::18110014 2018.
Andrew Brock. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1809 10..11000 11096, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive envi- ronments. arXiv preprint arXic:2402. 15391, 2024.
Shiyue Cao, Yueqin Yin, Lianghua Huang, Yu Liu, Xin Zhao, Deli Zhao, and Kaigi Huang. Efficient- vqgan: Towards high-resolution image generation with efficient vision transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 7368-7377, 2023.
Bryan Cardenas, Devanshu Arya, and Deepak Gupta. Generating annotated high-fidelity images containing multiple coherent objects. In 2021 IEEE International Conference on Image Processing (ICIP), pp.
IEEE, 2021.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint ar erXiv:2301.000 00704, 2023.
Tong Che, Yanran Li, Athul Jacob, Yoshua Bengio, and Wenjie Li. Mode regularized generative adversarial networks. In International Conference on Learning Representations, 2022.
Chi-Tam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Ti, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, al. Gr-2: A generative video-languuge-actiinn model with web-scale knowledge for robot, manipulation. Xiv preprinl arXie:2410.06155 061588, 2024.
Dongsheng Chen, Chaofan Tao, Lu Hou, Lifeng Shang, Xin Jiang, and Qun Liu. Litevl: Efficient video- language learning with enhanced spatial-temporal modeling. arXiv preprint arXiv:2210.11 11929, 2022.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint orXim:2310. 0.0242 2023.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.0460 2024.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pp. 1691-1703. PMLR, 2020.

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, llya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In International Conference on Learn- ing Representations, 2017.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsmail: An improved autore- gressive generative model. In International conference on machine learning, pp. 864-8 364-872. PMLR, 2018.
An-Chieh Cheng, Xueting Li, Sifei Liu, Min Sun, and Hsuan Yang. Autoregressive 3d shape gen- eration via canonical mapping. In European Conference on Computer Vision, pp. 89-104. Springer, 2022.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2021.
Abril Corona-Figueroa, Sam Bond- Taylor, Neelanjan Bhowmik, Yona Falinic A Gaus, Toby P Breckon, Hubert PH Shum, and Chris G Willcocks. Unaligned 2d to 3d translation with conditional vector- quantized code diffusson using transformers. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 14585-14594, 2023.
Nick Craswell. R-Precision, pp. 2453-2453. Springer US Boston, MA, 2009. ISBN 978-0-387-39944--9 doi: 11199988---88799944499 486. URL https://áoi
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vggan-clip: Open domain image generation and editing with natural language guidance. In European Conference on Computer Vision, pp. 88 105. Springer, 2022.
Giannis Daras, Yuval Dagan, Alex Dimakis, and Constantinos Daskalakis. Cousistent diffusson models: Mitigating sampling drift by learning be consistent. Advances in Neurol Information Processing Systems, 36, 2024.
Ingrid Daubechies. Ten lectures on wavelets. Society for industrial and applicd mathematics, 1992.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Teee, 2009.
Kangle Deng, Tianyi Fei, Xin Huang, and Yuxin Peng. Tro-gan: Introspective recurrent convolutional gan for text-to-video generation. In IJCAI, pp. 2216 2222, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidiroctional transformers for language understanding. arXiv preprint arXi::1810.02 04800,, 2018
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances neural information processing systems, 34:8780 11888008774,, 2021.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in neural information processing systems, 34:19822 19835, 2021.
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better texl-to-image generation via hierarchical transformers. Advances in Neural Information Processing Systems, 35: 16890-16902, 2022.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estima- tion. Xixio preprint or 1410:1400.855 2014.
40

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arviv preprint or Xrv::20010. 11000.11980 11929, 2020.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishck Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiu preprint ar arXi::2477.211 2000072.17877 2173, 2024.
Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua Susskind, and Armand Joulin. Scalable pre-training large autoregressive image models. Xiv preprint arXi::2401.085. 2024.
Arpad E. Elo. The Rating of Chessplayers, Past and Present. Arco Pub., 1978.
Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. Imagebart: Bidirectional con- text with multinomial diffusion for autoregressive image synthesis. Advances in newral informalion processing systems, 34:3518 3--2,532222 2021a.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 12873 12883, 2021b.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-forst International Conference on Machine Learning, 2024.
Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaim- ing He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint 20. 2024.
Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guilherme Turrisi da Costa, Louis Béthune, Zhe Gan, et al. Multimodal au- toregressive pre-training of large vision encoders. arXiv preprint arXiv:2411 14.42241.144022, 2024.
Ronald A Fisher. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical choracter, 368, 1922.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniy Taigman. Make-a- scene: Scene-based text-to-image generation with human priors. In European Conference on Com- puter Vision, pp. 89-106. Springer, 2022.
Kaifeng Gao, Jiaxin Shi, Hanwang Zhang, Chunping Wang, and Jun Xiao. Vid-gpt: Introducing gpt- style autoregressive generation in video diffusion models. arXio preprint 20. 2024.
Jiaxin Ge, Hongyin Luo, Siyuan Qian, Yulu Gan, Jie Fu, and Shanghang Zhang. Chain of thought prompt tuning in vision language models. ar preprint arXiv:2304.0700 079919 2023.
Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long vidco generation with time-agnostic vggan and timo-sensitive transformer. arXiv preprint arXiv:2204. 00638,,0338888 2022.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi- caatiigraaularity comprehension and generation. arXiu preprint arXiv:2404.14 14396, 2024.
Dhruba Ghosh, Hannanch Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024.

---

Published in Transactions on Machine Learning Rescarch (03/2025)
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in newral information processing systems, 27, 2014.
Jiatao Gu, Ying Shen, Shuanglei Zhai, Yizhe Zhang, Navdeep Jaitly, and Joshua M Susskind. Kaleido diff'usio: Improving conditional diffusion models with autoregressive latent modeling, ar preprinl or arXiv:2/55.2144. 2024.
Shuyang Gu, Dong Chen, Jiammin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/ /CF conference on computer vision and pattern recognition, pp. 10696 10706, 2022.
Ishaan Guirajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: A latent variable model for natural images. In International Con- ference on Learning Represenlations, 2017.
Baisong Guo, Xiaoyun Zhang, Haoning Wu, Yu Wang, Ya Zhang, and Yan-Feng Wang. Lar-sr: A local autoregressive model for image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1909-1918, 2022.
Jianyuan Guo, Zhiwei Hao, Chengcheng Wang, Yehui Tang, Han Wu, Han Hu, Kai Han, and Chang Xu. Data-efficient large vision models through sequential autoregression. arXiv preprint wxXiv2442.04844 2024.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2020.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In ICLR, 2021.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04 104, 2023.
Bo Han, Hao Peng, Minjing Dong, Yi Ren, Yixuan Shen, and Chang Xu. Amd: Autoregressive motion dilfusion. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 2022 2030, 2024.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In JCML, 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoen- coders are scalable vision learners. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Wanggui He, Siming Fu, Mushui Liu, Xierui Wang, Wenyi Xiao, Fangxun Shu, Yi Wang, Lei Zhang, Zhelun Yu, Haoyuan Li, et al. Mars: Mixture of auto-regressive models for fine-graince text-to-image synthesis. arXie preprint arXiv:2407.00614 2024.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. Xiviv preprint arXi::2010.147000 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint or 2207 12598. 2022.
42

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
Jordan Hoffinann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2208.1580 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for Lext-Lo-video generation via transformers. arXiv preprint arXiv:2205. 15868, 2022.
Anthony Hu, Lloyd Russell, Hudson Yoo, Zak Murez, George Fedosccv, Alex Kendall, Jamic Shotton, and Gianluca Corrado. Gaia-l: A generative world model for autonomous driving. arXiv preprinl arXin:2309.1 17080, 2023.
Yaosi Hu, Chong Luo, and Zhenzhong Chen. Make it move: controllable image-to-video generation with text descriptions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 18219 18228, 2022.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances Neural Information Processing Systems, 36:78723-78747, 2023a.
Mengqi Huang, Zhendong Mao, Zhuowei Chen, and Yongdong Zhang. Towards accurate image coding: Improved autoregressive image generation with dynamic vector quantization. In Proceedings of the IREE/CVF Conference on Computer Vision and Pattern Recognation, pp. 22696, 2023b.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
Moritz 1bing, Gregor Kobsik, and Leif Kobbelt. Octree transformer: Autoregressive 3d shape gen- cration on hicrarchically structured sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 2698 2707, 2023.
Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin Chen, Chenyi Lei, An Liu, et al. Unified language-vision pretraining in 11m with dynamic discrete visual tokeniza- tion. arxiv 2024. arXiv preprint arXiv:2309.0400
Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint XrXiv:24 10. 05954, 2024a.
Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient vidco generative modeling. Xiv preprint arXiv:24 10. 05955,,, 2024b.
Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. Video-lavit: Unified video-language pre-training with decoupled motianllotional tokenization. arXie preprint orXiv:2402. 03161, 2024c.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. In ICLR, 2020.
Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. Tn International Conference on Machine Learning, pp. 1771 1779. PMLR, 2017.
43

---

Published in Transactions on Machine Learning Research (03/2025)
Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognation, pp. 10124-10134, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXin preprint or 2001:2200..88311, 2020.
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 36, 2024.
DP Kingma. Auto-encoding variational bayes. arXie preprint
2013.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jose Lezama, Jonathan Huang, Grant Schindler, Rachel Hor- nung, Vighnesh Birodkar, Jimmy Yan, Soug-Chang Chiu, al. Videopoet: A large language model for zero-shot video generation. In Forty-first International Conference on Machine Learning, 2024.
Kuaishou. Kling ai. htt.:/iiigii.uiiioo...... kwwisshu..coo com/, 2024.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved pre- cision and recall metric for assessing generative models. Advonces in neural information processing systems, 32, 2019.
Black Forest Labs.
2024.
Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Hsuan Yang. Learning blind video temporal consistency. In Proceedings of the Europewn conference compuler vision (ECCV), pp. 170 170-185, 2018.
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 11523-11532 2022a.
Hyungyung Lee, Sungjin Park, Joonscok Lee, and Edward Choi. Unconditional image-text pair gen- eration with multimodal cross quantizer. arXiv preprint arXiv:2204.075597, 2022b.
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2.
Three insights towards enhancing aesthetic quality in text-to-image generation. ar preprint arXiv:2402. 17245, 2024a.
Haopeng Li, Jinyue Yang, Kexin Wang, Xucrui Qiu, Yuhong Chou, Xin Li, and Guoqi Li. Scalable autoregressive image generation with mamba. arXiv preprinl ar Xiv: 12245, 2024b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19744 PMLR, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-inage pre- training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023b.
Kun Li, Yanming Zhu, Jingyu Yang, and Jianmin Jiang. Video super-resolution using an adaptive superpixel-guided ruto-segressive model. Pallern Recognition, 51:59-77 71, 2016.
Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2142 2152, 2023c.

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprinl ar Xiv:24406. 11838, 2024c.
Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Zhe Lin, Rita Singh, and Bhiksha Raj. Controlvar: Exploring controllable visual autoregressive modeling. Xvv preprint orXiv:2406.. (1Xi:22406099700,, 2024d.
Zongming Li, Tianheng Cheng, Shoufa Chen, Peize Sun, Haocheng Shen, Longjin Ran, Xiaoxin Chen, Wenyu Liu, and Xinggang Wang. Controlar: Controllable image generation with autoregressive models. Xr preprint arXic:2410. 02705, 2024e.
Zongyi Li, Shujie Hu, Shujie Liu, Long Zhou, Jeongsoo Choi, Lingwei Meng, Xun Guo, Jinyu Li, Hefei Ling, and Furu Wei. Arlon: Boosting diffusion transformers with autoregressive models for long video generation. Xiv preprint ar 24.::2410000000 10.0002, 2024f.
Jian Liang, Chenfei Wu, Xiaowei Hu, Zhe Gan, Jianfeng Wang. Lijuan Wang, Zicheng Liu, Yuejian Fang, and Nan Duan. N Nuwa-infinity: Autoregressive over autoregressive generation for infinile visual synthesis. Advonces in Neural Information Processing Systems, 35:15420- -54422, 2022.
Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, et al. M6: A chinese multimodal pretrainer. arXiv preprint 2rXiv:2103.00823 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Compuler Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 19, pp. 740-755. Springer, 2014.
Bingchen Liu, Yizhe Zhu, Kumpeng Song, and Ahmed Elgammal. Towards faster and stabilized gan training for high-fidelity few-shot image synthesis. In International conference on learning representations, 2020.
Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina- mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pre- training. arXie preprint arXiv:240800060 2024a.
Hao Liu, Wilson Yan, and Pieter Abbeel. Language quantized autoencoders: Towards unsupervised text-image alignment. Advances in Neural Information Processing Systems, 36, 2024b.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. arXiv preprint 2rX2:22002 082668, 2024c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024d.
Haozhe Liu, Shikun Liu, Zijian Zhou, Mengmeng Xu, Yanping Xic, Xiao Han, Juan C. Pérez, Ding Liu, Kumara Kahalapitiya, Menglin Jia, Jui-Chieh Wu, Sen He, Tao Xiang, Jürgen Schmidhuber, and Junn-Mamuel Férez-Rúa, Mardini: Masked autoregressive diffusion for video generation at scale. arXiv preprint Xiv: 10. 202800 2024c.
Mushui Liu, Yuhang Ma, Xinfeng Zhang, Yang Zhen, Zeng Zhao, Zhipeng Hu, Bai Liu, and Changjie Fan. Llm4gen: Leveraging semantic representation of 1mms for text-to-image generation. arXiv preprint arXim:2407.00739 2024f.
Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), Computer Vision ECCV 2024, pp. 1-18. Springer Nature Switzerland, 2025. ISBN 978-3-031-72998-0 doi: 10. 10.0007997--3-311--299--- 1.

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Wenze Liu, Le Zhuo, Yi Xin, Sheng Xia, Peng Gao, and Xiangyu Yue. Customize your visual autore- gressive recipe with set autoregressive modeling, arXiv preprint or 4rX1:24400110111 2024g.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. Xiv preprint or (Xw::2209003000 2022.
Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference. on Computer Vision and Paltern Recognation, pp. 22139-22149 2024h.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 10012-10022, 2021.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified- io: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations 2022.
Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing rego-seeressvve visual generation. arXiv preprint arXiv:2409.04477 2024.
Hanjia Lyu, Jinfa Huang, Daoan Zhang, Yongsheng Yu, Xinyi Mou, Jinsheng Pan, Zhengyuan Yang, Zhongyu Wei, and Jiebo Luo. Gpt-4v (ision) as a social media analysis engine. arXiv preprint arXiv:2311.07547, 2023.
Xiaoxiao Ma, Mohan Zhou, Tao Liang, Yalong Bai, Tiejun Zhao, Huaian Chen, and Yi Jin. Star: Scale- wise Lex--lo-image generation via auto-regressive representations. arXiv preprinl or 2rxiv:2406.1070 2024.
Wael Mattar, Idan Levy, Nir Sharon, and Shai Dekel. Wavelets are all you need for autoregressive image generation. arXiv preprint Xix::24006. 100000..9900 19997, 2024.
Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. In RSS, 2023.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint 230:: 15505, 2023.
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models. In JCLR, 2023.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling
language models. In A. Oh, T. Naumann, A. Globerson, K. Sacnko, M. Hardt, and S. Levine (eds.), Advances in Newral Information Processing Systems, volume 36, pp. 00358-50376. Curran Associates, Inc., 2023. URL http://rooeednn....rrrrrssss neurips. httpp/scccpppeeeeeeeeee httpa///pppppp///////////////66. 228ca pdf.
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse reprosentations. arXiu preprint arXiv:2103.033 03,444 2021.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In NewrIPS, 2015.
OpenAI. Video generation models as world simulators. modeel--as-wordd-ssmmllle wowld- sim.ddsisssssorss 2024.
models-as-
46

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, pp. 4055 4064. PMLR, 2018
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEER/CVF International Conference on Compuler Vision, pp. 4195-4205, 2023.
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arxiv preprint or arXi:2208.06330 2022.
Yu Qi, Fan Yang, Yousong Zhu, Yufei Liu, Liwei Wu, Rui Zhao, and Wei Li. Exploring stochastic autoregressive image modeling for visual representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 2074-2081, 2023.
Xuelin Qian, Yu Wang, Simian Luo, Yinda Zhang, Ying Tai, Zhenyu Zhang, Chengjie Wang, Xi- angyang Xue, Bo Zhao, Tiejun Huang, et al. Pushing rege-segressive models for 3d shape generation at capacity and scalability. arXiv preprint arXix:2402. 1eee..2222 12225, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Πya Sutskover, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763 PMLR, 2021.
Tanzila Rahman, Hsin- Ying Lee, Jian Ren, Sergey Tulyakov, Shweta Mahajan, and Leonid Sigal. Make- a-story: Visual memory conditioned consistent story generation. In Proceedings of the IEEE/CW /VFF Conference on Compuler Vision and Pallern Recognilion, pp. 2493 2502, 2023.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. arXiv preprint arXiv:2006. 10704, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sulskever. Zero-shot lext-lo-image generation. In International conference on machine learning, pp. 1921-831. Pmir, 2021.
Adilya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXim:2244.00 06125, (2):3, 2022.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Scott Reed, Aäron van den Oord, Nal Kalchbrenner, Victor Bapst, Matt Botvinick, and Nando De Fre- itas. Generating interpretable images with controllable structure. In International Conference Learning Representations, Workshop track, 2016.
Scott Reed, Aäron Oord, Nal Kalehbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando Freitas. Parallel multiscale autoregressive density estimation. In International conference on machine leorning, pp. 2912-29211 PMLR, 2017.
Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J Guibas. Humor: 3d human motion model for robust pose estimation. In Proceediings of the IEEE/CV international conference on computer vision, pp. 11488 11499, 2021.
Sucheng Ren, Xiaoke Huang, Xianhang Li, Junfei Xiao, Jieru Mei, Zeyu Wang, Alan Yuille, and Yuyin Zhou. Medical vision generalist: Unifying medical imaging tasks in context. arXiv preprint 0rxi::2406.0550 05565, 2024a.
47

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Sucheng Ren, Xianhang Li, Haoqin Tu, Feng Wang, Fangxun Shu, Lei Zhang, Jieru Mei, Linjie Yang, Peng Wang, Heng Wang, et al. Autoregressive pretraining with mamba in vision. ar preprint arXiv:2406.00 07537, 2024b.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pp. 1530-1538. PML.R, 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-3644 36494, 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Im- proved techniques for training gans. Advances in newral information processing systems, 29, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint (rXiv:17010051/ 2017.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1 1-10, 2022.
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastoring atari, go, chess and shogi by planning with a learned model. Nature, 588(7839))604 (00-)996664400000 609, 2020.
Christoph Schuhmann. Clip+mlpp aesthetic score predictor. tttps://gitte htt//gtttuubcccom wtwaapporeeddseeeet--eeeeeeee
httpps:/ggihabb..cmcrartttttttt haaiubb.commccrsssppphchhhh 2022. Accessed: 2022-04-08.
Rico Sennrich. Neural machine translation of rare words with subword units. arXiv preprint
orXiv:1508.0740 07909, 2015.
Younggyo Seo, Kimin Lee, Fangchen Liu, Stephen James, and Pieter Abbeel. Harp: Autoregressive latent vidco prediction with high-fidelity image generator. In 2022 IEEE International Conference on Image Processing (ICIP), pp. 3943 3947. IEEE, 2022.
Ying Shen, Yizhe Zhang, Shuangfei Zhai, Lifu Huang, Joshua M Susskind, and Jiatao Gu. Many- to-many image generation with regressieee diffusion models. arXiv preprint 4rXiv:2404.03109 2024.
Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Upop: Unified and progressive pruning for compressing vision-language transformers. In International Conference on Machine Learning, pp. 113111 PMLR, 2023.
Xingjian Shi, Zhourong Chen, Hao Wang, Din Young, Wii-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Advances in newral information processing systems, 28, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rxiv:1409. 1556, 2014.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputinng 568: 127063, 2024.

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. ar preprint arXiv:2406.0663 2024a.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality, In The Twelfth International Conference on Learning Representations, 2023.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14398
14409, 2024b.
Haotian Tang, Yecheng Wu, Shang Yang, Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, and Song Han. Hart: Efficient visual generation with hybrid autoregressive transformer. arXiv preprint arXiv:2410.11 10.111110, 2024.
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. arXiv preprint arXiv:2407.1962) 2024.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXie preprint
arXiv:2405.090 09818, 2024.
Zachary Teed and Jia Deng. Raf:: Recurrent all-pairs field transforms for optical flow. In Compuler Eision-ECCY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 402-419. Springer, 2020.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prodiction. arXiv preprint 2rXi::244.00295, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anrr Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient. foundation language models. arXiv preprint or Xiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xr preprinl arXin:2307.0922
2023b.
Potru-Danicl Tudosiu, Walter Hugo Lopez Pinaya, Mark S Graham, Pedro Borges, Virginia Fernandez, Dai Yang, Jeremy Appleyard, Guido Novati, Disha Mehra, Mike Vella, al. Morphology-preservvie autoregrossive 3d generative modelling of the brain. In International Workshop Simolation and Synthesis in Medical Imaging, pp. 66 78. Springer, 2022.
Petru-Daniel Tudosiu, Walter HL Pinaya, Pedro Ferreira Da Costa, Jessica Dafflon, Ashay Patel, Pedro Borges, Virginia Fernandez, Mark S Graham, Robert J Gray, Parashkev Nachev, el al. Realistic seaphology-preervvine generative modelling of the brain. Nature Machine Intelligence, pp. 1 1-9, 2024.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for vidco generation. In Proceedings of the IEEE conference on computer vision and pattern recognilion, pp. 1526-1535, 2018.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXin:1812.01111 2018.

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukeuoglu. Pixel recurrent neural networks. In International conference machine learning, pp. 1747-1756. PMLR, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
A Vaswani. Attention is all you need. Advances in Newral Information Processing Systems, 2017.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mo- hammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2022.
Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with vqvae. arXiv preprint NNi::2103.01955 2021.
Gregory K Wallace. The jpeg still picture compression standard. Communications of the ACM, 34(4): 30 44, 1991.
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, et al. Efficient large language models: A survey. ar preprint 2xXi:23122038860 2023.
Zhongwei Wan, Che Liu, Xin Wang, Chaofan Tao, Hui Shen, Zhenwu Peng, Jie Fu, Rossella Arcucci, Huaxiu Yao, and Mi Zhang. Meit: Multi-modal electrocardiogram instruction tuning on large language models for report generation. arXiv preprint arXiv:2403.044 04945, 2024.
Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. Larp: Tokenizing videos with a learned autoregressive generative prior. arXio preprint arXiv:2410.2126, 2024a.
Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image-video tokenizer for visual generation. arXiv preprint SSSi:2446.099900 2024b.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple cewtuenc--t-sequeeee learning framework. In International conference on machine learning, pp. 23318 23340. PMLR, 2022a.
Siwen Wang, Churan Wang, Fei Gao, Lixian Su, Fandong Zhang, Yizhou Wang, and Yizhou Yu. Autoregressive sequence modeling for 3d medical image representation. arXiv preprint or (09.022409.08611, 2024c.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arxiv:2208. 10442, 2022b.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprind arXiv:2409.18869 2024d.
Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip S Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal Istms. Advances in neural information processing systems, 30, 2017.
50

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Yunbo Wang, Lu Jiang, Ming- Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3d Istm: A model for video prediction and beyond. In International conference learning representations, 2019.
Yuqing Wang, Tianwei Xiong, Daquan Zhou, Zhijie Lin, Yang Zhao, Bingyi Kang, Jiashi Feng, and Xihui Liu. Loong: Generating minute-level long videos with autoregressive language models. arXiv preprint ar 0rXi:2410.02755 2024e.
Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkorcit. Scaling autorogressive video models. arXiv preprint or Xiv: 10049.00.02344 2019.
Wenming Weng, Ruoyu Feng, Yanhui Wang, Qi Dai, Chunyu Wang, Dacheng Yin, Zhiyuan Zhao, Kai Qiu, Jianmin Bao, Yuhui Yuan, et al. Art-v: reto-regressive text-to-video generation with diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 7395 7405, 2024.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. ar preprint orXiv:2104.14806, 2021.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang. Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In Europeon conference on computer vision, pp. 720 736. Springer, 2022.
Chengyue WU, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unifidd multi- modal understanding and generation, 2024a. URL https: //arxiv.///xii.///// org/abs httg/abb/2410.. 2410. 13848.
Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot ma- nipulation. In International Conference on Learning Representations, 2024b.
Jialong WU, Haoyu Ma, Chaoyi Deng, and Mingsheng Long. Pre-training contextualized world models with in-the-wild videos for reinforcement learning. In NewrIPS, 2023a.
Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpl: Interactive videogpts are scalable world models. arXiv preprint arXiv:2405. 15223, 2024c.
Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models. arXiv preprint arxiv:2405. 15223, 2024d.
Ronghuan Wu, Wanchao Su, Kede Ma, and Jing Liao. Teonshop: Text-guided vector icon synthesis with autoregressive transformers. ACM Transactions on Graphics (TOG), 42(6):1 14, 2023b.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 11m. arXiv preprint arXiv:2309.055 05519,, 2023c.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning texl-to-image models with human preference. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 2096-2105, 2105, 2023d.
Yecheng WI, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual under- standing and generation. ar Xixio preprint Xrxi 04429, 2024e.

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Hsugn Yang. Gan inversion: A survey, IEEE transactions on pallern analysis and machine intelligence, 45(3):3121 3138, 2022.
Jiannan Xiang, Guangyi Liu, Yi Gu, Qiyue Gao, Yuting Ning, Yuheng Zha, Zeyu Feng, Tianhua Tao, Shibo Hao, Yemin Shi, et al. Pandora: Towards general world model with natural language actions and video stales. Xite preprint urXiv:2406. 09455, 2024.
Desai Xie, Zhan Xu, Yicong Hong, Hao Tan, Difan Liu, Feng Liu, Arie Kaufman, and Yang Zhou. Progressive autoregressive video diffusion models. arXiv preprint arXtv:2410.0815. 2024a.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unily multimodal understanding and generation. arXie preprinl or arv:22408. 12528, 2024b.
Yilun Xu, Gabriele Corso, Tommi Jaakkola, Arash Vahdat, and Karsten Kreis. Disco-diff: Enhancing contimuous diffusion models with discrete latents. arXiv preprint #xXic:2407033300 2024.
Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. Advances in Neural Information Processing Systems, 36, 2024.
Wilson Yan, Yunzhi Zhang, Picter Abbecl, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint orXiv:2104.10157, 2021.
Shuai Yang, Yuying Ge, Yang Li, Yukang Chen, Yixiao Ge, Ying Shan, and Yingcong Chen. Seed-story: Multimodal long story generation with large language model. ar preprint orXic:2407.08683 2024.
Kai Yao, Penglei Gao, Xi Yang, Jie Sun, Rui Zhang, and Kaizhu Huang. Outpainting by queries. In European Conference on Computer Vision, pp. 153-169. Springer, 2022.
Ziyu Yao, Jialin Li, Yifeng Zhou, Yong Liu, Xi Jiang, Chengjie Wang, Feng Zheng, Yuexian Zou, and Lei Li. Car: Controllable autoregressive modeling for visual generation. arXiv preprint arXiv:2410.0460 2024.
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovee, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. netrieval-augmennt multimodal language modeling. XrXiu preprint ar arxiv:2211.1 11.122211.221,,, 2022.
Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, et al. X-vila: Cross-modality aligument for large language model. arXiv preprint. or xrxv:22405. 19335, 2024.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:2110.0462 2021a.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint (tt::2206.107800 2(3):5, 2022.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Haupt- mann, Ming- Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video trans- former. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 10459 10469, 2023a.

---

Published in Transactions on Machine Learning Rescarch (03/2025)
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauplmann, et al. Language model beats diffusson tokenizer is key to visual generation. arXiv preprint arXiu:2310.05737 2023b.
Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolfgang Macherey, Yanping Huang, David Ross, Trfan Essa, Yonatan Bisk, Miua-Hsuaa Yang, et al. Spae: Semantic pyramid autoencoder for multimodal generation with frozen Ilnss. Advances in Neural Information Processing Systems, 36, 2024a.
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupla, Xiuye Gu, Alexander G. Hauplmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation. In The Twelfth International Conference on Learning Representations, 2024b. URL attps://openrewee
net cost/forum?idee A2g22
Lili Yu, Bowen Shi, Ramakanth Pasumuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXic:2309.0055 2(3), 2023c.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chieh Chen. An image worth 32 tokens for reconstruction and generation. arXiv preprint arXiv:2406. 00..5500000500 2024c.
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Vidco probabilistic diffusion models in pro- jected latent space. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 18456- 18456-18466, 2023d. URL https: //tp:///ppppaa oppnacccssccssscceee stpeenccesssthhecvf thecvf. com/ con2ent/CVPR2222 /html/Yu, Videe_erooaseeeeeeeeeeeeeeeaeeee Probabilistic Diffiiisitiiiiffioooooooooooooit paper. html.
Diffusion Soooolssoooooooooooooaeeeeeellee in Projected Sttpaaaattttttt_t___eeee_ Sypace_CVPR_2222
Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jianxiong Pan, Kaiwen Cui, Shijian Lu, Feiying Ma, Xuansong Xie, and Chunyan Miao. Diverse image inpainting with bidirectional and autoregressive transformers. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 69-78, 2021b.
Shenghai Yuan, Jinfa Huang, Yujun Shi, Yongqi Xu, Ruijic Zhu, Bin Lin, Xinhua Cheng, Li Yuan, and Jiebo Luo. Magictime: Tape-lapse video generation models as metamorphic simulators. arXiv preprint S55::24440001.. 2024a.
Shenghai Yuan, Jinfa Huang, Yonggi Xu, Yaoyang Liu, Shaofeng Zhang, Yujun Shi, Ruijie Zhu, Xinhua Cheng, Jiebo Luo, and Li Yuan. Chronomagic-bemn bench: A benchmark for metamorphie evaluation of text-to-time-lapse vidco generation. arXiv preprint arXiv:2406. 1 18522, 2024b.
Han Zhang, Weichong Yin, Yewei Fang, Lanxin Li, Boqiang Duan, Zhihua Wu, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-vilg: Unified generative pre-training for bidirectional vision-language generation. arXiv preprint ar 1xXiv:2112.1220 2021.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 14730- 14740, 2023a.
Qian Zhang, Xiangzi Dai, Ninghua Yang, Xiang An, Ziyong Feng, and Xingyu Ren. Var-clip: Text- to-image generator with visual ruto-regressive modeling. arXiv preprint arXiv:2408.. 01181, 2024a.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586 595, 2018.
53

---

Published in Transactions on Machine Learning Research (03/2025)
Shaofeng Zhang, Jinfa Huang, Qiang Zhou, Zhibin Wang, Fan Wang, Jiebo Luo, and Junchi Yan. Continuous-multiple image outpainting in one-step via positional query and a diffusion-based ap- proach. arXiv preprint orXiv:2401.15652 2024b.
Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xi- angyu Yue. Meta-transformerr A unified framework for multimodal learning. orXiv preprint orXiv:2307110002 2023b.
Kaifeng Zhao, Gen Li, and Siyu Tang. Dart: A diffusion-based autoregressive motion model for real-time text-driven motion control. ar preprint orXiv:24 0.005000 2024.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovac: Balancing learning and inference in variational autoencoders. In Proceedings of the aaci conference on artificial intelligence, volume 33, pp. 5885 5885-5892, 2019.
Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems, 35: 23412-23425, 2022.
Chunting Zhou, Lili Yu, Arum Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuue images with one multi-modal model. Xi preprint arXiv:2408. 1 11039, 2024.
Hongjian Zhou, Fenglin Liu, Boyang Gu, Xinyu Zou, Jinfa Huang, Jinge Wu, Yiru Li, Sam Chen, Peilin Zhou, Junling Liu, et al. A survey of large language models in medicine: Progress, application, and challenge. arXiv preprint (xXi::811105111 2023a.
Meng Zhou and Farzad Khalvati. Conditional generation of 3d brain tumor regions via vegan and temporal-agnostic masked transformer. In Medical Imaging with Deep Learning, 2024.
Meng Zhou, Matthias W Wagnor, Uri Tabori, Cynthia Hawkins, Birgit B Ertl- Wagner, and Farzad Khalvati. Generating 3d brain tumor regions in mri using vector-quantization generative adversarial networks. arXiv preprint ar XrXim:231000 0122511, 2023b.
Lei Zhu, Fangyun Wei, and Yanye Lu. Beyond text: Frozen large language models in visual signal comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27047 27037, 2024a.
Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vggan to 100,000 with a utilization rate of 99%. arXiv preprint arxiv:2406. 11837, 2024b.


Reviews:
[{"review_id": "C6Af3PVXKs", "reviewer": "Reviewer_N3GT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Clear organizational structure that helps readers understand the progression of autoregressive models in vision\n- Strong technical depth with detailed mathematical formulations and architectural explanations\n- Comprehensive coverage of recent developments up to 2024\n- Quantitative comparisons across different approaches through benchmark results\n- Thorough analysis of relationships with other generative approaches (GANs, Diffusion, VAEs)\n\nWeaknesses:\n- Limited analysis of computational complexity trade-offs between different approaches\n- Evaluation metrics focus mainly on standard benchmarks without sufficient discussion of task-specific metrics\n- Discussion of future challenges and research directions could be more concrete", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey provides a comprehensive review of autoregressive models in computer vision, with three major contributions:\n1. A systematic categorization of visual autoregressive models into pixel-based, token-based, and scale-based approaches, with detailed technical analysis of each framework\n2. A thorough examination of applications across multiple domains including image generation, video synthesis, 3D generation, and multimodal tasks, supported by ~250 references\n3. A comparative analysis of existing models through standardized benchmarks and evaluation metrics", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical changes:\n1. Add quantitative analysis of computational costs and efficiency trade-offs across different frameworks\n2. Expand evaluation metrics discussion beyond standard benchmarks\n3. Include more concrete future research directions with specific technical challenges\n\nBeneficial changes:\n1. Expand discussion on scaling behavior and resource requirements\n2. Add comparisons of training stability across different approaches", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kZbLRG4FbS", "reviewer": "Reviewer_TBJj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The proposed survey is comprehensive with many relevant methods cited and presented. The methods are well organized into modalities, then tasks, then finally types of auto-regressive approaches. Overall, the survey is of good quality and will be useful for the community working on developing and scaling auto-regressive models.\n\n* The survey is well illustrated and the presentation is clear. In particular, the tree diagrams are very informative and present a clear taxonomy of the methods. Additional figures explain well the different auto-regressive approaches.\n\n\nWeaknesses:\n\n* There is a “Video Tokenizer” paragraph in Section 3.1: Image Generation. Why not keep this for Section 3.2: Video Generation ? In result, MAGVIT and MAGVIT-v2 are discussed twiced in both 3.1 and 3.2, which makes the organization confusing. Also, both MAGVIT papers appear under “Unconditional Image Generation” in Figure 2, which is confusing as these are primarily designed to generate video.\n\n* Make-A-Scene is discussed both in “3.1.2.a Token-wise Generation” and in “3.3.3 Scene Generation”, which similarly to MAGVIT makes the structure of the paper confusing.\n\n* The survey is very much focused on generative models and misses on some interesting papers that are more focused on learning representations and world models rather than generating pixels:\n\n         * GAIA-1: A Generative World Model for Autonomous Driving, https://arxiv.org/abs/2309.17080. This paper presents in my opinion one of the most interesting auto-regressive world models. The model is trained on unlabelled driving data and can predict many time-consistent frames into the future starting from a context.\n\n         * The AIM serie: Scalable Pre-training of Large Autoregressive Image Models, https://arxiv.org/abs/2401.08541,  Multimodal Autoregressive Pre-training of Large Vision Encoders, https://arxiv.org/abs/2411.14402. These papers demonstrate the power of self-supervised learning and auto-regressive models to learn strong visual representations.\n\n\n\n* Three auto-regressive categories are presented, pixel-based, token-based and scale-based. What about auto-regressive approaches in the embedding space, without discretization, are there such methods ? Also, is it worth having a “scale-based” category for a category for so few methods ? And in Figure 2, why only the “Unconditional Image Generation” and “Text-to-Image Synthesis” tasks have the categorization pixel/token/scale ?\n\n* I have spotted several typos:\nSec 2.3 first line of the text -> “Variational Autoencoders a(VAEs)”\nPage 11, paragraph 2, weird formulation -> “Zhu et al. (2024b) presents VQGAN-LC.”\nBottom of page 13 -> “RQ-VAE (?).”", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is a survey on auto-regressive models in computer vision, mostly focused on generative models. A very wide variety of methods is presented, divided into modalities: image, video, 3D, ect, but also into types of auto-regressive approaches: pixel-based, token-based and scale-based. Many illustrations help understand and categorize the different approaches. Finally, the paper proposes a discussion on challenges and future work for auto-regressive models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested changes:\n- Fix the typos.\n- Improve the paper structure to eliminates redundant presentation of the same method.\n- Incorporate the suggested papers.\n\nQuestions and remarks:\n\n- Are all token based approaches based on what is described in 2.2.2 ?\n\n- “Pixel-based models (Sec. 2.2.1) generate images pixel by pixel, capturing the most fine-grained spatial details.” This is not necessarily always the case, self-supervised models such as DINO / DINOv2 are known to be extremely good for fine-grained tasks (segmentation for example), without reconstructing pixels.\n\n- Regarding the length of the paper, I believe it is appropriate for a survey, which are generally much longer than regular papers.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9n8cfaLCm8", "reviewer": "Reviewer_Fc6G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* As noted in the contribution, it is a comprehensive survey on autoregressive models, covering diverse topics in the area. Grouping by methodologies and tasks sounds reasonable to me with much details with regards to the task specific models in Section 3.\n\nWeaknesses\n\n* Section 3 is lengthy in that most techniques are similar and potentially shared even in different tasks. I think this section could be shorten further by introducing the key techniques relevant to particular task settings.\n\n* The pros and cons of autoregressive models when compared with other non-autoregressive models are not clear. Similarly, the challenges in Section 5 sound not specific to autoregressive models, but shared with other modeling approaches. \n\n* Similarly, evaluation methods detailed in Section 4 do not sound quite specific to the autoregressive models, and it is better to differentiate what are unique to the autoregressive models or not.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The major contribution of this work is the comprehensive survey on autoregressive models for vision tasks. This survey covers methods, tasks and evaluation in vision tasks with a particular focus on autoregressive models. The methodologically, the models are categorized as three, pixel-based, token-based and scale-based. The tasks are grouped by image generation, video generation, 3D generation and multimodal generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Shorten section 3, since many are duplicated across several tasks.\n\n* Clearly discuss the pros and cons of autoregressive models and its challenges when compared with non-autoregressive models.\n\n* Similarly, differentiate evaluation methodologies which are specific to autoregressive modeling approaches.", "broader_impact_concerns": "None primarily because it is a survey paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the survey is a valuable contribution to the community. It is timely, well-written, and well-illustrated.
Decision: Accept (poster)