NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale
Authors: Chunrui Han
Guopeng Li
Jingwei Wu
Quan Sun
Yan Cai
Yuang Peng
Zheng Ge
Deyu Zhou
Haomiao Tang
Hongyu Zhou
Kenkun Liu
Shu-Tao Xia
Binxing Jiao
Daxin Jiang
Xiangyu Zhang
Yibo Zhu
Venue: ICLR 2026 Oral (2026)
Keywords: Generative Models
Autoregressive Models
Diffusion Models
Text-to-image
Primary area: foundation or frontier models, including LLMs

Abstract: Prevailing autoregressive (AR) models for text-to-image generation either rely on heavy, computationally-intensive diffusion models to process continuous image tokens, or employ vector quantization (VQ) to obtain discrete tokens with quantization loss. In this paper, we push the autoregressive paradigm forward with NextStep-1, a 14B autoregressive model paired with a 157M flow matching head, trained on discrete text tokens and continuous image tokens with next-token prediction objectives. NextStep-1 achieves state-of-the-art performance for autoregressive models in text-to-image generation tasks, exhibiting strong capabilities in high-fidelity image synthesis. Furthermore, our method shows strong performance in image editing, highlighting the power and versatility of our unified approach. To facilitate open research, we have released our code and models to the community at https://github.com/stepfun-ai/NextStep-1.

Published as a conference paper at ICLR 2026
NEXTSTEP-1: TOWARD AUTOREGRESSIVE IMAGE GENERATION WITH CONTINUOUS TOKENS AT SCALE
Chunrui Han
Guopeng Li
Jingwei Wu!
Yan Cai 33.1*
Yuang Peng .2.1
Deyu Zhou! 1*
Haomiao Tang 1,2
Hongyu Zhou'
Kenkun Liu -
Binxing Jiao
Daxin Jiang'
Xiangyu Zhang'
Yibo Zhu'
Zheng Ge
Shu-Tao Xia
Quan Sun 1.9
SStepFun
TTsinghua University
SPeking University
GitHub
HuggingFace
NextStep-1
JF NextStep-1.1
ABSTRACT
Prevailing autoregressive (AR) models for text-to-image generation either rely on heavy, computationally-inennii
diffusion models to process continuous image tokens, or employ vector quantization (VQ) to obtain discrete tokens with quan- tization loss. In this paper, we push the autoregressive paradigm forward with NextStep-1, 14B autoregressive model paired with a 157M flow matching head, trained on discrete text tokens and continuous image tokens with next-token pre- diction objectives. NextStep-1 achieves state-of-the-arr performance for autore- gressive models in rext-to-image generation tasks, exhibiting strong capabilities in high-fidelity image synthesis. Furthermore, our method shows strong perfor- mance in image editing, highlighting the power and versatility of our unified ap- proach. To facilitate open research, we have released our code and models to the community at htttps: /tttt::://ggthbb.... wwwpsawb..utttttt.uuot........ae wtwpsun-ai//eeeta.t..
INTRODUCTION
Diffusion models have firmly established themselves as the dominant paradigm for high-fidelity image synthesis (Podell et al., 2024 Esser al. 2024 Labs, 2024, Wu et al. (025a). Despite this success, recent state-of-the-wee art architectures typically operate in decoupled manner, relying on separate, pre-trained text encoders (e.g., T5 (Raffel et al. 2020) or CLIP (Radford et al. 2021) to feed semantic features into a Multimodal Diffusion Transformer (MMDiT) (Esser et al.) 2024) backbone via cross-attention. This non-end-to-end design imposes rigid constraints: it fixes the input context window, restricts deep multimodal fusion, and hinders the model's ability to handle arbitrary interleaved image-text sequences naturally.
Inspired by the scalability of Large Language Models (LLMs)) 2025a), the "next-token prediction" paradigm offers a compelling alternative for unified multimodal generation. Existing approaches generally fall into two categories, each with distinct limitations. The first category, rep- resented by discrete autoregressive models(Sun et al. | 2024a Chen et al. [2025b] Wang et al. (2024)) relies on Vector Quantization (VQ (Zhen al. [202 Eslami al.) 2021) discretize images into visual tokens. This process introduces an information bottleneck manifesting as reconstruction ar- dffccts-add suffers from exposure bias(Han et al. (025).. Furthermore, mitigate information loss, models like Emu3 (Wang et al. 2024b) must employ low compression rates, resulting in excessively long token sequences that inflate computational costs and training difficulty. The second category involves hybrid architectures. such as Transfusion (Zhou et al. (2025) and Bagel (Deng et al. 2002), which attempt to integrate noisy inputs and diffusion losses directly into the LLM. While promising. these methods often compromise training efficiency. Unlike pure next-token prediction, they typi- cally require processing both noisy latents and clean conditioning signals (or ground truth images) simultancously within bidirectional attention blocks. This data duplication significantly increases the sequence length and attention overhead, undermining the efficiency benefits traditionally associ- ated with sparse autoregressive modeling. Besides, while recent explorations into continuous latent
Equual contribution (alphahetical by first name). 'Itermship at StepFun. CCorresponding authors.

---

Published as a conference paper at ICLR 2026
Mean Square
Next Token Prediction
1.M Head
Flow Matching Head
Velocity Prediction
Target Patch
Flow Matching
Target Flow
Causal Transformer
1-t
Hidken State
Nošse Sumple
Text
Image Tokenizer
Matching
Figure I: Overview of NextStep-1 Framework. NextStep-1 employs causal transformer to process tokenized text and image tokens. During training, Flow Matching Head predicts the continuous flow from a noise sample the target image patch, conditioned the oulput hidden state. At inference, this allows for generating images by iteratively guiding noise to create the next patch.
autoregression (Li et al. 2024c Fan al. 2024) attempt to bypass quantization, a substantial per- formance gap persists. To date, no continuous autoregressive model has matched the visual quality and consistency of state-of-the -te--rr diffusion models while retaining the simplicity and scalability of standard LLM.
In this work, we demonstrate that the key closing the performance gap lies in the image repre- sentation. We introduce NextStep-1, an autoregressive model built around novel image tokenizer that is specifically designed to create a well-dispersed and normalized latent space, thereby enabling stable training with high- sigh-dimensional continuous latents. By addressing the challenges in image tokenization, we demonstrate that a simple causal transformer with a lightweight flow matching head can achieve state-of-the-an results that rival top diffusion models in both quality and prompt adherence. We showcase the generation quality of our model in Section A Our contributions are highlighted follows:
1. We present NextStep-1, an autoregressive text-to-image model that achieves state-of-the-art performance with minimalist architecture, outperforming previous autoregressive methods as well as many strong diffusion- based methods.
2. We reveal key design principles for training robust image tokenizer that enables stable autoregressive generation with dineensinnii continuous latents.
3. We demonstrate the state-of-the-ar performance and versatility of NextStep-1 through comprehensive evaluations benchmarks for both text-to-image generation (e.g.. WISE, GenAl-Benchh and instruction based image editing (e.g., GEdit-Bench), highlighting its superior capabilities in compositional reasoning and prompt fidelity.
METHOD
2.1 UNIFIED MULTI-MODEL GENERATION WITH CONTINUOUS VISUAL TOKENS
NextStep-1 extends the well-established autoregressive language modeling paradigm to image gen- eration through simple and intuitive architecture, illustrated in Figure I To unify multi- modal inputs into a single sequence, the images will be tokenized to continuous image tokens by the image tokenizer and combined with discrete text tokens. For a multimodal token sequence = {zo, Ii, .... }. where I; is either a discrete text token or a continuous visual token, the autoregressive objective under the unified sequence formalized as:
p II p(z, x<)).
(I)
The unified multi-modas generation task proceeds by sampling the next token .r from the conditional distribution p(z; xz4i) modeled by a network. Discrete text tokens are sampled via a language modeling head, while continuous image tokens sampled by now-matching head.
2

---

Published as a conference paper at ICLR 2026
Our training objective consists of two distinct losses: standard cross-entropy loss for discrete text tokens, and a flow matching loss (ipman et al., 2023b) for continuous image tokens. Specifically, the flow matching loss is the mean squared enror between the predicted and target velocity vec- tors that map a noised patch to its corresponding clean patch. The model is trained end-to-end by optimizing weighted sum of these two losses:
(2) where Liext and Lvisual denote the loss for text and image tokens respectively, which are balanced by the hyperparameters Aea and Avisual.
Luaal = Auex + Cvisuall
2.2 MODEL ARCHITECTURE
Image Tokenizer. Our image tokenizer is fine-tuned from flux VAE (Lass] 2024) with only construction and perceptual losses. The tokenizer first encodes an image into 16-channel latents 2, applying an 8x spatial downsampling factor. To stabilize and normalize the latent space, apply token-wise normalization (detailed in Appendix [9, standardizing each channel to zero mean and unit variance. Furthermore, to enhance the robustness of the image tokenizer and encourage more uniform latent distribution, we introduce stochastic perturbation to the normalized latents. This technique is adapted from a-VAE (Sun et al. 2024c)) where it was employed to prevent variance collapse.
3 == Nooomaaizatio(()) a. where ~ 2/[0, > and (0, 1)
(3)
where & is standard Gaussian noise, and its magnitude is scaled by random factor a sampled uniformly from [0, 71. The y is hyperparameter controlling the maximum noise intensity.
The latents from the image tokenizer are pixel-shuffled into more compact sequence. This is achieved by applying space-to-deptl transformation with 2x2 kernel, which flattens 2x2 spatial latents into the channel dimension. For example, this converts the latents of a x56x256 image into 16x 16 grid of 64-channel tokens. This grid is then flattened into 1D sequence of 256 tokens to serve as input for the following Causal Transformer.
Causal Transformer. We initialize our model from the decoder-only Qwen2.5-14B (Yang et al.) 2024), leveraging its strong language understanding and reasoning capabilities for text-to-image generation. We organize the multimodal input sequence in the following format:
[text) <image area>>h* <boi>>(imgee <eoi>...
where [text} denotes discrete text tokens, and {image} represents continuous image tokens. <boi>> and <eoi> are special tokens marking the beginning-of-image and end-of-image. <im- age_area>>h#w represents the metadata about the spatial dimensions of the 2D image tokens.
Then the output hidden states from LLM passed to lightweight heads for modality-speciffi loss:
Language Modeling Head. We compute Cross-Entropy loss for hidden states of texts.
Patch-wise Flow Matching Head. Following (Li et al. 2024c), we use each patch-wise image hidden states as condition, denoise target patch at timesteps t, and compute the patch- wise flow-matoer matching loss (Lipman et al.] 2023a) with a 157M, 12-layer, and 1536 hidden- dimensions MI.P
For positional information, we use the standard ID RoPE (Su 2024
Despite the availability
of more complex 2D or multimodal RoPE alternatives (Bai et al., found that the simple ID formulation remains highly effective for mixed text-image sequences, and thus retain it for simplicity and efficiency.
2025 Wang et al.] (2224)), we
3 TRAINING RECIPES
TRAINING IMAGE TOKENIZER
Our image tokenizer is initialized from the Flux.1-dev VAE (Labs 2024). selected for its strong reconstruction performance. We fine-tune this model on the image-text dataset detailed in Sec- tion B.2 to adapt it to specific data distribution. For optimization, employ the AdamW
3

---

Published as a conference paper at ICLR 2026
Table 1: Training recipe of NextStep-1.
Pre- Pre-niainin
Post- Post-Training
Stagel
Stuge2
Annealing
Hyperparameters
Leaming
10
10 Constaat 0.1 (0.01
10"*)
x 10"*) Cosine
-9
LR Scheduler
Constant
Cosine
Constant 0.1
Weight Decay
0.1
0.1 (0.01 1) 20K
0,1 (0:01 1) 10K
Weight (CE: MSE)
(8.01
Training Steps
200K
LO0K
300
Warm-up Steps
200
Sequence Length per Rank
16K
16K
SK
Image Area (Min, Max)
256 256
(256 256, 512x
(256 x 256, 512x
(256 x 256, 512x 512)
(256 x 256, 512)
Image (Min, Max)
(256, 1024)
(256, 1024)
(256, 1024)
(256,
Training Tokens
1,23T
0.61T
408
SB
Data Ratio
Text-only Corpus
Image-Texl Text Pair Data
0.5
0.6
0.6
0.9
Image-to-Image
Interleaved Data
0.2
0.2
optimizer (Loshchilov and Hutter 2019) with (31 0.9, 0 gence stability. The model is trained for 50K steps with a total batch size 512, using a constant learning rate of 10-5 preceded by linear warm-up of 1000 steps.
10=*) for its conver-
3.2 PRE-TRAINING
The specific hyperparameters and data ratios for our pre-training are detailed in Table [ Specifi- cally, the pre-training follows a three-stage curriculum designed to progressively refine the model's s capabilities. Throughout these stages, all model parameters are trained end-to-end except for the pre-trained image tokenizer.
Stagel. In this initial stage, the model learns a foundational understanding of image structure and composition. For computational efficiency, images are resized and randomly cropped fixed 256x resolution. The training curriculum is composed of diverse data mixture: 20% text-only corpora, 60% image-text pairs, and 20% interleaved data. This stage consumed approximately 1.23T tokens.
Stage2. We implement a dynamic resolution strategy to train the model on range of higher res- olutions, targeting 256 and 512x512 base areas. This strategy utilizes different aspect ratio buckets for computational efficiency. In this stage, enrich the data mixture with more text-rich and video-interleaved data, leveraging the model's 's enhanced capacity to process fine details at these resolutions.
Annealing. In the final stage of pre-training, we perform annealing phase sharpen the model's capabilities on a highly curated dataset. This is achieved by training the model for one epoch high-quality subset of 20M samples, which were selected from Section (B.2 applying stricter filtering thresholds for aesthetic score, image clarity, semantic similarity, watermark, and so on. This annealing step significantly improves the model's final output, enhancing overall image structure, composition, texture, and aesthetic appeal.
3.3 POST-TRAINING
Following pre-training broad corpus establish generalist model, post-training serves to align the model's 's output with human preferences and downstream tasks. We achieve this alignment via two-stage process: Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO) (Rafailov et al. 2023). The hyperparameters for each stage are Table
Supervised Fine-Tunie Fine-Tuning (SFT). The SFT stage enhances the model's 's instruction-followe followwng capa- bilities and aligns its outputs with human preferences. The SFT dataset, comprising total of 5M
4

---

Published as a conference paper at ICLR 2026
samples, is organized into three components: 1) corpus of human-selected image-text pairs with high semantic consistency and visual appeal, augmented by images from other generative models to improve the model's handling of complex and imaginative prompts through distillation; 2) Chain- of-Thoughh (CoT) data (Wei et al.) 2022: Deng et al. 2025), improving text-to-image generation by incorporating language-based reasoning step before the final image is created; 3) high-quality instruction-guided image-to-image data from Section B.3 to strengthen the model's image editing capabilities.
Direct Preference Optimization (DPO). To align our model with human preferences, we employ Direct Preference Optimization (DPO) (Rafailov et al. [2024) method inspired by Diffusion-DPO (Wallace et al. 2024). To this end, we construct two distinct types of preference datasets from curated set of approximately 20,000 diverse prompts.
1. Standard DPO Dataset: For each prompt c. we directly use the SFT model to generate 16 candidate images. These images is then scored by ImageReward (Xu al. (023) to form preference pair (y"", y²), where the winning image y" is randomly sampled from the top 4 candidates, while the losing image y' is randomly sampled from the remaining 12.
2. Self-CoT DPO Dataset: To enhance the model's reasoning capabilities, we introduce an explicit reasoning step. For each prompt c, we first prompt our model generate de- tailed textual CoT, which is then extended to the original prompt. Using this CoT-enhanced prompt, we follow the identical pipeline as above form preference pair .
3.4 DATA
We construct a diverse training corpus designed to foster both robust text-to-image generation and versatile image editing capabilities. Detailed pipelines are described in Section
4
MODEL PERFORMANCE
4.1 PERFORMANCE OF TEXT-TO-IMAGE GENERATION
We comprehensively evaluate the text-to-image (T2I) generation performance of NextStep-1 on several representative benchmarks, each targeting different aspects of image generation, including visual-textual aligmment and world knowledge (full results in section A). As shown in Table 2 we assess NextStep-1 I's prompt-following ability across three key benchmarks, GenEval (Ghosh et al.) 2023) and GenAl-Bench (Li et al. (024a),, and OnelG-Bench (Chang et al.] (025). Results demon- strate that NextStep-1 is competitive with leading diffusion models. Furthermore, its outstanding performance long-context and multi-object scene, confirms its reliable compositional fidelity under complex prompts.
4.2 PERFORMANCE OF IMAGE EDITING
Quantitative Results on Editing Benchmarks. We developed fextStep-1-Edit by finetuning N on IM high-quality edit-only data Section demonstrates competitive performance against advanced efffusion-basele models. As shown in Table NextStep-1 - achieves scores of
6.58 on GEdit- Beatit-BBnch-BBe EN (Liu et al. (225c) and 3.71 on BrgEdit-Bende (Ye et al. (202).. indicating its strong practical editing capabilities.
DISCUSSIONS
5.1 WTIAT GOVERNS IMAGE GENERATION: THE AR TRANSFORMER OR THE FM HEAD?
A key architectural distinction of our framework is its direct, autoregressive modeling of continuous image tokens using flow matching objective. Prevailing autoregressive models for image gener- ation (Sun et al. 2023) 2024b Dong et al., 2024, Zhou et al. 2025 Chen et al. 2025a) typically rely heavy, diffusion models for entire image: autoregressive model first produces semantic
5

---

Published as a conference paper at ICLR 2026
Table 2: Comparison image-tex. alignment on GenEval Ghosh et al. 2023). GenAl-Benc (Lin al. 2024
and DPG-Bench (Hu et al. 2024). result with rewriting. i result is with Self-CoT.
GenAI-Benchh
Method
GenEvall
DPG-Bencch
Basic
Advanced
Proprictary
[Betker
2023
0.70
Seedream 30 Cao 2025
0.84
88.27
GPT4o (OpenAT 30256
Diffinsion
Stable Dilfusion 1.5 Rombuch et al. 2022
0.43
Stablc Diffusion
0.55
0.83
0.63
74.65
Stable Dilfusion Mctium
Esser cl 2024
0.74
0.88
0.65
84.08
Diffusion Lurge
0.71
PixArt-Alhha Chen et al.
2024
0.48
71.11
dev
Transfusion Zhoni ml
2025
0.63
Cog
2025
0.73
85.13
Lumina- Imane 2.0 QTh et
2025
0.73
87.20
HiDieam-II-PP Car 0
0.83
0.66
85.89
Mogao Liso et 2025
0.89
0.68
84.33
BAGEL Deng al. 2022
0.82/0.88
0.89/0.86
0.69/0.75
85.07
Show-o2
0.76
OmniGen2
Wi ar 21256
0.80/0.86*
83.57
Qwen-Image
AuloRegressive
SEED-X
GG ot al.
2024
0.49
0.86
0.70
Show-o
VILA-U T/wwa ai 2029
0.76
0.64
Want
0.54/0.65
Fluid
( Jin al. 2024
0.69
Infinity
al 2025
0.79
Janus-
Pro-71
C Nen d
2025h
0.80
0.86
0.66
84.19
Token-Shuffle
20230
NextStep-1
0.63/0.73
0.88/0.90'
0.67/0.74
85.28
Table 3: Comparison of image editing performance GEdit-Bench (Full Set) (Liu et al. 2025c and ImgEdit- Bench (ee al. 2025 G.SC, G_PQ. and G.O refer the metries evaluated by GPT-4.1 (OpenAl 2025a
Performance is evaluated based on the NextStep- Edii with 1:1 aspect ratio.
Model
GEdit-Bench- (Full Set)
GEdit-Bence CedittBench-oo CN (Full Seti
ImggEditBenchh
G.SC
G_PQ
G.O
G.SC
G.PQ
G.O
Proprietary
Gemini 2.0 Gemini2 2025
6.87
7.44
6.51
5.26
7.60
5.14
Doubae Shi ct J0A
7.22
7.89
6.98
7.17
7.79
6.84
OpenAl
7.74
8.13
7.49
7,30
4.20
Flux.
Kot.l----occ--- 0B% 2025
7.02
7.60
6.56
1.11
7.36
1.23
Open-source
mnstruct-ix22P4 Brooks et al. 2023
3.30
6.19
3.22
1.88
(Zang
1.83
AnyEdic Yu et 21144
3.05
3,88
2.85
2.45
OmniGen
20124
5.88
5.87
5.01
2.96
OmniGenn Wa S
201150
7.16
6.77
6.41
3.44
SteplX-Edit VT.0 Liu
2025c
7.13
7.00
6.44
7.30
7.14
6.66
3.06
StepIX-Edit
v1.1 Lib
A12200
7,66
7.35
6.97
7.40
698
BAGEL [Deng
2175
7.36
6.83
6.52
7.34
6,85
6.50
3.42
Hlux.1-kontcii-ee
2025
GPT-Imige-Fdi
Wing of M0233
7.24
3,80
NextStep-1
7.15
7.01
6.58
7.02
6.40
3.71
embedding, which is then used to condition diffusion model that generates entire image in sin- gle denoising process. contrast, our model autoregressively generates the image patch-by-pater modeling the distribution of each patch with significantly more lightweight flow matching model. We argue that this establishes our framework under the pure autoregressive paradigm with next-token prediction (NTP) modeling, rather than diffusion model merely orchestrated by Transformer.

---

Published as a conference paper at ICLR 2026
Large
Table 4: Configurations for different flow-matching heads.
Layers
Hidden Size
Parameters
FM Head Small
1024
40M
FM Base
157M
Head Large
2048
528M
Table 5: Quantitative results for different flow-macching head configurations. All variants are finetuned from the base- line with newly initialized head.
GenEval
GenAT-Bench
DPG-Bench
Baseline
0.59
0.77
85.15
Figure 2: Images generated under dif- ferent fow-matching heads.
w/ FM Head Small
0.55
0.76
83.46
FM Head Base
0.55
0.75
84.68
w/ FM Head Large
0.56
0.77
85.50
Mesa
Variance Values
OP-
2.-
2.0
1.8
20
100
250
350
200
251
CFG-10
Summpingg Steps
Sumppiing Stepe
channcl-wise isput Intent normalieation clanncl-wise impai normalization
Mesa Values Sampling Stops
Variance Values Sumpling Stups
1A02
1.01
CFG:59
Vaniance
6:99
CFc
Sempling
Figure 3: Evolution per-token mean variance over sampling steps under two CFG settings. CFG 1.5. the mean and variance stay close and 1, respectively, indicating stability. At CFG 3.0, they drift significantly, causing image quality degradation. With normalization, the distributions of output latents remain stable across all CFG settings.
A key finding from our experiments is the model's surprising insensitivity to the size of its flow- matching head. We ablated this on three heads with different sizes (small, base, and large). For each experiment, we re-initialized and trained only the head for 10k steps. Despite the significant variation in model size, all three heads produced remarkably similar results (Table 5 Figure 2 This insensitivity to the head's size strongly suggests that the transformer backbone performs the core generative modeling the conditional distribution p(z; F<s). The flow-matching head, akin to the LM head in language models, primarily acts lightweight sampler that translates the transformer's s contextual prediction into a continuous token. Consequently, the essential generative logic resides within the transformer's autoregressive NTP process.

---

Published as a conference paper at ICLR 2026
rELD va Nolan Slandard Uiviation
PANR (4B) va Nolae Standard Deviation
XSIM sx Notie Standard Deviation
Koles Sundard Dendetie
Sete Standard Detate
New Stedert Dedatie
Original
Diffesium
Diffisium Medium
Flax.1-ayy
NeaSSap
Figure 4: Impact of Noise Perturbation on Image Tokenizer Performance. The top panel displays quantitative metries (rFIDJ, PSNR1, and SSIM)) versus noise intensity. The hottom panel presents qualitative reconstruc- tion examples noise standard deviations of 0.2 and 0.5.
5.2 TOKENIZER THE KEY TO IMAGE GENERATION
Mitigating Instability under Strong Classifier-Fres Guidance. A known failure mode in VAE- based autoregressive models is the emergence of visual artifacts, such as gray patches, particularly under strong faasifier-ffree guidance scales (Fan et al. 2024). While prior work hypothesized this instability stemmed from discontinuities in ID positional embeddings, our analysis reveals that the root cause lies in the amplification of token-level distributional shifts under high guidance scales.
At inference time, CFG is introduced to enhance conditional fidelity. The guided prediction is computed via interpolation:
(<(yy) = (1 (1-w) w) (v(()))) + w vo(2ly)
(4)
where uy (2(2) and vo(xly) are the unconditional and conditional predictions, and is guidance scale. In diffusion models, inference with high guidance scale is stable because latent variables typically normalized, ensuring that conditional and unconditional predictions maintain sistent scale. However, in token-level autoregressive models, global normalization the entire latent tensor does enforce per-token statistical consistency. Consequently, small discrepancies between conditional and unconditional predictions are magnified by a large guidance scale, leading significant drift in the statisties of generated tokens over the sequence.
We empirically demonstrate this phenomenon in Figure 3 At moderate guidance scale of 1.5, the per-token mean and variance remain stable throughout the generation process. In contrast, a high guidance scale of 3.0, both statistics diverge significantly for later tokens, distributional shift that corresponds directly to the appearance of visual artifacts. Our tokenizer design, which incorporates token-wise normalization (see Equation (3), directly addresses this issue by enforcing per-token statistical stability. This simple but critical design choice mitigates the instability, enabling the use of strong guidance without degrading image quality.
Regularized Latent Space is Critical for Generation A key finding of our work a counter- intuitive inverse correlation between the generation loss and the final synthesis quality of the autore- gressive model. Specifically, applying higher noise intensity (y in Equation (3) during tokenizer training increases generation loss but paradoxically improves the quality of the generated images.
8

---

Published as a conference paper at ICLR 2026
Table 6: Comparison reconstruction performance on ImageNet- IK 236x256
(Deng al.
2009
Tokenizer
Latent Shape
PSNR
SSIM 1
Discrete Tokenizer
BBER-MoVQGAN (270M). Zheng
2022
32x32
27.04
0.74
LlamaGen (Son et 2024g
32x32
24.44
0.77
VAR (Tam
TTTok's Yu
2024b
128
17.52
0.44
Sellook (Wang el
1024
26.30
0.81
Continuous Tokemizer
Stable Diflusion 1.5
(Rombach et
2022
32x32x4
25.18
0.73
Stable Diffusion XL Poúell MTA4
32x32x4
26.22
0.77
Stable Diffasion Madium
2024
32x32<16
30.00
0.88
Flux 1-dcv (Labs 2024
32x32x16
31.64
0.91
NextStep-1
For instance, NextStep-1 uses tokenizer trained at 0.5, which incurred the highest generation loss yet produced the highest-fidelity images. Conversely, tokenizers trained for low generation loss caused the autoregressive model to yield outputs resembling pure noise.
We attribute this phenomenon noise regularization cultivating well-conditioned latent space. This process enhances two key properties: the tokenizer decoder's robustness to latent perturbations (Figure 4 and more dispersed latent distribution (Figure A3) property prior work has also found beneficial for generation (Yang et al., 2025 Yao et al. 2025 Sun et al., 2024c). While it remains unclear whether robustness or dispersion plays critical role, these results underscore the practical benefits of noise-based regularization and highlight promising directions for future analysis.
Reconstruction Quality is the Upper Bound of Generation Quality. The reconstruction fidelity of the image tokenizer fundamentally determines the upper bound for the quality of the final gener- ated image, particularly for fine details and textures. This principle has been validated numerous recent studies (Esser et al. 2024 Labs 2024 Dai et al. 2023), leading to a trend in the diffusion paradigm of building generative models on top of VAEs with exceptional reconstruction perfor- mance (e.g.. PSNR >30). In contrast, VQ-based autoregressive models have historically struggled to surpass this threshold, as shown Table While trade-off between reconstruction and genera- tion quality is often debated (Yao et al.) 2025). ), our work successfully applies autoregressive models to high-fidelity continuous VAEs, bridging this gap.
RELATED WORK
Diffusion and Flow Matching Models. Diffusion models have established themselves the dom- inant paradigm for high-fidelity image synthesis. While early foundations relied on U-Net (Ron-) neberger al., (015) architectures, recent advancements have shifted towards Diffusion Transform- ers (DiTs) (Peebles and Xiel (023) to leverage the scalability of attention mechanisms. More re- cently, Flow Matching (Lipman et al. 2202a Liu et al., (002)) has been adopted to rectify generation trajectories, with models like Flux (Labs) 2024) demonstrating SOTA visual quality. However, these systems typically rely on separate, pre-trained text encoders (e.g., T5-XXL(Rafie) CadPoRd et al. (02T)) to process prompts. This decoupled design imposes fixed context window, limiting deep multimodal fusion and the model's ability to handle interleaved inputs or long-context reasoning naturally. Furthermore, the generation process in these models-ofee in- volving bidirectional attention over the entire image latent space-is computationally intensive and distinct from the token-by- beknnebyttoken reasoning inherent to Large Language Models (LLMs).
al., 2020).
Discrete Autoregressive Models. Another line of research treats image generation as a sequence modeling problem, aiming to unify vision and language under single transformer backbone. Pio- neering works such as LlamaGen (Sun et al. 2024a)). Janus-Pro (Chen et al. 2025b), Emu3 (Wang et al. 2024b), and InfinityStar (Liu et al. 2025a) have successfully adapted the "next-token predic- tion" paradigm to vision. However, these methods predominantly rely on Vector Quantization (VQ) (Zheng et al. 2022) discretize continuous images into finite codebook. This introduces fun-

---

Published as a conference paper at ICLR 2026
damental bottleneck: the discretization process incurs information loss, limiting the reconstruction upper bound.
Unified and Hybrid Architectures. To bridge the gap between the generative quality of diffusson and the scalability of LLMs, recent studies have explored hybrid architectures, Models like Dream- LLM (Dong et al. 2024), Emu2 (Sun et al. 2024b), and Qwen-Imaged Wu al.) 2025a) achieve SOTA generation by appending a heavy diffusion head (often a MMDiT (Esser et al. 2024)) to an LLM. While effective, relying on the separate text encoder and diffusion model limits the potential for unified multi-modality fusion and making performance sensitive to the interleaved generation. Another direction, represented by Transfusion (Zhou al.) 2025) and Bagel (Deng et al. 2025), integrates continuous noisy inputs and diffusion loss directly into the LLM training. However, these methods often compromise the "pure" autoregressive nature of I.LMs for image data. They typi- cally employ bidirectional attention and require feeding both the noisy latent and the clean image (or condition) into the model simultaneously. These special designs increase the attention cost and sequence length during training, thereby hindering the efficient application of scaling laws compared to pure next-token prediction.
In contrast to the aforementioned approaches, NextStep-1 adheres to strict causal autoregressive objective on continuous image tokens (Li et al., 2024c Fan et al. 2025). NextStep-1 demonstrates that a standard LLM backbone, equipped with lightweight flowmmatching head, can serve as the primary generative engine. This approach avoids the quantization artifacts of discrete AR mod- els while maintaining the causal reasoning benefits and infrastructure compatibility of LLMs. By eliminating the need for bidirectional attention or noisy input duplication, NextStep-1 achieves high- fidelity generation with a unified, scalable, and simplified architecture.
CONCLUSION
We propose NextStep-1, a fully autoregressive model with versatile image generation and editing capabilities that sequentially predicts the next continuous image patch token via lightweight flow matching head. To address the stability and quality issues associated with digeenimensional con- tinuous image tokens, we develop robust autoencoder using noise perturbation and token-wise input latent normalization. We show that this design crucial for the performance of autoregres- sive model, and demonstrate its competitive results across a wide variety of image generation and editing tasks, not only achieving state-of-the-ee results among existing autoregressive image gener- ation models, but also showing competitive performance when compared to leading diffusion-based methods. We plan to release the model and code to inspire further research, foster collaboration, and accelerate progress in this exciting frontier.
ACKNOWLEDGMENTS
We thank Ailin Huang, Bin Wang. Changxin Miao, Deshan Sun, En Yu, Fukun Yin, Gang Yu, Hao Nie, Haoran Lv, Hanpeng Hu, Jia Wang, Jian Zhou, Jianjian Sun, Kaijun Tan, Kang An, Kangheng Lin, Liang Zhao, Mei Chen, Peng Xing, Rui Wang, Shiyu Liu, Shutao Xia, Tianhao You, Wei Ji, Xianfang Zeng, Xin Han, Xuelin Zhang, Yana Wei, Yanming Xu, Yimin Jiang, Yingming Wang, Yu Zhou, Yucheng Han and Ziyang Meng from StepFun for their strong support.
We would also like to sincerely thank Tianhong Li and Yonglong Tian for their insightful discus- sions.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang. J. Tang. H. Zhong, Y. Zhu, M. Yang, Z. Li, Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, Ye, X. Zhang. T. Xie,
Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
J. Baldridge, J. Bauer, M. Bhutani, N. Brichtova, A. Bunner, L. Castrejon, K. Chan, Y. Chen,
S. Dieleman, Du, et al. Imagen 3. arXiv preprint arXiv:2408.07009 2024.
J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo, et al. Improving image generation with better captions. OpenAl blog, 2023.
T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instruc- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Q. Cai, J. Chen, Y. Chen, Y. Li, F. Long, Y. Pan, Z. Qiu, Y. Zhang, F. Gao, P. Xu, et al. Hidream- il: A high-efficient image generative foundation model with sparse diffusion transformer. arXiv preprint arXiv:2505.22708 2025.
Chang, Y. Fang. P. Xing. S. Wu, W. Cheng, R. Wang, X. Zeng, G. Yu, and H.-B. Chen. Oneig- Oneig-bench: Omni-dimensionall nuanced evaluation for image generation. arXiv preprint arXiv:2506.0797 2025.
J. Chen, C. Ge, E. Xie. Y. Wu, L. Yao, X. Ren, Z. Wang, P. Luo, H. Lu, and Z. Li. Pixart-sigma Weak-to-strong training of diffusion transformer for 4k text-to-image generation. European Conference on Computer Vision (ECCV), 2024.
Chen, Z. Xu, X. Pan, Hu, C. Qin, T. Goldstein, L. Huang, Zhou, S. Xie, S. Savarese, et al. Blip3-o: A family of fully open unified multimodal arXiv preprint arXiv:2505.09: 0.9568, 2025a
training and dataset.
X. Chen, C. Wu, Z. Wu, Y. Ma. X. Liu, Z. Pan, W. Liu, Z. Xic, X. Yu, C. Ruan, and P. Luo. Janus-pro: Unified multimodal understanding and generation with data and model scaling, arXiv preprint arXiv:2501.17811 2025b.
X. Dai, J. Hou, Ca, S. Tsai, J. Wang, R. Wang, P. Zhang, S. Vandenhende, X. Wang, A. Dubey, et al. Emu: Enhancing image generation models using photogenic needles in haystack. arXiv preprint arXiv:2309.15807 2023.
G. deepmind Imagen4 team. Imagen4, 2025, URL wwwph:tpps.cc http////tt.////ttrge...gooeaaaaa deepmind a/Mode Caaell-aarrd//a Imag/mmaagennn 4 -oodel-Card.pda
M. Deitke, R. Liu, M. Wallingford, H. Ngo, O. Michel, A. Kusupati, A. Fan, C. Laforte, V. Voleti,
S. Y. Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems (NeurIPS), 2023.
C. Deng, D. Zhu, K. Li, C. Gou, F. Li, Z. Wang, S. Zhong, W. Yu, X. Nie, Z. Song, et al. Emerging properties unified multimodal pretraining. arXiv preprint arXiv:2505.1463: 2025.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
R. Dong, C. Han, Y. Peng, Z. Qi. Z. Ge, J. Yang. Zhao, J. Sun, H. Zhou, H. Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. In International Conference on Learning Representations (ICLR), 2024.
M. A. Eslami, S. Liu, A. d. Oord, O. Vinyals, M. J. Wainwright, and I. Sutskever. Taming trans- formers for high-resolution image synthesis. In International Conference on Machine Learning (ICML), 2021.
11

---

Published as a conference paper at ICLR 2026
P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning (ICML). 2024.
L. Fan, T. Li, S. Qin, Y. Li, C. Sun, M. Rubinstein, D. Sun, K. He, and Y. Tian. Fluid: Scal- ing autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410.1386. 2024.
L. Fan, L. Tang, S. Qin, T. Li, X. Yang, S. Qiao, A. Steiner, C. Sun, Y. Li, T. Zhu, et al. Uni- fied autoregressive visual generation and understanding with continuous tokens. arXiv preprint arXiv:2503.13336, 2025.
Y. Gao, L. Gong, Q. Guo, X. Hou, Z. Lai, F. Li, L. Li, X. Lian, C. Liao, L.. Liu, al. Seedream 3.0 technical report. arXiv preprint arXiv:2504. 1346, 2025.
Y. Ge, S. Zhao, J. Zhu, Y. Ge, K. Yi, L. Song, C. Li, X. Ding, and Y. Shan. Seed-x: Mul- timodal models with unified multi-granularity comprehension and generation. arXiv preprint arxiv:2404.14300 14396, 2024.
G. Gemini2.
Experiment
with gemini 2.0 flash native image
genera-
tion, 2025.
URL https:://dvee wwweeeeewopers..oooo googleeboooglblogg.. com/en/ exper ment with gemi mimiin-i 20- wwa.lahh-aaae mat
ve flt image generation
D. Ghosh, H. Hajishirzi, and L. Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. In Advances in neural information processing systems (NeurIPS), 2023.
E. Gloeckle, B. Y. Idrissi, B. Rozière, D) Lopez-Paz, and G. Synnaeve. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404. 19737, 2024.
J. Han, J. Liu, Y. Jiang. B. Yan, Y. Zhang, Z. Yuan, B. Peng, and X. Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), 2025.
X. Han, Y. Wu, L. Shi, H. Liu, H. Liao, L. Qiu, W. Yuan, X. Gu, Z. Dong, and S. Cui. Mvimgnet2. 0: A larger-scale dataset of multi-view images. arXiv preprint ar20v:2412.01430, 2024.
X. Hu, R. Wang, Y. Fang, B. Fu, P. Cheng, and G. Yu. Ella: Equip diffusion models with Ilm for enhanced semantic alignment. arXiv preprint arXiv:2403.05133. 2024.
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y Lo, P. Dollár, and R. B. Girshick. Segment Anything. In IEEE International Conference on Computer Vision (ICCV), 2023.
B. F. Labs. Flux, 2024. https:::: http://hhh..oooooloooofffeeeaaa/
B. F. Labs.
Flux. f-ill-dee 2025. flack-forest- labs/ FFUUX. 1-Fill-- 1evdev
URL https: Sttpss//huggingffcc.. co/
B. F. Labs, S. Batifol, A. Blattmann, F. Boesel, S. Consul, C. Diagne, T. Dockhorn. J. English,
Z. English, P. Esser, et al. Flux. kontext: Flow matching for in-context image generation and editing in latent space. arXiv preprint arXiv:2506.155 15742, 2025.
Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286 PMLR, 2023.
B. Li, Z. Lin, D. Pathak, Li. Fei, K. Wu, X. Xia, P. Zhang, G. Neubig, and D. Ramanan. Eval- uating and improving compositional text-to-visse voiuul generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024a.
D. Li, A. Kamko, Akhgari, A. Sabet, L. Xu, and S. Doshi. Playground v2. 5: Three insights to- wards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.1724. 2024b.
Li, Y. Tian, H. Li, M. Deng, and K. He. Autoregressive image generation without vector quanti- zation. In Advances in neural information processing systems (NeurIPS), 2024c.
12

---

Published as a conference paper at ICLR 2026
C. Liao, L. Liu, X. Wang. Z. Luo, X. Zhang, W. Zhao, J. Wu, L. Li, Z. Tian, and W. Huang. Mogao: An omni foundation model for interleaved multi-modal generation. arxiv preprint arXiv:2505.05472, 2025.
Z. Lin, D. Pathak, B. Li, Li, Xia, G. Neubig, P. Zhang, and D. Ramanan. Evaluating text-to- visual generation with image-to- te-text generation. arXiv preprint arXiv:2404.01291, 2024.
Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative model- ing. In International Conference on Machine Learning (ICLR), 2023a.
Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2023b.
J. Liu, Han, B. Yan, H. Wu, F. Zhu, X. Wang. Y. Jiang, B. Peng, and Z. Yuan. Infinitystar: Uni- fied spacetime autoregressive modeling for visual generation. arXiv preprint atXiv:2511.04777 2025a.
J. Liu, G. Liu, J. Liang, Z. Yuan, X. Liu, M. Zheng, X. Wu, Q Wang, W. Qin, M. Xia, et al. Improving video generation with human feedback. arXiv preprint arXiv:2501.1311. 2025b.
Liu, Y. Han, Xing, F. Yin, R. Wang, W. Cheng, J. Liao, Y. Wang, Fu, C. Han, er Steplx- edit: A practical framework for general image editing, arXiv preprint arXiv:2504. 17761, 2025c.
X. Liu, C. Gong, and Q Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 2022.
I. Loshchilov and F. Hutter. arXiv:1711.05101 2019.
Decoupled weight decay regularization.
arXiv preprint
C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling around 10 steps. Advances in neural information processing systems, 35:5775-5787, 2022.
C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. Machine Intelligence Research, pages 1-22, 2025.
G. Ma, H. Huang, K. Yan, L. Chen, N. Duan, S. Yin, C. Wan, R. Ming, X. Song, X. Chen, Y. Zhou, D. Sun, D. Zhou, Zhou, K. Tan, K. An, M. Chen, W. Ji, Q. Wu, W. Sun, X. Han, Y. Wei, Z. Ge, A. Li, B. Wang, B. Huang, B. Wang. B. Li, C. Miao, C. Xu, C. Wu, C. Yu, D. Shi, D. Hu, E. Liu, G. Yu, G. Yang, G. Huang, G. Yan, H. Feng, H. Nie, H. Jia, H. Hu, H. Chen, H. Yan, H. Wang,
H. Guo, H. Xiong, H. Xiong, J. Gong. J. Wu, J. Wu, J. Wu, J. Yang, J. Liu, J. Li, J. Zhang. J. Guo, Lin, K. Li, L. Liu, Xia, L. Zhao, L. Tan, L. Huang. L. Shi, M. Li, M. Li, M. Cheng, N. Wang, Q Chen, Q. He. Q. Liang, Q Sun, R. Sun, R. Wang, S. Pang, S. Yang, S. Liu, S. Liu, S. Gao,
T. Cao, T. Wang, W. Ming, W. He, X. Zhao, X. Zhang, X. Zeng, X. Liu, Yang, Y. Dai, Y. Yu, Li, Y. Deng, Y. Wang, Y. Wang. Y. Lu, Y. Chen, Y. Luo, Y. Luo. Y. Yin, Y. Feng. Y. Yang, Z. Tang, Z. Zhang, Z. Yang, B. Jiao, J. Chen, J. Li, S. Zhou, X. Zhang, X. Zhang, Y. Zhu, H.-Y. Shum, and D. Jiang. Step-video-12y technical report: The practice challenges, and future of video foundation model, 2025a. URL http/aa.vvvvv//////v//ea./aa.avv org/ www.a//...2222222 10248
X. Ma. P. Sun, H. Ma, H. Tang, C.-Y. Ma, J. Wang, K. Li, X. Dai, Y. Shi, X. Ju, et al. Token- shuffle: Towards high-resolution image generation with autoregressive models. arXiv preprint 2504 17789, 2025b.
C. Meng, R. Rombach, R. Gao, D. Kingma, S. Ermon, J. Ho, and T. Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CV conference on computer vision and pattern recognition, pages 14297-14306, 2023.
Y. Niu, M. Ning, M. Zheng, B. Lin, P. Jin, J. Liao, K. Ning, B. Zhu, and L. Yuan. Wise: A world knowledge-informed semantic evaluation for text-to-image generation. arXiv preprint arXiv:2503.07265. 2025.
D. A. Oliveira and D. M. de Matos. Storyreasoning dataset: Using ch thought for scene understanding and grounded story generation. arXiv preprint arXiv:2505. 102292, 2025.
13

---

Published as a conference paper at ICLR 2026
OpenAl. Introducing gpt-4.1 1 in the api. OpenAl Blog, 2025a. URL https::/openai.coo
index/apt-411
OpenAl. Introducing 4o image generation, 2025b. URL http:://opemal.comm/mme introducing 40 -mage--eneration
X. Pan, S. N. Shukla, A. Singh, Z. Zhao, S. K. Mishra, J. Wang, Z. Xu, J. Chen, K. Li, F. Juefei-Xu, Hou, and Xie. Transfer between modalities with metaqueries. arXiv preprint wxiv:2504.06256, 2025.
W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), 2023.
Y. Peng, Y. Cui, H. Tang, Z. Qi, R. Dong, J. Bai, C. Han, Z. Ge, X. Zhang, and S.-T. Xia. Dreambench++: A human-aligned benchmark for personalized image generation. arXiv preprint arXiv:2406. 16855, 2024.
D. Podell, Z. English, K. Lacey, Blattmann, T. Dockhorn, J. Müller, Penna, and R. Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations (ICLR), 2024.
Q. Qin, L. Zhuo, Y. Xin, R. Du, Z. Li, B. Fu, Y. Lu, J. Yuan, X. Li, D. Liu, et al. Lumina-image 2.0: A unified and efficient image generative framework. arXiv preprint arXiv:2503.21758 2025.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning (ICML), 2021.
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems (NeurIPS), 2023.
R. Rafailov, A. Sharma, E. Mitchell, Ermon, C. Manning, and C. Finn. Direct preference optimization: Your language model is secretly reward model. arXiv preprint arXiv:2305, 181,:993..,18290. 2024.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified tet-ooo-exttransormmme Journal of machine learning research (JMLR), 2020.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
O. Ronneherger, Fischer, and Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, 2015.
Y. Shi, P. Wang, and W. Huang. Seededit: Align image re-generation to image editing. arXiv preprint T..:24111.06686 2024.
Stability-AL. stable-diffusion-355.. 3.5- 2024.
URL https:///itthu com/
Stability Stability-AT/dd...
Su, Ahmed, Y. Lu, S. Pan, Bo, and Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 2024.
Sun, Y. Jiang, S. Chen, S. Zhang, B. Peng, Luo, and Z. Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint XXiv:2400.06525, 2024a.
Q. Sun, Q. Yu, Y. Cui, F. Zhang, X. Zhang, Y. Wang, H. Gao, J. Liu. T. Huang, and X. Wang. Emu: Generative pretraining in multimodality. In International Conference on Learning Representations (ICLR), 2023.
14

---

Published as a conference paper at ICLR 2026
Q. Sun, Y. Cui, X. Zhang. F. Zhang, Q. Yu, Y. Wang, Y. Rao, J. Liu, T. Huang, and X. Wang. Gen- erative multimodal models are in-context learners. In Proceedings the IEEE/CVF Conference Computer Vision and Pattemn Recognition (CVPR), 2024b.
Y. Sun, H. Bao, W. Wang, Z. Peng, L. Dong, Huang. Wang, and F. Wei. Multimodal latent language modeling with next-token diffusion. arXiv preprint arXiv:2412.08635, 2024c.
K. K. team. Kolors2.0, 2025. URL httph:httthpppt www.kkiiggiicoom
R. team.
Recraft v3. 2024.
URL
https:/www.reeraff.aalbooo
K. Tian, Y. Jiang, Z. Yuan, Peng, and L.. Wang. Visual autoregressive modeling: Scalable im- age generation via next-scale prediction. Advances in neural information processing systems (NeurIPS), 2024.
B. Wallace, M. Dang. R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik. Diffusson model alignment using direct preference optimization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
B. Wang, B. Wang, C. Wan, G. Huang, H. Hu, H. Jia, H. Nie, M. Li, N. Chen, S. Chen, et al. Step- large yet affordable: Model-system co-design for cost-effective decoding. arXiv preprint arXiv:2507.19422 2025a.
B. Wang, Z. Yue, Zhang, S. Chen, L. Bi. J. Zhang. X. Song, K. Y. Chan, J. Pan, W. Wu, et al. Selftok: Discrete visual tokens of autoregression, by diffusion, and for reasoning. arXiv preprint arXiv:2505.0755 2025b.
P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and Lin. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXiv:2409.21191, 2024a.
X. Wang, X. Zhang, Z. Luo, Q. Sun, Y. Cui, J. Wang, F. Zhang, Y. Wang, Z. Li, Q. Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arxiv:2409. 2024b.
Y. Wang, S. Yang, B. Zhao, L. Zhang, Q. Liu, Y. Zhou, and C. Xie. Gpt-image-edit-1155 m: A million-scale, gpt-generated image dataset. arXiv preprint arXiv:2507.21033 2025c.
Wei, X. Wang, Schuurmans, M. Bosma, Xia, Chi, Q. V. Le, D. Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems (NeurIPS), 2022.
C. Wu, J. Li, J. Zhou, J. Lin, K. Gao, K. Yan, S.mm. Yin, S. Bai, X. Xu, Y. Chen, et al. Qwen-image technical report. arXiv preprint arXiv:2508. 122224500.02224, 2025a.
C. Wu, P. Zheng, R. Yan, S. Xiao, X. Luo, Y. Wang, W. Li, X. Jian, Y. Liu, J. Zhou, et al. Omnigen2: Exploration to advanced multimodal generation. arXiv preprint arXiv:2506. 18871, 2025b.
Y. Wu, Z. Zhang, J. Chen, H. Tang, D. Li, Fang, L. Zhu, E. Xie, Yin, Yi, et al. Vila- unified foundation model integrating visual understanding and generation. arXiv preprint M4Xi:22409004299, 2024.
S. Xiao, Y. Wang, J. Zhou, H. Yuan, X. Xing, R. Yan, C. Li, S. Wang. T. Huang, and Z. Liu. Omnigen: Unified image generation. arXiv preprint arXiv:2409.11 13340, 2024.
E. Xie, J. Chen, Y. Zhao, J. Yu, L. Zhu, C. Wu, Y. Lin, z. Zhang, M. Li, J. Chen, et al. Sana 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer. arXiv preprint 2501v 10..2501..14444 8427, 2025a.
J. Xie, W. Mao, Z. Bai, D. J. Zhang, W. Wang, K. Q. Lin, Y. Gu, Z. Chen. Z. Yang, and M. Z. Shou. Show-o: One single transformer to unify multimodal understanding and generation. Xiv preprint arxiv:2408.
0e.4448..2225 12528, 2024.
15

---

Published as a conference paper at ICLR 2026
J. Xie, Z. Yang, and M. Z. Shou. Show-o2: Improved native unified multimodal models. arXiv preprint arXiv:2506.0 155664 2025b.
J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems (NeurIPS), 2023.
A. Yang, B. Yang, B. Zhang. B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, Yang, Tu, J. Zhang, J. Yang. Yang. Zhou, J. Lin, K. Dang. Lu, Bao, K. Yang. L. Yu, M. Li, M. Xue, P. Zhang, Q Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang. Y. Wan, Y. Liu, Cui, Z. Zhang, and Z. Qiu. Qwen2.5 technical report. arXiv preprint 2024.
arXiv:2412.11111
J. Yang, T. Li, L.. Fan, Y. Tian, and Y. Wang. Latent denoising makes good visual tokenizers. arxiv preprint mrXiv:2507.15855 2025.
J. Yao, B. Yang, and X. Wang. Reconstruction vs. generation: Taming optimization dilemma in la- tent diffusion models. Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), 2025.
Y. Ye, X. Hc, Z. Li, B. Lin, S. Yuan, Z. Yan, B. Hou, and L. Yuan. Imgedit: A unified image editing dataset and benchmark. arXiv preprint arXiv:2505.20275 2025.
Q. Yu, W. Chow, Z. Yue, K. Pan, Y. Wu, X. Wan, J. Li, S. Tang, H. Zhang, and Y. Zhuang. Anyedit: Mastering unified high-quality image editing for any idea. arXiv preprint arXiv:2411.15738, 2024a.
Q. Yu, M. Weber, X. Deng, X. Shen, D. Cremers, and L.-C. Chen. An image is worth 32 tokens for reconstruction and generation. Advances Neural Information Processing Systems (NeurIPS), 2024b.
T. Z.i. Cogview4, 2025. URL
K. Zhang, L. Mo, W. Chen, H. Sun, and Y. Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. In Advances in neural information processing systems (NeurIPS), 2023a.
Zhang, Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference computer vision (ICCV), 2023b.
W. Zhang, H. Zhang. X. Li, J. Sun, Y. Shen, W. Lu, D. Zhao, Y. Zhuang, and L. Bing. 2.5 years in class: A multimodal textbook for vision-language pretraining. arXiv preprint arXiv:2501.009: 2025.
C. Zheng, T.-L. Vuong, J. Cai, and D. Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems (NeurIPS). 2022.
C. Zhou, L. Yu, A. Babu, K. Tirumala, M. Yasunaga, L. Shamis, J. Kahn, X. Ma, L. Zettlemoyer, and Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. International Conference on Learning Representations (ICLR), 2025.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
A MORE RESULTS
Qualitative Results. Figure AT visually demonstrates NextStep-1's ability generate diverse, high-quality images with fine details, perform precise edits, and carry out complex free-form ma- nipulations that alter context and actions while preserving subject consistency.
Choers
Image Generation
The future
Next
Noxtstop-1
ArdCCleect
Mavrisl
Harkyrend
Malio
Tine
Clang
Image Editing
NEXT EDIIT
TRANQUII LIVING
Coler Charee
Corerte Ltds
StarHub
LEAGUE
Free-form Manipulation
Jamsping the lch
Ainugh
Steciig
Figure Al: Overview of NextStep-1 in high-fidelity image generation, diverse image editing, and complex free-form manipulation.
17

---

Published as a conference paper at ICLR 2026
Table Al: Comparison OnelG-Benge
Chang et al. 2025
English prompts.
Method
Alignment
Text
Reasoning
Style
Diversity
Overall
Proprictary
Imagen3 (Baldridge al. 2024
0.843
0.343
0.313
0.359
0.188
0.409
Recraft V3 (eem 4024
0.810
0.795
0.323
0.378
0.205
0.502
Kolors 2.0 Ieam 2025
0.820
0.427
0.262
0,360
0,300
0.434
Seedream Gao
2025
0.818
0.865
0.275
0.413
0.277
0.530
Imagen4 ddcepiid Imagent Team 2025
0.857
0.805
0.338
0.377
0.199
0.515
GPT-4o OpenAl 7025b
0.851
0.857
0.345
0.462
0.151
0.533
Diffiusion
Stable Diffusion
Rombach 2022
0.319
Stuble Diffusion XL Podell ct 2024
0.688
0.029
0.237
0.332
0.296
0.316
Stable Diffusion 3.5
Large Ata 2024
0.809
0.629
0.294
0.353
0.225
0,462
Flux.
[Labs.
2024
0.786
0.523
0.253
0.368
0.238
0.434
Vog
au 2025
0.786
0.641
0.246
0.353
0.205
0,446
SANA-1.5 TB
Xic 2025a
0.762
0.054
0.209
0.387
0.222
0.327
SANA-1.5 4.8B (PAG) Xie et 2025a
0,765
0.069
0.217
0.401
0.216
0.334
Lumina-Image 2.0 (Qin al.
0.819
0.106
0.270
0.354
0.216
0,353
HiDreeamII-FFu (Cai al. 2025
0.829
0.707
0.317
0.347
0.186
0.477
HI.IP3-0 (Chen
202N3
0.307
BAGEL [Dng
0.769
0.244
0.173
0.367
0.251
0.361
Show-o2-T.3B Xie
2025b
0.798
0.002
0.219
0.317
0,186
0.304
Show-o2-7B Xic ct al. 20256
0.817
0.002
0.226
0.317
0.177
0,308
OmniGen2
Wu ol 20256
0.804
0,680
0.271
0.377
0.242
0.475
Qwen-Image
0.882
0.891
0.306
0.418
0.197
0.539
Regesseees
Emu3 Wang et al.
2024b
0.737
0.010
0.193
0.361
0.251
0.311
Janus- 2025b
0.001
0,276
0.267
NextStep-1
0.826
0.507
0.224
0.332
0.199
0.417
Table A2: Comparison of world knowledge reasoning on WISE Niu al.
2025
result is with Self-CoT.
Model
Cultural
Time
Space
Biology
Physies
Chemistry Overall"
Overall (Rewrite)f
Proprictary
GPT--o (OpcnAl 2025b
0.81
0.71
0.89
0.83
0.79
0.74
0,80
Diffusion
Stable Diffusion 1.5 Rombuch et
2022
0.34
0.35
0.32
0.28
0.29
0.21
0.32
0.50
Stable Diffusion
Podel
Stable Difiusion 3.5 Lave (Sabbiiy--A) AT. 2024
0.44
0.50
0.58
0.44
0.52
0.31
0.46
0.72
PixAit-Alpha (Chen et 20741
0.45
0,50
0.48
0.49
0.56
0.34
0.47
0.63
Playground
Flux. d-dev [Lbs
0.48
0.58
0.62
0.42
0.51
0.35
0.50
0.73
MetaQuery-XI
2025
0.56
0,55
0.62
0.49
0.63
0.41
0.55
BAGEL Deng al
2125
0.44X0,76'
0.55/0.691
0.68/0.55
0.44/0.55
0.6000.5
0.39X0.58
0.52/0.70f
0.71/0.77'
Qwen Imtge Wu 2025u
0.62
0.63
0.77
0.57
0.75
0.40
0.62
AntoRegressive
Sbow-o-512 $12 Xie et 2024
0.28
0.40
0.48
0.30
0.46
0.30
0.35
0.64
VILA-U Wireri
0.26
0.33
0.37
0.35
0.39
0.23
0.31
Emuš
Wang
m 20240
0.45
0.41
0.27
Janus fro-D Cill 0E 20256
0.30
0.37
0.19
0.36
0.12
0.26
0.35
0.71
NextStop-1
0.5100.70
0.5410.65
Quantitative Results. We also conduct fine-grained analysis on OnelG-Bene Bennchhech (Chang et al. 2025) with English prompts. reffectss NextStep-1's significant advantage across areas such as alignment, text rendering, reasoning and stylistic control over existing autoregressive models. To evaluate NextStep- I's ability to integrate world knowledge into image generation, we use the WISE benchmark (Niu et al. 2025), which emphasizes factual grounding and semantic understanding. As shown in Table A2 NextStep-1 achieves the best performance among autoregressive models, also exceeding most diffusion models, with protocol. Collectively, these results demonstrate NextStep-11 I's robust knowledge-awan semantic alignment and cross domain reasoning capabilities.
more significant boost under the prompt rewrite
18

---

Published as a conference paper at ICLR 2026
B DATA PIPELINES
To comprehensively equip our model with broad and versatile capabilities, we construct diverse training corpus composed of four primary data categories: text-only corpus, image-text pair data, image-to-image data, and interleaved data. Each category is curated to serve a distinct role in fos- tering different aspects of the model's generative abilities.
B. TEXT-ONLY CORPUS
To preserve the extensive language capabilities inherent in the large language model (LLM), we incorporate 400B text-only tokens sampled from Step-3 (Wang al. 2025a) during training.
B.2 TEXT PAIR DATA
Data consisting of image-text pairs forms the foundation of the model's text-to-image generation capabilities. We developed a comprehensive pipeline to curate a high-quality, large-scale dataset from diverse set of initial sources.
1. Data Sourcing: We collected a large-scale dataset from diverse sources, including web data, multi-task VQA data and text-rich documents.
2. Quality-Based Filtering: We then applied rigorous filtering process, evaluating each im- age on aesthetic quality, watermark presence, clarity, OCR detection, and text-image se- mantic alignment.
3. Re-captioning: After deduplicating the filtered images, we used the Step-lo-turkes lo-turbo gen- erate rich and detailed captions for each image in both English and Chinese.
This multi-stage pipeline yields a final dataset of 550M high-quality image-text pairs, providing a foundation for training model with both strong aesthetic sense and broad world knowledge.
B.3 INSTRUCTION-GUDDE
DATA
To enable a wide range of practical applications, we curated high-quality dataset for instruction- guided image-to-ing tonggee---mage tasks, such as visual perception (Kirillov et al. (223)), controllable image generation (Zhang et al. 20236), image restoration (Labs) 2025), general image editing (Peng al.] 2024), and more.
For visual perception and controllable image generation tasks, we synthesized 1M samples by ap- plying the annotator of ControlNet (Zhang et al. (023b) to part of our high-quality image-text pair data. For image restoration and general image editing, we collected 3.5M samples, comprising data from GPT-Image-Edii (Wang et al. 2025c). Step IX- X-Edit (Liu et al. (025c)), and proprictary in-house dataset. Following StepIX-Edit (Liu et al. 2025c), all editing data were subjected to a rig- orous VLM-based filtering pipeline that assessed both image-pair quality, rationality, consistency, and instruction alignment, resulting in about 1M high-quality instruction-guidee image-to-image data for training.
B.4 INTERLEAVED DATA
Interleaved data seamlessly integrates text and images, offering rich and nuanced sequential asso- ciations between modalities. Specifically, our knowledge-rich interleaved dataset is primarily com- posed of four distinct categories: general video-interleaved data, tutorials, character-centric scenes, and multi-view data.
To endow our model with extensive world knowledge, we first constructed a large-scale, 80M- sample video-interleaved dataset. This was achieved through meticulous curation pipeline, in- spired by Step-Video Video (Ma et al., 2025a), which encompasses frame extraction, deduplication, and captioning. Furthermore, following the methodology of mmtextbook (Zhang al. 2025), col- lected and processed tutorial videos by leveraging ASR and OCR tools. This component specifically targets text-rich real-world scenes, enhancing the model's textual understanding and generation in
19

---

Published as a conference paper at ICLR 2026
Face detection & character binding
Stepl::
Stppll::
Web data collection
unicocto sompliing
Char Bbox array
Andrace
Coarse filtering
Gel face embeldings within
Matching faces based on cosine similarity
Saving mela infos for each character
Face detection & character binding
Frame extraction & MLLM captioning
Step I:
Sep
Step III:
Step
Crer
J
Stop 1o
"The camera shifts the outdoors,
Checklist:
Frame extraction & MLLM captioning
where
Object Consistency
talking woman"
A triiutte Consistency
extraction using mela infos
Relation Consistenry
Figure A2: Data processing of character-centrie data.
context. A key contribution, detailed in Figure A2 is our character-centric dataset, NextStep- Sextttee-Viieo- Video- Interleave-5M. For this dataset, we extracted video frames centered around specific characters and generated rich, storytelling-style captions akin to (Oliveira and de Matos (025), thereby signifi- cantly improving the model's capacity for multi-turn interaction. Finally, to bolster geometric rea- soning, we curated multiview data from two open-source datasets, MV-ImageNet--2 (Han et al. 2024) and Objaverse-XL (Deitke et al. 2023). which enhances the model's ''s ability to maintain mul- tiview consistency.
C IMAGE TOKENIZER
Our image tokenizer is initialized from the Flux. 1-dev VAE Labs 2024), selected for its strong
reconstruction performance. Differently, as shown in Figure 2, we apply a token-wiie normalization operation on the Encoder output, and adopt the noise schedule like a- --E Sun al. (2024c) during
tuning. For better understanding. the normalization operation is detailed in the code of Algorithm I
We collect latent distributions in 16 channels for three VAE variants in Figure A3 aligns best with the normal distribution, reflecting a dispersed latent distribution.
NextStep-1 VAE
Algorithm 1: Python code for the Token-wise Normalization of NextStep-VAr VAE.
X: input latent tensor of shape (B, H, H, CJ # eps: stability constant, default 1e-6 def normalization_oper cerrriiiooperiiiinnooo (X, eps-le-6)
Compute mean and std along the channel dimension (-1)
mu m.me (dim=-1, keepdin=Truc) sigma x.std (dim=-1, keepdim-True, unbiased-Falee)
Apply no normal ization ion: zero-mean and unit-variance
__rrrm (x mu) (sigma + eps)
return X norm
D LIMITATIONS AND CHALLENGES
Artifacts. While NextStep-1 successfully demonstrates that autoregressive models can operate on high-dimensional continuous latent spaces, achieving generation quality comparable to diffusion models, this approach also introduces unique stability challenges. We observed the emergence of several distinct generative artifacts when transitioning from a VAE with a dimensiinni latent space (e.g., spatial downsample factor is 8 and number of latents channel is 4) to with higher-
20

---

Published as a conference paper at ICLR 2026
Latent Distribution of Flax, I-dev VAE
Latent Distribution NextStep-1 VAE w/o Noise
Distribation NextStep-1
Figure A3: Latent distributions in 16 channels for three VAE variants: Flux. I-dev, NextStep-1 w/o noise, and NextStep-1. Blue bars show empirical histograms; lines indicate the standard normal distribution.
dimensional space (e.g., spatial downsample factor is 8 and number of latents channel is 16). While the former configuration produced stable outputs, the latter occasionally exhibited failure modes, as illustrated in Figure A4
While the underlying causes remain an open question, we identify several plausible contributing fac- tors: (1) Local noise or block-shaped artifacts emerging in the later stages of generation may arise from numerical instabilities; (2) Global noise across the image may reflect under-convergence, implying that additional training could mitigate the issue; and (3) Subtle grid-like artifacts could reveal limitations of the ID positional encoding in capturing 2D spatial relationships.
Mitigating Artifacts in NextStep 1.1. Bagel (Deng et al. 2025 is trained approximately 5T tokens, significantly exceeding the 2T tokens used for NextStep-1. Motivated by this, we incorpo- rated additional high-quality data to continue training N, in Figure A5 by leveraging superior data over extended training iterations (approximately 500K) and an improved version of Flow-GRPO (Liu al. 2025b) in NextStep- most artifacts observed in N are successfully eliminated.
NextStep-1.1. As shown
Inference Latency of Sequential Decoding. theoretical analysis per-token latency on an H100 GPU (983 TFLOPS, 3.36 TB/s bandwidth) with batch size of 1, as detailed in Table A3 decomposes the contributions of individual components. The results show that the dominant bottle- neck lies in the serial decoding of the LLM, while the multi-step sampling in the flow-matching head also constitutes a substantial portion of the por-token generation cost. These observations suggest two promising directions for accelerating inference. First, the efficiency of the flow matching head
21

---

Published as a conference paper at ICLR 2026
Bottom noise
CAUTICONS
PaRti fox
Magnum
Qpi
Bottom solid color
Noisy image
JOHN GALACTIC ADVENTUR
WHAT YOU A M9
Grid
Figure A4: Failure cases for high-dimensional continuous tokens.
NextStep-1
NextStep-1.1
EUISOON
Figure A5: Comparisons between NextStep-1 and NextStep-I.I.
could be improved by reducing its parameter count, applying distillation to achieve few-step genera- tion (Meng et al. 2023), or using more advanced few-step samplers eLu et al. 2022 2025). Second, the autoregressive backbone could be accelerated by adapting recent advances from the LLM field, such as speculative decoding (Leviathan et al., (023) or multi-token prediction (Gloeckle et al. 2024), to the domain of image token generation.
22

---

Published as a conference paper at ICLR 2026
Table A3: Inference latency breakdown at 983 TFLOP/s compute and 3.36 TB/s memory bandwidth.
Sequence Length
Last-token Latency (ms)
Accumulated Latency (8)
I.I. Decoder
LM Head
FM Head
Total
w/o FM Head
256
7.20
0.40
3.40
2.82
1.95
1024
7.23
0.40
3.40
11.31
7.83
7.39
Challenges in High-Resolution Training. Our framework faces two primary challenges in scal- ing to high-resolution image generation, particularly when compared to diffusion models, which benefit from well-established techniques (Esser et al. al.] 2024) Chen al. (2024) in this domain. First, the strictly sequential nature of autoregressive generation requires substantially more training steps to converge at higher resolutions. In contrast, diffusion models refine the entire image in parallel at each iteration, enabling more direct exploitation of 2D spatial inductive biases. Second, tech- niques recently developed for high-resolution diffusion models, such as timestep shift, are difficult adapt to setting. This limitation arises because the Flow Matching Head acts primarily lightweight sampler, while the transformer backbone performs the core generative modeling; thus, modifications the sampling process have only marginal impact the final output. Designing high-resolution generation strategies specifically for patch-wise autoregressive models remains an important direction for future research.
Challenges in SFT. SFT in our autoregressive framework poses unique challenges compared to diffusion models. We observe that fine-tuning on small, high-quality datasets exhibits unstable dynamics. In contrast to diffusion models, which can often adapt to target distribution and main- tain stable and general image generation with only a few thousand samples, our SFT process yields substantial improvements only when trained datasets the datasets, the model remains in a precarious equilibrium; either improves marginally with negligi- ble impact abruptly overfits to the target distribution. Consequently, identifying intermediate checkpoint that achieves alignment with the target distribution while preserving general generative capability remains significant challenge.
scale. With smaller
E
LLM USAGE
This paper only uses large language models (LLMs) for language polishing and improving the clarity of writing. No part of the research design, experiments, data analysis, or results was generated by LLMs.
23


Reviews:
[{"review_id": "uyuobmeryO", "reviewer": "Reviewer_BTzv", "summary": "This paper presents NextStep-1, an autoregressive image generation framework that combines patch-level flow matching with a continuous-token VAE. The approach demonstrates strong performance on both generation and editing benchmarks, supported by solid ablation studies and high-quality visual results.", "questions": "1.Is the autoregressive generation performed patch-by-patch? If so, how is spatial consistency ensured across adjacent patches in the final reconstructed image?\n\n2.Could the authors provide a diagram illustrating the tokenizer architecture, especially the normalization and latent regularization mechanisms?\n\n3.Given the claim that reconstruction quality upper-bounds generation quality, does this imply that achieving high-fidelity controllable generation fundamentally requires near-perfect reconstruction?\n\n4.Is this the first work to combine autoregressive modeling with patch-level flow matching? A clearer comparison of competing paradigms (e.g., VQ-based AR vs. diffusion vs. continuous AR) would help contextualize the novelty and advantages of the proposed approach.\n\n5.The ablation focuses on the flow-matching head—how sensitive is performance to the design or capacity of the encoder (i.e., the VAE encoder)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is clearly written and technically sound, presenting a coherent and well-motivated method.\n\n2.The ablation studies are particularly valuable—for instance, the analysis clarifying the relative contributions of the backbone versus the flow-matching head provides useful architectural insights.\n\n3.Figures and tables are of high quality and effectively support the claims made in the text.", "weaknesses": "1.Several key technical details remain underspecified. For example, it is unclear whether the autoregressive process decodes one patch at a time and how global consistency is maintained if patches are stitched together.\n\n2.The design of the tokenizer—including its normalization scheme—is described only textually; a schematic illustration would greatly improve clarity.\n\n3.While the paper asserts that “reconstruction quality is the upper bound of generation quality,” it does not explicitly discuss the implications for controllable or conditional generation tasks.\n\n4.The positioning of this work within the broader literature is incomplete: it is not clear whether this is the first AR model to employ patch-level flow matching, nor does the paper compare the trade-offs between alternative design choices (e.g., tokenization strategies, training objectives), making it difficult to assess the method’s ultimate potential or limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d7DZ40MJni", "reviewer": "Reviewer_es99", "summary": "The paper proposes an autoregressive text-to-image model that achieves state-of-the-art performance with minimalist architecture with revealing some key design principles.", "questions": "NA", "limitations": "", "rating": 2, "confidence": 5, "soundness": 4, "presentation": 1, "contribution": 4, "strengths": "1. I really appreciate the research taste of the paper. The paper uses some extreme simple method to achieve state-to-the-art performance.\n\n2. The paper can give many take aways to the sequent researchers.", "weaknesses": "1. I really suggest the authors to hire someone who is adept at academic paper writting to re-write the whole paper. For instance, the current paper is very unclear. For instance, the introduction is too short. The related work paper is too short without fully respecting the former authors. Given this, I have to lower my score to 4. This is a paper needs major revision.\n\n2. Many parts are unclear. For instance, what GPUs are used in training. The pre-training and post-training sections are also unclear.\n\n3. I do not know what the “Next” in the paper presents?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vucxBrQYKK", "reviewer": "Reviewer_UGjn", "summary": "This manuscript introduces NextStep-1, a large (14B) autoregressive (AR) model for text-to-image generation. Unlike prevailing AR models that rely on Vector Quantized (VQ) discrete tokens, NextStep-1's core innovation is its direct autoregressive prediction on **continuous image tokens**.\n\nIts architecture consists of a large AR Transformer (Qwen2.5-14B) and a lightweight (157M) Flow Matching (FM) head. The model processes tokens sequentially: for text tokens, it uses a standard LM head (cross-entropy loss); for each continuous image token (patch), it uses the AR model's hidden state as a condition to drive the FM head, which generates the patch.\n\nThe paper's central claims include:\n\n1. This \"minimalist architecture\" (AR-LLM + lightweight FM head) can achieve SOTA performance on T2I tasks, competitive with top-tier diffusion models.\n\n2. The key to this success is a **novel image tokenizer** (fine-tuned from Flux VAE). This tokenizer, through channel-wise normalization and stochastic perturbation, creates a robust latent space that enables stable training on high-dimensional continuous tokens.\n\n3. The model achieves SOTA or competitive results on T2I benchmarks (e.g., GenEval, WISE) and image editing benchmarks (e.g., GEdit-Bench).", "questions": "To validate the paper's core claims and increase its impact, I strongly recommend the authors address the following key questions with experiments:\n\n**Question 1: Does the reliance on million-sample SFT imply a fundamental flaw in the model's \"alignability\" and \"customizability\"?**\n\nThe paper admits SFT requires massive data. Does this mean the model is far less efficient or practical for aligning with human preferences and performing style customization (e.g., LoRA) compared to the diffusion models it claims to rival?\n\n**Question 2: How should we evaluate the trade-offs of this AR paradigm, given its SOTA quality but practical limitations in speed and scalability?**\n\nQuery: The paper demonstrates impressive SOTA quality, yet Appendix D and Table A2 confirm this comes at the cost of inference latency and high-resolution scaling challenges. Could the authors elaborate on how they view this trade-off? Is the goal to prioritize quality and contextual dependency over speed, and what are the most promising future directions to mitigate these latency and scaling bottlenecks?\n\n**Question 3: Regarding the artifacts in Appendix D (Figure A3) for high-dimensional tokens: Are these a side effect of the normalized VAE?**\n\nThe paper shows new stability challenges (e.g., bottom noise, solid blocks) in high dimensions. Are these artifacts a direct side effect of the channel-wise normalization or noise perturbation introduced in the VAE? Does forcing a normalized latent space, while solving CFG drift, create new optimization challenges for the AR model in high-dimensional spaces?\n\n**Question 4: Is the 1D Raster-scan order a bottleneck for capturing 2D spatial dependencies?**\n\nWhen using a 1D row-first serialization, the last token of one row is not spatially adjacent to the first token of the next. Does this hinder the model's ability to learn vertical spatial relationships across rows? Why not employ a more spatially-aware serialization (e.g., Z-order, Hilbert curve) or 2D positional encodings to address this?\n\n**Question 5: Is the 14B scale of the AR Transformer a necessary condition for high-quality generation?**\n\nThe paper successfully adapts a 14B LLM (Qwen2.5) for continuous image generation. But is this massive scale essential? Could a smaller AR model (e.g., 2B or 7B), designed specifically for image context, achieve similar results with the FM head? What role does model scale play in the \"core generative modeling\"?\n\n**Question 6: What is the advantage of this \"AR context + FM patch\" paradigm over architectures that predict global velocity (e.g., Transfusion, BAGEL)?**\n\nModels like Transfusion or BAGEL use a Transformer to parallelly predict the flow velocity for the entire image. What are the conceptual and practical advantages of NextStep-1's \"serial AR prediction + serial FM generation\" architecture? Does it trade parallelism (speed) for stronger contextual dependency or finer-grained local control?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **SOTA Autoregressive Performance**: The paper's primary contribution is demonstrating, for the first time, that an AR model based on continuous tokens (NextStep-1) can achieve SOTA performance on T2I tasks, rivaling top-tier diffusion models. This architecture, combining a large AR Transformer for context prediction with a lightweight FM head for continuous token generation, is proven to be a very successful and promising technical direction.\n\n2. **Deep Insights into Tokenizer and Latent Space**: One of the paper's most outstanding merits is its deep analysis of the tokenizer in Section 4.2. The paper correctly identifies a key bottleneck in generative modeling: the quality and properties of the latent space.\n\n3. **Excellent Analysis of CFG Instability**: The paper's insight in Section 4.2 and Figure 3, which attributes instability at high CFG scales to \"per-token distributional shift\" (rather than 1D RoPE issues), is an excellent observation. The proposed solution (channel-wise normalization in the VAE) is simple and effective.\n\n4. **Strong FM Head Ablation**: The finding in Section 4.1—that the size of the FM head has minimal impact on final quality—is strong evidence supporting the authors' core argument that the AR Transformer is performing the \"core generative modeling,\" while the FM head acts primarily as a lightweight sampler.\n\n5. **Clarity and Completeness**: The paper is exceptionally well-written, complete, and clear. The detailed appendices (e.g., data pipelines in Appendix B, training recipes in Appendix C) are thorough, transparent, and provide significant reference value for the community.", "weaknesses": "1. **Inherent Bottleneck in Inference Latency**: The paper admits in Appendix D and Table A2 that inference latency is a major weakness. The AR sequential decoding is the first bottleneck, and the FM head's multi-step sampling is the second. A 1024-token image requiring 11.31 seconds of accumulated latency (Table A2) is likely far slower in practice than parallel diffusion models.\n\n2. **Significant Challenges in High-Resolution Scaling**: The paper frankly states in Appendix D that the model faces challenges in scaling to high-resolution training (e.g., 1024x1024) and admits that techniques developed for high-res diffusion (like timestep shift) are not applicable to this framework. This severely limits the model's practical ability to \"compete\" with SOTA diffusion models, which commonly excel at 1024x1024.\n\n3. **Contradictory Information on Stability**: The paper's narrative on stability is confusing. Section 4.2 claims the tokenizer solves CFG instability. However, Appendix D (Limitations) opens by stating that high-dimensional continuous latents introduce \"unique stability challenges,\" showing failure cases like bottom noise, solid color blocks, and grid artifacts (Figure A3). This suggests a trade-off: does solving CFG drift by forcing a normalized VAE latent space introduce new, severe generative artifacts in high dimensions?\n\n4. **\"Data-Hungry\" SFT Process**: Appendix D notes that the SFT process is unstable and requires \"million-sample scale\" datasets to show significant improvement, a stark contrast to diffusion models that can be fine-tuned with a few thousand samples. This drastically undermines the model's practical utility for alignment and customization (e.g., LoRA), which is a core strength of modern diffusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6aXyaB8MFk", "reviewer": "Reviewer_taan", "summary": "This work introduces NextStep-1, an AR model with flow matching head for image generation. The architecture desire of NextStep-1 is a lot like MAR[1] / FLUID[2] with the diffusion head replaced by flow matching. The work also investigates tokenizer design and figures out the importance of regularized latent space. Experiments on multiple image generation/editing benchmarks show strong performance of NextStep-1 compared to previous diffusion and AR models. \n\nReference:\n[1] Li, Tianhong, et al. \"Autoregressive image generation without vector quantization.\" Advances in Neural Information Processing Systems 37 (2024): 56424-56445.\n[2] Fan, Lijie, et al. \"Fluid: Scaling autoregressive text-to-image generative models with continuous tokens.\" arXiv preprint arXiv:2410.13863 (2024).", "questions": "1. In pre-training and tuning, how are the objectives of LM head and flow-matching head balanced?\n2. How much compute overhead does the flow matching head add in inference compared to standard AR models?\n3. Do the authors compare the convergence rate of NextStep-1 with standard AR models? Since usually flow-matching/diffusion models are less efficient in convergence. \n4. Will channel-wise normalization weaken the expressivity of learned latents since it can break the correlation between different channels? Also, does it lead to instability in training since it's a per-sample normalization?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow. \n2. The paper investigates building strong AR visual generative models which is an important direction. \n3. The paper includes comprehensive experiments on multiple image generation / editing benchmarks and shows strong performance.", "weaknesses": "1. The paper claims state-of-the-art performance of NextStep-1. Though it does show strong performance on multiple benchmarks, it doesn't dominate all the metrics. For example in table 1, there are better baselines in GenEval. \n2. The paper introduces a natural extension to MAR/FLUID, that is replacing the diffusion head with flow matching, and proves its effectiveness. This weakens the methodological contributions in this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)