Selftok-Zero: Reinforcement Learning for Visual Generation via Discrete and Autoregressive Visual Tokens
Authors: Bohan Wang
Mingze Zhou
Zhongqi Yue
Wang Lin
Kaihang Pan
Liyu Jia
Wentao Hu
Wei Zhao
Hanwang Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Tokenizer
MLLM
Reinforcement Learning
Primary area: deep_learning

Abstract: Reinforcement learning (RL) has become an indispensable post-training step for unlocking the full potential of Large Language Models (LLMs). Its core motivation is to incentivize the model’s inference trajectory via a reward model, effectively balancing the exploration–exploitation trade-off in scenarios where collecting exhaustive input–output ground-truth pairs is infeasible. This motivation naturally extends to visual generation, where perfect alignment between an image and a textual prompt is inherently ambiguous and often unattainable. However, existing visual generative models are not yet ready for RL due to the following two fundamental drawbacks that undermine the foundations of RL: 1) For diffusion-based models, the actual generation trajectories of sampled images cannot be reliably rewarded, as diffusion inversion is notoriously difficult. 2) For autoregressive (AR) models, we show that the widely used spatial visual tokens do not satisfy the Bellman equation and thus violate the policy improvement theorem of RL. To this end, we propose to use Selftok (Self-consistency Tokenizer), which represents each image as a sequential 1D stream of discrete, autoregressive tokens. Together with language, we train a pure AR vision-language model (VLM) for visual generation. Impressively, without using any text-image training pairs, a simple policy gradient algorithm applied to Selftok tokens significantly boosts visual generation performance, surpassing existing models by a large margin. Implementation details are provided in the Appendix.

Selftok-Zero: Reinforcement Learning for Visual Generation via Discrete and Autoregressive Visual Tokens
Bohan Wang?*
Mingze Zhou Wentao Hu
Zhonggi Yue
Wang Lin'
Kaihang Pan
Liyu Jia
Wei Zhao?
Hanwang Zhang
'Zhejiang University
Huawei Central Media Technology Institute
*Nanyang Technological University
{mingze, linwanglw, kaihangpan) eww.neeu..uuu....ee.mm fliyu002, wentao002) attu.edu.se bohan. wang977@maill com (zhaowei82, (ragghanwang)huuaeeee
edu.
Abstract
Reinforcement learning (RL) has become an indispensable post-training step for unlocking the full potential of Large Language Models (LLMs). Its core motivation is incentivize the model's inference trajectory via reward model, effectively balancing the eexpoittoonnllliitttiinnn trade-off in scenarios where collecting ex- haustive input-output ground-truth pairs is infeasible. This motivation naturally extends to visual generation, where perfect alignment between an image and a textual prompt is inherently ambiguous and often unattainable. However, exist- ing visual generative models are not yet ready for RL due to the following two fundamental drawbacks that undermine the foundations of RL:
For diffusion- models, the actual generation trajectories of sampled images cannot be reliably rewarded, as diffusion inversion is notoriously difficult.
For autoregressive (AR) models, we show that the widely used spatial visual tokens not satisfy the Bellman equation and thus violate the policy improvement theorem of RI.
To this end, propose to use Selftok (Self-consistency Tokenizer), which rep- resents each image as a sequential ID stream of discrete, autoregressive tokens. Together with language, train pure AR vision-language model (VLM) for vi- sual generation. Impressively, without using any text-image training pairs, a simple policy gradient algorithm applied Selftok tokens significantly boosts visual gen- eration performance, surpassing existing models by a large margin. Implementation details are provided in the Appendix.
Introduction
Recent advances in visual generative models have been driven by large-scale training on paired text-image datasets [3]. To produce high-fidelity image given textual prompt these models decompose the complex image generation process into sequence of simpler steps. In diffusion- based models 36 [5]. this decomposition unfolds over continuous time-steps t E [0. 1] by the reverse diffusion process: starting from Gaussian noise Xg E NN 1) with
0. the model iteratively denoises each X₃ to form the trajectory X ~4 X1, where X₃ x. In autoregressive (AR) models 665 (74, x is represented
sequence of discrete visual tokens Vx [u...... uk] (reasons not considering continuous visual tokens in this work included in Appendix A. D). and image generation proceeds by sequentially predicting each token given previous ones. both cases, models
Equal Contribution.
Coorespponing Author.
Research done during internship at Huawei.
39th Conference Neural Information Processing Systems (NeurIPS 2025).

---

Time-step
(b) Spatial visual tokens
Spatial Tokens
X DPO D Dataset
(a) Diffusionn-tas based
behavior policy (x,
mismatch
AR Tokens
target pollcy Po(K:|X,,) (X,
m
<M2>
<2
<D3>
<V4
Na ~ N(,1))
X
X1
Figure 1: (a) The mismatch between the behavior policy g(x,|xi and target policy (the reverse trajectory of a diffusion model parameterized by () in Diffusion-DPO [69]. This leads to poor action-space coverage, e.g., X₁ along the target policy trajectory is often outside the shaded 95% confidence interval of (xx, [x))). (b) Due to the anti-causal links (red) for spatial tokens, learning the locally optimal policy ni for later token (e.g., 04) can propagate backward and interfere with earlier tokens that were already optimized (e.g., U1, U2 02). In contrast, AR tokens without such links do not have this issue. A more formal illustration is in Section 2.2
are trained to follow the alignment of each paired training sample (x, y) over the entire generation process: given y, each generation step is directly supervised to match ground-truth specified by x, ie., its forward path in diffusion-basee models or token sequence in AR models.
However, this form of supervised learning is fundamentally limited due to the one-to-many nature of text-to-image mapping. For instance, single text prompt may correspond to infinitely many plausible images, yet the training dataset only includes a finite set of its typical looks (e.g., x =round apple, y ="an apple"). Because perfect aligmment between x and y is inherently ambiguous, models trained through supervision eventually resort to mimicking the training distribution, rather than faithfully following prompt y to generate a corresponding x. For example, we empirically observe that in early-stage training, models can still follow prompts about an atypical look, such as "a squared apple", ; but as training converges, their generated images eventually collapse to typical examples in the training dataset, such round apples.
straightforward remedy is to curate a supervised dataset that exhaustively covers all possible alignments. Yet, this approach is unsustainable.
This motivates us train visual generative models with Reinforcement learning (RL), which offers proven solution to this challenge, as demonstrated extensively on Large Language Models (LLMs) [66] 24 Instead of supervising each generation step by the alignment in the training dataset, RL instead imposes a task-specific reward only after the full generation process is complete, e.g., computing the CLIP-based image-prompt similarity as the reward. This shift enables the model to explore diverse generation trajectories and exploit those that yield high rewards, thereby incentivizing prompt- following visual generation. For example, given y ="a square apple". a model trained with RL is incentivized to produce any image that is semantically consistent with the prompt, without being constrained by the alignment in the training dataset.
Unfortunately, existing image representations have the following limitations for visual RL:
Diffusion-based models induce infinite Markov Decision Process (MDP) formulation with high-dimensional, continuous state-action spaces (i.e., X2 as a state, a denoising step as an action), which complicates optimization in RL. Recent attempt [69] explores an off-policy approach, where the key challenge the lack of state-action trajectories available in the original Direct Preference
Optimization formulation [55]. due the intractable diffusion inversion (48). As workaround, it samples from the forward diffusion process to approximate the behavior policy. However, as shown in Figure Ia this introduces a large mismatch between the behavior policy linear forward path) and the target policy (a non-linear reverse trajectory), leading to poor action-space coverage and hence inefficient learning (63].
Current AR models use spatial visual tokens 9 75 74,, where images are represented as grids of
patches, a convention dating back to early computer vision. However, we show that these spatial tokens lack the true AR structure, which violates the policy improvement optimality in RL (63 (Section 2.2 , as illustrated in Figure Tb First, spatial pixels (the cause) collectively form the image (the effect), and observing any part of the image during encoding induces spurious dependencies

---

SelftokSFTT
Selo
Janus-Pro-7B
Janus-Pro-7BB Pro-7B- MidJourney V6.1
FLUXI.1 PRO
a striking portrait that captures the essence of Salvador balf, with one side of his face depicted his iconic, surrealistic style, and the other half transformed into metallic, robotic visage. The pointing features vivid array of colors, with the robotic side incorporating shodes of silver and hints of circuitry, contrasting with the flesh ball's human side. background simple, solid color details Dall's representattion.
remains
contemporary bathroom. features white hanging shelf, securely mounted on a pale blue wall, showcasing an assortment toiletries. The items include glass container of cotton swabs, small plant in terracot pot, and an arrey of bottles with chrome cups ref reflecting the soft overhead light. Below shelf, plush cotton towel, perfectly fluffed and of gentle lovender hangs elegantly polished towel
the midst of an enigmatic scene, a mystical cat sits enveloped by undulating curls of vibrant smoke hues purple, blue, and green. Its eyes shine luminously, emitting aura of enigmatic choos magic that seems to dance around its sleek, shadowy fur. At the forefront visual emblematic seeing eye' prominently featured, adding blur, creating shollow depth field places the focus squarely on eenie yet coptivating feline figure.
theme, background fodes
detailed Persion metal engraving vasi, showcasing intricate petterns and designs, rests to the left side of a vibrunt bouquet of orange flowers. The with its silver sheen traditional craf trmasshhphhppp sits polished wooden table. The orange petals of the flowers provide stark controst to the metallic tones the vase, creating visually appealing composition.
Figure 2: Text-to-lmage generation results by Selftok using the prompts of DPG-Bench.
among tokens due to the collider effect [51], leading to non-AR causal graph. Second, predicting a token at later step (action) affects the tokens predicted in earlier steps (carlier states), so the later policy may contradict earlier policies that have already been optimized. Hence, RL applied to spatial tokens expected to significantly less effective than when applied to AR tokens.
In this paper, we build an AR model that supports effective RL-based post-training for visual generation. First, we completely abandon the long-standing spatial prior and introduce Selftok: Self- consistency Tokenizer (70), which leverages the AR nature of the reverse diffusion process encode an image into autoregressive tokens corresponding to its diffusion generation trajectory (Section 2.1). AR property, Selftok produces visual tokens that satisfy the optimality condition
Next, thanks of the policy improvement (Section 2.2p. Motivated by this, we build Seltok-Zero (Section B a Selftok-based AR model post-trained with visual RL. Without using any pairwise supervision, Selftok-Zero achieves impressive image generation performances GenEval: 92% (Table T and DPG-Bench: 85.57 (Table 2). Comparisons of text-to-image generation with existing VLMss are given in Figure

---

X1
Va
V
2 1
DK-1
Vk
Vi-2
D K-11
VK
Xo
X
V=k(t)
(a)
(b)
(c)
Figure 3: (a) The causal graph for AR, where each dotted direct edge represents causation. (b) The recursion of the AR causal graph. (c) The causal graph for learning V2R(1) in Eq. (5).
2 Problem Formulation
We begin by introducing the Selftok tokenizer 770 which encodes images into autoregressive (AR) token sequences derived from the reverse diffusion process-and an AR model based on its visual tokens (Section p. We then formulate the visual reinforcement learning (RL) problem and show why AR tokens are necessary for policy improvement in this setup (Section 2.2
2.1 Selftok: Ceff-Consistency Tokenizer
Selftok encode image into K discrete tokens, ie., Enc(T) VK [V1, .2,... Uk], which can be decoded to reconstruct / while adhering to an autoregressive (AR) prior. We formulate the following constrained optimization:
min Eno(T)= VR, | - (Vx) 11?
s.l. P(Vx) P(vi) P(valui).... P(wxloi......................... UK-I),
where we define as special equality indicate that the tokens Vx conform to the AR causal graph in Figure 3a i.e., each token is generated from its predecessor This definition is necessary, as the factorization is always valid for any token sequence through the chain rule of probability and does not necessarily imply an AR structure per se.
As with other discrete compression problems [78], solving the constrained optimization in Eq. T inherently NP-hard due the combinatorial nature token assignment. To make this tractable, we introduce an inductive bias grounded in the reverse diffusion process, which jointly satisfies the AR constraint and the reconstruction objective. In particular, the term "Consistency" comes from Consistency Model [60]. Similarly, we use a diffusion model and make the decoder consistent with image generation path, i.e., reconstructing x from any noisy inputs x, along the path.
Specifically, we show Figure 35 that AR structure has an equivalent recursion, enabling a divide- and-conquer approach that decomposes the challenging constraint in Eq. into simpler ones:
P(Vx) P(V=)) ((2))Vee)
(2)
where Ve+- [01, U2.... 1[-1] and V2 [U,, Ui+1 ..., UK] For example, we can recursively apply Eq. 22) until it becomes trivial learning problem P(Vek) P(uK [V<)): if V is provided, is easy encode the last token uk. Interestingly, the reverse diffusion process (in ODE form) has
similar decomposition 43 61
dx; di
= (x(),, t [0
solution
X destination
X midway point
J. (x.)ds
(3)
path midway to destinatio:: X; --X0
where Vx(xe) is the velocity field at time-step that transports the noisy midway Xr, starting from Xo CV 1), towards the clean image xi I. This shows that, if the midway x, is provided, the reconstruction of x starting from x, is easier than directly moving from Zo to x1
'Tii can be written mathematically as P (Voe)do(V2e)) = P(Ve) Vi E (L...., K} using the do- calculus [51)

---

Hence, we can establish correspondence between the two recursions by aligning the provided midway point (part 1) and what comes after (part 2), respectively:
J. vx(xx))s
P(VK) = = (
+
P(Vzileed
(4)
Whole
Part
Part 2
Motivated by this, we aim to compose the AR constraint into the reconstruction in Eq. T Specifically, decompose the entire reconstruction (from pure noise Xo to x1) into two parts with similar recursion: Part I: A given X,, sampled from the diffusion path (xx, X1), encapsulates Ves. which is assumed to be already encoded; and Part 2: The reconstruction from to X1 for learning the Ux] Now, we present the Selftok training objective for an image sample
tokens V21- = [0t, X1 I:
Selftok objective
min Eno(x.) xxxx)-VV VK. [E[0,1]
E
[|x] Dec(xx, V22(e))11]
(5)
Lx xi)
Doc
where V/z(1)) [(k(t)) +1(0)+1, ..y un] and k(() is token schedule with k(1) K +1 and k(0) = 1,
which maps each continuous time-step to discrete token index in Eq. 44). The choices of (xx x1) and k(t) discussed in Section C When the context is clear, use k(t) and i interchangeably.
We highlight that our Sefltok is indeed non-spatial: Vx discretizes the continuous velocity field of the entire image generation path, which is beyond the naïve spatial visual cues. Seltok objective in Eq. (5) optimizes the original one in Eq. ①1) from three aspects: (1 Reconttruttii,; (2)AR Constraint by Recursive Design; (3)AR Constraint by Causal Identification. Detailed explanation are provided
in Appendix A.2 Thus, the inner expectation of Eq. (5) can be rewritten as:
IE (0,1((011)
[[|x] Dec (a) Xo + p(t) X1, V24(o) [13]
We pre-train a VLM based on the AR tokens produced by Selftok. First, we initialize the VLM from the pretrained Llama3-8B 22 model and expand its vocabulary with an additional |C] 32, 768 Selftok visual words. As result, the model's vocabulary integrates both textual and visual tokens into a unified embedding space. Next, the VLM is pre-trained using the standard language modeling objective on interleaved language and visual tokens. We include additional details in Appendix
2.2 Visual RL
In visual RL, we aim to fine-tune VLM (policy) that selects the next token (action) based on the cur- rent sequence (state) to maximize task-specific reward (e.g., the consistency between the text prompt and generated image). Without loss of generality, we limit our discussion to visual tokens [vi..... Uk].
the same principle applies to language tokens. We discuss the recipe for visual RL in detail:
State: The state 8k [ui,...., Uk] is the token sequence generated by VLM at step k E (1....,K] K), and the initial state
Sk
50 is defined as empty sequence.
V k+1
2) Action: An action at step selects the next token Uk+1 from
the visual codebook i.e., each step, there |C] possible actions to choose from.
2
lel
S Sk+1
3) State transition: P(s Uk+1) 1 because Sk+11
[sk, U..1].
Reward: Generally, the reward r(sk, Uk+11) received step k + 1 depends on the previous state (where the reward is from) and the action as the next token Uk+1, predicted at the previous state (how the reward is obtained). With the state and [8k, U++1], we can also write
V
Figure The recursive Bellman equation fails when child node vn (i.e., future token) anti-causally affects parent node Uk41.
action defined above, 8k+11
r(8++1) = r(sg, 11+1).
5) Policy: Given the current state 8s, the policy π a (8k)
predicts action the next token Ux+1- The goal of RL is to find optimal policy x, which
5

---

generates a trajectory so ~ SK that maximizes the cumulative reward (with omitted discount factor):
K--1
max V (50),
where (sx) = E
Σ
r(st
vo+1)
(sx) is the value function, accounting for the expectation of all the possible cumulative rewards received along the trajectory 8k: ~ 8K generated by π.
We show that only AR tokens derive the Bellman equation, which underpins the optimality of policy update that guarantees effective RI
V (50) in Eq. (7) can be rewritten as:
(50)
Σ EC
. ( so) [r(s1) + Va (s1)).
(8)
Therefore, we can recursively apply Eq. (8) and derive the Bellman equation:
(5()
Σ
T(U441|8E)
[r(8k+1) Va (8++1)]
UC+IEC
Thanks to the above equation, the optimized π in Eq. 7 can be step-by-step obtained:
argimax n (18++188)) < argmax (r(sk+1) + (*x+1)].
(10)
2k+1
Although the above policy update is greedy, its optimality is guaranteed by the policy improvement theorem (63,, which shows that the locally optimal action t at step & does not affect the earlier improved actions due to the AR property. Note that non-AR spatial tokens cannot satisfy the Bellman equation, and therefore cannot support the policy update that relies on it. The key reason that Eq. (A15) cannot be derived, as the future action where > k + 1, influences earlier actions through the anti-causal links (shown red in Figure 4). Therefore, spatial tokens are not compatible with RL.
3 Selftok-Zero: Selftok-based Visual RL
We now describe the implementation details for Selftok-based visual RL for visual generation, such text-to-inn text-to-image and visual editing tasks, without using any pairwise supervision, including two reward models for evaluating the quality of the generated images and training objectives for updating the policy network.
3.1 Reward Model
The overall design philosophy of our reward model is to utilize visual comprehension models to evaluate the generated image in visual RL and provide feedback for optimization. For tasks such as text-to-image generation, the comprehension model should understand and evaluate the consistency between the generated image and the textual prompt. In this paper, categorize the comprehension- based reward into two major types:
Program-baset Reward: This type is useful for more structured tasks like object identification, counting, and spatial relationships [20), where the prompt explicitly and unambiguously states the desired generation, e.g., "3 clocks and / dog", and thus generation quality. For example, we count the clocks based on the detector's confidence, returning if the count is correct and otherwise. Each prompt has its item sets to be tested, and the average of the scores for each test is used as the reward score.
visual detectors 7 evaluate the
QA-based Reward: For more complex and ambiguous prompts, challenging rely solely on automated programs. To this end, we resort to more powerful visual comprehension models like Intern [I] GPT-4o [32] which can comprehend nuanced prompts and generate accurate answers. Specifically, inspired by [1). we first decompose the prompt to semantic tuples (e.g., entity, attribute, and relation) and then generate questions (e.g.. "Is the car red?"). The MLLMs are asked to perform a VQA task for the prompt and generated image, returning a score of O to 1 (e.g.. wrong to correct) for each question. The reward is obtained by averaging the evaluation of the MILMs
Details of the derivation provided in Appendix
A.3

---

on multiple questions for prompt. We can also fine-tune such models to obtain more task-specifie reward functions.
As preliminary study, we only validate the feasibility of the above two types. However, we believe that there should be more effective comprehension tasks reward models for better performance, and we leave the exploration of them for future work.
3.2 Policy Gradient
We adopt a simplified version of GRPO [58] without importance sampling and encourage readers to explore more advanced alternatives. For each prompt, the policy network generates batch of outputs {s Ne=1 where B represents the batch size and each st denotes the final state [vo, Vi,..., uk] of the i-th visual sequence. For a batch, we calculate the total rewards {r(s')} where we slightly abuse the notation that the total reward r(s") r(sk)) all the intermediate rewards r(s() = 0, vk < K. We also caleulate the advantages [A, where each A, measures the relative quality of output s² compared to the average reward:
A r(s') mean -me
(11)
std(fr(s'), ..., "()),))))
where mean(-) and (.d are the mean and standard deviation of all rewards, respectively. Then, we update the policy network parameters by the following training loss:
1 C= C B Σ [A (
(12)
where the KL divergence D1 [KL((T| measures the difference between the new policy x and the old policy Told- where the new policy π is the up-to-date one after policy gradient; the old policy wold refers to the one used to generate the token sequences before the policy gradient update.
log - 1 is to maintain training stability. It
4 Related Work
Most visual generative models are trained purely on paired text-image datasets via large-scale pre-training 36 18 68), or additionally with supervised fine-tuning on curated high-quality data
pairs [9 19 Recent efforts have attempted to transition from supervised learning to reinforcement
learning (RL) to better align visual outputs with textual prompts. In the context of diffusion models,
DPOK 17 and DDPO [5] are the first to formulate RL training frameworks, but they do not demonstrate visual generation capabilities in fully open-vocabulary setting. Subsequent works adapt Direct Preference Optimization (DPO)69) to diffusion-based generation, but has mismatched behavior and target policy, leading to poor action-space coverage and inefficient learning. More recent methods based on Guided Reward Policy Optimization (GRPO) 42 77 report promising results, but their policy networks impose restrictive Gaussian assumptions over denoising actions and operate under limited RL horizons (e.g.. 10 denoising steps), which we hypothesize may hinder exploration and constrain the model's ' maximum potential. For AR models, existing approaches are typically built
on either spatial visual tokens [74] 68, (85 or unstructured 1D token sequences (56) that lack explicit constraints to enforce causal ordering. These designs violate the policy improvement optimality in RL, which leads to significantly diminished gains from RL training. In contrast, method builds on AR visual tokens, which enables tractable RL by defining a proper policy via softmax over a discrete, fixed-size action space. These ultimately lead to state-of-the-an performance in open-vocabulary visual generation, as demonstrated in Section
5 Experiment
In this section, we experimentally evaluate the text-to-image generation capabilities of the Selftok- Zero, demonstrating the effectiveness of visual RI. We also provide details the visual RL. training and analyze the impact of various factors the model performance.
7

---

Table 1: Evaluation of text-to-image generation ability GenEval benchmark. represents the result of our evaluation. Janus-Pro- Pro-7B-Zer represents model that has undergone the same visual RL process as Selftok- Peettok-Pre--ae Pre- and Selftok-Zero.
Type
Method
Single Obj.
Two Obj.
Counting
Colors
Position
Color Attr.
Overall
PixArt-a
[
98
50
44
48
SDXL 62
98
74
39
85
15
23
55
FLUX.
36
98
79
73
77
22
45
66
Diffusion Only
DALL-E
59
96
87
47
83
43
45
67
SD3-Medium
15
99
94
72
33
74
CogView4-61B
99
86
66
79
48
73
HiDream-I1 20
100
98
79
91
60
72
83
SEED-X 19
97
58
26
80
19
14
49
Transfusion
Hybrid Model
D-DIT 40
97
80
54
76
32
65
Show-o 76
98
80
666
84
31
50
68
GPT-4o 44
99
92
85
91
75
85
Emu3-Gen
98
34
17
54
TokenFlow-X0 XI.
53
95
60
41
16
24
.5
ILLUME+ 31
99
88
62
84
42
53
72
Infinity 25
85
49
73
Janus-Pro-P7
99
89
80
Pure dAR
Janus-Pro-7HB
98
88
58
88
76
79
98+0
95_7
58.0
8941
90,11
814-1
85..6
Selftok-Pre
99
57
58
81
22
43
60
Scftok-Pre-Zeroo
99
94
58
89
89
73 4:90
84 421
Selftok-SFT
100
79
66
91
45
62
74
Sclftok-eero
99_1
95 -5_
,8,22
94.3
96..5
92
5.1 Implementation details
We perform visual RL (Section 3.1 pre-training checkpoints--eelfeeee trained purely image-text intericaved data and Selftok-SFT with additional fine-tuning on curated dataset-leading two final models SelftokPrre-eero and Selftok-Zens respectively. We evaluate their performance on Geneval [20] and DPG-Bench [30]. For program-based reward, we use MM-Detection [7] as the detectors and the threshold for detection to 0.6. For QA-based reward, we utilize InternVL ( and mPLUG 37 as the comprehension model. Note that we carefully deduplicate the training prompts to ensure that there no overlap with the set. For the sake of reproducibility. after the visual RL training, we do not incorporate any test-time scaling techniques during inference.
5.2 Main Results
The quantitative experimental results summarized in Table and Table 2 which evaluate the performance of our Selftok- based approach on the GenEval and DPG-Bench benchmarks.
Selftok-Zero achieves satte-of-the-rt' performance in text-to-image generation. As shown in Table Selftok-Zero obtains the highest overall score of 92 on the GenEval benchmark, surpassing all previous models, including strong baselines such Cog View4-6B (73) and HiDream-11 (83). Selftok-Zero also outperforms across all major sub-tasks, e.g., Colors (94) and Position (96). Sim- (Table 2 Selftok-Zer -ero achieves overall score of 85.57, outperforming SD3-Medium (84.08) and Janus-Pro-7B 19). The qualitative results are presented in Figure 5 the images generated by Selftok-Zero exhibit high-quality alignment with the textual descriptions.
ilarly,
Visual RL significantly enhances image-text consistency. A direct comparison of Selftok-SFT vs Selftok-Zero and Janus- Janus-Pro-7B Janus Pro-orroo---- 7B-Zero highlights the benefits of visual RL. On GenEval, Selftok-Z4 Zero improves upon its supervised counterpart nearly every metric, with notable gains in Position (45+96) and Counting (66-+88) On DPG-Bench, visual RL leads to a +3.77 increase overall score, with improvements in Entity (from 88. 18.157)91.78 and Relation (from 13.68-995.261 These results indicate that visual RL is effective in closing the gap between generated images and complex textual prompts.
Selftok is more effective than spatial tokens in visual RI. The results in Table and Table show that Selftok significantly outperforms spatial bosen-based methods in visual reinforcement learning (e.g., Janus-Pri Prous--rro-70 Be-Zero +6 vs Selftok-Zero +18 on GenEval). We illustrate the reward score changes

---

Jamus-Pra-22 -Zaoo
Copllewe-dB
Infinity
photo of three pizzas
Counting
photo of four donuts
photo of four TVs
photo yellow broccoli
Colors
photo of dog
photo of zebra
photo of wine glass right of hot dog
Position
photo of dog right of reddy bear
photo of zebra right of porking meter
Color Attribute
photo of red orange and purple broccoli
photo of orange and purple sandwich
photo of white banana and black elephant
Figure 5: Qualitative experimental results of Gelftok-based visual RL. Compared existing text-to- image generation models, the images generated by Selftok demonstrate better alignment with the given prompts.

---

Table 2: Performances DPG-Bench. The methods in this table are all generation-specific models except Show-o, Janus-Pro, and Selftok.
Type
Method
Global
Entity
Attribute
Relation
Other
Overall
PixArt-a
74.97
79.32
78.60
82.57
76.96
71.11
SDXL
52
83.27
82.43
80.91
86.76
80,41
74.65
DALI-E.
59
90.97
89.61
88.39
90.58
89.83
83.50
Diffusion Only
SD3-Medium 15
87.90
91.01
88.83
80.70
88.68
84.08
FLUX.I-dev 36
85.80
86.79
89.98
90.04
89.90
83.79
CogView4-6B 3
83.85
90.35
91.17
91.14
87.29
85.13
HiDream-11
26
76.44
90.22
89.48
93.74
91.83
85,89
Hybrid Model
Show-o
67.48
Emu3-Gen 74
85.21
86.68
84
90.22
83.15
80.60
Janus 75
82.33
87.38
87.70
85.46
86.4
79.68
Infinity 25
93.11
90.76
83.46
Pure dAR
Janus-Pro- 7B
9
86.90
88.90
89.40
89.32
89.48
84.19
Banus-Pro-7B+
83.59
89.74
87.51
92,94
81.20
83,48
Manus-Pro-BB-Zom
84.50..0.91
90.13
87.29 -0.22
93,44.40.50
82,40 41.20
84,49 +1.00
Selftok-Pre
87.41
87 09.09
88.08
87.89
87 $7.42
80.37
SelftokSFFT 82.07
88.15
87.69
93.68
80.40
81.80
Selfiok-Zero
83.59
91.78
89.04....39
95.26
82.80.42.40
85,57.43.77
during visual RL evaluation GenEval and DPG-Bench in Appendix. It is evident that although Janus-Pro- Pro-7B (79) outperforms Selftok-SFT (74) before visual RL, Selftok-Zero comes from behind to surpass Janus-Pro- Prouus-Pro-7B-.. -ero (e.g., +7 on Geneval), thanks to the AR properties of Selftok (see Section 2.I. These results further highlight the significant impact of the image tokenizer design on visual RL.
Program-based reward yields more substantial gains in visual RL. We observe that the improve- ments GenEval (program-based reward) are more pronounced than DPG-Bench (QA-based reward). While Selftok-Zero outperforms Selftok-SFT by +18 in overall score on GenEval (74-92)), the improvement on DPG-Bench is slightly smaller (+3.77, 81.00-85.57) This suggests that program-based rewbed-enabled by structured detectors and precise and more reliable training signals during reinforcement learning, especially for attributes like object counting, color, and spatial layout.
stronger
Qualitative Examples. In Figure 2 we visualize the performance of Selftok-Zeru the DPG test prompt. We also compare our model with MidJourney (67] and FLUX 36. showing that Selftok- Zero performs well in both adhering to complex semanties and generating aesthetically pleasing images. However, it should be noted that the current model can only generate images at a resolution of 256 256, indicating significant potential for improvement in image detail in future work.
Conclusion
In this paper, we introduce Selftok-Zero, an autoregressive (AR) visual generative model trained with reinforcement learning (RL)). built upon Selftok's AR visual token representation. Unlike prior models based on spatial or unstructured token sequences, Selftok- leverages the AR dependency among Selftok tokens enable stable and theoretically grounded policy improvement under RL. By defining well-structured policy over tractable discrete action space, Selftok-Zero eliminates the need for pairwise supervision and enables efficient, end-to-end specific reward signals. Empirically. Selftok-Zent achieves strong results, significantly outperforming existing models on GenEval and DPG-Benc benchmarks. To our knowledge, this is the first work to demonstrate that RL-based post-training substantially enhance the open-vocabulary visual generation capabilities of AR models. As future work, we aim to improve the token generation speed of Selftok-22 -Zero by spatial-temporal compression, and extend Selftok-Zera toward high-resolution generation and physics-anee aware visual reasoning.
optimization using task-
10

---

References
[1] Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.0333 2025.
[2] Meta AI. Llama: Open and efficient foundation language models, Accessed: 2025-04-07.
com
[3] Zhipu AL. Cogview4: Next-generation image creation. https://cogviewt..n 2025.
[4] Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini, Oğuzhan Fatih Kar, Elmira Amirloo, Alaaeldin El-Nouby, Amir 7amir, and Afshin Dehghan. Flextok: Resampling images into 1d token sequences of flexible length. arXiv preprint arXiv:2502.13967 2025.
[5] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. 2024.
[6] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo. Huchuan Lu, et al. Pixart-a: Fast training of diffusion transformer for photorealistic rext-to-image synthesis. arXiv preprint urXiv:2310.00426 2023.
[7] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint XXiv:1906.07155 2019.
[8] Liang Chen, Zekun Wang, Shuhuai Ren, Lei Li, Haozhe Zhao, Yunshui Li, Zefan Cai, Hongcheng Guo, Lei Zhang, Yizhe Xiong, et al. Next token prediction towards multimodal intelligence: A comprehensive survey. arXiv preprint arXiv:2412.18611 2024.
[9] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint 1.K.::22001.781111 2025.
[10] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open- source multimodal models with model, data, and test-time scaling, 2024.
[11] Zhe Chen, Jiannan WU, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/VV conference computer vision and pattern recognition, pages 24185-244 24198, 2024.
[12] Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. arXiv preprint arXiv:2310.188 2023.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.
[14] Shivam Duggal, Phillip Isola, Antonio Torralba, and William T Freeman. Adaptive length image tokenization via recurrent allocation. In First Workshop Scalable Optimization for Efficient and Adaptive Foundation Models, 2024.
[15] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-fire first international conference on machine learning, 2024.
[16] Patrick Fsser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-1222 12883, 2021.

---

[17] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models, 2023.
[18] Yu Gao, Lixue Gong, Qiushan Guo, Xiaoxia Hou, Zhichao Lai, Fanshi Li, Liang Li, Xiaochen Lian, Chao Liao, Liyang Liu, Wei Liu, Yichun Shi, Shiqi Sun, Yu Tian, Zhi Tian, Peng Wang, Rui Wang. Xuanda Wang, Xun Wang, Ye Wang. Guofeng Wu, Jie Wu, Xin Xia, Xuefeng Xiao, Zhonghua Zhai, Xinyu Zhang. Qi Zhang, Yuwei Zhang, Shijia Zhao, Jianchao Yang, and Weilin Huang. Seedream 3.0 technical report, 2025.
[19] Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi multi-granularity comprehension and generation. arXiv preprint arXiv:2404.1453 2024.
[20] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object- framework for evaluating text-to to-ig alignment. Advances in Neural Information Processing Systems, 36:52132-52152, 2023.
[21] Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Syn- naeve. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:240.199337 2024.
[22] Ian Goodfellow, Jean Pouget-Abadie Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 (3(11):139) 13:13-144 2020.
[23] Xin Gu, Ming Li, Libo Zhang, Fan Chen, Longyin Wen, Tiejian Luo, and Sijie Zhu. Multi- reward as condition for instruction-based image editing, 2024.
[24] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948. 2025.
[25] Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis, 2024.
[26] HiDream-ai. Hidream-il. https://hidreance com 2025.
[27] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loie Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230, 2018
[28] Kyle Hsu, William Dorrell, James Whittington, Jiajun Wu, and Chelsea Finn. Disentanglement via latent quantization. Advances in Neural Information Processing Systems, 36:45463-45488, 2023.
[29] Tianyang Hu, Jun Wang, Wenjia Wang, and Zhenguo Li. Understanding square loss in train- ing overparametrized neural network classifiers. Advances in Neural Information Processing Systems, 35: 16495-16666 2022.
[30] Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with 11m for enhanced semantic alignment. arXiv preprint 15X 2024.
[31] Runhui Huang, Chunwei Wang, Junwei Yang, Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Lanqing Hong, Hengshuang Zhao, and Hang Xu. Illumet: Illumi nating unified milm with dual visual tokenization and diffusion refinement. arXiv preprint arXiv:2504.0192: 2025.
[32] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card, arXiv preprint arXiv:2410.21276 2024.
12

---

[33] Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao. Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: physical law perspective. arXiv preprint arXiv:2411 .23350 2024.
[34] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and 2019.
pattern recognition, pages
[35] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Shing-Chagg Chiu, al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312. 2023.
[36] Black Forest Labs. Flux. https:
www.cotcocc......cc.. forestt--foretto. law.orresttlaattttoeeeee ffux 2024.
[37] Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient vision-language learning by cross-modie skip-connections. arXiv preprint arXiv:2205. 2022.
[38] Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37:56424-56445, 2024.
[39] Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregressive image generation with folded tokens. arXiv preprint arXiv:2410.01756, 2024.
[40] Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang. Dual diffusion for unified image generation and understanding. arXiv preprint arXiv:2501.0022 2024.
[41] Wang Lin, Liyu Jia, Wentao Hu, Kaihang Pan, Zhongqi Yue, Wei Zhao, Jingyuan Chen, Fei Wu, and Hanwang Zhang. Reasoning physical video generation with diffusion timestep tokens via reinforcement learning. In submission, 2025.
[42] Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl, 2025.
[43] Xingchao Liu, Chengyue Gong, al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023.
[44] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ractsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14 4124. PMLR, 2019.
[45] Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al. Janusflow: Harmonizing autoregres- sion and rectified flow for unified multimodal understanding and generation. arXiv preprint arXiv:2411.00 109975, 2024.
[46] Edan Meyer, Adam White, and Marlos C Machado. Harnessing discrete representations for continual reinforcement learning. arXiv preprint atXiv:2312.01200 2023.
[47] Keita Miwa, Kento Sasaki. Hidehisa Arai, Tsubasa Takahashi, and Yu Yamaguchi. One-d-piece: Image tokenizer meets contoly-e compression. arXiv preprint arX1v:2501.10064, 2025.
[48] Ron Mokady, Amir Hertz, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/VF conference on computer vision and pattern recognition, pages 6038 6047, 2023.
[49] OpenAl. Introducing 4o image generation. https://openai. ctapiiiccoomiiii////e/e
introducing www.ooinnno---i----eeeeeeeeeeeee image-generation/ 2025. Accessed: 2025-04-22.
13

---

[50] Jiachun Pan, Xingyu Xie, Hanwang Zhang, and Shuicheng YAN. Vr-sampling; Accelerating flow generative model training with variance reduction sampling. 2024.
[51] Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, NY, USA, 2nd edition, 2009.
[52] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Mülllr, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
[53] Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.0336) 2024.
[54] Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language understanding by generative pre-training, OpenAI Blog, 1(8), 2018.
[55] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024.
[56] Kyle Sargent, Kyle Hsu, Justin Johnson, Li Fei-Fei, and Jiajun Wu. Flow to the mode: Mode-seeking diffusion autoencoders for state-of-the-art image tokenization. arXiv preprint arXiv:2503.1 1056, 2025.
[57] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278 252244, 2022.
[58] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
[59] Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu. Improving image captioning with better use of captions. arXiv preprint arXiv:2006.1 1807, 2020.
[60] Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. In International Conference on Machine Learning, pages 32211-32252 PMLR, 2023.
[61] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[62] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525,, 2024.
[63] Richard S Sutton, Andrew G Barto, al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[64] Haotian Tang, Yecheng Wu, Shang Yang, Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, and Song Han. Hart: Efficient visual generation with hybrid autore- gressive transformer. In The Thirteenth International Conference on Learning Representations, 2025.
[65] Chameleon Team. Chameleon: Mixed-mouee modal carly-fusion foundation models, arXiv preprint S.Kiv:2405.091111 2024.
[66] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao. Huabin Zheng, Huan Yuan, Jia Chen, Jianhang
14

---

Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye. Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang. Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang. Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with lmms, 2025.
[67] Midjoumey team. Midjourney. https://www.mdd hww.eurreey....coooo..e.ee.. com/home 2025.
[68] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37:84839-8448 84865, 2024.
[69] Bram Wallace, Meihua Dang, Rafael Rafailov, Lingi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong. Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization, 2023.
[70] Bohan Wang, Zhongqi Yue, Fengda Zhang, Shuo Chen, Li'an Bi, Junzhe Zhang, Xue Song, Kennard Yanting Chan, Jiachun Pan, Weijia Wu, Mingze Zhou, Wang Lin, Kaihang Pan, Saining Zhang, Liyu Jia, Wentao Hu, Wei Zhao, and Hanwang Zhang, Discrete visual tokens of autoregression, by diffusion, and for reasoning, 2025.
[71] Bohan Wang. Zhongqi Yue, Fengda Zhang, Shuo Chen, Li'an Bi, Junzhe Zhang, Xue Song, Kennard Yanting Chan, Jiachun Pan, Weijia Wu, Mingze Zhou, Wang Lin, Kaihang Pan, Saining Zhang, Liyu Jia, Wentao Hu, Wei Zhao, and Hanwang Zhang. Selftok: Discrete visual tokens of autoregression, by diffusion, and for reasoning, 2025.
[72] Kai Wang, Mingjia Shi, Yukun Zhou, Zekai Li, Zhihang Yuan, Yuzhang Shang, Xiaojiang Peng, Hanwang Zhang, and Yang You. A closer look time steps worthy of triple speed-up for diffusion model training. arXiv preprint arXiv:2405. 17403, 2024.
[73] Tan Wang, Zhongqi Yue, Jianqiang Huang, Qianru Sun, and Hanwang Zhang. Self-supervised learning disentangled group representation as feature. Advances in Neural Information Processing Systems, 34: 34:18225-18240. 2021.
[74] Xinlong Wang, Xiaosong Zhang. Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang. Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409. 18869, 2024.
[75] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.13848, 2024.
[76] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528. 2024.
[77] Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong. Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, and Ping Luo. Dancegrpo: Unleashing grpo visual generation, 2025.
[78] Yibo Yang. Stephan Mandt, Lucas Theis, et al. An introduction to neural data compression. Foundations and Trends@ in Computer Graphics and Vision, 15(2):113-200 2023.
[79] Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Randomized autore- gressive visual generation. arXiv preprint 2.11:2211100076,, 2024.
15

---

[80] Qihang Yu. Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang Chich Chen. An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 37: 12814224421266,, 2024.
[81] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
[82] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 1039, 2024.
[83] Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vq-gan to 100,000 with utilization rate of 99%. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[84] Zheng Zhu, Xiaofeng Wang, Wangho Zhao, Chen Min, Nianchen Deng, Min Dou, Yuqi Wang. Botian Shi, Kai Wang, Chi Zhang, et al. Is sora world simulator? comprehensive survey on general world models and beyond. arXiv preprint arXiv:2405.03520, 2024.
[85] Xianwei Zhuang. Yuxin Xie, Yufan Deng, Dongchao Yang, Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou. Vargpt-v1. 1::: Improve visual autoregressive large unified model via iterative instruction tuning and reinforcement learning, 2025.
16

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In the abstract and introduction, We discussed the flaws in current RL in visual generation fields. And introduce new method to improve it in this paper, which accurately match the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In section we point out that our work is limited to the speed of token generation and image resolution. In the future work we will try to improve them.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
17

---

Answer: [Yes]
Justification: As shown in section we provide the full proof that our AR token can derive the Bellman equation.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We introduce the details of our experiment in sectior which can help the readers to reproduce results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the relevant materials, including the source code, to help the community reproduce our result in supplemental materials. The benchmark data comes from the Geneval and DPG open-source benchmark.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
y /nips. /nips.cc//ppp http:/ccccccccguueeeeeeeeeeeeeee wwaccc/guiies/eee httpubii//gi.iee/ddeeeaaaaiiaiia
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide the experiment details in sectior5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: To ensure a fair comparison with the baseline method, we strictly adhere to the testing settings of the Geneval and DPG benchmark.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the detail about the compute resources in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: fttttps://eeuriiie hww./ps......p. weeeurrps.ccpuubl//Bhiitt cc/ppullc/Bthicccuueeellll.
Answer: [Yes]
Justification:
We carefully conform with the NeurIPS Code of Ethics in our code in every
respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the Broader Impacts in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work has no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We include the license of each asset in Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We use LLM as our backbone model and reward model, the details introduced in section[5.1
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hettap//moeursss.ccooooorrrerree for what should or should not be described.
23

---

This appendix is organized as follows:
In Section
We include additional problem formulation, including the use of discrete visual tokens and the Self-consistency Tokenizer (Selftok) in Section
in Section
In Section
we discuss additional implementation details, including the evaluation dataset in
Section B. Selftok implementation in Section [.2] autoregressive (AR) model pre-training in
Section and visual reinforcement learning (RL) in Section [3.4
show additional results that demonstrate the AR structure of Selftok tokens, ablation study on Selftok to justify our choice of hyperparameters, reward progression in visual RL, qualitative results showing that visual RL reduces hallucination, and potential of visual RL in image editing.
In Section
In Section D and
we discuss the current limitations of our work and broader impacts, respectively,
A
Additional Formulation
A.1 Why Discrete?
XE loss
XE loss
MSE loss
<w
<BOV>
CW2
CBOVS
(a) pure dAR
(b) hybrid model (dAR + cAR)
<B08>
<W/
CW2
<BOV>
2V2
<F>>
<B0S>
CW2
<B0V>
<V2
language tokens
discrete visual tokens
language
continuous visual tokens
Figure A6: Comparison of (a) pure discrete autoregressive model (dAR) and (b) hybrid model that combines dAR and continuous autoregressive model (cAR). <BOS>/<BOV> indicates the start of sentence/image. <w;>/<;>> denotes the i-th language/visual token. Both models predict the next token given all previous ones, e.g., [<BOS>,.. <vz>] <u4?
We advocate the of pure discrete autoregressive model (dAR) (Figure Aba), rather than hybrid approach that combines a dAR for language and a continuous autoregressive model (cAR) for images (Figure A6b)
82 38) The latter is widely adopted by proponents who argue that visual data
should be encoded as continuous tokens to minimize the compression loss, but this is just a minor concern-there are many post-progssiing methods available to ensure the precision [14 47 64 39] However, using cAR (or hybrid) leads to major issues that cannot be fundamentally resolved without adopting pure dAR:
cAR cannot inherit the successful infrastructure and training paradigm of LLMs. This is the most common reason cited in existing dAR-based VLMs [65 74
35 Yet, the following three
justifications often overlooked by the community.
cAR is more error-prone in next-token prediction. While dAR functions as a sequential token classifier trained with cross-entropy (XE) loss, cAR operates a sequential vector regressor trained with mean squared error (MSE) loss, which is less stable and harder to optimize than XE 22] g Perhaps this is the key reason why most cARs abandon the causal next-token prediction and revert to bidirectional modeling, such as demasking [38, (79 or holistic reconstruction [82 45)
Unfortunately, they undermine the core design philosophy of the decoder-only AR: the causal dependency of tokens [54].
cAR introduces unnecessary complexity into reinforcement learning (RL). is widely known that RL is an indispensable post-traie post-training step unleash the power of LLMs 24 However, cAR turns the finite Markov Decision Process (MDP) formulation of dAR-with discrete state-action into an infinite MDP with continuous state-action space, thereby complicating policy optimization (46).
space sence-into optimizalon
Continuous representations are less disentangled than discrete ones. Disentanglement uncovers the modular and true generative factors of data 227 733, which are critical for: 1) Unbiased visual comprehension, e.g., if "color" and "object" disentangled, the model can still recognize

---

black swan as swan, even if all the training examples of swans are white; and 2) Controlled generation, if such disentanglement holds, the model can generate a black swan without seeing one in training. Since a real-valued vector is infinitely countable, single continuous token may theoretically entangle all the factor combinations. As a result, achieving disentanglement would require an impractically large amount of training data to cover all the combinations [44] e.g., we need ((N MM) images, where is #values per factor and M is the #factors per image. In contrast, discrete tokens, with their limited information bandwidth, serve as strong inductive bias that encourages disentanglement 228
A.2 Selftok: Self-consistency Tokenizer
Here, we verify that the Seltok objective in Eq. optimizes the original one in Eq. ! from the following three aspects:
1) Reconstruction: When t = 0. Eq. (5) already includes the reconstruction objective in Eq. ①I
by considering || Dee (Vx)
X1 Dec(Xo, VK = V2*(0)=1)]11 because the latter decoder
only takes in new non non-informettve input: the white noise Xo-
2) AR Constraint by Recursive Design: Due to the correspondence between AR and diffusion recursion in Eq. (4) Eq. (5) is recursive breakdown of Eq. (1) by time-step t: V/i is learned from reconstruction X1 -Dec(x, V>*(e))] )1² that completes the path x; ~> X1; whereas the midway point X, encapsulates Vea. which is considered to be already identified by the probability factorization in Eq. (2) and the causal structure in Figure 3a]
Xx. This satisfies
3) AR Constraint by Causal Identification: To ensure that the learned V/ indeed of AR structure,
Le., the encoder identifies the causal effect from Ves to Vzi. we need to justify that Eq. (5) is an unbiased estimate of Vz3 from x; (i.e., Vei) for all t E [0, 1]. To this end, show that Eq. (5) induces the causal graph in Figure Bc| Causation Xg - X <- X1 denotes that x2 is sampled from ((x+(x)) by mixing noise Xo and image X causation x, + V2k(0) +-I denotes that the tokens VzR(1) are learned from X1 and x,. In this way, Xo serves as an instrument variable (IV) [51], independent of the confounder X₁. Recall the re- re-paammttrizaion: x, = ott) X. + put) X1, where (t) and u(t) can be considered as time-specific constants [43] Thus, the inner expectation of Eq. (5) can be rewritten as:
E
[||:
Dec (o(t) Xo + p(t) x1, V2n(o) [13]
(A13)
(0,11
which implies that V22(1) be directly estimated from the IV xo, ensuring that V2*(1)) learned from X₃ is unbiased, even in the presence of the confounder X1.
A.3 RL theoretical derivation
We now show that only AR tokens can derive the Bellman equation, which underpins the optimality of policy update that guarantees effective RL. We start by rewriting our goal V (5a) in Eq. (
V= V=(s))
E
[r(s0, v1) + r(s1, 12) +... +.. r(sk-1,
[e) (60) 81 b on (1)
(A14)
E r(so.01) + E [))))
(rsi, 02) + .... r(sk-1,
Va(s1)
(A15)
Σ T(ui(50) [f(s1) + Va1))
(A16)
VIEC
holds because the transition probability P(8++1|8x: u++1) = 1. As shown in Figure 4 holds because of the causal dependency of AR, where the choice of action #*+1 only depends on and does not affect the former action es that has already been chosen. Therefore, we can recursively apply Eq. (A16) and derive the Bellman equation:
Eq. (A
Eq. A15
V (sx) = Σ T(vk41|sK) [r(sk+1) + Va (su+1)l.
(A17)
VE+iEC
25

---

one-stop renderer
multi-stop diffusion sampler
Image/
512 tokens
1024 talions
seed
seed 2
Encoder
Quantizer
lay B
& MiR
Bay & Wrel
Bi
tokens
Vy
Renderer
MSE
LPPPS
GAN
Figure B7: Left: (a) Reconstructions with one-step renderer (512 or 1024 tokens) and multi-step diffusion sampler (512 tokens, two seeds); Right: (b) Renderer architecture diagram.
B
Additional Implementation Details
B.1
Dataset
We conducted experiments on GenEval 20 and DPG-Bench [30]. GenEval is an object-focused framework to evaluate compositional image properties such as object co-occurrence, position, count, and color, which is under MIT License. models to follow complex prompts, which is under Apache License. It contains diverse and complex prompts and constructs QA pairs based on these prompts. These QA pairs are answered using MLM and then the final score is calculated.
benchmark used to evaluate the ability of
B.2 Selftok
For encoder, We use a dual-stream transformer backbone like MMDiT [5], which consists an im- age stream (blue modules) and token stream (yellow modules). Each stream has its own parameters, specialized for processing patch- patch-based image embeddings and ARsbased token embeddings. The backbone consists of N blocks with identical architecture. For quantizer, similarity, which is updated through an exponential moving average instead of gradient descent. For decoder, we use diffusion model initialized from SD3 [15]. It is dual-stream transformer MMDiT architecture, where the input to the original language token stream is replaced with the quantized embeddings. To remove the original language influence and better adapt Selftok tokens, the weights of our token stream are trained from scratch.
one based on cosine
After training, we apply standard multi-step diffuusinn sampler 15 56] to decode our tokens V into a reconstructed image. However, this process slow as it requires multiple sequential forward passes. To accelerate this, build renderer R(Vx) that reconstructs I in single forward pass. We initialize R with the decoder weights. To remove its dependency on Xt, we replace with a sequence of learnable "canvas" token embeddings shown in Figure B7 (b), which becomes part of the model parameters of R. Then with the learned token embeddings V = EEne(T 7) frozen, we optimize jointly with MSE loss for pixel-level reconstruction, LPIPS [81 and GAN 22 loss for perceptual
quality, as including the latter two resolves the well-known blurry reconstruction issue when training decoder with the MSE loss alone [34 [6]:
min max R(VK)=I'
|I- LLPIIS(I,, A2 (log D(I) + log(1 - D(I')
(B18)
MSE loss
perceptual loss
GAN loss
where X1, A2 are loss weights, D is the discriminator of the GAN. To improve training stability, set A1 = 0.1, Au = O for the first 30k training iterations and X = 0.5, A2 = 0.5 afterwards. As shown in Figure B (a), besides the improved visual perception, the one-step renderer brings two benefits: 1) significantly reduces the image generation time, and 2) it eliminates the randomness introduced by the random seed in diffusion-based generation (see Figure B7 (a)). We train the tokenizer 32 Ascend 910B for 96 hours.
Please refer to 71 for the rest of details.
26

---

Text-to-lmage
Image-to-Ters
<RDE>
Image
<ROV>
<V
<V
<R00>
Image- Onlly
Text-Only
<BOV>
<vi>
<EOV>
<208>
Image Editing
Visual Understanding
<BOV>
<V1>>
<EOV>
qURa t.oon
Figure B8: Illustration of the proposed data format for cross- coss-modality and cross-task pre- pre-training.
B.3 Pre-training
We initialize the VLM from the pretrained Llama3-8B [2] model, which is under META LLAMA COMMUNITY LICENSE, and expand its vocabulary with an additional 32.768 Selftok visual words. As a result, the model's vocabulary integrates both textual and visual tokens into a unified embedding space. The VLM is trained using the standard language modeling objective, which aims to maximize the log-likelihood of multimodal token sequences in an AR fashion:
D
P()) log Po (ve()e)ss
where the sequence Y may consist of interleaved language and visual tokens, and thus y/s y denotes either language token (wa) or visual token (vt)- Since both text and image content represented as discrete token IDs, the prediction head is shared and supervised at each position using cross-entropy loss. The training consists of the following two stages:
Stagel: Cross-modality Pre-training. In this stage, we aim to learn the correspondence between visual tokens and language tokens, thereby facilitating the transition of the pre-trained Llama3 model from LLM to VLM. To achieve this, we introduce four data formats designed address the challenges of cross-modalie alignment. Each format helps the model process and integrate vision and language inputs for coherent multimodal understanding and generation. The Texa-to-Image format aligns caption with visual data, enabling image generation from textual descriptions. Conversely, the Image- to Text format facilitates understanding tasks by associating visual data with textual descriptions. To address potential misalignments that can occur during text-to- rext-to-image tasks, the Image-Onie format is introduced, allowing the model to learn visual structure independently. Finally, the Text- Only data ensures the preservation of the model's linguistic capabilities, maintaining its ability process and generate text. These formats and their functions are summarized Figure B8 with special tokens such as [BOS] and [EOS] marking the sequence boundaries, and [BOV] and [EOV] indicating the start and end of visual data. The training data is comprised of 530 million high-quality image-text pairs and text sequences and we train the model on 400 Ascend 910B for 120 hours.
Stage2: Cross-task Pre-training. this stage, we perform cross-task pre-training to enable the model to learn human instructions across various tasks. This is accomplished through supervised fine-tuning (SFT) on datasets from three distinct tasks: 1) text-to-image generation, 2) image editing, and 3) image understanding. The instruction format follows the structure "USER: iinsrruciinss ASSISTANT: <Answers>", where only the content of <Answer> contributes to the loss function, optimizing the model's is ability provide accurate responses. In Stage2, fine-tune the model using Ascend 910B for hours.
We denote the VLM after stage and 2 as Selftok-Pre and Selftok-SFT, respectively.
B.4 Visual RL
We adopt GRPO 588 setting the coefficient of KL divergence 3 to 0.04 and the size of the group to 24 with synchronous sampling and update frequency. During training, the model is optimized using the AdamW optimizer with learning rate of 1.5e-6. We train the model on 64 Nvidia A800 GPUs for 96 hours.
27

---

Stride-one Shuffle
Stride-two Shuffle
Normal
330
Aooroged bocrage
Iawrgee treeey
Selftok
6 g
12
540
100
233
105
20.
Poditin
Poction
VQGAN-LC
Aaceoyed Erever
Iwersged Erentry
300
11.0
11.0
dverged brewpe
33.
Accoppa Broups
11.0
FlowMo
100
dnoraged trevers
Avwrage Entnds
E
VAR
5
202
753
300
4.10
Woien Pocition
10-07 Rosttion
Thkor Pistiion
Figure C9: Plots of the next-token prediction entropy versus token position for our Selftok, 2D spatial tokens (VQGAN-LC (8). ID tokens (FlowMo (56)), multi-scale 2D tokens (VAR (68)), using the original or shuffled sequences. Only Selftok exhibits a segmented decreasing trend that aligns with the three sequence orders. Although VQGAN-LC also displays segmented trend, each segment is not decreasing. Conversely, while FlowMo shows a decreasing trend, it is not segmented under the shuffled orders.
C Additional Results
Semantic interpretability. We find that tokens corresponding to smaller time-steps tend to capture the overall background, color tone composition of the image, those at middle ones tend to capture object shapes and those at larger ones tend to capture fine- ane-grained details and textures. This is because the diffusion process itself is tightly linked with visual semanties [1,2,3], and Selftok simply encode the process as tokens, as shown in Figure CT0
AR structure. We empirically verify that the structure of Selftok is AR by plotting the token predic- tion entropy curves w.r.t. token positions under three generation orders using dAR model (Llama 3.1). Besides the normal sequential order [VI, U2, u3., we use another two orders: 1) stride-one shuffle, which a concatenation of subsequence [Ui, .... followed by subsequence [V2, 14, U6, and 2) stride-two shuffle, which is concatenation of subsequence [11, 74, u... [V2, Us, ..., and [va, Up. ..... The design principle of these orders is simple: an ordered subsequence of an AR sequence is still AR. As entropy measures the uncertainty in token prediction, if the sequence is AR, the entropy trend is generally decreasing. Therefore, if the token sequence is AR, the two shuffled orders should demonstrate segmented decreasing curve. As shown in Figure C9 can see that only Selftok demonstrates such a segmented decreasing trend corresponding to the three sequence orders.
Time sampler. For time sampler, besides the simple uniform sampling, SD3 [5] introduces the logit-normal time-step sampler by assigning higher probability density to mid-range time-steps 0.5). We compared the reconstruction performance when using uniform and logit-normal sampling in Table C3 which shows that the simple uniform sampling performs the best for Selftok.
28

---

Seiftok
VOGAN-LC
Progressive Reconstruction
Flowhlo
VAR
Seittok
VGGAN-LC
Interpolation
FlowMo
VAR
Figure C10: Progressive reconstruction (left to right): Reconstructions by progressively masking out shorter sequence of tokens before inputting the decoder. Interpolation (left to right): Reconstruc- tions by gradually replacing tokens of the left image with those the right one. All methods except Selftok exhibit strong spatial characteristics (i.e,, tokens<>patches).
29

---

500
Table C3: Ablation on time sampler and token schedules. 'sampl'' and 'sched.' denote 'sampler' and 'schedule'.
lognorm
Time sampl.
Token sched.
PSNR
SSIM
LPIPSI
uniform
custom
21.86
0.600
0.150
200
uniform
uniform
21.10
0.564
0.177
uniform
logit-normal
20.78
0.555
0.180
logit-normal
custom
20.98
0.561
0.170
logit-normal
uniform
19.89
0.498
0.205
0.25
logit-normal
logit-normal
20.08
0.513
0.196
Figure C11: Token schedule k(t).
"lognorm" denotes logit-normas
Table C4: Reconstruction performance of different tokenizers on 256 256-resolution ImageNet 50k validation set. I Results from the original paper.
Tokenizer
Type
#Token
#Code
FFID
PSNRT
SSIM
LPIPS
LlamaGen 662
2D
16x 16
214
2.19
20.67
0.589
0.132
Cosmos
2D
32x32
2 216
0.87
24.82
0.763
0,070
VAR 668
2D
680
212
0.99
22.12
0.624
0.109
TYTok-S-122
80
ID
128
212
1.71
17.52
0.437
0,210
FlexTok 4
ID
256
64,
000
1.45
18,53
0.465
0,222
FlowMo-Hii
[66
ID
1,024
2
0.56
24.93
0.785
0.073
Selftok (Ours)
ID
1,024
2
0.54
26.30
0.805
0.063
Token schedule k(tt.. Recall that the AR constraint in Eq. D requires that every token must
conform to the decomposition P(Vx) AR P(Ve)) P(Vz,(Ves). Vi [1, 1]. We achieve this decomposition by diffusion time-steps, thanks to the recursive nature of the reverse diffusion process in Eq. 4 denoted as Viz + ~ X1 and Ves + ~4 Xx. That is to say. the second-half tokens Vz4 can be learned recursively by the diffusion decoder, conditioned on Xt. which represents the already identified first-half tokens Vei. As we uniformly sample & E [0, in training, the best token schedule should be uniform assignment h (t) = [t x K1 is involved in the recursive diffusion time-step. To better understand this, we provide three failure cases: If we allocate all the tokens V21, i.e., k(t) 1, E [0, 1), this corresponds
ensure that every token
trivial []) P(Vx(V<1 = [). Vx < ~4 X1, and + Xg. where we always input the full VK to the decoder. So, Vx loses the AR property. This case reduces to the FlowMo approach [56]. 2) If we always allocate all tokens to Ve1, i.e. k(() = K + 1, 1I (0, 1]. this corresponds another trivial decomposition P(Vk). P(DVK)) + Xg X1 and VK < Xo- where we always send empty sequence to the decoder. This case reduces to the unconditional diffusion generation without learning Vx at all. 3) Consider a non-extreme case where &(t) is not uniformly aligned with e.g., kit 0.8) [0.2 KJ, we disrespect the decomposition because the majority of tokens 22.00xxKK corresponds to dense time-steps in the short interval E [0.8, 1], while the rest in V V< R corresponds sparse time-steps, violating the balanced recursive correspondence in Eqq. 4 We explored three different choices for k<()) 1) the uniform one with k(t) = [t x KJ + 1; 2) custom schedule that allocates few tokens to small t; and 3) a logit-normal schedule that allocates few tokens to both small and large t. We plot k(t) in Figure and compare the performance of the models trained with each schedule in Table C3 However, in practice, we empirically observe better reconstruction quality by designing schedule k<tt) that allocates fewer tokens to smaller t, i.e., k<(t) < k- (t) for t < 0.5. This aligns with the well-known trait of diffusion models: the early path
decomposition P(V=1
X2 for a small t has minimal impact on the reconstruction x; X1, which can be omitted 72 50
Tokenizer metrics. Encoding and decoding single image with the Selftok tokenizer requires only 0.86 and incurs computational cost 2.59 TFLOPs, underscoring the efficiency our approach. Quantitative comparisons with other tokenizers are provided in Table C4 our tokenizer achieves
30

---

(a) GenEval scores of different methods.
(b) DPG scores of different mod- (c) Geneval scores of models trained
cls. *Model trained only with the with different KL-divergence coeffi- cients.
program- based reward.
Methods
GenEvval Score
SDXL
53.8
Model
DPG Score
KLL coefficient
Geneval Score
SDXL. Diffusion-PPO
56.3 (+2.5)
Sef
81.80
0
SelftokSFFT
74 92 (+18)
Selfiok.P'
82.43
0.05
92
Selftok- Zero
Selftok-Zero
85.57
87
40
Reoordd
Rewerd
SelfBbk Zero
Zero
70-
Stepp (a) The reward progression evaluated on GenEval.
Step (b) The reward progression evaluated DPG-Bench.
Figure C12: Comparison of reward progression over steps for Selftok (Selftok-Zero) and spatial tokens (Janus-rro-7B--eero) on GenEval and DPG-Bench.
state-of-the-arr performance, attaining the best results to date rFID (0.54), PSNR (26.30), SSIM (0.805), and LPIPS (0.063).
Selftok is more effective than spatial tokens in visual RL. Figure C12 illustrates the reward score changes during visual RL evaluation on GenEval and DPG-Bench. It evident that although Janus-Pro-7B (79) outperforms Selftok-SFT (74) before visual RL, Selftok-Zero comes from behind to surpass Janus- Pro-7B-Ze Zero (e.g., +7 on Geneval), thanks to the AR properties of Selftok. These results further highlight the significant impact of the image tokenizer design visual RL.
Ablation Results. We conducted three categories of ablation studies: 1) Online vs. offline policy. Using SDXL (52) with Diffusion-DPP 669] as the offline baseline, we observe in Table C5a that Diffusion- underperforms our method, likely because the sample trajectories are misaligned with the model's optimization trajectories. 2) Reward function comparison. We compare training with only the program-based reward against training with both types of rewards. As shown in Table C5b combining the two rewards provides more comprehensive learning signal and yields superior performance. 3) Kl--divegeence ablation. We examine the effect of the KI. divergence (Table (5c):: removing the KL term leads to highly unstable training, whereas increasing the KL coefficient slows convergence. Accordingly, we set the KL coefficient to 0.05 in our experiments.
Hallucination in Text-to-Image Generation. One of the challenges in text-to-image to-image generation the "hallucination" issue, where Vision-Language Model (VLM) tends to generate images that closely follow the training data distribution rather than genuinely reason about the text prompt. This can lead to the model failing to generate certain objects or scenes that are less common or not well- represented in the training set. In Figure|CT CT3 we provide examples where the Selftok-SFT model fails to generate certain objects due to the rarity these combinations in the training data. However, after applying visual RI. (Selftok-Zero), the model is able to generate these previously missing combinations, showing a significant improvement in handling rare or complex prompts. The ability of Sclftok- -Zro to generate these images after the visual RL phase highlights how reinforcement learning can effectively overcome the hallucination problem, improving the model's generalization and reasoning capabilities beyond the initial supervised training.
Visual RL for Image Editing. To further unlock the potential of our model, we also incorporate Vi- Model (LLM)-specffically
sual RL into the image editing task, where we utilize Intern VLB -as the reward model. This model evaluates whether the generated image strictly follows the instructions and accurately modifies the source image. Inspired by the work of 23 18) we ask the reward model return score between 0 and 5, with indicating the highest level of adherence to the instructions. In a few hundred steps, our Selftok-Zert model shows
31

---

Selftok- SFT
Selftok-Zers
photo of purple
photo of orange
photo of white arange
photo of square shaped orange
opple tree
handbog
square shoped square-shoped opples
green carrot
Figure C13: More examples of failures in the Selftok-SFT model due to distributional biases in the training data during vision- iision-language supervised training.
Input
SeltokkIFT
Selitok-Zero
Input
SelftokSFTT
Selftok-Zaro
Replace
color
Chonge the off atting lncation
equirai reading
woman's har straght insteod carly
Remove
Change the surface from
roAbbe Fing
Figure C14: Qualitative experimental results of Selftok- visual RL on image editing. Compared to the Selftok-SFT, the images generated by Selftok- -Zro demonstrate better alignment with the given instructions and better visual fidelity.
significant improvement over the Selftok-SFT model. As shown Figure C14 our model can correctly correspond to the instructions and generate appropriate edited images.
Unlike text-to-ine tomgge generation, image editing involves more nuanced transformations, making significantly more challenging to evaluate automatically. The complexity arises from the need to assess both the fidelity of the edits to the original image and the accuracy of the applied changes according to the given instructions. Therefore, to provide a more general and accurate reward for image editing tasks, plan explore more sophisticated reward models that handle the intricacies of image modification. Additionally, we aim to develop refined evaluation principles that better capture the subtlety and precision required in image editing. This will be key focus in future work, where we hope to improve the reliability of automated assessments and provide more meaningful feedback.
D Limitations
The primary limitation does not lie in Selftok itself, but rather in the significantly slower token generation speed LLMS compared diffusion models. For instance, when using 512 tokens per frame, generating a one-minute video clip at 24 fps would require generating x12x224 60 = 737, 280 tokens-posing a substantial throughput challenge. Fortunately, we are optimistic that this issue will be mitigated by introducing spatial-tempora compression, in conjunction with the rapid progress in real-time massive token generation within the L.M community 21). Another limitation of this work stems from the restricted model scale. Due to limited capacity, we have not yet demonstrated Selftok's
32

---

Original Image
(512x512)
DO
Selftok ($12 tokens, 258x256)
O0
Selftok (768 tokens, 512x512)
O
Figure C15: Qualitative results of 512 x 512 resolutions.
ability to transfer visual knowledge language and realize multimodal emergent capabilities. resources permit, we plan to investigate the scaling laws of multimodal training with Selftok, aiming validate its potential for cross-modal synergy. Next, we highlight our two ongoing works for Selftok:
Multi-resolution Selftok. The current resolution of Selftok limited 256 256, which constrains the quality of visual generation. Our design follows an incremental principle: higher-resolution images are supported by increasing the number of tokens, while reusing the tokens extracted from their lower-resolution counterparts. This enables efficient scalability, allowing resouutresolution data to leverage dAR model pre-trained on resoltienolution inputs. This approach particularly appealing, as it parallels the practice in LLM training, where longer document training benefits from presents our preliminary results, which will be included in
prior training on shorter texts. Figure C15 the future work.
Physics-aware Post-training Inspired by the impressive performance gains of visual RL by using the program-based reward, our next step is to incorporate physical laws into Selftok-based video generation. For example, we can track the trajectories of moving objects and evaluate whether they conform to fundamental motion principles. This direction has great potential in addressing the ever-lasting criticisms that large visual models struggle learn a true world model [84 333 In our
recent work, we demonstrated that Selftok can achieve near- wear-perfect object motion generation in a toy visual environment 41
E Broader Impacts
Ethical Impacts. Our work does not raise any ethical concerns. The research does not involve subjective assessments or the use of private data. Only publicly available datasets and models are utilized for experimentation.
Expected Societal Implications. Our work proposes an effective method apply reinforcement learning in visual generation. A major societal concern with this method lies in its potential for misuse. For example, some malicious individuals may exploit our method to train model generate violent or pornographic images. To counteract such threats, it is crucial to develop strong ethical standards and stricter regulation.
33


Reviews:
[{"review_id": "fmTSSBpRa3", "reviewer": "Reviewer_EWpd", "summary": "This paper introduces Selftok-Zero, an autoregressive (AR) visual generative model trained with reinforcement learning (RL)), built upon Selftok’s AR visual token representation. Unlike prior models based on spatial or unstructured token sequences, Selftok-Zero leverages the AR dependency among Selftok tokens to enable stable and theoretically grounded policy improvement under RL.", "questions": "See weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n- The proposed Selftok-Zero method addresses the violation of the Bellman equation in the traditional AR tokenizers, enabling stable and theoretically grounded policy improvement under RL.\n- The Selftok-Zero model performs well on image generation benchmarks like GenEval and DPG-Bench.\n\n**Weakness**：\n- Robustness of the Tokenization Scheme. The core of Selftok lies in establishing a correspondence between the Autoregressive (AR) structure and the diffusion process. To what extent does this correspondence depend on the choice of the diffusion process q(xt|x1) and the token schedule k(t)? Could different scheduling strategies (e.g., linear, cosine) or noise types significantly impact the quality and structure of the final AR tokens? \n- Semantic Interpretability of Tokens. While traditional spatial tokens may be causally flawed, their positional information makes them somewhat interpretable. For the AR token sequence generated by Selftok, do individual tokens or consecutive token segments correspond to specific, interpretable semantic concepts in the image (e.g., object contours, textures, or parts)? Could the authors provide a visual analysis to reveal the correspondence between these AR tokens and the image content?\n- Reward Model Design and Bias. The two proposed reward models (Program-based and QA-based) are clever, but they may introduce jejich own biases. For instance, the Program-based Reward depends on the performance of a specific detector; if the detector performs poorly on certain objects, the RL process could be misguided. Similarly, the QA-based Reward relies on the inherent biases and knowledge limitations of the \"judge\" VLM (e.g., GPT-4o). How do the authors assess and mitigate the potential biases introduced by these reward models themselves?\n- Direct Comparison with Competing Works. A core argument of the paper is that methods like Diffusion-DPO are inefficient due to the mismatch between the behavior and target policies. Could the authors design an experiment to directly compare the performance of Selftok+RL against Diffusion-DPO under identical settings?\n- Quantitative Evaluation of Encoder Quality and Efficiency. To provide a comprehensive evaluation of the proposed tokenizer, could the authors report on several key quantitative metrics(rFID, the time or FLOPs required to encode a single image)?", "was_revised": true, "final_justification": "Based on the discussion with the authors, most of my concerns are resolved, except for the absence of a fair comparison with online DPO algorithms. I will raise my score, but I still strongly recommend the authors to provide such fair comparison in later versions.", "rebuttal": "Thank you very much for your constructive comments and invaluable evaluation. We are pleased to see that you have raised very critical questions regarding our work, and our responses are as follows:\n\n## W1：Robustness of the Tokenization Scheme.\n\nNice observation! The choice of the diffusion process $q(x_t|x_1)$ and the token schedule $k(t)$ should ensure that all decompositions $P(V_K)=P(V_{{\\lt}i})P(V_{{\\geq} i}|V_{{\\lt}i})$ are accounted for in training. For example, consider a failure case where $x_t=x_0$ for any $t\\neq1$, or a token schedule such that all tokens are allocated to $V_{{\\ge}1}$, $i.e.$, $k(t)=1$, $∀𝑡 ∈ [0, 1)$. This corresponds to a trivial decomposition$P(V_{{<}1} = [\\space])·P(V_K|V_{{<}1} = [\\space])$, $V_K ⇔ x_0 ⇝ x_1$, and $[\\space] ⇔ x_0$ where we always input the full $V_K$ to the decoder. So $V_K$ loses all the AR property.\n\nFor the forward process, we adopt rectified flow ($i.e.$, $x_t=tx_1+(1-t)x_0$), as we initialize our decoder from SD3. We are happy to try other scheduling strategies and noise types as future work.\n\nFor token schedule, as we uniformly sample $𝑡 ∈ [0, 1]$ in training, the best token schedule should be a uniform assignment $𝑘^∗(𝑡) = ⌈𝑡 × 𝐾⌉ + 1$ to ensure that every decomposition is equally respected in the training objective. We perform ablations in **Token schedule** of **Appendix C**. Interestingly, we observe a slightly better reconstruction quality by designing a schedule $𝑘(𝑡)$ that allocates fewer tokens to smaller $𝑡$, $i.e.$, $𝑘(𝑡) < 𝑘^∗(𝑡) \\space for \\space 𝑡< 0.5$. This aligns with the well-known trait of diffusion models: the early path $x_0 ⇝ x_𝑡$ for a small $𝑡$ has minimal impact on the reconstruction $x_𝑡 ⇝ x_1$, which can be omitted.\n## W2：Semantic Interpretability of Tokens\n\nYes, we find that tokens corresponding to smaller time-steps tend to capture the overall background, color tone or composition of the image, those at middle ones tend to capture object shapes and those at larger ones tend to capture fine-grained details and textures. This is because the diffusion process itself is tightly linked with visual semantics [1,2,3], and Selftok simply encode the process as tokens.\n\nWe would love to show you some visualizations that demonstrate the interpretability of our tokens, but we are unable to do so due to the constraints of the conference policy. We will follow your suggestion to include those figures in the revision. Meanwhile, you can check similar results in **Figure 1** of [1].\n\n[1] Yue Z, Wang J, Sun Q, et al. Exploring diffusion time-steps for unsupervised representation learning[J]. arXiv preprint arXiv:2401.11430, 2024.\n\n[2] Preechakul K, Chatthee N, Wizadwongsa S, et al. Diffusion autoencoders: Toward a meaningful and decodable representation[C]. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10619-10629.\n\n[3] Zhang Z, Zhao Z, Lin Z. Unsupervised representation learning from pre-trained diffusion probabilistic models[J]. Advances in neural information processing systems, 2022, 35: 22117-22130.\n\n## W3：Reward Model Design and Bias\n\nGreat point! We agree that this could introduce potential biases. In this work, we did not explicitly address those biases, as our focus is to demonstrate that AR tokens enable effective visual RL, whereas spatial tokens are less effective in this regard. And it is worth noting that even with this potential deficiency, we still achieve state-of-the-art performance. This highlights the potential of visual RL with Selftok tokens, and we will improve the RL pipeline in our future work.\n\n## W4：Direct Comparison with Competing Works\n\nWe fine-tuned a diffusion model (SDXL) with Diffusion-DPO using the same reward model on the GenEval benchmark. To ensure the identical settings, we train SDXL with the same amount of data, and we used prompts and images to construct positive and negative pairs, whereas Selftok-Zero only use prompts. The results are as follows:\n\n|       Methods        | GenEval Score |\n| :------------------: | :-----------: |\n|         SDXL         |     53.8      |\n| SDXL + Diffusion-DPO |  56.3(+2.5)   |\n|     Selftok-SFT      |      74       |\n|     Selftok-Zero     |    92(+18)    |\n\nThe table above shows that Diffusion-DPO does not exhibit a significant improvement(+2.5) on the GenEval benchmark, while our method leads to a +18 increase in overall score, which further demonstrates the effectiveness of our method.\n\n## W5：Quantitative Evaluation of Encoder Quality and Efficiency\n\nThe comparisons with other works, as well as the results of quantitative metrics related to the Tokenizer, are as follows:\n\n|     Tokenizer      | Type | #Token |   #Code   |  rFID↓   |   PSNR↑   |   SSIM↑   |  LPIPS↓   |\n| :----------------: | :--: | :----: | :-------: | :------: | :-------: | :-------: | :-------: |\n|      LlamaGen      |  2D  |  1024  | $2^{14}$  |   0.59   |   24.44   |   0.768   |   0.064   |\n|       Cosmos       |  2D  |  1024  | $≈2^{16}$ |   0.87   |   24.82   |   0.763   |   0.070   |\n|        VAR         |  2D  |  680   | $2^{12}$  |   0.99   |   22.12   |   0.624   |   0.109   |\n|    TiTok-S-128     |  1D  |  128   | $2^{12}$  |   1.71   |   17.52   |   0.437   |   0.210   |\n|      FlexTok       |  1D  |  256   |   64000   |   1.45   |   18.53   |   0.465   |   0.222   |\n|     FlowMo-Hi      |  1D  |  1024  | $2^{14}$  |   0.56   |   24.93   |   0.785   |   0.073   |\n| **Selftok (Ours)** |  1D  |  1024  | $2^{15}$  | **0.54** | **26.30** | **0.805** | **0.063** |\n\n| Time(s) | TFLOPs |\n| :-----: | :----: |\n|  0.86   |  2.59  |\n\n\nFrom the table above, it can be observed that our tokenizer has achieved the best performance to date, reaching optimal levels in metrics such as rFID (0.54), PSNR (26.30), SSIM (0.805), and LPIPS (0.063). Additionally, it boasts high computational efficiency, requiring only 0.86 seconds to encode and decode an image."}, {"review_id": "YSpUnjEj7j", "reviewer": "Reviewer_c2No", "summary": "This paper identifies and addresses a fundamental challenge in applying reinforcement learning (RL) to visual generation. The authors argue that existing paradigms are ill-suited for RL: diffusion models suffer from intractable policy trajectories, while standard autoregressive (AR) models using spatial tokens violate the Bellman equation, undermining the optimality of policy improvement. To resolve this, the paper proposes leveraging \"Selftok,\" a tokenizer that represents images as truly autoregressive 1D token sequences. By training a Vision-Language Model on these tokens and then applying a policy gradient algorithm with a hybrid (program- and QA-based) reward function, the resulting model, \"Selftok-Zero,\" achieves state-of-the-art performance on text-to-image generation benchmarks without requiring paired data during the RL phase.", "questions": "1.  Could you provide a more intuitive explanation of why Selftok's diffusion-based encoding ensures a stricter AR property compared to a simple raster-scan ordering of spatial patches? What is the key mechanism that prevents the \"collider effect\" mentioned in the introduction?\n\n2.  The reward function is a critical component. Did you perform any ablations on the composition of the reward model? For instance, how does the model perform on a complex benchmark like DPG-Bench if trained only with the program-based reward, or vice-versa?\n\n3.  Training large models with RL can often be unstable. Did you encounter any significant stability challenges during the policy gradient optimization? How critical was the KL-divergence term (`DKL(π||πold)`) in Eq. 10 for achieving stable convergence?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\n1.  The core strength of this paper is its clear and insightful diagnosis of why conventional visual tokens are problematic for RL. The argument that spatial tokens induce anti-causal dependencies that violate the policy improvement theorem (Section 2.2) is a fundamental contribution and provides a strong theoretical motivation for the proposed approach.\n\n2.  This work is among the first to demonstrate that RL can be a highly effective post-training step for AR visual generative models. The \"Selftok-Zero\" concept—achieving significant performance gains purely through RL without supervised fine-tuning on new pairs—is powerful and opens a promising direction for aligning generative models with complex human preferences.\n\nWeakness\n\n1.  The success of the method is heavily dependent on the Selftok tokenizer, which is a complex system introduced in prior work. The main paper provides limited intuition on *how* Selftok's diffusion-based process creates truly autoregressive tokens, making the core component feel somewhat like a black box.\n\n2.  The quality of Selftok-Zero is ultimately capped by the capabilities of the external models used for reward (e.g., InternVL, MM-Detection). Any biases, errors, or limitations in these reward models will be directly inherited and amplified by the policy. This is a general limitation of RLHF-style methods but remains a key consideration.\n\n3. The current model generates images at a 256x256 resolution, which is low by modern standards.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable comments and insightful questions. Our responses are as follows:\n\n## W1：Supplementary Content for Tokenizer\n\nWe apologize for the insufficiency of content related to the tokenizer; our work focuses on emphasizing that autoregressive (AR) tokens is the key for reinforcement learning (RL) in visual generation. Thank you again sincerely for your suggestion—we will include more details about the tokenizer in the main text.\n\n## W2：Reward Model Design and Bias\n\nGreat point! We agree that this limitation has potential influence on the policy. And we apologize that in this work, we did not explicitly address those biases, as our focus is to demonstrate that AR tokens enable effective visual RL, whereas spatial tokens are less effective in this regard. And it is worth noting that even with this potential deficiency, we still SOTA. This highlights the potential of visual RL with Selftok tokens, and we will improve the RL pipeline in our future work. This issue may need to be addressed in future work through better reward models and large-scale RL.\n\n## W3：Image Resolution\n\nThanks for pointing that, actually our tokenizer can be extended to 512x512, as shown in **Figure C10** in **Appendix**. We also plan to extend it to higher resolutions in future work. Furthermore, achieving competitive results at 256x256 resolution is more challenging, as generating image details is more difficult; this further illustrates the effectiveness of our method.\n\n## Q1：Explanation of Selftok \n\nWe would love to provide a causal graph that makes the explanation easier, but couldn't do so due to rebuttal policies. We will try our best to explain intuitively to you. Recall that the collider effect arises as spatial tokens are encoded by conditioning on the original image $x_1$. In our work, the key mechanism to circumvent this problem is to introduce a random noise $x_0$ independent from $x_1$. From this $x_0$, we learn tokens that gradually guide the reverse diffusion process from $x_0$ to $x_1$. Specifically, each token is learned to enable an additional step towards $x_1$, from a position $x_t$ that is dictated by **previous tokens**. This naturally leads to causal dependencies among all tokens.\n\n\n## Q2：Reward Ablation\n\nThis issue is indeed a critical component. We will show that models trained solely with program-based rewards do have improvements on DPG benchmark, with the specific results as follows:\n\n|     Model     | DPG Score |\n| :-----------: | :-------: |\n|  Selftok-SFT  |   81.80   |\n| Selftok-P$^*$ |   82.43   |\n| Selftok-Zero  |   85.57   |\n\n$*$ represents model trained only with the program-based reward\n\nWe will also continue to explore more settings related to rewards in future work. Thanks again for your valuable question.\n\n## Q3：Training Stability\n\nNice question! We have indeed encountered challenges that significantly impact the stability of RL training. Specifically, we found that training becomes highly unstable when the learning rate is high; therefore, we set the learning rate to $1.5e−6$. Furthermore, training also becomes unstable when there is a mismatch between the model's sampling and update frequencies, so we maintain consistency between these frequencies to avoid this issue.\n\nWe also explored the effects of three different KL divergence coefficients, with the results shown as follows:\n\n| KL coefficient | Geneval Score |\n| :------------: | :-----------: |\n|       0        |       \\       |\n|      0.05      |      92       |\n|      0.1       |      87       |\n\nWe found that when the KL coefficient was 0, training became very unstable. We tried our best to set hyperparameters, but we were unable to stabilize the training. When the KL coefficient was 0.1, the model's convergence speed slowed down, and it was unable to exceed the GenEval score when the KL coefficient was 0.05."}, {"review_id": "106aUKvw5l", "reviewer": "Reviewer_EyKQ", "summary": "The paper proposes to use Selftok (Self-consistency Tokenizer), which represents each image as a sequential 1D stream of discrete, autoregressive tokens. Then, the authors train a pure AR VLM and claim that a simple policy gradient algorithm applied to Selftok tokens significantly boosts model performance.", "questions": "1. I think the description in Section 2.1 lines 92–96 is too brief, which makes it difficult for a broader audience to understand. I believe that incorporating the analogy and explanation from Appendix lines 67–76 into the main paper would significantly improve comprehension. \n\n2. Meanwhile, Section 2.2 might be overly detailed. For example, lines 135–141 do not present new content introduced by this paper, but rather a classic derivation in reinforcement learning.\n\n3. In Appendix A line 87, there is a minor typo.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Overall, I believe this is a technically solid paper, with high impact on visual generation, with novel method, excellent evaluation and reliable reproducibility. \n\nHowever, there is also a weakness. Currently, the design of the Selftok tokenizer includes some interesting analogies, such as those in equations (A3) and (A4) in Appendix A. If the paper could provide rigorous theoretical justification or explanation for the validity of these analogies, it would make the work even more compelling.", "was_revised": true, "final_justification": "Overall, I believe this is a technically solid paper, with high impact on visual generation, with novel method, excellent evaluation and reliable reproducibility.\n\nIn the rebuttal, the authors resolved all my concerns, and therefore I will maintain my score of 5.", "rebuttal": "Thank you very much for your valuable suggestions and positive feedback. We will revise our paper in accordance with your recommendations.\n\n## W1：Supplementary Content for Tokenizer\n\nThank you for pointing that out! For the theoretical justification similar to our work, please refer to [1]. The theorem in this paper proves that the loss of image attributes increases as more noise is added. This thus enables adding tokens in a reverse-autoregressive manner to make up for the loss of attributes.\n\nWe apologize for the insufficient coverage of content related to the tokenizer. Our work aims to emphasize that AR tokens can still support effective RL-based post-training for visual generation, so we condensed the content regarding the tokenizer. Thank you again sincerely for your suggestions. We will clarify this part in the revised version.\n\n[1] Yue Z, Wang J, Sun Q, et al. Exploring diffusion time-steps for unsupervised representation learning[J]. arXiv preprint arXiv:2401.11430, 2024.\n\n## Q1：Supplementary Content for Tokenizer\n\nWe sincerely apologize for the insufficient coverage of content related to the tokenizer. We will add more details about the tokenizer in the main text to make this section more comprehensive. Thank you again for your suggestion.\n\n## Q2：RL-related Content\n\nWe sincerely apologize for the redundancy in the proof of RL theory. We will correct it in the revised version. Thank you very much for your suggestion.\n\n## Q3：Typo\n\nThank you for pointing out the typo, we will correct it in the revised version."}, {"review_id": "tEjBnx9dCv", "reviewer": "Reviewer_Bvhq", "summary": "The paper proposes an autoregressive tokenization formulation for images that relies on the reverse diffusion process. It also proposes a visual RL based generation method based on its tokenization formulation. Through experiments, it shows that the proposed method achieves improved text-to-image generation capabilities and the tokenization formulation is effective for visual RL, enabling them to apply RL-based post-training for autoregressive models.", "questions": "- Why is a raster-scan based solution problematic for the policy improvement optimality in RL?\n- What is the key reason behind Selftok-Zero's results?", "limitations": "N/A", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "First, I would like to point out that after multiple attempts the paper felt very foreign to me, and I am aware it could simply be that I am not familiar with some of the related works. I am open to discussing with ACs on outcomes based on my review.\n\nOne strength of the paper that stands out is that learning AR tokenization with reverse diffusion _makes sense_ and the authors showcase this idea backed by good results. The writing felt very _formal_, but for the right people, this could be appreciable. From my perspective, I would expect a clearer explanation of the two motivation put in the introduction. From its current portrayal, the motivation did not make sense to me. Many important parts are in appendix, e.g., the authors mention in line 92 \"To make this tractable, we derive in Appendix a\ncorrespondence between AR structure and reverse diffusion process by writing them in a recursive form\", for the discussion of building a tokenizer, I would expect a clear understanding of the method without going to the appendix.", "was_revised": true, "final_justification": "I have decided to keep the my original score because the writing was hard to follow.", "rebuttal": "Thank you very much for your constructive questions and valuable suggestions. We are delighted to discuss our work with you, and our responses are as follows:\n\n## W1：Clarification on tokenizer and motivation \n\nWe are sorry that due to page limitations, we had to place some content related to the tokenizer in appendix. Thank you for your suggestion; we will revise the paper to include more details about the tokenizer in the main text to make it easier for readers to understand.\n\nThe main point of the paper is to demonstrate that autoregressive (AR) tokens is the key for reinforcement learning (RL) in visual generation. In the introduction, we explain why RL is less effective for diffusion models and AR models based on spatial tokens. To address this gap, we reveal that the reverse diffusion process is inherently AR through a recursive decomposition. This allows us to learn AR tokens that support effective visual RL. \n\n\n## Q1：Why is a raster-scan based solution problematic for the policy improvement optimality in RL?\n\nAs shown in **Figure 1** of our paper, the raster-scan based tokens do not follow the AR causal dependencies, as they are essentially inter-dependent, $e.g.$, knowing one image patch will leak information of other patches. We prove that this violates Bellman equation in **Section 2.2**.\n\n## Q2：What is the key reason behind Selftok-Zero's results?\n\n\nAs discussed in **W1**, our work aims to show that AR tokens enable effective RL-based post-training for visual generation. We constructed AR tokens as visual representations based on the autoregressive diffusion process, aligning with the policy improvement theorem. Experiments on GenEval and DPG benchmarks show its effectiveness: only AR tokens meet the conditions for the existence of an optimal solution, while other non-AR tokens yield weaker RL results—highlighting our work's efficacy."}]

Author rebuttal:

Meta-review: The authors introduce SelfTok-Zero, an AR model that builds on top of previous tokenizer SelfTok, enhanced with RL-based post-training. Effectiveness of the proposed method is demonstrated on GenEval and DPG-Bench.

Initially, the reviewers raised several concerns, which are briefly outlined below:

* Reviewer EWpd: Tokenization robustness, interpretability of tokens, reward model design, method comparison, tokenizer evaluation.

* Reviewer c2No: Dependency on prior work Selftok tokenizer and reward models, limited resolution generation.

* Reviewer EyKQ: Writing and method clarification.

* Reviewer Bvhq: Method's motivation.

The rebuttal and subsequent author-reviewer discussions effectively addressed most of the reviewers' concerns. After carefully considering the reviews, rebuttal, and discussion, the AC concurs with the reviewers’ assessment and thus recommends acceptance of the paper. Finally, the authors are encouraged to incorporate the rebuttal experiments into the manuscript and address the reviewers’ feedback (e.g., the writing refinement) in the final revision.
Decision: Accept (poster)