Title: Selftok-Zero: Reinforcement Learning for Visual Generation via Discrete and Autoregressive Visual Tokens
Venue: NeurIPS 2025
Accepted: yes
Keywords: Tokenizer

MLLM

Reinforcement Learning
Abstract: Reinforcement learning (RL) has become an indispensable post-training step for unlocking the full potential of Large Language Models (LLMs). Its core motivation is to incentivize the model’s inference trajectory via a reward model, effectively balancing the exploration–exploitation trade-off in scenarios where collecting exhaustive input–output ground-truth pairs is infeasible. This motivation naturally extends to visual generation, where perfect alignment between an image and a textual prompt is inherently ambiguous and often unattainable. However, existing visual generative models are not yet ready for RL due to the following two fundamental drawbacks that undermine the foundations of RL: 1) For diffusion-based models, the actual generation trajectories of sampled images cannot be reliably rewarded, as diffusion inversion is notoriously difficult. 2) For autoregressive (AR) models, we show that the widely used spatial visual tokens do not satisfy the Bellman equation and thus violate the policy improvement theorem of RL. To this end, we propose to use Selftok (Self-consistency Tokenizer), which represents each image as a sequential 1D stream of discrete, autoregressive tokens. Together with language, we train a pure AR vision-language model (VLM) for visual generation. Impressively, without using any text-image training pairs, a simple policy gradient algorithm applied to Selftok tokens significantly boosts visual generation performance, surpassing existing models by a large margin. Implementation details are provided in the Appendix.


Peer reviews:
comments: SUMMARY:
The paper proposes an autoregressive tokenization formulation for images that relies on the reverse diffusion process. It also proposes a visual RL based generation method based on its tokenization formulation. Through experiments, it shows that the proposed method achieves improved text-to-image generation capabilities and the tokenization formulation is effective for visual RL, enabling them to apply RL-based post-training for autoregressive models.

STRENGTHS AND WEAKNESSES:
First, I would like to point out that after multiple attempts the paper felt very foreign to me, and I am aware it could simply be that I am not familiar with some of the related works. I am open to discussing with ACs on outcomes based on my review.

One strength of the paper that stands out is that learning AR tokenization with reverse diffusion _makes sense_ and the authors showcase this idea backed by good results. The writing felt very _formal_, but for the right people, this could be appreciable. From my perspective, I would expect a clearer explanation of the two motivation put in the introduction. From its current portrayal, the motivation did not make sense to me. Many important parts are in appendix, e.g., the authors mention in line 92 "To make this tractable, we derive in Appendix a
correspondence between AR structure and reverse diffusion process by writing them in a recursive form", for the discussion of building a tokenizer, I would expect a clear understanding of the method without going to the appendix.

QUESTIONS:
- Why is a raster-scan based solution problematic for the policy improvement optimality in RL?
- What is the key reason behind Selftok-Zero's results?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 1
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-28

comments: SUMMARY:
The paper proposes to use Selftok (Self-consistency Tokenizer), which represents each image as a sequential 1D stream of discrete, autoregressive tokens. Then, the authors train a pure AR VLM and claim that a simple policy gradient algorithm applied to Selftok tokens significantly boosts model performance.

STRENGTHS AND WEAKNESSES:
Overall, I believe this is a technically solid paper, with high impact on visual generation, with novel method, excellent evaluation and reliable reproducibility. 

However, there is also a weakness. Currently, the design of the Selftok tokenizer includes some interesting analogies, such as those in equations (A3) and (A4) in Appendix A. If the paper could provide rigorous theoretical justification or explanation for the validity of these analogies, it would make the work even more compelling.

QUESTIONS:
1. I think the description in Section 2.1 lines 92–96 is too brief, which makes it difficult for a broader audience to understand. I believe that incorporating the analogy and explanation from Appendix lines 67–76 into the main paper would significantly improve comprehension. 

2. Meanwhile, Section 2.2 might be overly detailed. For example, lines 135–141 do not present new content introduced by this paper, but rather a classic derivation in reinforcement learning.

3. In Appendix A line 87, there is a minor typo.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies and addresses a fundamental challenge in applying reinforcement learning (RL) to visual generation. The authors argue that existing paradigms are ill-suited for RL: diffusion models suffer from intractable policy trajectories, while standard autoregressive (AR) models using spatial tokens violate the Bellman equation, undermining the optimality of policy improvement. To resolve this, the paper proposes leveraging \"Selftok,\" a tokenizer that represents images as truly autoregressive 1D token sequences. By training a Vision-Language Model on these tokens and then applying a policy gradient algorithm with a hybrid (program- and QA-based) reward function, the resulting model, \"Selftok-Zero,\" achieves state-of-the-art performance on text-to-image generation benchmarks without requiring paired data during the RL phase.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n1.  The core strength of this paper is its clear and insightful diagnosis of why conventional visual tokens are problematic for RL. The argument that spatial tokens induce anti-causal dependencies that violate the policy improvement theorem (Section 2.2) is a fundamental contribution and provides a strong theoretical motivation for the proposed approach.\n\n2.  This work is among the first to demonstrate that RL can be a highly effective post-training step for AR visual generative models. The \"Selftok-Zero\" concept—achieving significant performance gains purely through RL without supervised fine-tuning on new pairs—is powerful and opens a promising direction for aligning generative models with complex human preferences.\n\nWeakness\n\n1.  The success of the method is heavily dependent on the Selftok tokenizer, which is a complex system introduced in prior work. The main paper provides limited intuition on *how* Selftok's diffusion-based process creates truly autoregressive tokens, making the core component feel somewhat like a black box.\n\n2.  The quality of Selftok-Zero is ultimately capped by the capabilities of the external models used for reward (e.g., InternVL, MM-Detection). Any biases, errors, or limitations in these reward models will be directly inherited and amplified by the policy. This is a general limitation of RLHF-style methods but remains a key consideration.\n\n3. The current model generates images at a 256x256 resolution, which is low by modern standards.\n\nQUESTIONS:\n1.  Could you provide a more intuitive explanation of why Selftok's diffusion-based encoding ensures a stricter AR property compared to a simple raster-scan ordering of spatial patches? What is the key mechanism that prevents the \"collider effect\" mentioned in the introduction?\n\n2.  The reward function is a critical component. Did you perform any ablations on the composition of the reward model? For instance, how does the model perform on a complex benchmark like DPG-Bench if trained only with the program-based reward, or vice-versa?\n\n3.  Training large models with RL can often be unstable. Did you encounter any significant stability challenges during the policy gradient optimization? How critical was the KL-divergence term (`DKL(π||πold)`) in Eq. 10 for achieving stable convergence?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Selftok-Zero, an autoregressive (AR) visual generative model trained with reinforcement learning (RL)), built upon Selftok’s AR visual token representation. Unlike prior models based on spatial or unstructured token sequences, Selftok-Zero leverages the AR dependency among Selftok tokens to enable stable and theoretically grounded policy improvement under RL.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n- The proposed Selftok-Zero method addresses the violation of the Bellman equation in the traditional AR tokenizers, enabling stable and theoretically grounded policy improvement under RL.\n- The Selftok-Zero model performs well on image generation benchmarks like GenEval and DPG-Bench.\n\n**Weakness**：\n- Robustness of the Tokenization Scheme. The core of Selftok lies in establishing a correspondence between the Autoregressive (AR) structure and the diffusion process. To what extent does this correspondence depend on the choice of the diffusion process q(xt|x1) and the token schedule k(t)? Could different scheduling strategies (e.g., linear, cosine) or noise types significantly impact the quality and structure of the final AR tokens? \n- Semantic Interpretability of Tokens. While traditional spatial tokens may be causally flawed, their positional information makes them somewhat interpretable. For the AR token sequence generated by Selftok, do individual tokens or consecutive token segments correspond to specific, interpretable semantic concepts in the image (e.g., object contours, textures, or parts)? Could the authors provide a visual analysis to reveal the correspondence between these AR tokens and the image content?\n- Reward Model Design and Bias. The two proposed reward models (Program-based and QA-based) are clever, but they may introduce jejich own biases. For instance, the Program-based Reward depends on the performance of a specific detector; if the detector performs poorly on certain objects, the RL process could be misguided. Similarly, the QA-based Reward relies on the inherent biases and knowledge limitations of the \"judge\" VLM (e.g., GPT-4o). How do the authors assess and mitigate the potential biases introduced by these reward models themselves?\n- Direct Comparison with Competing Works. A core argument of the paper is that methods like Diffusion-DPO are inefficient due to the mismatch between the behavior and target policies. Could the authors design an experiment to directly compare the performance of Selftok+RL against Diffusion-DPO under identical settings?\n- Quantitative Evaluation of Encoder Quality and Efficiency. To provide a comprehensive evaluation of the proposed tokenizer, could the authors report on several key quantitative metrics(rFID, the time or FLOPs required to encode a single image)?\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}