Group Critical-token Policy Optimization for Autoregressive Image Generation
Authors: Guohui Zhang
Hu Yu
Xiaoxiao Ma
JingHao Zhang
Yaning Pan
Mingde Yao
Jie Xiao
Linjiang Huang
Jie Huang
Feng Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: Autoregressive Image Generation
Text-to-Image Generation
Reinforcement learning
Primary area: generative models
TL;DR: A GRPO method designed for autoregressive image generation

Abstract: Recent studies have extended Reinforcement Learning with Verifiable Rewards (RLVR) to autoregressive (AR) visual generation and achieved promising progress.
However, existing methods typically apply uniform optimization across all image tokens, while the varying contributions of different image tokens for RLVR's training remain unexplored. 
In fact, the key obstacle lies in how to identify more critical image tokens during AR generation and implement effective token-wise optimization for them. 
To tackle this challenge, we propose $\textbf{G}$roup $\textbf{C}$ritical-token $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{GCPO}$), which facilitates effective policy optimization on critical tokens. 
We identify the critical tokens in RLVR-based AR generation from three perspectives, specifically: 
$\textbf{(1)}$ Causal dependency: early tokens fundamentally determine the later tokens and final image effect due to unidirectional dependency;
$\textbf{(2)}$ Entropy-induced spatial structure: tokens with high entropy gradients correspond to image structure and bridges distinct visual regions;
$\textbf{(3)}$ RLVR-focused token diversity: tokens with low visual similarity across a group of sampled images contribute to richer token-level diversity. 
For these identified critical tokens, we further introduce a dynamic token-wise advantage weight to encourage exploration, based on confidence divergence between the policy model and reference model.
By leveraging 30\% of the image tokens, GCPO achieves better performance than GRPO with full tokens.
Extensive experiments on multiple text-to-image benchmarks for both AR models and unified multimodal models demonstrate the effectiveness of GCPO for AR visual generation.

Published as a conference paper at ICLR 2026
GROUP CRITICAL-TOKEN POLICY OPTIMIZATION FOR AUTOREGRESSIVE IMAGE GENERATION
Guohui Zhang', Hu Yu', Xiaoxiao Ma', Jinghao Zhang Yaning Pan', Mingde Yao", Jie Xiao', Linjiang Huang Jie Huang'. Feng Zhao"!
'University of Science and Technology of China, 2Shanghai Innovation Institute
Fudan University, *The Chinese University of Hong Kong, 'Beihang University
ABSTRACT
Recent studies have extended Reinforcement Learning with Verifiable Re- wards (RLVR) to autoregressive (AR) visual generation and achieved promising progress. However, existing methods typically apply uniform optimization across all image tokens, while the varying contributions of different image tokens for RLVR's training remain unexplored. In fact, the key obstacle lies in how to iden- tify more critical image tokens during AR generation and implement effective token-wise optimization for them. To tackle this challenge, we propose Group Critical-token Policy Optimization (GCPO), which facilitates effective policy op- timization on critical tokens. We identify the critical tokens in RLVR-based AR generation from three perspectives, specifically: (1) Causal dependency: early tokens fundamentally determine the later tokens and final image effect due uni- directional dependency; (2) cntrropy-idduee spatial structure: tokens with high entropy gradients correspond to image structure and bridges distinct visual re- gions; (3) RIVRRfocused token diversity: tokens with low visual similarity across a group of sampled images contribute richer token-level diversity. For these identified critical tokens, we further introduce dynamic token-win wise advantage weight to encourage exploration, based on confidence divergence between the pol- icy model and reference model. By leveraging 30% of the image tokens, GCPO achieves better performance than GRPO with full tokens. Extensive experiments multiple text-to- text-to-image benchmarks for both AR models and unified multimodal models demonstrate the effectiveness of GCPO for AR visual generation. Code is available https: thub. http:/thw...c.ccoh.cet..eece httpm//ghhui//GGG
INTRODUCTION
Visual generative models (Sun et al. 2024 Liu et al. 2024a Ma et al. 2024) based on the autore- gressive (AR) paradigm have made significant progress in the field of high-quality image genera- tion. Meanwhile, Reinforcement Learning (RL) with Verifiable Rewards (RLVR), demonstrated by OpenAl-o1 (OpenAI] 2024) and DeepSeek R1 (Guo et al. 2025a) to enhance the reasoning abilí- ties of large language models (LLM) (Yang et al. [025 Team et al. 2025). is now being gradually introduced into the visual generation to improve preference alignment and task controllability.
Recent works apply RLVR, especially Group Relative Policy Optimization (GRPO) (Shao et al. 2024) for text-to-image generation by designing visual SraiinsoffTThuglhe (Co'T) (Tang et al.) 2025). optimizing reward functions (Yuan et al. 2025). and constructing customized datasets (Pan et al., 2025a). Despite these advances, these methods typically assume that each token contributes equally to the RLVR's s training objective and apply uniform policy optimization across the entire image token sequence. While different tokens play distinct roles in text-to-image generation: some tokens deter- mine and correspond to the global structure of the image, while others correspond to backgrounds or details. Concurrent RLVR- LLM reasoning works (Wang et al. 2025d a) also realized such a functional distinction between tokens, and split them into reasoning ceaeniing-rlllttee related critical tokens and remaining knowledge-related tokens, where the former have higher entropy and dominate reasoning
"Corresonniigg author.

---

Published as a conference paper at ICLR 2026
ability. While this analogy highlights a shared imbalance in token importance, visual generation bears higher complexity due to the causal AR modeling and bidirectional image structure.
In this paper, we idemtifl critical tokens in -LVR-based AR generation from three perspectives: 1) Causal dependency of AR; 2) Entropy-indueee spatial structure; and 3) RLVR- R- token diversity. Specifically, 1) Early generated tokens continuously influence subsequent tokens and have a significant impact on the overall image structure due to the causal attention mechanism, as shown in Fig. [ 2) For the token sequence of each image, we initially attempt correlate entropy with image token, similar to the operations in (Wang et al. 2025d). While, we find that the distribution of high/low entropy tokens didn't 't consistently correspond to certain parts of the image, like structure or background. Pushing one step further, we observe that the token entropy gradient map demonstrates a consistent spatial pattern among images, with the high value approximately corresponding to the structures and bridging distinct visual regions, see Fig. 2 which is sensitive to RI. 3) Within the group of GRPO, we observe that tokens deliver varying diversities for the same position along images. As shown in Fig. 3 tokens corresponding to background and texture regions tend to exhibit higher similarity, while tokens with lower similarity correspond to more complex region structures.
On the basis of our critical token selection strategy, we introduce Group Critical-token Policy Optimization (GCPO), a novel RLVR framework for AR image generation that facilitates effec- tive policy optimization critical image tokens. During each optimization step, GCPO first selects the critical tokens following the above three perspectives. Then, we further devise a dynamic token- level advantage weight for critical tokens better encourage exploration. This dynamic weight is based on confidence divergence of critical tokens between the updating policy model and reference model, which differs from the standard GRPO that allocates advantage uniformly for each token. Finally, we only retain the policy gradients of critical tokens to perform policy optimization.
By utilizing only critical tokens (30% of the tokens), GCPO achieves better performance than GRPO with full tokens on multiple text-to-image generation benchmarks. Extensive experiments demonstrate the effectiveness of GCPO, including Geneval, 12I-CompBench, and Human Prefer- ence Benchmark, which is also verified on both AR models and unified multimodal models. In summary, our contributions are as follows:
We identify critical tokens in RaVe- AR visual generation from three perspectives: Causal dependency of AR, Entropy-induced spatial structure, and RLVR-focused token diversity, achieving a structure-centric and comprehensive critical-tokes selection strategy.
We propose Group Critical-token Policy Optimization (GCPO), a RLVR framework for autoregressive image generation that facilitates effective policy optimization on critical image tokens. We further devise the dynamic advantage weight strategy for critical tokens to enable reasonable exploration and stabilize the inherently generative prior, based on their confidence divergence between the policy model and the reference model.
GCPO is applicable to both AR models and unified multimodal models. Extensive exper- iments demonstrate that GCPO, by optimizing only critical tokens (30% of the tokens), achieves better performance than GRPO with full tokens across multiple T2I benchmarks.
2 RELATED WORK
Autoregressive image generation models (Sun al. 2024 Team 2024 Ma et al. 2024 Pan et al. 2025b) adopt the next token prediction paradigm, which has been widely applied in LLMs to enable text-to-image generation. Representative works (Ramesh) et al. 2022 Liu et al., 2024a) typically first use an image tokenizer (Esser et al.) 2021) to discretize continuous image data into sequence of tokens, and then employ transformer architecture to model visual tokens. Furthermore, recent research (Huang et al., (025) has focused on unifying image generation and image understanding within a single architecture. These models are capable of accepting diverse types of input (e.g., text, image, video) and producing one more modalities as output. MetaMorph (Tong et al. 2024) utilizes SigLIP to extract visual embeddings and introduces moddalit-sppecfii adapters for more efficient cross- modal alignment. Janus-Pro (Chen et al. 2025a) adopts a dual-encoded structure that separately processes textual and visual data. Transfusion (Zhou et. 2024h, Show-o (Xie et al. 2024), and BAGEL (Deng al. 2025) further combine the strengths
Autoregressive Visual Generation.
2

---

Published as a conference paper at ICLR 2026
of Transformer and Diffusion architectures for multimodal understanding and generation, achieving superior performance actoss various tasks.
Reinforcement Learning for Visual Generation. Reinforcement Learning with Verifialle Re- wards (RLVR) has achieved significant progress in the field of large language models (LLMs). A series of open-source models (Guo et al. 2025a Team et al. 2025) and RLVR methods (Yu et al., 2025c Yue et al. 2025 (Shrivastava et al. 2025 Zeng et al. [0025a] Pan et al. (025c) have been pro-
posed, further advancing the development of this field. Meanwhile, recent efforts (Liu et al. 2025 2025 Pan et al. 2025a) have increasingly explored the potential of RLVR, especially Group Relative Policy Optimization (Shao et al., 2024) (GRPO), in the field of visual generation. SimpleAR (Wang et al., (0025b) has demonstrated that GRPO can significantly enhance the aesthetic quality and prompt alignment of AR models. T2i-R1 (Jiang et al. (025) leverages GRPO to jointly optimize both semantic-level and token-level Chain-of-TThuuu Thought (CoT)) reasoning processes, thereby improving the generative capabílities of unified multimodal model. In addition, (Gallici Borde, 2025) introduces GRPO into scale-wise visual generative models that achieve high-quality image generation by predicting the "next scale" and demonstrating great potential (Han et al. 2025).
Xue et al.
Reinforcement Learning for Critical tokens. Recently, several studies (Li et al. 2025 Vassoyan
ct al. 2025| Wang et al. 2025c; Zhao et al. 2025 Zeng et al. 20225b] Ma al. (025) have focused on the deeper analysis and exploration of the role of RL in LLMs' reasoning task, especially at the token level. Critical Tokens Matter (Lin et al. 2024) suggests that identifying and replacing critical tokens can significantly improve the model's 's accuracy, and proposes contrastive estimation method to accurately locate these tokens. ConIPO (Yonn et al. 2025) further investigates the effectiveness of selectively optimizing only low-confidence and information-rich tokens. Additionally, (Wang al. 2025dl) a) points out the existence of "fork" tokens in LLM reasoning paths, indicating that these tokens typically have high entropy and are related to logical reasoning. They further observe that these fork tokens (critical tokens) can be identified by entropy and play more critical role than other tokens in enhancing the I.I.'ss reasoning ability. However, identifying critical tokens in AR visual generation and implementing effective token- wise optimization for them remains unexplored.
3 PRELIMINARY
Autoregressive Image Generation. A common autoregressive (AR) model includes two main com- ponents: an image tokenizer and an AR transformer (Esser et al. 2021 Yu et al. 202T 2025b a). For image tokenizer, typically VQ-VAE (Van Den Oord et al. 2017). it converts images I E #xxxx into discrete tokens sequence Z ( 2N), where each token ze E V. V and t represent the VQ-VAE codebook and token index in sequence, respectively. Next, the AR model autoregres- sively predicts the joint distribution of the next image token conditioned the text and previously generated tokens: P(a|c) = I P(zelzce.c) c). where represents text embedding.
Token Entropy. The AR model outputs probability distribution over the codebook V for each token. Therefore, we can use the following formula calculate the entropy (Shannon 1948) of this distribution, which is referred to as the token entropy H:
H(2,) = Σ Pt. log Pt,k- k=1
Group Relative Policy Optimization (GRPO). For each prompt GRPO (Shao et al. 2024 sam- ples a group of image outputs for ,o? ....... from the old policy model Te...- Then, computes the corresponding rewards (r ....... for each output within each group. The advantage A is calcu-
lated from the rewards of the group, and each output the group shares the same advantage. Then optimizes the policy model Test by maximizing the following objective:
TaRPO ( 1 ( ~M0
1
lo'
min (ri(0)A), clip (r1 A+) - 33 -KD (ooi(tollleet) )
(2)
i=
where
A' mean(fr)) (fr', ,r..... (10
To q, g'o2
ri(0)=
(3)
std(fr) ,2,... G
To (o) (g,0=2)
3

---

Published as a conference paper at ICLR 2026
4 OBSERVATION AND ANALYSIS IN AR VISUAL GENERATION
4.1 CAUSAL DEPENDENCY OF AR
The capability of AR models stems from the causal atten- tion mechanism in transformers. Under this next token prediction paradigm, the initially generated tokens continuously influence the generation of all subsequent tokens, thereby significantly impacting the overall structure and layout of the image. To further validate this point, we inject additional noise into the tokens at different posi- tions during the generation process (Beyer al. 2025). This causal influence is visibly illustrated in Fig. [ where perturbations to the 58) introduce substantial
SOUrCE trage
Perturb Index 0-56
early 58 tokens (token index from changes to the image's global structure, while perturbations to the middle 58 tokens (token index from 250 to 308) only affect local details. This empirical evidence confirms that early tokens serve as global priors and structural guides. In contrast, later tokens are constrained by both preceding tokens and local consistency, making them more focused on generating local content and details. There- fore, the initial tokens should be part of critical tokens, these early decisions propagate throughout the entire AR generation and establish the foundation for high-quality visual structures.
500-558
Figure 1: Visual results of perturbing different tokens.
4.2 ENTROPY-INDUCHS SPATIAL STRUCTURE
Concurrent RLVR-based LLM reasoning works (Wang al. 2025d a) have found a functional dis- tinction between text tokens, and split them into reasoning related critical tokens and knowledge- related tokens based on token entropy. The former with high entropy (such as "wait", "however") serve as logical connectors that bridge consecutive reasoning parts in CoT, while the latter with low entropy primarily capture factual or sppecic knowledge.
Inspired by this, we first analyze the entropy distribution of image tokens. As shown in Fig. 2 we observe that the distribution of high/low entropy tokens corresponds to different parts of the image in different prompts: tokens with high entropy mainly correspond to the background in Fig. 2 (a), while corresponding to the subject in Fig. (b). This phenomenon gradually becomes obvious in RL training. We argue that it is likely influenced by image and prompt complexity, and we provide more analysis results in Sec. B Furthermore, we find that this entropy distribution in images exhibits a re- gional and spatial pattern, where tokens within the subject background regions have approximate entropy values. We argue that image tokens exhibit strong spatial locality, with neighboring pixels sharing similar visual characteristics and entropy values (He et al. 2024 Xiang & Fan 2025
Prompt: The imoge depicts gold filigree tree of life in detailed fantasy painting.
Prompt: photo of bird
7:00
1.5
125
100
Cenerntee Image
RL step
RL. step 1800
Genwneed
xtap.
800
Rl. xhop 1600
Figure 2: The entropy maps of images exhibit distinct spatial patterns. Tokens with high entropy gradient consistently correspond to image structure, which gradually strengthens with RL training.
Pushing one step further, we analyze the 2D gradient of entropy and observe consistent spatial pat- terns among different images. Specifically, tokens with high entropy gradients typically correspond to subject structure or regions with significant structural variation, and this pattern becomes more pronounced with RL training. These tokens exhibit large entropy changes in neighboring tokens, serving as connectors to link distinct visual regions. Based on this insight, point out that tokens with high entropy gradients are critical and sensitive for spatial structures, and entropy gradients can serve universal and reliable proxy for identifying tokens associated with image structure.
4

---

Published as a conference paper at ICLR 2026
4.3 RLVR-FOCUSED TOKEN DIVERSITY
GRPO typically relies on sample-level reward signals, where the differences between samples guide the direc- tion of policy optimization. A group of similar sam- ples provides limited reward information, which restricts the model's s performance improvement and training effi- ciency. In light of this, (He al. 2025 Yu al. 2025c Chen et al. 20256) focus on enhancing sample diversity through entropy regularization or encouraging semantic diversity. In AR visual generation task, we further fo- cus on token-level diversity. Within the group of GRPO, observe that tokens deliver varying diversities for the same position along images. As shown in Fig. 3] tokens in background and texture regions tend to exhibit higher similarity and hardly reflect the visual differences among images. While tokens with lower similar- ity correspond to more complex regional structures and contribute richer information for GRPO- based policy optimization. Therefore, we select tokens with low similarity as part of critical tokens.
prompt: A white palor bear cub wecninn sunglasics site in noodow with flowers.
03
(6) Croup images
(b) Group Tokan simiarlly
Figure 3: The average cosine similar- ity of tokens at corresponding location across group of images.
4.4 DYNAMIC ADVANTAGE WEIGHT
Balancing exploration with stabilization generative pri- is crucial for different tokens during RLVR train- ing (Wang et al. 2025a). We argue that different criti- cal tokens should have dynamic and distinct exploration constraints. Specifically, initial tokens should explore more moderately prevent global structural collapse, while tokens with high entropy gradients and low simi- larity should have stronger exploration. Considering that the reference model itself serves as the starting point for training, we analyze the confidence divergence of each to- ken between the training policy model and the reference model. As shown in Fig. we observe that this divergence is not only dynamic as the policy model updating, but also has distinct constraints: initial tokens exhibit smaller confidence divergence, while tokens corresponding to structures show larger divergence. Based on this, we utilize this dynamic divergence weight eliminate complex manual specification.
prompt: a photo block hot dog
a2
Figure 4: Confidence divergence be- tween the Policy and Reference model
GROUP CRITICAL-TOKEN POLICY OPTIMIZATION
5.1 TOKEN SELECTION
The Sec. discusses the critical image tokens selection from three aspects. Here, we detail the token selection strategy, as illustrated in Fig. Specifically, denote the image token sequence as = {21...,, 2N), where N is the total length of the sequence. The initial image tokens is represented Zinit {21....; ZKmah with Kinit indicating the number of initial tokens.
Subsequently, we reshape the entropy sequence {H associated with these tokens into 2D entropy map select structure-related tokens. To mitigate the influence of noise in the entropy map, we perform a local averaging operation. Considering the local spatial and causal dependency of image tokens, the averaging is performed as follows:
H. mean H, + (L.))) H!") + (T.M) + H
(4)
where (1, uI,u), (u), (r, u) and (1) denote the upper-left, upper. upper-right and left neighboring posi- tions of H,, respectively. Then, in the average entropy map. We select the K,truet tokens with the largest gradients as Zstruct-
the central difference to calculate the gradient of each token
Next, we calculate the cosine similarity of token embeddings at each sequence position within group of images. Specifically, for each token sequence position t, we consider the group of token embed- dings (er,1, ...2..... } derived from G images. The pairwise cosine similarity between token
5

---

Published as a conference paper at ICLR 2026
See 4.1
Prompt: white deg
depandinnc
Moroo
Nlve Yates Bood, (Pe))
Model
2k-(8)
C
CP 0.
Estropy-iddioed spotial structure
GCPO
Rechape
Sec 5.2
Satract
Fatrasy
Zeleet Zra Zxtrast Ziem
Mean
Sec. 52
Grod (11) Getrrll Gred
Decock
A
RLVR facued
diseriity
Zate
Sas
Group Computation
Reward Model
mosked token
Zist
intial Tokers
Zerut
Tuklons with higher ertinapy gradien!
Zain
Tokees with lower
Siniaanlly
Figure 5: Overview GCPO. GCPO first generates a group of images for each prompt and obtains the corresponding reward. For token selection, first select the initial token Zini- Then, calculate the local mean and central difference gradient in 2D entropy map, and select the token with the higher gradients Zstruct- Subsequently, based on intra-gronn cosine similarity at each position. For dynamic advantage weight, we calculate the cumulative mean confidence difference of each token the advantage weight w. Finally, only retain the policy gradients of critical tokens sellect to perform policy optimization.
select the tokens with lower similarity Zomi
embeddings at position / is calculated as follows:
S(1)
Ct,j Ct.k
i
(5)
| ex
We then calculate the average pairwise similarity S, for each sequence position t. We select the K Kiimilaiity tokens with the lowest average based on S.
Finally, the overall critical token selection set is defined as the union of the three subsets:
Zrelect Zinit U Zetruct U Zeam-
(6)
By default, the size of each subset Kinit Kstruct and Ksim is set to 10% of the total token sequence length, ensuring balanced and representative selection.
5.2 DYNAMIC ADVANTAGE WEIGHT
The Sec. [.44 discusses the motivation of the dynamic advantage weight. Furthermore, considering that the token at position is predicted by its preceding tokens, we further employ the cumulative average of confidence divergence the weight for each critical token. This can be formalized as:
W1 Σ clip (Cratiey -Cjet, -wr Fw)
where wi denotes the advantage weight at position t. Opolicy and Chef represent the confidence (model's log probability) of the j-th token on the policy model and the reference model, respectively. ew is the clip coefficient prevent excessive weights from influencing training stability.
5.3 OBJECTIVE FUNCTION
The overall objective of GCPO is formulated as follows:
G
Io'l
[z E Zuelea)] (wt min (r)
(O) = E fof mew
G
1=1
(8)
clip (ri(()) 1 1, A') 31DxI (Te (ree) ]

---

Published as a conference paper at ICLR 2026
photo tie right of matoncycle
mountain and small tree
real life photogrophy of super mario, 8k
photo four hot dogs
photo of white sandwich
scoring scarecrow.
Janus-Pro 78
Janus-Pro 7B GRPO
Janus-Pro 7B 6CPO (our)
Reward: Geneval
Reward: HPS
Figure 6: Visualization Results. We provide image generation results on Counting, Color, Position, and Shape tasks, as well as image quality.
where I[-] is the indicator function that evaluates to if the condition inside holds and O otherwise.
The differences are highlighted red between Equation (8) and Equation (3): (i) The optimization
term of each token is multiplied by I [2, E seelect), ensuring that only critical tokens in Zeelect are involved the overall optimization objective; (ii) The advantage term of each critical token multiplied by allocating token-wise advantage weight encourage exploration.
EXPERIMENTS
In this section, evaluate GCPO to improve the performance of AR model and the unified multi- modal model on two representative tasks. (1) Composition Image Generation: We report the results on GenEval and 12i-Compbench, which primarily evaluate the models' ability on spatial relation- ships and color attributes. (2) Image quality and Human Preference Alignment: We report DEQA, ImageReward, and PickScore metrics, reflecting the visual quality and human preference images.
6.1 EXPERIMENTAL SETUP
Training Settings. For the composition image generation task, our training dataset is sourced from 50,000 training prompts generated by Geneval pipeline, following (Liu et al. 2025). For the image quality and human preference alignment task, we utilize 15000 prompts from the HPSv2 training dataset. We conduct evaluations on LlamaGen (Sun et al. 2024). Janus-Pro 1B (Chen et al., (025a), and Janus-Pro 7B (Chen al. 2025a Please refer to the Appendix for detailed training settings.
Benchmark. We evaluate our method on Geneval (Ghosh et al. 2023). T21-CompBench (Huang al.) 2023), and DrawBench (Saharia et al., 2022) comprehensively validate effectiveness. T2I- CompBench is comprehensive benchmark for open-world compositional text-to-image generation, covering six compositional task categories. DrawBench contains comprehensive and challenging prompts designed to assess the generative capabilities of T2I models. We use it to evaluate DEQA- Score YYou et al. 2025). ImageReward (Xu ct 2023 and Pick Score (Kirstain et al. 2023) metrics. We also report HPSv2 score (Wu et al. 2023) HPSv2 Benchmark test data.
Reward Model. For the composition image generation task, following (Liu et al. 2025), we adopt Geneval reward as reward model. For the image quality and human preference alignment task, we use HPSv2 as our reward model. Notably, given the relatively lower performance LlamaGen on Geneval, only use HPSv2 as the reward model for this base.

---

Published as a conference paper at ICLR 2026
Table
Quantitative comparison results on the GenEval benchmark. The best result in green
Method
Overallt
Sing Obj-t
Two Obj.t
Counting
Colort
Position"
Color Attr
Diffusion hased Method
PixAri-a (Chen et nl.
2024
0.48
0.98
0.50
0.44
0.80
0.08
0.07
SDXL (Podell
0.55
0.74
0.39
0.85
0.15
0.23
SD3 Esser el ar 21ZA
0.63
0.98
0.78
0.50
0.81
0.24
0.52
DALL-E [Bcker
2023
0.67
0.87
0.47
0.45
FLUX.1 dey Lbb
0.66
0.98
0.81
0.74
0.79
0.22
0.45
AR baxed method
LlamaCien (Snn et
21124
0.32
0.71
0.34
0.21
0.58
0.07
0.04
Emu3
Wang
0.54
0.98
0.71
0.34
0.81
0.17
0.21
Show-o
Xie cl al
2114
0.68
0.98
0.80
0.66
0.84
0,31
0.50
GPT-4a
OpenAT
2125
Janus-Prot 1B
Chen
2025g
0.73
0.98
0.82
0.51
0.89
0.65
0.56
Jamus- Pro--B-
Chen el
20255
0.80
0.99
0.89
0.59
0.90
0.79
0.66
AR-baase
Show Show-otPARK --PPM (Guo et al.
202.5
0.69
0,97
0.75
0.60
0.83
0.54
0.53
T21-R1 Tiang et al. 2025
0.79
0,99
0.91
0.53
0.91
0,76
0.65
Llanausa+GG11
0.39
0.41
0.28
0.06
Janus-Pro-IB-GER
0.84
0.95
0.77
Janus 7B-GRPP
0.87
0.99
0.92
0.71
0.94
0.92
0.73
LlamaGen=GCPO
0.42
0.83
0.49
0.25
0.71
0.13
0,08
Manus-Pr-IB-GCRR Janus-Pro7B-GGCP
0.85
0.96
0.63
0.88
0.73
0.90
0.95
0.90
0.90
0.95
0.76
Comparison results on T21-CompBench and DrawBench, evaluated by DEQA -ccore, IeageReward, and PickScore. The best result is green Time: Time consumption of each RL-
Table 2:
training step. We achieve state-of-the-art performance without introducing additional time overhead.
Method
Color
Shape
Texture?
Spatial" Non-Spal. Complex!
DEQA
HPS
ImgRvdd
PickScore
Time (s)l
AR-baaed
Llammaiion
al
2024
0.3967
0,5103
0.0772
0.2908
2.70
Jamus-Pro-IB
Chen
2025a
0,3439
0.2363
0.2788
0.0969
0,2813
0.3733
3,38
25.46
-0.19
20.79
Jams-Pro-7B
Chen
20250
0.6359
0.3828
0.4936
0.2061
0.3085
0.3559
3.55
28.00
0.68
21.82
AR-bouod Mechost
LlmmsGen+GRPO
Moms-PPo.IB-GRRP
0,7050
013150
0.4621
0.3020
0.2963
0.3159
3.67
29.16
0.73
21.59
9.54
Banu-PPo-78-CRPP
0.7478
0.3999
0,5849
0.5481
D. 3090
0.3744
3.70
30.64
8.99
22.13
14.75
LlamsSien+GCPO
0.5736
0.1015
0.3199
20.59
Mauu-Pro-BB-GCPR GCPO
0,7373
0.3001
0.4803
0.3270
0.2948
0. 1.3009
3.73
29.61
0.73
21.60
9.69
Jamus-Poo7B..GCRP
0,7508
0.5173
0.7030
0.3824
0.3133
0.3 0.3888
3.79
30.90
1.01
22.10
14.89
6.2
MAIN RESULTS
We compare our method with leading AR models and diffusion models on GenEval, as shown in Table Only utilizing critical tokens (30% of the total tokens), GCPO achieves significant improve- ments over GRPO across all three base models. Notably, Janus-Pro-7B+GCCT overall score of 0.90. This is primarily attributed task (+0.19). Furthermore, Table 2 shows results of our method on T21-CompBench, which differs substantially from the GenEval-style training data. GCPO consistently outperforms GRPO in the majority of tasks, achieving up to 20% performance gains on Shape (+0.1174), Texture (+0. (1811, and Spatial task (+0.1343), thereby demonstrating strong generalization. We analyze the time con- sumption of our method in Table 2 Our approach achieves the aforementioned performance gains without introducing additional computational overhead (GCPO only increases time consumption by 1%). As shown in Fig. 6 the model with our method accurately understands the shape of the mountain (big) and the tree (small). contrast, the model with GRPO generates a forest.
attained the highest substantial improvement in the Counting
As shown in Table 2 under the HPS-based reward setting, our method consistently outperforms GRPO and demonstrates superior performance across all three models. Notably, Janus-Pro- 7B+GCPO achieves the best scores image quality and human preference alignment. Meanwhile, the images generated by our method exhibit more natural and vivid details, as illustrated in Fig, For more comparison results, please refer to Appendix
8

---

Published as a conference paper at ICLR 2026
30-
OarDON)
0.00
0.64
29.5-
0182
28
27.5
Gooeal
0.78
0100
27
0.76
253
20
irtill tokin
20-
70% One
0.14
tikess with Highe entrgy gad
loketa with lowar artuarty
20h Concal Sdcr with DAY
0.72
103% Backt:
Strpola Toien AISS
0.70
0% 15% 25%
200
400
650
1001
i000 1200 1900 1860
2001
-00
600 ROU 1000 1230 1900 1800
Steps
RL-Sleps
(a) Comparison token selection strategies and selection ratios.
(b) Comparison of critical tokens and other tokens HPS
(c) Comparison of critical tokens and other tokens Geneval
Figure 7: (a): All three types of critical tokens deliver clear performance gains at 10% initial selec- tion ratios and outperform random selection, with further increases yield only limited improvements. (b) & (c): GRPO with critical tokens (30%) has more performance improvement than GRPO with remaining tokens (70%). Our GCPO further achieves performance improvement. We further ana- lyze the settings for the overall selection ratio and the selection ratio for each type in Appendix
6.3 ABLATION STUDY
We conduct an analysis study on different token-selection strategies and selection rates, as shown in Fig. 7a Within the initial selection ratios range (10% of tokens). GRPO performance rises rapidly (+2.8T) with each of the three types of critical tokens, respectively; beyond this threshold, the gains are limited (+0.95). Moreover, combining all three types of critical tokens clearly outperforms se- lecting any single type at the same token budget (30% of tokens), which demonstrates the correct identification of critical tokens in RLVL training. We further verify that all three strategies signifi- cantly outperform random selection. As shown in Table we present more experimental results on multiple benchmarks under Geneval and HPS reward settings. Our critical token strategies consis- tently achieve balanced performance improvements. Furthermore, with the introduction of dynamic advantage weight, all metrics reach optimal values, demonstrating the effectiveness of our designs.
Table 3: Ablation results on critical tokens and dynamic advantage weight. Init-T: initial tokens, HG-T: high entropy gradient tokens, LS-T: low similarity tokens; DAW: dynamic advantage weight.
Inie-T
HG-T
LS-T
DAW
GenEval
121-CompBench
DEQA
HPS
ImgRwdt
Colort
Shape?
Texture"
Spotialt
Non-Spat-1 1
Complex
0.82
0.6084
0.2820
0.3496
0.2374
0.2865
0.2877
3.66
28.90
0.63
0.81
0.5440
0,2712
0.3368
0.2262
0.2832
0.2794
3.63
25.22
0.63
0.82
0.6721
0.2942
0.4101
0.2569
0.2938
0.3109
3.64
28.78
0.66
8
0.83
0.6802
0.2924
0.3991
0.2936
0.2943
0.3015
3.70
29.33
0.71
/
0.85
0.7373
0.3201
0.4803
0.3220
0.2948
0.3059
3.73
29.61
0.73
We also consider the comparison between 30% critical tokens and 70% of the remaining tokens. As shown Fig. 7b and [c GRPO with critical tokens are comparable with the GRPO bascline, while GRPO with other tokens leads to a drop in performance. Despite the fact that the other tokens account for 70% of the total tokens used in training and more than twice the count of critical tokens, they still have significant performance gap. This finding highlights the greater importance of critical tokens based on our selection strategy for effective model training.
In addition, to further illustrate the rationality of the selection ratio for each type and the overall selection ratio, we provide more experiments the Appendix E that the imbalanced selection for each type significantly degrades performance compared to balanced selection (10%: 10%:: which is also supported by the experimental results in Fig. 7a
As shown Table we observe
Notably, increasing the total selection ratios from 30% to 45% yields almost no performance im- provement, but decreasing it from 30% to 15% results in a significant performance degradation, as shown in the Table. 10 Furthermore, increasing total selection ratios to the entire sequence of to- kens leads to a decline in performance. This phenomenon arises from the use of DAW on the entire sequence of tokens, for which conduct additional experiments and analyses in the Appendix E.4

---

Published as a conference paper at ICLR 2026
Next Token Prediction (NTP) is currently the most general paradigm in NLP and AR image gener- ation. Nevertheless, based on the intrinsic spatial structure of images, previous works (Tian et al.) 2024 Han et al. 2025 Ma et al. 2024p have explored next-scale prediction to achieve high-quality and efficient image generation. We successfully extend our method to the next-scale prediction gen- eration paradigm. Specifically,
expand method follows: the importance of initial tokens (AR causality) is extended to the importance of early scales, while cntropy-gradiente
structure and similarity-based diversity mechanisms can be seamlessly adapted. Our approach demonstrates strong generalizability and achieves superior performance to GRPO across multiple metrics. This suggests that the key concept of critical tokens is general and worthy of further investigation. No- tably, our GCPO utilizes only critical tokens (25 % of the tokens). The comparison results are shown Table. and Fig.
Table 4: Results scale-wise model. The best result is in green
Method
DEQAT
HPPS
ImgRewardf
PickScuret
GenEvalf
Star
Ma
2024
3.70
26.38
0.52
21.84
0.47
Star
Ma et al.
2024
+ GRPO
4.11
30.17
0.76
22.19
0.50
Star
Ma et al.
2024
GCPO (our)
4.15
30.32
0.78
22.17
0.49
CONCLUSION
In this paper, we introduce Group Critical-token Policy Optimization (GCPO), novel RLVR frame- work for autoregressive image generation. We identify critical tokens in RLVR-based AR visual generation from three perspectives: Causal dependency of AR, Entropy-induced spatial structure, and RLVR-focused token diversity. We devise the dynamic advantage weight for critical tokens to enable reasonable exploration, based their confidence divergence between the policy model and reference model. Extensive experiments demonstrate that by leveraging critical tokens (30% of the image tokens), GCPO achieves better performance than GRPO, which operates on the full tokens.
ACKNOWLEDGMENTS
This work was supported by the Anhui Provincial Natural Science Foundation under Grant 2108085UD12. We acknowledge the support GPU cluster built by MCC Lab of Information Sci- ence and Technology Institution, USTC. The Al-driven experiments, simulations and model training were performed on the robotic Al-Scientist platform of Chinese Academy of Sciences.
ETHICS STATEMENT
Data Usage. All training and evaluation datasets used in this study are publicly available and have been widely adopted in prior research, including Geneval, T2I-CompBench, DrawBench, and HPSv2. We do not collect or distribute any new human or proprictary data. The prompts and images used do not contain personally identifiable information or sensitive content.
Content and Bias. We do not introduce any new annotations or external knowledge sources that could inject additional biases beyond those already present in the original models and datasets. To further promote responsible use, we encourage the integration of existing safety filters, content- moderation tools, and bias-detection techniques when deploying models enhanced with our method.
10 REPRODUCIBILITY STATEMENT
Here, we provide content for better Reproducibility.
1. The detailed method is described in Sec.
2. Hyperparameters for all used models in this paper (see Sec.
Besides, all training data, checkpoints, and code will be released within the scope of the conference.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dallle-3. pdf. 2(3):8, 2023.
Lao Beyer, Tianhong Li, Xinlei Chen, Sertac Karaman, and Kaiming He. Highly compressed tokenizer generate without training. arXiv preprint arXiv:2506.082. 250.008276. 2025.
briaai. Rmbg v2.0 is a new state-of-the-arr background removal model.
https://
huggingface co
2. 07] 2025.
Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart-5: Fast and controllable image generation with latent consistency models. arXiv preprint
arXiv:2401.05252, 2024.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint orXiv:2501.1785 2025a.
Xiwen Chen, Wenhui Zhu, Peijie Qiu, Xuanzhao Dong, Hao Wang, Haiyu Wu, Huayu Li, Aristeidis Sotiras, Yalin Wang, and Abolfazl Razi. Dra-grpo: Exploring diversity- reward adjustment for rl-zero-like training of large language models. arXiv preprint 200502555..09655 2025b.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie. Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. 14683, 2025.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 12873-12883 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Matteo Gallici and Haitz. Sáez de Ocáriz Borde. Fine-tuning next-scale visual autoregressive models with group relative policy optimization. arXiv preprint Sttii:225052233:11 2025.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances Neural Information Processing Systems, 36: 52132-52152, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12941 2025a.
Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Rui Huang, Haoquan Zhang, Manyuan Zhang, Jiaming Liu, Shanghang Zhang, Peng Gao, et al. Can we generate images with cot? let's verify and reinforce image generation step by step. arXiv preprini 250.::2200.1392 2025b.
Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis, 2024. URL http::/aarr..orr/a////222244111 2025.
Andre He, Daniel Fried, and Sean Welleck. Rewarding the unlikely: Lifting grpo beyond distribution sharpening. arXiv preprint arXiv:2506.02355 2025.
Yefei He, Feng Chen, Yuanyu He, Shaoxuan He, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipar: Parallel auto-regressive image generation through spatial locality. arXiv preprint arXiv:2412.0 04062, 2024.
11

---

Published as a conference paper at ICLR 2026
Jian Hu, Mingjie Liu, Ximing Lu, Fang Wu, Zaid Harchaoui, Shizhe Diao, Yejin Choi, Pavlo Molchanov, Jun Yang. Jan Kautz, et al. Brorl: Scaling reinforcement learning via broadened exploration. arXiv preprint arXiv:2510.01180 2025.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A compre- hensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723-78747 2023.
Wenxuan Huang, Shuang Chen, Zheyong Xie, Shaosheng Cao, Shixiang Tang, Yufan Shen, Qingyu Yin. Wenbo Hu, Xiaoman Wang, Yuntian Tang, et al. Interleaving reasoning for better text-to- image generation. arXiv preprint arXiv:2509.0644 06945, 2025.
Dongzhi Jiang. Ziyu Guo, Renrui Zhang, Zhuofan Zong, Hao Li, Le Zhuo, Shilin Yan, Pheng-Ann Heng, and Hongsheng Li. T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot. arXiv preprint arXiv:2505.00703, 2025.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-3666666 2023.
Black Forest Labs. Flux.
hom.tbaa...om//acc.... -abss /fabs/ 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint wrXiv22408.03322 2024.
Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G Patil, Matei Zaharia, al. Lims can easily learn reason from demon- strations structure, not content, is what matters! arXiv preprint arXiv:2502.07377. 2025.
Zicheng Lin, Tian Liang, Jiahao Xu, Qiuzhi Lin, Xing Wang. Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang. and Zhaopeng Tu. Critical tokens matter: Token-level contrastive estimation enhances Ilm's 's reasoning capability. arXiv preprint 1 2024.
Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng I Lin, Yi Xin, Xinyue Li, Qi Qin, Yu Qiao, Hong- sheng Li, and Peng Gao. Lumina-mgpt: Illuminate flexible photorealistic text-to-image genera- tion with multimodal generative pretraining. arXiv preprint arXiv:2408.0266. 2024a.
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.0044 2025.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference Computer Vision pp. 38-55. Springer, 2024b.
Xiaoxiao Ma. Mohan Zhou, Tao Liang, Yalong Bai, Tiejun Zhao, Biye Li, Huaian Chen, and Yi Jin. Star: Scale-wise text-conditioned autoregressive image generation. arXiv preprint 2rXiv:2406.10797 2024.
Xiaoxiao Ma, Haibo Qiu, Guohui Zhang, Zhixiong Zeng, Siqi Yang, Lin Ma, and Feng Zhao. Stage: Stable and generalizable grpo for autoregressive image generation. arXiv preprint arXiv:2509.25027, 2025.
OpenAl Learning to reason with 1lms.
http:///penaiiccom/nee
Learning-tan to- reason with-1lms/ 11ms/ 2024.
OpenAl. Introducing 4o image generation.
https://ppenal.ccmm
iintroducing-4o image-maeee.. generation/ 2025.
Kaihang Pan, Wendong Bu, Yuruo Wu, Yang Wu, Kai Shen, Yunfei Li, Hang Zhao, Juncheng Li, Siliang Tang, and Yucting Zhuang. Focusdiff: Advancing fine-grained text-image alignment for autoregressive visual generation through rl. arXiv preprint arXiv:2506.00 05501, 2025a.
12

---

Published as a conference paper at ICLR 2026
Kaihang Pan, Wang Lin, Zhongqi Yue, Tenglong Ao, Liyu Jia, Wei Zhao, Juncheng Li, Siliang Tang, and Hanwang Zhang. Generative multimodal pretraining with discrete diffusion timestep tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 26136-26146 2025b.
Yaning Pan, Qianqian Xie, Guohui Zhang, Zekun Wang, Yongqian Wen, Yuanxing Zhang, Haox- uan Hu, Zhiyu Pan, Yibing Huang, Zhidong Gan, al. Mt-video-bench: A holistic video un- derstanding benchmark for evaluating multimodal Ilms in multi-turn dialogues. arXiv preprint arXiv:2510.17722. 2025c.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint urXiv:2307.01952. 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen, Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-36493 -6494, 2022.
Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-4233 1948.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang. Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Vaishnavi Shrivastava, Ahmed Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. Sample more to think less: Group filtered policy optimization for concise reasoning. arXiv preprint arXiv:2508.09720 2025.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 0655:22406.06525, 2024.
Chameleon Team. Chamelcon: Mixed-modal early-fusion foundation models. arXiv preprint 00X1::2405.09818. 2024.
Kimi Team, Angang Du, Bofci Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 2025.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in Neural Information Processing Systems, 37:84839-8488 84869, 2024.
Shengbang Tong. David Fan, Jiachen Zhu, Yunyang Xiong. Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412. /4164, 2024.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
Jean Vassoyan, Nathanaël Beau, and Roman Plaud. Ignore the kl penalty! boosting exploration critical tokens to enhance rl fine-tuning, arXiv preprint arXiv:2502.06333 2025.
Jiakang Wang, Runze Liu, Fuzheng Zhang, Xiu Li, and Guorui Zhou. Stabilizing knowledge, pro- moting reasoning: Dual-token constraints for rlvr. arXiv preprint arXiv:2507.15778 2025a.
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint ar 22 2022.
13

---

Published as a conference paper at ICLR 2026
Junke Wang. Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu- Yu-Gang Jiang. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv preprint arXiv:2504.11455 2025b.
Qiuchen Wang, Ruixue Ding, Yu Zeng, Zehui Chen, Lin Chen, Shihang Wang. Pengjun Xie, Fei Huang. and Feng Zhao. Vrag-rl: Empower vision-perception- based rag for visually rich in- formation understanding via iterative reasoning with reinforcement learning. arXiv preprint arXiv:2505.22019 2025c.
Shenzhi Wang, Le Yu, Chang Gao, Chujic Zheng. Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.01939 2025d.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409, 18869, 2024.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. v2: A solid benchmark for evaluating human preferences of text-to-
Human preference image synthesis. arXiv preprint arXiv:2306.09344 2023.
Xunzhi Xiang and Qi Fan. Make it efficient: Dynamic sparse attention for autoregressive image generation. arXiv preprint arXiv:2506. 18226, 2025.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528, 2024.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: - 2023.
Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv 2025.
preprint arXiv:2505.0788
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. 2025.
Hee Suk Yoon, Eunseop Yoon, Mark A Hnsegawa-Johnson, Sungwoong Kim, and Chang D Yoo. Confpo: Exploiting policy model confidence for critical token selection in preference optimiza- tion. In Forly-second International Conference Machine Learning, 2025.
Zhiyuan You, Xin Cai, Jinjin Gu, Tianfan Xue, and Chao Dong. Teaching large language models to regress accurate image quality scores using score distribution. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14483-14494, 2025.
Hu Yu, Biao Gong, Hangjie Yuan, DanDan Zheng, Weilong Chai, Jingdong Chen, Kecheng Zheng, and Feng Zhao. Videomar: Autoregressive video generatio with continuous tokens. arXiv preprint a50iv:2506. 14168, 2025a.
Hu Yu, Hao Luo, Hangjie Yuan, Yu Rong, and Feng Zhao. Frequency autoregressive image genera- tion with continuous tokens. arXiv preprint 2Xii:2503.05305 2025b.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:21 10.:22277. 2021.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu. Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint ar0iv:2503. /4476, 2025c.
14

---

Published as a conference paper at ICLR 2026
Shihao Yuan, Yahui Liu, Yang Yue, Jingyuan Zhang. Wangmeng Zuo, Qi Wang, Fuzheng Zhang, and Guorui Zhou. Ar-grpo: Training autoregressive image generation models via reinforcement learning. arXiv preprint arXiv:2508.000 06924, 2025.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.0512 2025.
Yu Zeng, Wenxuan Huang. Shiting Huang. Xikun Bao, Yukun Qi, Yiming Zhao, Qiuchen Wang. Lin Chen, Zehui Chen, Huaian Chen, et al. Agentic jigsaw interaction learning for enhancing visual perception and reasoning in vision-language models. arXiv preprint arXiv:2510.01 2025a.
Yu Zeng, Yukun Qi, Yiming Zhao, Xikun Bao, Lin Chen, Zehui Chen, Shiting Huang, Jie Zhao, and Feng Zhao. Enhancing large vision-language models with ultra-detailed image caption gen- eration. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 26703-26729, 2025b.
Yu Zhang, Yunqi Li, Yifan Yang, Rui Wang, Yuqing Yang, Dai Qi, Jianmin Bao, Dongdong Chen, Chong Luo, and Lili Qiu. Reasongen-rl: Cot for autoregressive image generation models through sft and arXiv preprint arXiv:2505.24877 2025.
Yiming Zhao, Yu Zeng, Yukun Qi. Yang Liu, Xikun Bao, Lin Chen, Zehui Chen, Qing Miao, Chenxi Liu, Jie Zhao, et al. V2p-bench: Evaluating video-language understanding with visual prompts for better human-modes model interaction. arXiv preprint arXiv:2503.17736, 2025.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma. Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 1039, 2024.
15

---

Published as a conference paper at ICLR 2026
A DETAILED EXPERIMENTAL SETUP
A.1 HYPERPARAMETER
In this paper, we used three models (LlamaGen (Sun et al. 2024). Janus-Pro 1B (Chen et 2025a), and Janus-Pro 7B (Chen et al.) 2025a)) on the composition image task and image quality task. For different models and reward (HPS Reward (Wu al. 2023) and Geneval Reward (Liu et al. 2025 settings, our training configuration and parameters are as follows:
Table 5: GCPO training hyperparameters.
Name
LlamaGen for HPS
1B Geneval
for HPS
7B Geneval
7B for HPS
Learning rate
le-5
3e-6
le-6
3e-6
1e-6
Beta B3
0.04
0.01
0.01
0.01
0.01
Group Size
Classifier- Free Guidance Scale
Max Gradient Norm
1.0
1.0
1.0
1.0
1.0
Batchsize
Training Steps
900
1,600
1,600
1,600
1,600
Gradient Accumulation Steps
2
1
Dynamic advantage weight clip tw
Image Resolution
256 x 256
384 384
384 x 384
384 384
384 384
A.2 TRAINING EFFICIENCY
Compared with standard GRPO, additional operations involve entropy gradient calculation and token embedding similarity computation. We have implemented certain optimizations, so these operations do not introduce additional computational burden affect training efficiency.
B ENTROPY ANALYSIS IN RLVR-BASER AR VISUAL GENERATION
In section 4.2 discuss the entropy distribution of images in AR generation. We observe that the entropy distribution of image tokens exhibits a spatial pattern and fails to maintain consistency in different prompts. We further provide some examples and statistical results to illustrate this point, as shown in Fig.
Left: spoon dressed eyes Right: tsothbrush
Left: photo of mechorioal angel womon with crystal wings, in the sti-fi style of Stefon Kortic, created by Storley Lau and Artgemm
Right: plate topped with different kinds of fruit.
Jofernnce sddeet tadkgound
sageed bedigound
Moon Suppeyy
0.27
0.5
0.70
2.07
300
1000
(a) The entropy distribution of simple or compositional prompts. The entropy of the background is signifi- cantly greater than that of the subject.
(b) The entropy distribution of complex detailed prompts. The entropy of the subject is greater than that of the background.
Figure 8: In each figure, the blue line represents the average entropy of the subject regions, while the orange line represents the average entropy of the background regions. |difference) represents the average entropy difference between subject and background regions.
We argue that this entropy distribution is directly related to the complexity of the prompts. Specif- ically, for composition prompts, such as Geneval-style prompts, the entropy of the subject region lower than that of the background region (see Fig. Sap. These prompts cluding only the necessary keywords without excessive modifiers. As a result, the model has higher uncertainty when generating tokens in the background region due to the lack of sufficient prompt
relatively simple, in-
16

---

Published as a conference paper at ICLR 2026
information. For complex prompts, such the HPS testing data, the subject regions have higher information density, resulting higher entropy for tokens in these regions (see Fig. Sbb.
Based on this insight, conduct corresponding experiments to verify this point. We analyze the images generated by Janus-Pro and the corresponding entropy distribution, where prompts come from the Geneval Benchmark and HPS testing data. We RMBG v2.0 (briaai 2025). state- of-the-art background removal model, to separate the foreground and background regions of the generated images. Then, we calculate the average token entropy within the foreground region, as shown in Fig. The statistical results further verify our view.
In addition, further investigate the evolution of entropy distribution during the RL training pro- cess. We find that RL training does not alter the original entropy pattern, but instead further rein- forces it, which is similar observations LLMs (Li et al., 2025; Vassoyan et al., (025). This phenomenon means that the entropy difference between the subject and background regions grad- ually increases (see the histogram in Fig. B). resulting in more pronounced entropy (information) changes for high-gradient tokens in their neighborhood. Such information variation and complex structural regions are simultaneously coupled to these tokens, prompting us to include them in the scope of critical tokens for focused optimization.
C MORE COMPARISON RESULTS WITH RELATED WORK
In the field of AR image generation based on RLVR, (Jiang et al. 2025 and Pan et al. 2025a) are representative works that are close ours. The differences are that (Jiang al. 2025) introduces semantic-Cos CoT before image generation and uses a combination of multiple reward models, including HPS (Wu et al. 2023), GroundingDINO (Liu al. 2024b), GIT (Wang et al. 2022), and LLaVA- VieVinion-B (Li et al. 2024), and constructs training data that includes both Geneval-style and T2i-compeench-styy prompts. We only utilize Geneval rewards and Geneval-style data. (Pan et al,, 2025a) constructs fine-gred grained paired prompt-image training data and first trains the model on paired data with images, then operates GRPO training without images. In contrast, we only perform image- free RL training. Comparison results Geneval and T2i-comphench
shown in the Table 6 Our method achieves significant lead on Geneval and also obtains comparable results on T2- compbench.
Table 6: Comparison results with T2I-R1 and Focus-Diff on Geneval and T21-CompBench.
T21-CompBench
Method
GenEvalf
Colort
Shapet
Texturef
Spatialf
Non-Spat.
Complex
T21-R1 Jiang et al.
2025
0.79
0.8130
0,5852
0.7243
0.3378
0.3090
0,3993
Focus- Docus--Dillee (Pan el a. 2025a
0.85
0.7996
0.5748
0.7007
0,3789
0,3098
0,3912
Janus-Pro-7BBs Pro-7B-GKPO
0.87
0.7478
0.3999
0.5849
0.2481
0.3090
0.3744
Janus-Pro Pro-7B=GCO
0.90
0.7508
0.5173
0,7030
0,3824
0.3133
0,3888
(4) complex prompts from HPS
(b) simple prompts from GenEval
Figure 9: Visual results from different benchmarks
17

---

Published as a conference paper at ICLR 2026
D ANALYSIS BETWEEN SIMILARITY AND REGIONS IN AR GENERATION
We provide more examples from the HPS benchmark and GenEval benchmark to further illustrate the relationship between similarity and regions, shown in the Fig. [ It is obvious that simple background and texture regions in images within the same group tend to be more consistent (higher similarity), while complex regions show greater diversity (lower similarity).
E ADDITIONAL EXPERIMENTS
E. THE INFLUENCE OF DYNAMIC ADVANTAGE WEIGHT
Dynamic Advantage Weight (DAW) is designed to work in cooperation with the critical token strat- egy and improve performance by further encouraging diverse advantages among the critical tokens. Applying DAW to all the 100% tokens may amplify certain negative gradients among the remaining non-critical tokens. To verify this, we conduct the corresponding experiments, as shown in Tabel. 7
As shown in Table. 7 the experimental results show that our method (30% tokens with DAW) (100% tokens with DAW) > (100% tokens without DAW). This verifies that 1) DAW is also useful to the general GRPO. 2) Critical tokens and DAW work synergistically to achieve the best performance. We argue that DAW may amplify certain negative gradients among the remaining non-critical tokens. Therefore, the rational selection of critical tokens is essential for performance improvement, and combining this selection with DAW leads to even better results.
Table 7: Ablation results on dynamic advantage weight
Method
DEQA
HPSt
ImgRewardf
PickScoret
GenEvalf
100% Tokens without DAW (GRPO)
3.67
29.16
0.73
21.59
0.84
100% Tokens with DAW
3.71
29.54
0.72
21.62
0.85
30% Critical Tokens without DAW
3.70
29.33
0.71
21.58
0.83
30% Critical Tokens with DAW (our)
3.73
29.61
0.73
21.60
0.85
E.2 RESULTS ON ADDITIONAL. SELECTION RATIO SETTINGS
We further provide experiments with other selection ratios, as shown in Table X We observe that when the total selection ratio is 30%, and the selection ratio is set to 15%:10%:55 the performance decreases significantly even though the total number of selected tokens remains unchanged.
We compare multiple ratios for each type of critical token, shown in Fig. 7 For each type, 10% serves as a performance gain threshold. Beyond this point, performance improves slowly. Therefore, adopt 10% as the default for each type.
Table 8: Comparison results on additional selection ratio settings
Method
DEQAI
HPS
ImgReward
PickScoret
GenEvalf
30% (15%:10%:550 10%:5%)
3.66
29.54
0.62
21.54
0.83
30% (10%:109) 10%:10%) (our)
3.73
29.61
0.73
21.60
0.85
E.3 THE ABLATION ON LOCAL AVERAGING OF THE ENTROPY MAP
Local averaging of the entropy map is proposed to reduce the noise and outliers in the entropy map, enabling more robust and efficient critical token selection. We further provide corresponding ablation experiments on this strategy in the Table. 9 where removing the local averaging results in slight decrease in model performance.
18

---

Published as a conference paper at ICLR 2026
Table 9: Comparison results on local averaging
Method
DEQAI
HPSt
ImgReward
PickScore1
GenEvalf
w/o local averaging
3.71
29.56
0.71 0.73
21.58
0.84
local averaging (our)
3.73
29.61
21.60
0.85
E.4 THE ABLATION ON THE TOTAL SELECTION PROPORTION
In Table. 10 We provide experiments on the total selection proportion. including 15% (5%+5%+55% and 45% 115+++55++1599 We observe that reducing the selection proportion to 15% results in significant performance drop, while further increasing it does not bring notable gains. Therefore, we outperform the GRPO with all tokens using as few critical tokens as possible.
Table 10: Ablation results on total selection proportion
Method
DEQA
HPS
ImgReward
PickScoret
GenEvalf
15% (5%+5%+5%%
3.63
29.29
0.63
21.55
0.82
30% (10%+10%++0%)) (our)
3.73
29.61
0.73
21.60
0.85
45% (15%+159+15%)
3.70
29.65
0.75
21.62
0.85
E.5 THE INFLUENCE GROUP SIZE
We argue that for the GRPO algorithm itself, a larger group size is empirically verified to yield more
stable policy variance and a more diverse exploration space. Brorl (Hu et al.; 2025) points out that substantially increasing the group size of the LLM can further improve the performance of models
that are experiencing learning stagnation. Previous methods (Jiang et al. 2025 Zhang et al. 20255)
typically group size trade-off between performance and memory requirements.
To further validate this on AR visual generation, we provide more experiments of our method on different group sizes as shown in Table We find that group size and HPS have a strong positive correlation. DEQA, PickScore, and ImageReward show significant improvements from 4 to 8, but the improvement is minimal when the group size further increased (from 8 12). Conversely, the GenEval only shows an improvement when the group size increases (from 8 to 12).
Table 11: Comparison results on different group sizes
Method
GPU Memory.
Training Timel
DEQA
HPS
ImgReward
PickScoref
Genevalf
G=4
39G
4.5h
3.73
29.61
0.73
21.60
0.85
G=8
79G
9.2h
3.80
30.39
0.82
21.70
0.86
G=12
115G
11.4h
3.81
30.63
0.85
21.72
0.88
F MORE VISUAL COMPARISON RESULTS
In this section, we further provide visual comparison results on the composition image task and image quality task to better illustrate the effectiveness and compatibility of our method. We will provide prompts for images in Sec.
F.1 COMPARISON ON LLAMAGEN
We only use the HPS Reward LlamaGen and abandon the Geneval reward. This is because we find that the Geneval score for each image generated by LlamaGen is close to 0 and does not provide effective training rewards. Nevertheless, the model trained with the HPS reward not only demonstrates better generation quality, as shown in I0 but also further improves performance on Geneval.
19

---

Published as a conference paper at ICLR 2026
LlamoGen
LlamoSen
Llamasen +SRPO
Llamasen +SRPO
LlemaSen 5CCO
LlamaSen +CCO (our)
(8)
(b)
(c)
(d)
(e)
(d)
(a) Visual comparison results on HPS benchmark.
Visual comparison results on Geneval
Figure 10: Qualitative comparison between the LlamaGen + GRPO and LlamaGen GCPO (our) trained with HPS reward. Prompts are Sec.
E5
F.2 COMPARISON ON JANUS-PRO 1B
We provide more qualitative comparison results Janus-Pro 1B, as shown in 11 (HPS reward) and [2 (Geneval reward). These visual results further demonstrate the effectiveness of our approach.
Janus-Pro 1B
Janus-Pro 1B +GRPO
Janus-Pro 1B +GCPO (our)
(a)
(b)
(c)
(d)
(e)
(f)
Figure 11: Qualitative comparison between the Janus-Pro IB GRPO and -rous-Pro IB + GCPO (our) trained with HPS reward. Prompts are in Sec.
F5
F.3 MORE VISUAL RESULTS DURING RLVR TRAINING
To better understand the training dynamics of method, visualize the results of samples gen- crated by the same prompts during training, as shown Fig, 13 These qualitative results intuitively demonstrate how the model continuously optimizes towards the goal of improving image quality and Human Preference Alignment as training progresses.
F.4 COMPARISON STAR
We provide more qualitative comparison results on the scale-wise model. Star (Ma et al. 2024). as shown in Fig. 14 These visual results further demonstrate the extension of approach.
20

---

Published as a conference paper at ICLR 2026
Janus-Pro 1B
Janus-Pro +GRPO
Janus-Pro 1B +GCP (our)
(b)
(c)
(d)
(e)
Figure 12: Qualitative comparison between the Janus-Pry 1B + GRPO and Janus-Pro 1B + GCPO (our) trained with Geneval reward. Prompts are in Sec. F.5
Training Process on HPS Reward
The image features an anime girl in short skirt and thigh-high socks, with slim figure and accentuated hips.
Sonic the Hedgehog depicted muscular Greek god in highly detailed digital painting Greg Rutkowski and Alphonse Mucha Artstation.
jellyfish sleeping in space station pod.
Figure 13: We visualize the generated samples during the optimization of Janus-Pro 7B trained with HPS reward. As training progressed, the model is steadily optimized towards the target of Human Preference Alignment.
21

---

Published as a conference paper at ICLR 2026
Star
GRPO
Our
Figure 14: Visualization Results on Star.
F.5 USED PROMPTS IN THIS SECTION
The prompts used Fig. 10a are:
(a) The image is of a raccoon wearing Peaky Blinders hat, surrounded by swirling mist and rendered with fine detail.
(b) A teddy bear mad scientist mixing chemicals depicted in oil painting style as fantasy concept art piece.
(c) A clock tower with lighted clock faces, against twilight sky.
The image features closeup portrait of stone angel statues, created with the Unreal Engine and featuring intricate details by various artists.
The prompts used Fig. 10b are:
(a) photo of teddy bears
photo of couch below a cup
(c) photo of white sandwich
(d) photo of blue handbag and white cell phone
The prompts used in Fig. 1
arr:
(a) Wicked witch casting fireball dressed in green with screaming expression.
(b) The image is a portrait of Homer Simpson as a Na'vi from Avatar, created with vibrant colors and highly detailed in cinematic style reminiscent of romanticism by Eugene de Blaas and Ross Tran, available on Artstation with credits to Greg Rutkowski.
(c) A small green dinosaur toy with orange spots standing on its hind legs and roaring with its mouth open.
Mila Kunis portrayed fire elemental in highly detailed digital painting.
22

---

Published as a conference paper at ICLR 2026
(e) A pizza is displayed inside pizza box.
(f) A portrait of dinner dish of protein and greens.
The prompts used Fig. 12 are:
(a) photo of person and bear
photo of fire hydrant and a train
(c) photo of four bowls
(d) photo of baseball glove right of bear
(e) a photo of green bus
(f) a photo of two trains
G THE USE OF LARGE LANGUAGE MODELS (LLMS)
In this paper, LLMs are employed to refine the writing, further enhancing the readability and quality of the paper.
23


Reviews:
[{"review_id": "ZZA9fXabbG", "reviewer": "Reviewer_MkTp", "summary": "This paper proposes GCPO, which selects critical tokens through multiple criteria to improve GRPO for T2I generation. The authors identify the critical tokens as initial tokens, tokens with high entropy gradients, and tokens with low average similarities. A dynamic advantage weight is also proposed to balance exploration and stabilization.", "questions": "1. How does the group size G affect performance? Does it have a strong positive correlation?\n2. Is the local averaging of the entropy map very important? It is intuitively reasonable, but it would be better to include an ablation study to verify its effect.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea is intuitive and well-motivated.\n2. Section 4 provides solid experimental observations, which support the proposed criteria for selecting critical tokens.\n3. The proposed method is simple and appears to be effective in improving generation performance.", "weaknesses": "1. Figure 7 shows that “30% Critical Tokens with DAW” performs better than the “100% baseline,” while “30% Critical Tokens” performs worse. How does the model perform when DAW is applied to the “100% baseline”? It is important to determine whether DAW or the critical tokens contribute more to the overall performance.\n2. To my understanding, $Z_{struct}$ is selected within a single image, while $Z_{sim}$ is selected across all images. Should $Z_{struct}$ therefore come from different positions for each image, and $Z_{sim}$ from the same positions across images? If so, why does $Z_{sim}$ come from different positions as shown in Figure 5?\n3. The selection ratio (along with the ratio of each token type) appears to be an important hyperparameter. The current setting is 10%:10%:10%. It would be better if the authors could provide results for additional selection ratio settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FhYhom5BcC", "reviewer": "Reviewer_z8AU", "summary": "Current RL methodologies typically assign similar weights to all image tokens, but different tokens are expected to play different roles. To distinguish among tokens, the authors identify three types of critical tokens based on the causal nature of autoregressive models, token entropy, and token diversity. In addition, they apply differentiated weights by comparing each identified token with the initialization model.", "questions": "Since critical tokens may overlap across the defined categories, does this situation require additional processing?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Considering that different image tokens play different roles in image generation, it is reasonable to focus on optimizing critical image tokens using RL.\n- The paper provides the valuable insight that image tokens with high entropy gradients convey spatial structural information of the generated images.\n- The proposed RL method demonstrates better performance than GRPO on both Janus-Pro and LLaMaGen models.", "weaknesses": "- Since critical tokens can change during training, they should be re-identified at each training step, which may introduce substantial additional computation. However, the paper lacks a comparison of training cost/consumption.\n- The performance of the 1B model drops on the GenEval benchmark in Table 1. What is the reason?\n- Figure 7(a) only shows the effect of the selection ratio for each token type, while the analysis does not address the choice of the total selection proportion (e.g., 30%).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pYbqe2Qn9w", "reviewer": "Reviewer_KyYg", "summary": "This paper introduces Group Critical-token Policy Optimization (GCPO), a novel RLVR framework that focuses on optimizing critical tokens, regarding to causal dependency, structure importance, and token diversity, for autoregressive image generation, achieving significant improvements in efficiency and performance over GRPO across benchmarks.", "questions": "1. Given the limited performance gains, could the authors clarify additional advantages, such as the impact on RL training time due to token filtering?\n2. How would GCPO perform on non-spatial vision tokenizers like VAR’s application on T2I [1] and DDT [2] for vision token extraction?\n\n[1] Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis\n\n[2] Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Novelty: Introduces a critical-token selection strategy based on causal dependency, entropy gradients, and token diversity.\n* Exploration: Provides intriguing insights into the role of structure in token entropy during text-to-image generation.\n* Comprehensive Validation: Demonstrates improvements across Geneval, T2I-CompBench and DrawBench benchmarks.", "weaknesses": "* Limited Gains: Improvements over GRPO are minor, especially for Geneval on 1B models and T2I-CompBench on 7B models, despite added complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p7boI30Pbk", "reviewer": "Reviewer_eipe", "summary": "This papers proposes Group Critical-token Policy Optimization for autoregressive image generation models drawing insights from Reinforcement Learning with Verifiable Rewards in the context of LLMs and Chain of Thought reasoning.", "questions": "The paper relies on heuristics which may make the method brittle and sensitive to hyper-parameters:\n\n- The assumption is that the first K tokens (e.g., 10% of the sequence length ) are critical because their initial generation decisions propagate and determine the overall image structure (why this particular number?)\n- The 2D gradient of the token entropy map. The authors observed empirically that tokens with high entropy gradients consistently correspond to important image structures and region-bridging elements. This is interesting, any theoretical insights or links to grad-cam and interpretability methods?\n- Token diversity within a group of generated images. The criterion selects tokens with the lowest average pairwise cosine similarity of their embeddings across a batch of images. The rationale is that low-similarity tokens are associated with more complex regional structures that contribute richer information for policy optimization, while high-similarity tokens are mundane background or texture. Any further experiments to check this? Perhaps and appendix with more examples and images.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper identifies a more efficient and insightful method to improve RL post-training for AR models.\n- The paper shows good empirical performance\n- The paper carries out extensive experiments", "weaknesses": "The paper does not discuss the recent Visual Autoregressive VAR paradigm: https://arxiv.org/pdf/2404.02905\nThere are works that have used Group Relative Policy Optimization for VAR such as 'Fine-Tuning Next-Scale Visual Autoregressive Models with Group Relative Policy Optimization': https://arxiv.org/abs/2505.23331\n\nThese should be discussed in the text, and extensions of GCPO to VAR should be mentioned as a future research direction. The problem with standard AR is that they impose a raster-scan inductive bias on image generation which does not align with the multi-scale nature of images: there is no reason to believe that the top left corner should condition the bottom right corner and not the other way around. As the authors mention 'early tokens serve as global priors and structural guides', which more closely aligns with a multi-scale approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)