Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation
Authors: Yifu Luo
Xinhao Hu
Keyu Fan
Haoyuan Sun
Zeyu Chen
Bo Xia
Tiantian Zhang
Yongzhe Chang
Xueqian Wang
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
masked autoregressive models
text-to-image model
Primary area: applications

Abstract: Reinforcement learning (RL) has garnered increasing attention in text-to-image (T2I) generation. However, most existing RL approaches are tailored to either diffusion models or autoregressive models, overlooking an important alternative: masked generative models. In this work, we propose Mask-GRPO, the first method to incorporate Group Relative Policy Optimization (GRPO)-based RL into this overlooked paradigm. Our core insight is to redefine the transition probability, which is different from current approaches, and formulate the unmasking process as a multi-step decision-making problem. To further enhance our method, we explore several useful strategies, including removing the Kullback–Leibler constraint, applying the reduction strategy, and filtering out low-quality samples. Using Mask-GRPO, we improve a base model, Show-o, with substantial improvements on standard T2I benchmarks and preference alignment, outperforming existing state-of-the-art approaches.

Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation
Yifu Luo 1...1, Xinhao Hu 1.1, Keyu Fan 1.1, Haoyuan Sun Zeyu Chen 1.2, Bo Xia',
Tiantian Zhang*.. Yongzhe Chang', Xueqian Wang*.
Equual Contribution
Project Lead, *Corresponiing Authors,
Tsinghua University
Technical University of Munich
Abstract
Reinforcement learning (RL) has garnered increasing attention in text-to-image (T2I) generation. However, most existing RL approaches are tailored to either diffusion models autoregressiive models, overlooking an important alternative: masked generative models. In this work, we propose Mask-GRPO, the first method incorporate Group Relative Policy Optimization (GRPO)-based RL into this overlooked paradigm. Our core insight is to redefine the transition probability, which is different from current approaches, and formulate the unmasking process a multi-step decision-making problem. To further enhance our method, we explore several useful strategies, including removing the Kullback-Leibler constraint, applying the reduction strategy, and filtering out low-quality samples. Using Mask-GRPOL improve base model, Show-o, with substantial improvements on standard T2I benchmarks and preference alignment, outperforming existing state-of-the-art art approaches. The code is available on https: //github. com/ jingzhejun//e junMask- -aatm/Mask-RRPPPP
Introduction
Generative text-to-image (T2I) models 115 have made tremendous progress in recent years, offering powerful and innovative methods for visual content creation. Broadly, existing T2I models can be
categorized into three main types: diffusion models [1 6), autoregressive (AR) models 7 [ and
masked generative models (MGMs) [H13. Diffusion models generate images by gradually refining random noise through a denoising process, while standard AR models treat image generation as sequential token- token-by-token prediction. MGMs, on the other hand, can be seen as hybrid [10 [4] they predict all masked tokens simultancously in parallel at each iteration, but only keep the most confident ones, defined newly unmasked tokens. The rest will be remasked for the next iteration, which we define as newly remasked tokens. This approach can be thought of discrete diffusion process through the absorbing state ([MASK]) progressively unmasking tokens while preserving the autoregressive nature, predicting masked regions based on predicted ones, similar to Bert [5]. This
hybrid makes MGMs show superior trade-offs between sampling quality and speed [3]
Meanwhile, reinforcement learning (RL) 16-18 has gained increasing attention for its strong capability to enhance the reasoning capabilities of Large Language Models (LLMs) [19 20 Inspired
by these successes, researchers have begun extending RL the visual domain 25 including
application in T2I generation [24] 255. Notably, Group Relative Policy Optimization (GRPO)-basee methods 2 have been applied both diffusion models 26 and AR models (55 However, MGMs remain largely unexplored in this context.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Unlike AR and diffusion models, applying RL to MGMs poses a unique challenge: defining the transition probability in RL. In AR models, the transition probability corresponds naturally to next-token distributions. In diffusion models, each step's reverse denoising distribution acts as the transition probability. Since MGMs predict all masked tokens in parallel, a naive solution to use the product of their probabilities at each iteration as the transition probability effectively blending AR (token-based) and diffusion (iteration-bet based) perspectives. Yet, when applying it to our base model Show-o [27] using GRPO (20], as shown in Figure we observe unsatisfactory performance.
This result prompts a deeper inquiry: What is the correct or proper transition probability for MGMs? Revisiting the core unmasking mecha- nism of MGMs, we find that it is the most confi- dent tokens at each iteration, which we defined as newly unmasked tokens above, that matter most to the transition probability. Building on this, propose two candidate definitions: (I) the product probability over all newly un- masked tokens at each iteration. (2) the product of probability over both newly unmasked and remasked tokens, but treated with probabilistic trick. We will elaborate on these formulations in Section 3.2
Ours
18
AR AR-style Transition Probability
Rewards
15
14
13
MM
80
100
120
140
Training Steps
Figure 1: The unsatisfactory performance using the AR-style probability.
Building on this insight, we propose Mask- GRPO, the first approach that integrates RI. into MGMs for T2I generation. Specifically, we model the unmasking process as multi-step decision-making problem, using both transition possibil- ity definitions we gave above. We then apply GRPO (20) into our base model, Show-o [27], resulting in significant boosting performance. To further enhance Mask-GRPO, we explore the impact of removing the Bellbacck-eibble (KL) constraint. We find that KL regularization unnecessary if a
sufficiently small learning rate is used.
This requirement presents two key challenges: (1) MGMs typically require number of iterative steps to generate a final picture, while GRPO 20 requires revisiting all these steps, which is significantly time- and resource-consuming. (2) RL training is always unstable, which leads to less
However, RL requires efficient but high-quality sampling 28
16
performance improvement. We term it as the *Vanishing Samples' problem. On the one hand, it resembles the vanishing advantages problem 29 observed in LLMs; on the other hand, it differs from that because we mainly use CLIP (30] as our reward model, and it is almost impossible to obtain the same rewards within a group. It is more like sample quality issue we will further discuss in Section 3.3
To address these issues, we introduce a reduction strategy and a sample filtering strategy, respectively, The former reduces the iteration steps for computation or sample generation during training, and the latter uses a reward-based filtering mechanism to discard low-quality samples. Experiments support these strategies and lead to further improvements.
Our contributions are summarized follows:
We are the first to introduce RL to MGMs. We model the unmasking process multi- step decision-making problem and propose two effective formulations of the transition probability. Based on this, we develop Mask-GRPO, the first GRPO-basee -asedd approach on MGMs for T2I generation.
We explore several enhancements to push Mask-GRPO further, including removing the KL constraint, applying the reduction strategy, and filtering out luw-quality samples. We analyze each and validate their effectiveness through extensive experiments.
Our method achieves superior performance on standard T21 benchmarks and preference alignment, improving over the base model and matching or surpassing existing -f-te- art approaches.

---

2 Preliminaries
2.1 Mechanism of Masked Generative Models
As summarized by 331, MGMs follow the encode-decode manner similar to AR models. In the encoding stage, an image is represented as sequence of discrete tokens [2 33 where each token corresponds a categorical label. In the decoding stage, the model aims to recover the full token sequence and map it back to image pixels. However, unlike AR models, MGMs are trained to predict tokens in parallel using a bidirectional transformer, rather than loken-by-toker in an autoregressive manner. MGMs are always seen as discrete diffusion process through the absorbing state (IMASK)). and we give a detailed analysis in Appendix
Specifically, MGMs use an iterative decoding strategy over T steps. Let Y w/ denote the latent token sequence at iteration t, where N is the length of the reshaped token matrix. We denote Y Ç Y2 and Y S Y as the subset of all masked tokens and unmasked tokens, respectively. Note that
YM
M n r = 0, Y UYY = Y,,
To generate image, the model starts from blank canvas where all tokens are masked:
YoM Yo. Yo 0,
and ends with a fully generated token sequence:
Y 0, Y = Yr.
(3)
At each iteration t, the model predicts all masked tokens in Y.M simultaneously and moves a subset of them, which we define as newly unmasked tokens in Section [ to Y, to obtain Y Y:4 and Y++1 for the next iteration. The number of tokens moved in each ierrttion is predefined as ny, satisfying:
Σ T4e m==N. N.
At each iteration f, the model first predicts the probabilities as P. E RN x for all masked tokens in Y,M in parallel, conditioned on all unmasked tokens in Y, Here, K denotes the codebook size and NrM is the size of Y.M, which means the number of masked tokens at iteration f. After that, for each mask token position 1 I i S N.M, token y² is sampled based on the prediction probabilities pt E R and a 'confidence" score is assigned based on this probability. Y4 is then formed by choosing the most confident tokens from Y,M and moving them to Y, according to their confidence scores. This Choose and Move (CaM) operation each iteration can be formulated as:
CoMp = [1, 10 otherwise,
if cst > sorted cs,, [NMM -ntr]
(5)
where as} is the confidence score for token y', and cs, is the list of all confidence scores at iteration t. Finally, the remaining tokens in Y:M, which we defined as newly remasked tokens in Section 1 are remasked to form Y and subsequently, Y4+1, for the next iteration.
2.2 Reinforcement Learning
RL is commonly formulated as discounted Markov Decision Process (MDP) specified by the tuple M = AS P, R, Po. 7), where S and A denote the state space and action space respectively, P represents the transition dynamics, R the reward function, Po refers the initial state distribution. and y denotes the discount factor. The agent interacts with the environment following policy म. which generates a trajectory 7 (So, Ag 81, 81,01, the expected return:
.., ...). ). The goal of RL. is to find policy that maximizes
n arg max E
R(s: az)]
(6)
3

---

GRPO Redefine ine the transition probability
Group Advantages
A
A2
A
Aa
"A monkey swimming the sea."
Policy Optimization
Sample Filter
Mask GRPO
Group Sampling
CLIP
Unmasking Process
Figure 2: The framework of Mask-GRPO. We redefine the transition probability formulate the unmasking process as a Markov decision-making problem. The reward is got from CLIP [30].
3
Methods
Our goal is to enhance MGMs for T21 generation using RL. To achieve this, we propose Mask- GRPO, the first RL. approach specifically designed for MGMs, illustrated in Figure We begin by formulating the unmasking process in MGMs as multi-step Markov decision-making problem in Section 3.1 Then, in Section 3.2 we propose
candidate definitions of transition probability building on our framework. Finally, we describe several enhancements for Mask-GRPO, including removing the KL constraint, applying reduction strategy, and filtering out low-quality samples.
3.1 Unmasking Multi-Step Markov Decision-Making
Online RL improves policy performance through direct interaction with the environment, guided by reward signals. To integrate this with MGMs, we first formulate their unmasking process a Markov decision-making problem.
Building on the mechanism in Section 2.1 we define the unmasking policy at each iteration t as:
(Y (Y,,)),
where e is the input prompt. This simplifies from the explicit form:
To TY 1 Y441/Y,M, Y, Y),
(8)
by leveraging the deterministic relationship between /y? and Y, (as detailed in Equation
I).
crucial property for modeling Markov decision-making problem is that each step must depend only on the current state and action. Fortunately, this holds in MGMs, where each iteration only depends on the current token sequence Y, and prompt c. Inspired by [26), we expand the T'iteration unmasking process into T T-step Markov decision process:
st P (Y,,c)
A Yt+1
p( 118,-118,,44 at) & po(844118,,a)
Po 2 Yo
(9)
A ()(Y),c),
=
R(s,, a,)
(R(ST-1,aT-)
otherwise
7 2 7:

---

where 0 s is a constant that will not be used in GRPO- methods 20 34 r(Y,, c)
refers to the reward assigned to the final image generation result, and (o St. a,) denotes the transition probability we will discuss later. Based on this formulation, we integrate GRPO 20 34 into MGMs and transform the learning objective from Equation (6) into the following:
(()
G
T-1
11 GT
Σ J=I
min (0) A2, clip (T! (0), -, 16) A! - -BDr (Tel (Tel(Tee))
1
(10)
where G the number image generations within a group per prompt, and the j-th generation is presented by the superscript of j. The Nef denotes the reference policy (e.g.. the base model). The likelihood ratio (0) and group-level normalized advantage A! are defined follows, respectively:
(0) Po Po(0/+1lo1..0
(11)
(aua(s s/ a²)"
A/ R( at) meanl (R(s), 0
(12)
std(( a
So far, we have framed the unmasking process as multi-step decision-makinn problem and in- corporated GRPO into it. However, how to choose the reward function r and model the transition probability po(s sts a₂) remains unknown. While we can use established perceptual models as our reward function (c.g.. CLIP 30), which we find effective, the second problem is more complex.
A straightforward idea is to mimic the autoregressive transition definition used in LLMs.. For MGMs, since all tokens in Y.M are predicted in parallel, we can define the AR-style transition probability as:
( ae) ne To (Y++1
II cs'-
(13)
'EY.
Here, cst is the confidence score (i.e., predicted probability) for the i-th token at iteration t, as defined in Section 2.1 It can be understood as: AR models predict a token each time and take its token probability as the transition probability, while MGMs predict all tokens in Y.M in parallel each iteration and take the product of every token probability as the transition probability.
While this definition is intuitive and consistent with AR models, unsatisfactory performance when applying it directly to MGMs (as shown in Figure [ This motivates deeper investigation into whether this definition inherently flawed and if so, what more principled alternative would be.
initial experiments reveal
3.2 Two Candidate Definitions of Transition Probability
To address the limitations discussed above, we revisit the unmasking mechanism of MGMs and offer a key insight: The newly unmasked tokens (those with CaM, the most critical role in determining the transi- tion probability. For simplicity here we denote Y.Ca C Y,M as the subset of them.
Unmasking Step
Sampled Token id
UmMask
in Equation (5) play
3
Prediction Distribution
T 0
2
T
=
1
SM M
Taken Sampling
3
Some Transition
V
JM
To better illustrate this insight, we present an example in Figure 3 depicting the case of iter- ation 0. For simplicity, assume the image consists of only four tokens, each with predic- tion distribution Př E K where K is the size of the codebook. We also assume that only one token needs to be predicted at this iteration (i.e., = 1). Under this setup, we observe that multiple different token samplings and confidence scores (i.e., predicted probabilities) can lead the same transition (St+1, St, a,), provided the following conditions hold:
3
A
M
3
2
The illustration of transition probability. It is observed that different token samplings finally can lead to the same next state.
Figure 3:
5

---

The sampling for the newly unmasked tokens Y, remains the same.
The remaining tokens in Y yo which will be remasked, have lower confidence scores cs; than any of newly unmasked tokens in Y.C.M.
These lead us define the transition probability pe p at) by explicitly modeling these two factors. Let min(cse) be the minimum confidence score among Y.Ca Then, our first candidate definition of transition probability can be defined as:
(
II (EE
(1))* )
po (8 Se, ae) =
II VEYYAMM
caf
E
(14)
(E(XMMY k:(Di)* <min(ese)
Here, (pj)* refers to the probability of sampling the k-th token at i-th token position at iteration t, which is the k-th column in pi. While Equation (14) provides a mathematically grounded definition of transition probability, also propose second, empirically motivated definition:
po($441 a1)
II
(15)
VEYCCU
We refer the definition in Equation (44 and Equation (15) as per and poz. respectively. Note that Pe2 is simplification of P81. focusing solely on the newly unmasked tokens Y.CaM and ignoring others. Despite this simplification, both definitions proved effective in our experiments, with pel offering greater theoretical rigor and pe2 offering computational simplicity.
With these definitions in place, now possess all the essential components needed to apply RL to MGMs. In the next subsection, we introduce several strategies to further enhance the performance of proposed method.
3.3 Enhancing Strategies
Building upon our framework in Section 3.1 and the two candidate transition probabilities in Sec- tion 3.2 introduce several practical strategies to enhance performance. These include: removing the KL constraint, applying reduction strategy, and filtering out low-quality samples.
Removing Kullback-Leinn
Constraint. Inspired by recent advances in LLMs 229
35
experiment with removing the KL constraint, setting 3 = 0 in Equation (IO). The motivation lies in the size of our base model approximately 1.3 billion parameters such models, the KL constraint may hinder explorations. As demonstrated in Section 4.3) removing the KL term improves the generation performance. Note that this somewhat contrasts with the findings in [36], where the KL term was shown to be beneficial. We attribute this discrepancy to the difference in model size: our model uses the 1.3B Show-o (27), while theirs employs the 2.5B Stable Diffusion 3.5 Medium (SD3.5-M) (5) This aligns with trends observed in LLMs where larger models benefit from KI. regularization, whereas smaller models may suffer from it.
which is relatively small. For
Reduction Strategy, As denoted in Section image synthesis in MGMs requires multiple unmasking iterations. While MGMs are more efficient than standard AR models or diffusion models, revisiting every step to compute the objective in Equation (TO) introduces significant computational overhead. Inspired by (37), which shows that even the first diffusion step captures key trajectory trends, we introduce the two following reduction strategies:
1. Computational reduction strategy, which computes the object in Equation 10) over only subset of the total iterations (e.g., the first or last 25 out of 50 steps).
2. Unmasking reduction strategy, which reduces total number of unmasking iterations during training (e.g., from T 50 to T 20), while maintaining the full unmasking schedule during evaluation.
The computational reduction strategy aims to prevent revisiting iterations, while the unmask- ing reduction strategy focuses on reducing the total number of unmasking iterations. As shown in Section 4.3] the second strategy accelerates training while only suffering slight performance degradation.

---

Table 1: Model Comparison GenEval.
Model
Params.
Single Obj.
Two Obj.
Counting
Colors
Position
Color Attri.
Overallf
Diffusion
Matching Models
PixArt-alpha
0.6B
0.98
0.50
0.44
0.80
0.08
0.07
0.32
SD1.5 39
0.9B
0.97
0.38
0.35
0.76
0.04
0.06
0.43
SD2.1 39
0.9B
0,98
0.51
0.44
0.85
0,07
0.17
0.50
LDM 39
1.4B
0,92
0.29
0.23
0.70
0.02
0.05
0.37
SD-XL
0.39
0.15
DALLE-2 41
6.5B
0.94
0.49
0.10
0.19
0.52
SD3.5-L 5
8B
0.98
0.89
0.73
0.83
0.34
0.47
0.71
DALLE-3
0.67
Antoregressive Models
LlamaGen 8
0.8B
0.32
0.71
0.34
0.21
0.58
0.07
0.04
JanusFlow 22
1.3B
0.97
0.59
0.45
0.83
0.53
0.42
0.63
Janus +
1.3B
0.97
0.68
0.30
0.84
0.46
0.42
0.61
SimpleAR-I.5B
lnfinity (+Rewiiter)
0.85
0.49
0.73
Chameleon
35
7B
0.39
Emu3 (+ (eReeriter)
46
8.5B
0.99
0.81
0.42
0.80
0,49
0.45
0.66
Masked Generative Models
MaskGen-XL
11
1.1B
0.99
0.61
0.55
0.81
0.13
0.31
0.57
Meissonic 12
0.99
0.66
0.42
0.86
0.10
0:22
0.54
Show-o 27
1,3B
0.95
0.52
0.49
0.82
0.11
0.28
0.53
Shov-o+Mask-GRoO
1.3B
0.99
0.69
0.35
0.73
Results models other than our approach are or original papers.
Sample Filtering. During RL training, observed frequent instability: reward scores often degrade temporarily, and the performance the final model stagnates. We identify this critical limitation as "Vanishing Samples'. We notice that similar trends have been concurrenlyy observed in [29] 21,, and they are attributed to equal rewards within the group of samples, resulting in:
Rd mean (RR)) 0.
(16)
This leads to zero advantages A in Equation 12 However, our setting differs from theirs in that we use CLIP-based reward model, not rule-based one. Therefore, exact reward duplication within group is unlikely. Instead, is more like a sample quality issue, where important positive samples within group become indistinguishable due to low standard deviation in rewards. This is particularly problematic when the reward model fails to accurately distinguish good from bad generations.
To mitigate this, we propose a dynamic sampling strategy that filters out low-quality samples. be specific, we set a dynamic filtering threshold based on the history standard deviation of the rewards within each group. If newly generated group reward standard deviation below the threshold, then we resample the group. In practice, we set as the lowest 10th percentile of the history standard deviation.
Table 2: Model Comparison MSCOCO 30K FID.
Model
Params.
FID-30K]
PixArt
0.6B
7.32
GigaCian
48
0.9B
9.09
SDL.5
0.9B
9.62
Janus
1.3R
7.11
LDM
39
1.4B
12.64
To summarize above, we present the framework of our Mask-GRPO with sample filtering and removing the KL constraint, which achieves the best performance in our experiments. In the following sections, the default Mask-GRPO ap- plies the poi defined in Equation (14). employs
DreamL.MM
49
7B
8.76
LWM
50
7B
12.68
DALL.E 51
12B
27.50
SEED-X
17R
14.99
Show-o
1.38
9.24
Show-o+Mask-GRPO
1.3B
8.32
sample filtering, removes the KIL constraint, and retains the full iteration steps during both training and evaluation.
7

---

Experiments
4.1 Experiment Setup
Training Settings. We conduct experiments using the training set of LAION dataset [59]
without accompanying images. Inspired by [25], we select prompts of varying lengths both short and long learning. The base model we selected is Show-o [77]. We employ CLIP (30) as the primary reward model. In ablation studies Section 4.3 additionally validate our approach with IeageReward [54)
they have been shown to provide more informative signals for reinforcement
Additional implementation details are provided in Appendix
Evaluation Details. Following Stable Diffusion [39 [5), we evaluate MASK-GRPO with standard T2I benchmarks GenEval [35] and FID [56] on the MSCOCO dataset [57]. In ablation studies
Section 4.3 additionally evaluate our method with ImageReward [54] for preference alignment. Notably, our training set does not contain any GenEval-style prompts. As a result, the GenEval [55]
prompts serve zero-shot evaluations, thereby providing robust assessment of generalization performance.
4.2
Main Results
Table [ presents the GenEval results, while Table reports the MSCOCO-30K FID score for our proposed Mask-GRPO. It can be observed that our approach significantly enhances the T2I capabilities of the base model, with an improvement GenEval to 0.73 and MSCOCO-300 FID to 8.32. These validate the effectiveness of
Qualitative comparisons illustrated in Figure 4 with additional visualization results included in Appendix D
Remarkably, Mask-GRPO outperforms all ex- isting state-of-the-an methods on GenEval [55] despite using a smaller base model (1.3B) than most competing approaches. Moreover, Mask-
appennafa af apharnafand ayhot minosss Soe apples clack vvda bick hoedlgg eede
baro aada gralle
Mack
well phone
Base
GRPO shows strong zero-shot generalization, GenEval (55) is evaluated under zero-shot set- tings in experiments. This also highlights the effectiveness
employing CLIP as the re- ward model in our framework. A further discus- sion on CLIP as the reward model provided in Section 4.4
SD1.
Onr
Moreover, we find that both of Pøi and P82 are useful, with the former converging faster.
Figure 4: Qualitative Comparisons on GenEval.
4.3 Ablation Studies
The results of all ablation studies are presented in Table
Removing KL. It be observed that removing the KL term significantly boosts performance, aligning with our hypothesis that KL may hinder exploration for small models. However, we also observe that the absence of the KL term makes Mask-GRPO more sensitive to hyperparameters, particularly the learning rate. A more detailed analysis provided in Appendix C
Reduction Strategy. Among the two reduction strategies, the unmasking approach proves to be more effective, achieving better balance between computational efficiency and perfor- mance. Although it entails slight performance drop, it reduces computational requirements by approximately 3-4x, making it highly practi-
Table 3: Ablation Results.
Method
GenEval
Show-o
0.53
Mask-GRPO
0,73
* Mask-GRPO w/ KI.
0.62
Mask-GRPO w/o sample filter
0.60
cal choice for RL fine-tuning of large models.
+ Mask-GRPO w/ Computation Reduction Strategy
0.56
Unmasking Reduction Strategy
Sample Filtering. The results indicate that filter- ing out low-quality samples improves the final
performance, supporting key insight that data quality plays crucial role in reinforcement learning

---

for text-to-image generation. Unlike LLMs, where curated RL fine-tuning datasets are increasingly available. the T2I domain currently lacks specialized datasets for RL fine-tuning. We hope that this gap will be addressed in future work to further advance the field.
Reward Model. Finally,
Mask-GRPO's
robustness under different reward models by re- placing CLIP with ImageReward [54] the reward model. Results in Table confirm that Mask- GRPPO consistently enhances the T2I generation performance across different reward models, thereby validating its generality.
Table 4: Results with ImageReward [4
Model
ImageReward
Show-o
1.02
Show-o ImageReward
1.23
Table 5: Additional Experiments on Relative Correctness.
Original Prompts
N
N
2N
2
'a photo of six clocks'
32.5298
35.8555
33.2857
'a photo of four handbags'
29.7800
33.0426
32.4762
'a photo of two backpacks'
28.4991
34.1256
29.7021
'a photo of two frisbees'
32.1401
36.8218
29.6494
'a photo of four toothbrushs
33.8095
38.2953
34.8197
'a photo of four vases'
28.2428
37.1507
29.9521
"a photo of two computer keyboards'
32.3883
37.2445
31.1868
'a photo of four baseball gloves'
29.5035
33.6917
31.3151
a photo of two beds'
33.5974
38.9766
31.7446
a photo of four giraffes'
29.2052
36.3132
32.4882
Overall
30.9696
35.1873
31.6620
4.4 Further Discuss
In this subsection, we further discuss the underlying reward mechanisms CLIP when used as the reward model. Although CLIP is known to have limitations in counting and attribute binding, nevertheless observe notable improvements these aspects, as shown in Table T
We believe the success of method stems from the design of GRPO, where we do not rely on absolute correctness, but rather on relative correctness within sample group. GRPO-based methods, we do not use raw rewards from the reward model directly. Instead, they are transformed into group-level normalized advantage (as shown in Equation 12) ). which effectively compares the relative correctness among samples within each group. Therefore, accurate "absolute correctness' for counting or attribute correctness, as long as it captures the relative trend, it can still drive the model to improve.
if CLIP cannot provide
To verify this, we conduct an additional experiment. We randomly select 10 CeeuEval-oountin-stye prompts (e.g. 'a photo of N items") and generate 10 correct images accordingly. We then modify each prompt to 'a photo of N. (correct) images. We computed the ELIP-scores for each prompt-image pair. As shown in Table 5 CLIP consistently assigns higher scores to prompts matching the correct count, demonstrating that can indeed capture 'relative correctness'. This justifies the use of CLIP within our framework and explains why Mask-GRPO can improve performance in counting and sttribute-binding tasks.
I items' and 'a photo of 2.N items', while keeping the originally generated
5 Conclusion and Future Work
In this paper, present Mask-GRPO, GRPO-based approach to improve MGMs in T2I generation. the first to introduce online RL to MGMs. We revisit the definition of transition probability meticulously, finding that the corresponding definitions in AR and diffusion models not feasible for MGMs. By redefining two candidate transition possibilities, we successfully integrate GRPO 20 into MGMs. Consequently, explore several useful strategies to further improve method.
While achieving superior results, Mask-GRPOY potential still not fully revealed. On the one hand, applying such methods on larger base models to explore possible scaling-up up will be interesting work.

---

On the other hand, generation always more complicated as it has to generate coherent outputs that are both contextually and temporally consistent.
potential on video generation is valuable, while text-to-vides
The reward model remains key challenge for T2I generation. First, unlike RL. post-training in L.Ms,, the application of RL to T2I generation depends on effective reward models for evaluating generated images. In this work, we adopt CLIP as the reward model, which proves effective. However, similar to other reward models, CLIP is still narrow in scope and not sufficiently accurate to support further improvements in RL post- post-training. Developing more robust and fair reward models, or jointly training them during RL, may help mitigate these limitations.
Acknowledgments and Disclosure of F Funding
This work supported in part by the Natural Science Foundation of Shenzhen (No. JCYJ202308 071 07111604008, No. JCYJ2024081 111100240011120000000 the Natural Science Foundation of Guangdong Province (No. 2024A151 2024A1515010033) National Key Research and Development Program of China (No. 2022YFB44004000 and Cross-disciplinary Fund for Research and Innovation of Tsinghua SIGS (No. JC2024002).
References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[2] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[3] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
[4] Keyu Tian, Yi Jiang, Zchuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 1788889--4886 2024.
[5] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[6] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
[7] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, et Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
[8] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint wrXiv:2406.06525 2024.
[9] Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2142-2152, 2023.
[10] Huiwen Chang. Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11315-11325, 2022.
[11] Dongwon Kim, Ju He, Qihang Yu, Chenglin Yang, Xiaohui Shen, Suha Kwak, and Liang- Chieh Chen. Democratizing text-to-image masked generative models with compact text-aware one-dimensional tokens. arXiv preprint arXiv: 2025.
10

---

[12] Jinbin Bai, Tian Ye, Wei Chow, Enxin Song, Qing Guo Chen, Xiangtai Li, Zhen Dong, Lei Zhu, and Shuicheng Yan. Meissonic: Revitalizing masked generative transformers for efficient high-resolution text-to-image synthesis. arXiv preprint arXiv:24
2024.
[13] Jiwan Hur. DongJae Lee, Gyojin Han, Jachyun Choi, Yunho Jeon, and Junmo Kim. Unlocking the capabilities of masked generative models for image synthesis via self-guidance. Advances in Neural Info Information Processing Systems, 37:130977-130999, 2024.
[14] Huiwen Chang. Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint 2xXi:23301000000 2023.
[15] Jacob Devlin, Ming-Woit Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language Technologies, volume I (long and short papers), pages 4171-4186, 2019.
[16] Richard Sutton, Andrew Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[17] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint vrXiv:2005.01445 2020.
[18] Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[19] Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412. 2024.
[20] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
[21] Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. VI- rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.088377 2025.
[22] Yi Yang. Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. Rl-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503. 1061, 2025.
[23] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.0/785, 2025.
[24] Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng. Can we generate images with cot? let's verify and reinforce image generation step by step. arXiv preprint arXivv250l. 2025.
[25] Junke Wang, Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv preprint arXiv:2504.1 1455, 2025.
[26] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023.
[27] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation, arXiv preprint arXiv:2408, 1528, 2024.
11

---

[28] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:100700637, 2017.
[29] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu. Xin Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. /4476, 2025.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 1748-87642 PmLR, 2021.
[31] Kevin P Murphy. Probabilistic machine learning: Advanced topics. MIT press, 2023.
[32] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883 2021.
[33] Yuchao Gu, Xintao Wang. Yixiao Ge, Ying Shan, and Mike Zheng Shou. Rethinking the objectives of vector-quantized tokenizers for image synthesis. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 7631 2024.
[34] Zhihong Shao, Peiyi Wang, Qihao Zhu, Rumxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 2xXv:2402.033300, 2024.
[35] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2078. 2025.
[36] Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.054700 2025.
[37] Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint arXiv:2412.. 14803, 2024.
[38] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426 2023.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[40] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprini arXiv:2307.0193: 2023.
[41] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. urXiv preprint arXiv:2204.06/25, 1(2):3 2022.
[42] Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al. Janusflow: Harmonizing autoregres- sion and rectified flow for unified multimodal understanding and generation. arXiv preprint urXiv:2411.0797. 2024.
[43] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint 2xXv:22410.18845 2024.
12

---

[44] Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. arXiv preprini arXiv:2412.04431, 2024.
[45] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. ar preprint 2405v:2405.098/1 2024.
[46] Quan Sun. Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. arXiv preprint arXiv:2307.0m 00522222 2023.
[47] Zhiyuan Yan, Junyan Ye, Weijia Li, Zilong Huang, Shenghai Yuan, Xiangyang He, Kaiqing Lin, Jun He, Conghui He, and Li Yuan. Gpt-imgeval: comprehensive benchmark for diagnosing gpt4o in image generation. arXiv preprint www.::2504.027777. 2025.
[48] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris. and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10124-10134, 2023.
[49] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang. Liang Zhao, Jian- jian Sun, Hongyu Zhou, Haoran Wei, et Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint urXivv2309... 1499, 2023.
[50] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv e-prints, pages arXiv-2402, 2024.
[51] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to- image generation. In International conference on machine learning, pages 8821-883 Pmlr, 2021.
[52] Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint 2tti::2044.44999 2024.
[53] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278-25299 2022.
[54] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neurul Information Processing Systems, 36: 1990--15935, 2023.
[55] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132-52152 2023.
[56] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[57] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
[58] Yifu Luo, Yongzhe Chang, and Xueqian Wang. Wavelet fourier diffuser: Frequency-aware diffusion model for reinforcement learning. arXiv preprint arXiv:2509, 19305, 2025.
[59] Haoyu Wang, Bingzhe Wu, Yatao Bian, Yongzhe Chang, Xueqian Wang, and Peilin Zhao. Probing the safety response boundary of large language models via unsafe decoding path generation. aixiv preprint 4rXiv:2408.10668 2024.
13

---

[60] Haoyu Wang, Zeyu Qin, Yifei Zhao. Chao Du, Min Lin. Xueqian Wang, and Tianyu Pang. Lifelong safety alignment for language models. arXiv preprint arXiv:2505.228 20259, 2025.
[61] Haoyu Wang, Guozheng Ma, Ziqiao Meng, Zeyu Qin, Li Shen, Zhong Zhang, Bingzhe Wu, Liu Liu, Yatao Bian, Tingyang Xu, et Step-on-feet tuning: Scaling self-alignment of IIms via bootstrapping. arXiv preprint arXiv:2402.07610, 2024.
[62] Haoyu Wang, Zeyu Qin, Li Shen, Xueqian Wang, Dacheng Tao, and Minhao Cheng. Safety reasoning with guidelines. arXiv preprint arXiv:2502.04044, 2025.
[63] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[64] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 1-.53728-537411 2023.
[65] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[66] Zining Zhu, Liang Zhao, Kangheng Lin, Jinze Yang, En Yu, Chenglong Liu, Haoran Wei, Jianjian Sun, Zheng Ge, and Xiangyu Zhang. Perpo: Perceptual preference optimization via discriminative rewarding. arXiv preprint arXiv:2502.0437/ 2025.
[67] Fei Wang. Wenxuan Zhou, James Y Huang. Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language models. arXiv preprint arXiv:2406. 1839, 2024.
[68] Liang Chen, Lei Li, Haozhe Zhao, and Yifan Song. Vinci. rl-v: Reinforcing super generalization ability in vision-language models with less than $3, 2025.
[69] Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai- Chang. Open- vithinker: An early exploration complex vision-language reasoning via iterative self- improvement. arXiv preprini 2503::2500.13322. 2025.
[70] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.0674 2025.
[71] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03000 03000, 2022.
[72] Yaron Lipman, Ricky TQ Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[73] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[74] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amn Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint wwXXv:2332.13977 2023.
[75] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018
[76] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.033 03326, 2024.
14

---

[77] Aaron Hurst. Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprini arXiv:2410.21276, 2024.
[78] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, pages 4055-4064. PMLR, 2018
[79] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high- figh-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[80] Minghui Hu, Chuanxia Zheng, Heliang Zheng, Tat-Jen Cham, Chaoyue Wang, Zuopeng Yang. Dacheng Tao, and Ponnuthurai N Suganthan. Unified discrete diffusion for simultancous vision-language generation. arXiv preprint arXiv:2211. 148422 2022.
14842,
[81] Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances Neural Information Processing Systems, 37:56424-56445, 2024.
[82] Haoyuan Sun, Bo Xia. Yongzhe Chang, and Xueqian Wang. Generalizing alignment paradigm of text-to-image generation with preferences through f-divergence minimization. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 27644-27652, 2025.
[83] Haoyuan Sun, Bo Xia, Yifei Zhao, Yongzhe Chang, and Xueqian Wang. Identical human prefer- ence alignment paradigm for text-to-image models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025.
[84] Haoyuan Sun, Bo Xia, Yifei Zhao, Yongzhe Chang, and Xueqian Wang. Positive enhanced preference alignment for text-to-image models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEFE, 2025.
[85] Haoyuan Sun, Bin Liang, Bo Xia, Jiaqi Wu, Yifei Zhao, Kai Qin, Yongzhe Chang, and Xueqian Wang.
rainbowpa: Improvements integrated preference aligmment for diffusion-basee text-to-image generation. Transactions on Machine Learning Research, 2025.
[86] Haoyuan Sun, Jiaqi Wu, Bo Xia, Yifu Luo, Yifei Zhao, Kai Qin, Xufei Lv, Tiantian Zhang, Yongzhe Chang, and Xueqian Wang. Reinforcement fine-tuning powers reasoning capability of multimodal large language models. arXiv preprint arXiv:2505.188 18536, 2025.
[87] Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang. et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.078.0 2025.
15

---

A Related Work
A.1 Reinforcement Learning in Large Language Models
Prominent approaches like Reinforcement Learning from Human Feedback (RLHF) 63 and Direct Prefer- ence Optimization (DPO) [64] have demonstrated remarkable success in enhancing the safety and instruction- ceruuciio-following capabilities of LLMs 663 655 and multimodal LLMs (MLLMs) [66] 67]. Re- cently, the emergence of OpenAI ol [9 and DeepSeek-R. R1 (20) has sparked renewed interest in incentivizing reasoning capabilities in LLMs via RL. Notably, the GRPO algorithm (34] utilizes the rule-based reward design, demonstrating the huge potential for large-scale RL applications on LLMs.
RL 16-18 58
has emerged as pivotal paradigm for refining LLMs 19
20
59
62
Inspired by this, recent works aim to improve the original GRPO in LLMs 22 or attempt to apply to MLLMs [68 77].
A.2 Visual Generation Models
Diffusion Models. In recent years, numerous diffusion-based methods [ 39 41 40 38 have
demonstrated exceptional capabilities in T2I generation. In this framework, the model is tasked with predicting the added Gaussian noise. Recently, flow matching 771 72) follows the core idea of diffusion models but treats generation as learning a continuous- time normalizing flow, achieving competitive visual generation results with fewer denoising steps [5]. However, these methods still suffer from their unstable controllability and low inference speed due to the multi-denoising steps.
Autoregressive Models. The transformer models with autoregressive output schemes have demon- strated great success modeling both language and multi-modality generation [73/77]. Inspired by such progress, series of works [78 32. [1 45 utilize such autoregressive paradigms with causal attention to learn the dependency of image pixels for image generation. More recently, (8) also demonstrated that images can be generated through the LLMs' architecture in autoregressive way. However, such raster-order autoregression suffers from high computational cost and performance bottlenecks when synthesizing high nighutessoltiin and high-fidelity images. [79].
Masked Generative Models. To address the challenges of diffusion models and standard AR models, TaskGiT [10 first introduces new bidirectional transformer for image synthesis modeling image generation as a mask prediction problem. During training, trained on a similar proxy task to the mask prediction in Bert [15), while at inference time it adopts novel paralleled decoding method to synthesize an image in a constant number of steps. Inspired by its success, more MGMs [14] 80 start to emerge and have gained significant success in T2I generation. Recently, this approach has been effectively extended by Show-o (27] and MAR ET, in the aspect of visual unification and continuous-van vallued tokens integration. Considering the efficiency and recent progress of MGMs, it is worthwhile to further improve their performance by RL.
ceesstandnnggeneraiti
A.3 Reinforcement Learning in Text-to-Image Generation
The success of RL for LLMs also incentivizes research to try RL on T2I generation. Prior works mainly focus on aligning pretrained T2I models with human preferences (82 improving aesthetics and semantic consistency. Very recently, inspired by GRPO 20 new approaches have been proposed
trying apply ERPO-based methods on T2I generation 86 36 87). However, all these works done in diffusion fefuusion-bbsedd models or standard AR models, overlooking the masked generative models. In this paper, we trying to fill this gap.
B Masked Generative Models as Discrete Diffusion Process
The theory of discrete diffusion in generative models focuses on applying a Markov process to corrupt data in discrete states, followed by a reverse process for reconstruction. The forward process ( (1) is represented by a categorical distribution:
g(
16

---

where Qi is the transition matrix, and the corruption process can be tailored with different structured matrices, such as uniform or absorbing state models.
B.1 Forward Process
The forward process gradually corrupts data by replacing each token with either a new token or an absorbing state (like [MASK] token). For discrete tokens, the transition matrix Qe is formed by a product of matrices over time Qt = Q1Q2 ... Qt. The forward process can be mathematically expressed as:
g @at(
where Qt may represent structured transition patterns like Gaussian kernels, nearest-neighbor map- pings, or absorbing states.
B.2 Reverse Process
The reverse process po po( (a4)) is learned to denoise the corrupted data, starting from a noisy state IT and moving towards zo. The reverse process can be parameterized by a neural network predict the most likely previous state given the current noisy state:
po pe(21-1(zz)
Σ (( (o)p)
20
This reverse process learns to undo the corruption step-by-step, ideally predicting the original data distribution.
B.3 Structured Denoising Diffusion in Discrete State Spaces
In Discrete Denoising Diffusion Probabilistic Models (D3PM), the forward process is generalized by incorporating structured transition matrices, allowing for richer and more controlled data corruption processes. These models do not require embedding continuous data into latent continuous spaces but instead operate entirely in discrete spaces.
B.3.1 Transition Matrices for Discrete Diffusion
The choice of the transition matrix Qt plays a crucial role in the quality of the model. There are several variants, such as:
Uniform Transition:
This is a simple case where each state has an equal chance of transitioning any other state.
Absorbing State Models: Here, token may either stay the same or transition into an absorbing state (e.g., [M ASK)).
Discretized Gaussian: For ordinal data like images, the transition probabilities are biased toward states that are closer in terms of similarity or embedding space.
The corruption process is defined Appendix B.I where the transition matrix Q2 is constructed from domain knowledge data-specific structure. The reverse process aims recover the original data 2o using learned model.
B.4
Variational Loss Function for D3PM
To train D3PM models, variational loss function is used, which combines both the log-likelihood of observed data and the KL divergence between the forward and reverse distributions. The loss function for training D3PMs is expressed as:
LEIBO Ey(co)
DRL. (xlll(())( po
17

---

For the discrete case, the transition matrices and their structure determine how the data is corrupted and reconstructed, and the objective is to minimize the difference between the corrupted and reconstructed data over all steps.
C Implementation Details
All experiments are conducted on 16 NVIDIA A 80GB GPUs. The learning rate is set as 3e and the group size G set is 6. We utilize Adam as optimizer and set beta to 0.95, and we set the batch size to 96 (6 rollouts per GPU for one prompt).
During training, we find that it is more sensitive when we remove the KL divergence constraint. Specifically, when we use the KL constraint, we can set the learning rate as 5e while at most 3e 6 when removing the KL divergence constraint. The training will be collapsed if we set learning rate of and remove the KL constraint at the same time.
D
Visualization
Some image generation results are shown in Figure 5 The corresponding prompts are (each line, from left to right):
Line 1:
Create an image of a cat as gardener, wearing a straw hat, gardening gloves, and surrounded by colorful flowers.
Anime illustration of Princess Mononoke from Studio Ghibli, by artgerm, stunning artwork.
Futuristic cyberpunk city at night, neon lights, high-tech car, vibrant colors, cinematic lighting, highly detailed, sci-fi atmosphere, 8k resolution, unreal engine.
A whimsical pink cloud-shaped building with minimalist windows and doors. floating above a vibrant blue sky with cotton- like clouds, Studio Ghibli-style animation movie texture.
Line 2:
A white puppy sitting playfully in autumn leaves, surrounded by fallen red apples, soft natural lighting.
Heroic elf warrior, golden glowing background, detailed fantasy armor, cinematic lighting, epic fantasy art, high detail.
Vibrant city skyline during sunset, modern skyscrapers, colorful abstract style, warm gradient sky, digital art, urban landscape, vivid colors.
Mystical forest with glowing mushrooms and a babbling brook.
Line 3:
Futuristic metallic humanoid robot, highly detailed face, sci-fi background, cinematic lighting, dystopian cityscape, 4K resolution.
A cute duck wearing a chef uniform covered in cookie batter, unreal engine render 8k.
A realistic Venusaur animal among the trees, forest lake, moss, cold weather, dark teal and amber, Sony A7 IV.
4d photographic image of full body image of cute little chibi boy realistic, vivid colors octane render trending on artstation, artistic photography, photorealistic concept art, soft natural volumetric cinematic perfect light, UHD no background.
18

---

Figure 5: Visualization results of Mask-GRPO.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
We provide details for our theory.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide our code in the github link.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
We provide our code in the github link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwiicccguie//eeeeeeeeeis CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See our code link and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Section
5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: See Section S
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All creators or original owners of assets (e.g., code, data, models), used in the paper, has been properly credited; the license and terms of use have been explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
All models will be released upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
25

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.M policy (https: hattop///aarrpppc.cc///offeerere for what should or should not be described.
26


Reviews:
[{"review_id": "SgjYLuTdxL", "reviewer": "Reviewer_QEdg", "summary": "This work introduces Mask-GRPO, a reinforcement learning (RL) method specifically designed for masked generative models in text-to-image (T2I) generation. The proposed method applies Group Relative Policy Optimization (GRPO) in a novel way by framing the image unmasking process as a multi-step decision-making problem, incorporating a new definition of transition probability. The model's performance is further enhanced through several key strategies, including the removal of the Kullback–Leibler constraint, a reduction strategy, and the filtering of low-quality samples.", "questions": "See Weakness.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The idea of formulating the progressively unmasking image content in MGM as a multi-step decision problem is novel, and also aligns with the RL framework naturally. \n2. In the state transition,  the proposed method treats newly unmasked tokens at each step as the decisive factor. Based on this, this paper proposes two new methods for calculating transition probabilities that are more suitable for MGM. The approaches seem effective from the experimental results.\n\n**Weaknesses**\n1. The paper claims a 38% performance improvement on GenEval over the baseline. The stated baseline appears to be Show-O 1.3B, but according to Table 2, the reported improvement is from 0.68 to 0.73. Could the 38% figure be incorrect?\n2. In the ablation study, both of the proposed reduction strategies appear to significantly degrade performance, with results even falling below the Show-O baseline. Does this suggest that these two strategies may not be effective or meaningful?\n3. Among the various evaluation metrics in GenEval, slight improvements are observed across all categories except for colors. However, the reward model used is CLIP, which is known to have limited capabilities in understanding object quantity and positional relationships, but a relatively strong ability to perceive color. How can this discrepancy be explained?\n4. There are some typos that the author should address. For example, on line 79, \"in-troduce\" is incorrectly hyphenated. In the references on line 522, there's a formatting error. Figure 1 appears to be incomplete. In Figure 4, the horizontal axis is labeled \"GenEval score,\" but the vertical axis represents the corresponding metrics; presenting this information in a table might be more effective.", "was_revised": true, "final_justification": "I thank the authors for their efforts on the manuscript and rebuttal. Before the rebuttal, I raised concerns about the statistical error, category discrepency, and \\etc. The authors have addressed these concerns well in their response. Therefore, I maintain the borderline accept rating.", "rebuttal": "Dear Reviewer QEdg,\n\nWe sincerely appreciate your thoughtful feedback and encouraging recognition of our novel idea and well-supported methodology. Your careful reading and insightful questions have been invaluable. Below we address each of your concerns in detail.\n\n# Q1: Table Statistics Error\n> The paper claims a 38% $\\cdots$ be incorrect?\n\nThank you for your careful review. The reported **38% improvement in GenEval performance is corret**, as shown in Figure 4. However, we sincerely apologize for an error in Table 2 of the original submission. Specifically, we **mistakenly reported an incorrect baseline score for Show-o**. The correct GenEval score of Show-o, as observed in our experiments, is 0.53, and our method achieves 0.73, resulting in the 38% improvement. Below is the corrected Table 2:\n\n**Table 2.** Model Comparison on Geneval. Results are from [1] or their original papers.\n| Model | Params | Single Obj. | Two Obj. | Counting | Colors | Position | Color Attri. | Overall |\n| ----- | ------ | ----------- | -------- | -------- | ------ | -------- | ------------ | ------- |\n| Show-o | 1.3B | 0.95 | 0.52 | 0.49 | 0.82 | 0.11 | 0.28 | 0.53 |\n\nNote that our experimental result is consistent with [1]. Similar variations are also found in other text-to-image (T2I) works[2,3]. We apologize for any confusion this may have caused and will revise Table 2 accordingly in the updated manuscript.\n\n# Q2: Reducing Strategy\n> In the ablation study $\\cdots$ effective or meaningful?\n\nThank you for raising this point. The GenEval scores for the two reduction strategies are:\n\n- Computation Reduction Strategy: 0.56  \n- Unmasking Reduction Strategy: 0.66\n\nBoth outperform the baseline (0.53), although they are below the Mask-GRPO (0.73). However, we emphasize that these strategies, especially the unmasking reduction strategy, are effective and meaningful due to **better sample effiency** and **lower training cost**. To be brief, **the unmasking reduction strategy achieves convergence using only 20% of the samples, resulting in a 25% performance improvement on GenEval while reducing training costs by 60%–80%.** \n\nWe begin with a brief recap of the definition of our unmasking reduction strategy: it reduces the total number of unmasking iterations during training (in our experiment, from $T$ = 50 to $T$ = 10), while maintaining the full unmasking schedule during evaluation. Importantly, in GRPO-based training (our method), one sample does not correspond to a single prompt or image prompt or one image, but rather to a single unmasking iteration, since the loss is computed iteratively at each step (see Equation (10)). Therefore, by applying the unmasking reduction strategy, we reduce both the number of unmasking iterations and the sample count to just 20%. Despite using only 20% of the samples, this strategy delivers a 25% performance gain onGenEval. Moreover, it leads to a 60%~80% reduction in training cost. For these reasons, we conclude that the unmasking reduction strategy offers superior sample efficiency and is indeed effective.\n\nThe computation reduction strategy also reduces training cost by computing loss over only  a subset of the total iterations. However, the unmasking conduction strategy is more effective than the computation reduction strategy, as detailed in Section 4.3.\n\nIn conclusion, both strategies provide meaningful trade-offs between efficiency and performance, especially in online reinforcement learning (RL) settings。\n\nWe appreciate your insightful question and hope these clarifications help. \n\n# Q3: Category Discrepancy\n> Among the various evaluation $\\cdots$ discrepancy be explained?\n\nThank you for pointing this out. First of all, we apologize again for the errors in Table 2. After correcting the baseline statistics (Q1), we confirm that our method improves GenEval performance across all categories, including colors. Please find the updated results below:\n\n**Table 2.** Model Comparison on Geneval. Results are from [1] or their original paper.\n|Model|Params|Single Obj.|Two Obj.|Counting|Colors|Position|Color Attri.|Overall|\n|-|-|-|-|-|-|-|-|-|\n|Show-o[2]|1.3B|0.95|0.52|0.49|0.82|0.11|0.28|0.53|\n|**Show-o + Mask-GRPO**|1.3B|**0.99**|**0.90**|**0.69**|**0.85**|**0.35**|**0.59**|**0.73**|\n\nHowever, we **acknowledge that the improvement in the 'Colors' category is relatively smaller**, while our reward model, clip model, has a strong ability to perceive color. We attribute this to the **limitation of the base model, which is the nature of RL-based post training**. \n\nMask-GRPO is a RL-based post-training method in T2I generation. As discussed in recent works [4,5,6] on large language models, the power of RL-based post training is limited by the base model. RL is improving the performance by unlocking the potenial of the base model, 'teaching' it to 'climb higher' to where it is capable but does not know how to reach. However, RL cannot create new capabilities beyond the base model’s inherent capacity. Therefore, we believe the smaller gains in 'Colors' category reflect a performance ceiling of the base model in handling color semantics, which our method can optimize but not fundamentally surpass. We notice that [3] has similary Geneval results after applying RL to the same base model, which supports our explainations.\n\nThank you again for your question. We hope we address the concern.\n\n# Q4: Typos and Figure 4\n> There are some typos $\\cdots$ be more effective.\n\nThank you for the suggestion. We will thoroughly review and correct all typos in the revision to ensure clarity and readability. We sincerely apologize for any inconvenience caused to your reading due to typos.\n\nRegarding Figure 4, we agree that presenting the results as a table improves clarity. We propose the following revised table:\n\n**Table 4.** Ablation performence on Geneval.\n| Method Varient | Geneval Score |\n| ----- | ------ |\n| Base Model | 0.53 | \n| + Mask-GRPO | 0.73 |\n| + Mask-GRPO w/KL | 0.62 |\n| + Mask-GRPO w/o Sample Filter | 0.60 |\n| + Mask-GRPO w/ Computation Reduction Strategy | 0.56 |\n| + Mask-GRPO w/ Unmasking Reduction Strategy | 0.66 |\n\n# References\n[1] Yan, Zhiyuan, et al. \"Gpt-imgeval: A comprehensive benchmark for diagnosing gpt4o in image generation.\" arXiv preprint arXiv:2504.02782 (2025).\n\n[2] Guo, Ziyu, et al. \"Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step.\" arXiv preprint arXiv:2501.13926 (2025).\n\n[3] Liu, Jie, et al. \"Flow-grpo: Training flow matching models via online rl.\" arXiv preprint arXiv:2505.05470 (2025).\n\n[4] Gandhi, Kanishk, et al. \"Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.\" arXiv preprint arXiv:2503.01307 (2025).\n\n[5] Yue, Yang, et al. \"Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?.\" arXiv preprint arXiv:2504.13837 (2025).\n\n[6] Shah, Darsh J., et al. \"Rethinking reflection in pre-training.\" arXiv preprint arXiv:2504.04022 (2025)."}, {"review_id": "uBrNjTf65D", "reviewer": "Reviewer_Vae1", "summary": "This paper proposes to apply Group Relative Policy Optimization (GRPO) to text-to-image Masked Generative Models (MGMs). Specifically, the authors frame MGM inference as a Markov decision process, and explore different options to implement the transition probability between decoding steps.\n\nIn addition, they revisit various reinforcement learning training practices and optimizations in this new masked generative modeling context.\n\nFor experiments, they start from a Show-o base model, use CLIP as the reward function, train on LAION text prompts, and evaluate performance on GenEval and MSCOCO-30K FID.", "questions": "Questions are moslty related to my concerns presented here-above:\n- Are there any regularities in the types of prompts that tend to fail the variance threshold? If a resampled group still fails to pass, what happens? Is it repeatedly resampled?\n- What mechanism do the authors believe allows Mask-GRPO to improve on GenEval metrics, and counting and attribute binding tasks, while only relying on CLIP as the reward model?\n- Could the authors provide the text prompts corresponding to the samples shown in Figure 5? This would help readers better interpret the qualitative results, especially w.r.t. the comment about diversity l.299.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n- Applying GRPO to masked generative models is, to my knowledge, a novel proof of concept. I would add that, unlike autoregressive and diffusion models -- where the training objective and inference process are naturally aligned -- masked generative models suffer from a training–inference mismatch. Reinforcement learning provides a principled way to bridge that gap, making this contribution particularly interesting.\n- The authors experiment with different formulations for the transition probability between decoding steps, and systematically reassess the impact of multiple RL training stabilizations and optimizations in this setting.\n- Mask-GRPO achieves very competitive improvements over the base Show-o model on standard text-to-image metrics.\n\nWeaknesses\n- I find the paper not very clearly presented, with some formulations that are confusing or imprecise. For instance, Equation 13 is described as “the product of every [masked] token probability” (line 165), but also as “the joint probability of all newly predicted tokens” (line 37), which seems contradictory. The second phrasing is potentially misleading, as it could be interpreted to mean the method is modeling the joint distribution of token values (as is the goal in generative modeling, e.g., autoregressive or diffusion models), whereas, as far as I understand, the policy is actually modeling the probability of selecting tokens for unmasking. Clarifying this distinction would avoid confusion.\n\n- The paper claims that low reward variance within a group motivates the sample filtering approach. While the filtering is ablated and shown to be effective, it would be useful to back this initial claim, maybe with qualitative examples, for example, showing sets of images of different variance.\n\n- Related to sample filtering, it is unclear whether resampled groups are repeatedly evaluated and rejected until they pass the variance threshold. If so, are there certain bad prompts that never pass and get stuck? \n\n- Using CLIP as a reward model could inherently limit the improvements Mask-GRPO can achieve, given CLIP’s well-established weaknesses such as counting and attribute binding. In fact, GenEval was explicitly designed to address the shortcomings of the CLIP score. As such, it seems surprising that the proposed method improves performance across the board on GenEval, and even shows gains in counting and attribute binding (as suggested by Figure 3). This apparent contradiction should be discussed in more detail.\n\n- There seems to be quite a few typos. Some I noted are\nl.296. presented, l.235  seems to be lacking a figure reference, l.229 includes an unnecessary dot inside a sentence", "was_revised": true, "final_justification": "I find the premise of the paper very interessing, and the discussions about GRPO could elevate CLIP as a judge particularly enlightning.\nMoreover, my comments have been very thoroughly adressed. \nI hope the paper is accepted.", "rebuttal": "Dear Reviewer Vae1,\n\nThank you very much for your comprehensive and detailed review of our paper, as well as for recognizing the novelty and effectiveness of our work. Please find our point-by-point responses below.\n\n# Q1: Sample Filter\n> Are there any regularities in the types of prompts $\\cdots$ Is it repeatedly resampled?\n\nThank you for your detailed question. Yes, if a sample group is rejected, it will **repeatdly resample itself until it either passes the varience threshold or reaches the maximum number of attempts** (set to 3 in our experiments to balance computation costs).\n\nIn our experiments, we observed that **certain prompts consistently fail to meet the variance threshold within the allowed number of attempts**. Below are several representative examples:\n\n- '6ad74e6708fc4 Dresses For 14 Year Girls Online Shopping | Dresses For 14 Year ...'  \n- 'Halloween-Images-For-Kids-Clip-Art.png'\n- 'Tags: K-Pop, Kim Chung-ha, Serious, Make Up, Lemon, Blue Background, Nail Polish, Fruits, Yellow Outfit, Yellow Dress, Blooming Blue'\n- '6947 Cooper Point Rd NW, Olympia, WA 98502 (#1486235) :: Platinum Real Estate Partners'\n- 'How to beat travel stress on family holidays'\n- 'Round Trivet - #25 - Tree of Life (3 colour option)'\n- 'Beautiful 1970s boho lace maxi dress'\n- 'How to Determine Your Body Type | Jalisa's Fashion Files'\n- 'Palm tree clip art'\n- 'Watercolor Holiday Characters - image 4 - student project'\n- '2018 Transit 150 Low Roof,  Empty Cargo Van #T80223 - photo 5'\n\nWe found that these prompts share some common characteristics:\n\n- **Non-Descriptive or Unstructured Prompts**: Many appear to be taken directly from web page titles or article headers (e.g., ’6ad74e6708fc4 Dresses For 14 Year Girls Online Shopping | Dresses For 14 Year ...‘), often including irrelevant or noisy content, and lacking clear visual instruction.\n\n- **Lack of Visual Specificity and High Ambiguity**: Some prompts are abstract or stylistically vague. They do not clearly specify the visual information, which causes high ambiguity. For instance, 'How to beat travel stress on family holidays' is conceptually broad with no clear visual target; 'Palm tree clip art' suggests a style but does not describe the image clearly; and 'Halloween-Images-For-Kids-Clip-Art.png' leaves many details undefined (e.g., number of characters, composition, setting).\n\nTo summarize, these 'hard-pass' prompts lack of specific visual guidance, which leads to high ambiguity and frequent rejection by our variance-based filter.\n\nThank you again for the question. We hope we clarify the details well.\n\n# Q2: CLIP Model\n> What mechanism do  $\\cdots$ only relying on CLIP as the reward model?\n\nThank you for this insightful question. It is well known that the CLIP model has limited ability in counting and attribute binding. However, we believe the success of our method stems from the design of GRPO, where **we do not rely on absolute correctness, but rather on relative correctness within a sample group.**\n\nIn GRPO-based methods (ours), we do not use raw rewards from the reward model directly. Instead, we transform them to group-level normalized advantage $A_t^j$ (see Equation (12)), which is effectively **comparing the relative correctness among samples within a group** [1].  Therefore, even if CLIP model cannot provide an accurate 'absolute correctness' for counting or attribute correctness, as long as it **capture the relative trend**, it can still drive the model to improve.\n\nTo verify this, we conducted an additional experiment. We randomly selected 10 GenEval-counting-style prompts (e.g. 'a photo of $N$ items') and generated 10 correct images accordingly. We then modified each prompt to 'a photo of $\\frac{N}{2}$ items' and 'a photo of $2N$ items', while keeping the originally generated (correct) images. We computed the CLIP scores for each prompt-image pair using our reward model. The results are as follows:\n\n**Table 5.** Additional Experiments on Relative Correctness\n| Original Prompts | $\\frac{N}{2}$ | $N$ | $2N$ |\n| ----- | ------ |------ |------ |\n| 'a photo of six clocks' | 32.5298 | 35.8555 | 33.2857 |\n| 'a photo of four handbags' | 29.7800 | 33.0426 | 32.4762 |\n| 'a photo of two backpacks' | 28.4991 | 34.1256 | 29.7021 |\n| 'a photo of two frisbees' | 32.1401 | 36.8218 | 29.6494 |\n| 'a photo of four toothbrushs' | 33.8095 | 38.2953 | 34.8197 |\n| 'a photo of four vases' | 28.2428 | 37.1507 | 29.9521 |\n| 'a photo of two computer keyboards' | 32.3883 | 37.2445 | 31.1868 |\n| 'a photo of four baseball gloves' | 29.5035 | 33.6917 | 31.3151 |\n| 'a photo of two beds' | 33.5974 | 38.9766 | 31.7446 |\n| 'a photo of four giraffes' | 29.2052 | 36.3132 | 32.4882 |\n| **Overall** | **30.9696** | **35.1873** | **31.6620** |\n\nAs shown, CLIP consistently assigns higher scores to prompts matching the correct count, demonstrating that it can indeed capture 'relative correctness'. This justifies the use of CLIP within our GRPO framework and explains why Mask-GRPO can improve performance in counting and attribute-binding tasks.\n\nWe appreciate your valuable question and hope this additional experiment provides further clarity.\n\n# Q3: Text Prompts of Figure 5\n> Could the authors provide the text prompts $\\cdots$ the comment about diversity l.299.\n\nThank you for your careful suggestion. Below are the text prompts corresponding to each image in Figure 5 (each line, from left to right):\n\nLine 1:\n\n- Create an image of a cat as a gardener, wearing a straw hat, gardening gloves, and surrounded by colorful flowers\n\n- Anime illustration of Princess Mononoke from Studio Ghibli, by artgerm, stunning artwork\n\n- Futuristic cyberpunk city at night, neon lights, high-tech car, vibrant colors, cinematic lighting, highly detailed, sci-fi atmosphere, 8k resolution, unreal engine\n\n- A whimsical pink cloud-shaped building with minimalist windows and doors, floating above a vibrant blue sky with cotton-like clouds, Studio Ghibli-style animation movie texture\n\nLine 2:\n\n- A white puppy sitting playfully in autumn leaves, surrounded by fallen red apples, soft natural lighting\n\n- Heroic elf warrior, golden glowing background, detailed fantasy armor, cinematic lighting, epic fantasy art, high detail\n\n- Vibrant city skyline during sunset, modern skyscrapers, colorful abstract style, warm gradient sky, digital art, urban landscape, vivid colors\n\n- Mystical forest with glowing mushrooms and a babbling brook\n\nLine 3:\n\n- Futuristic metallic humanoid robot, highly detailed face, sci-fi background, cinematic lighting, dystopian cityscape, 4K resolution\n\n- A cute duck wearing a chef uniform covered in cookie batter, unreal engine render 8k\n\n- A realistic Venusaur animal among the trees, forest lake, moss, cold weather, dark teal and amber, Sony A7 IV\n\n- 4d photographic image of full body image of a cute little chibi boy realistic, vivid colors octane render trending on artstation, artistic photography, photorealistic concept art, soft natural volumetric cinematic perfect light, UHD no background\n\nWe sincerely apologize for the omission and will include these prompts in the revised version.\n\n# W1: Unclear Presentation\n> I find the paper not very clearly presented $\\cdots$ Clarifying this distinction would avoid confusion.\n\nThanks for your suggestion. We agree that the phrasing in l.37 may be misleading. We will revise this sentence to clarify the distinction and avoid potential confusion. We sincerely apologize for any inconvenience caused.\n\n# W2: Qualitative Presentation\n> The paper claims that low reward variance $\\cdots$ showing sets of images of different variance.\n\nThank you for this valuable suggestion. We agree that visualizing image sets with different reward variances would better convey our ideas - particularly in combination with the discussion in Q2. We will include such examples in the revised version.\n\n# W5: Typos\n> There seems to be quite a few typos $\\cdots$ inside a sentence.\n\nThank you for pointing this out. We will thoroughly proofread the manuscript and correct all typographical and grammatical errors in the revision. We sincerely apologize for any disruption this may have caused to your reading experience. \n\n# References\n[1] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv preprint arXiv:2501.12948 (2025)."}, {"review_id": "IE8Npl2WB5", "reviewer": "Reviewer_Ydx6", "summary": "This paper introduces Mask-GRPO, a method to incorporate Group Relative Policy Optimization (GRPO)-based online reinforcement learning (RL) into masked generative models (MGMs) for text-to-image (T2I) generation. The authors redefine the transition probability in RL for MGMs, formulating the unmasking process as a multi-step decision-making problem. They propose two candidate definitions for transition probability and explore several enhancement strategies, including removing the Kullback-Leibler (KL) constraint, applying a reduction strategy, and filtering out low-quality samples. The method shows improvement compare with the base model (Show-o) on the GenEval benchmark and  MSCOCO-30K FID.", "questions": "1. Can you offer more details on equation (14)?\n2. Can you provide more evaluations on T2I tasks other than Geneval?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. The experiment explores different probability calculation methods when treating the generation process of MGM as a Markov process, with a clear motivation.  \n\n2. According to the results reported in the paper, Mask-GRPO improves the performance of show-o on Geneval and COCO-30K FID.  \n\n3. The paper is well-structured, with clear explanations of the methodology.  \n\n### Weaknesses\n1. The font size in the figures is too small—it should ideally match the font size of the main text.  \n\n2. Equation (14) appears to be the core part of the proposed method. However, the details of Equation (14) in the paper are not sufficiently clear. Could more details about Equation (14) be provided?  \n\n3. In addition to the evaluation on Geneval, could more evaluations on the latest text-to-image (t2i) benchmarks, such as DPG-Bench and WISE, be included? Geneval has actually been around for a long time, and evaluation on newer benchmarks would better demonstrate the method's effectiveness.  \n\n4. There are some spelling errors in the paper, such as \"in-troduce\" in line 79 and \"settingβ=0\" in line 212.", "was_revised": true, "final_justification": "Authors address my concerns, so I maintain my score.", "rebuttal": "Dear reviewer Ydx6,\n\nThank you for your comprehensive review of our paper. We greatly appreciate your recognition of the clear motivation and strong empirical results of our method. Please find our detailed responses to your concerns below.\n\n# Q1: More Details on Equation (14)\n> Can you offer more details on equation (14)?\n\nThank you for your question. We would like to offer a more in-depth explanation of Equation (14), in conjunction with Figure 2 (in our paper). In brief, Equation (14) **computes the sum of probabilities over all possible sampling paths that result in the same transition. The first term of the right-hand side represents the product of probabilities for newly unmasked patches (tokens), while the second term corresponds to the product of probabilities for newly remasked patches**.   \n\nLet us first revisit the definition. In Figure 2, we compute the transition probability for iteration $t = 0$, which is $p_\\theta (s_{1} | s_0, a_0)$ in Equation (14). For simplicity, assume the image consists of only 4 patches (index from 1 to 4), and let $i$ denote the patch index. Also assume that the visual codebook size is 4, with token ids from 1 to 4, denoted by $k$. We define $p_t^i$ as the softmax distribution at index $i$ at iteration $t$, and $(p_t^i)^k$ as the probability of sampling token id $k$ at index $i$. At $t = 0$, the initial state $s_0$ is fully masked, and only one patch needs to be unmasked. \n\nAs shown in Figure 4, assume the softmax distributions are: \n\n- $p_0^1$ = $[0.1, 0.2, 0.6, 0.1]^T$\n\n- $p_0^2$ = $[0.5, 0.2, 0.1, 0.2]^T$\n\n- $p_0^3$ = $[0.3, 0.4, 0.2, 0.1]^T$ \n- $p_0^4$ = $[0.15, 0.2, 0.15, 0.5]^T$. \n\nThe key insight is that **as long as the patch $i = 1$ samples token $k = 3$ with probability 0.6, regardless of what other patches sample, the resulting transition is always the same - first patch remains unmasked while the others are remasked**. This is because when patch $i = 1$ samples token $k = 3$, its confidence score $cs_0^1$ = 0.6 is the highest among all $cs_0^i$, regardless of the other samples. According to Equation (5), this leads to the following Choose and Move (CaM) operations: \n\n- $CaM_0^1 = 1$\n\n- $CaM_0^2$ = $CaM_0^3$ = $CaM_0^4 = 0$\n\nIn other words, the first patch is unmasked while the others are remasked.\n\nReturning to Equation (14), we compute the $p_\\theta (s_{1} | s_0, a_0)$ based on Figure 2. Here we have:\n\n- $Y_0^{CaM}$ = {1}, which means the newly unmasked patch index with $CaM_0^i = 1$\n\n- $Y_0^M \\backslash Y_0^{CaM}$ = {2, 3, 4}, which means the newly remasked patch index with $CaM_0^i = 0$\n\nThen the first term of Equation (14) is simply $cs_0^1$ = 0.6, representing the 'probability product of newly unmasked patches'. For the second term, we can rewrite it as $\\prod_{i = 2, 3, 4} (\\sum_{k} (p_0^i)^k)$, where k needs to be $(p_0^i)^k < min (cs_0) = cs_0^1$ = 0.6. However all $k = 1 , 2, 3, 4$ meet this requirment, no matter what $i$ is. Therefore, the second terms equals to $(0.5+ 0.2 + 0.1 + 0.1) \\times (0.3 + 0.4 + 0.2 + 0.1) \\times (0.15 + 0.2 + 0.15 + 0.5) = 1$, which is the 'probability product of newly remasked patches'. Putting it all together we have $p_\\theta (s_{1} | s_0, a_0) = 0.6 \\times 1 = 0.6$, which is obvious illustrated in Figure 2.\n\nTo summrize, Equation (14) accurately computes the real transition probability, which AR-style (see Equation (13)) cannot capture. It explicitly distinguishes between newly unmasked and newly remasked patches and calculates their probabilities accordingly.\n\nWe sincerely appreciate your question and apologize for the lack of detail of Equation (14). These clarifications will be incorporated into the revised version.\n\n# Q2: Other Benchmarks\n> Can you provide more evaluations on T2I tasks other than Geneval?\n\nThank you for this valuable suggestion. We have conducted additional experiments on WISE[1], The results are as follows:\n\n**Table 7.** Model Comparison on WISE[1].\n|Model|Params|Culture|Time|Space|Biology|Physics|Chemistry.|Overall|\n|-|-|-|-|-|-|-|-|-|\n|Show-o|1.3B|0.28|0.40|0.48|0.30|0.46|**0.30**|0.35|\n|**Show-o + Mask-GRPO**|1.3B|**0.53**|**0.48**|0.46|0.41|0.35|0.21|**0.45**|\n|vlla-u-7b-256|7.0B|0.26|0.33|0.37|0.35|0.39|0.23|0.31|\n|Orthus-7B-base|7.0B|0.07|0.10|0.12|0.15|0.15|0.10|0.10|\n|Orthus-7B-instruct|7.0B|0.23|0.31|0.38|0.28|0.31|0.10|0.10|\n|Janus-Pro-1B|1.0B|0.20|0.28|0.45|0.24|0.32|0.16|0.26|\n| Janus-Pro-7B|7.0B| 0.30     | 0.37 | 0.49  | 0.36    | 0.42    | 0.26      | 0.35    |\n| JanusFlow-1.3B |1.3B | 0.13     | 0.26 | 0.28  | 0.20    | 0.19    | 0.11      | 0.18    |\n| Harmon-1.5B|1.5B  | 0.38     | **0.48** | **0.52**  | 0.37    | 0.44    | 0.29      | 0.41    |\n| SD-v1-5  |1.0B  | 0.34     | 0.35 | 0.32  | 0.28    | 0.29    | 0.21      | 0.32    |\n| SD-2-1   |1.5B     | 0.30     | 0.38 | 0.35  | 0.33    | 0.34    | 0.21      | 0.32    |\n| SD-XL-base-0.9 |N/A     | 0.43     | **0.48** | 0.47  | **0.44**    | 0.45    | 0.27      | 0.43    |\n| SD-3-medium |N/A    | 0.42     | 0.44 | 0.48  | 0.39    | **0.47**    | 0.29      | 0.42    |\n| FLUX.1-schnell|12B      | 0.39     | 0.44 | 0.50  | 0.31    | 0.44    | 0.26      | 0.40    |\n\nAs shown, Mask-GRPO achieves substantial gains in WISE, with up to 29% improvement, demonstrating the generalizability of our method beyond GenEval.\n\nHowever, due to time and computational constraints, we have not yet completed the evaluation on DPG-Bench. We will update the results in the second phase of the rebuttal. Thank you for your understanding.\n\nWe will include these new benchmarks in the revised version of our paper. Thank you again for the helpful suggestion.\n\n# W1: Font size\n> The font size in the figures $\\cdots$ match the font size of the main text.\n\nThank you. We will revise all figures to ensure the font sizes are consistent with the main text in the updated manuscript.\n\n# W4: Spelling Errors\n> There are some spelling errors $\\cdots$ in line 212.\n\nThank you for pointing this out. We will thoroughly proofread the manuscript and correct all typographical and grammatical issues. We sincerely apologize for any inconvenience this may have caused in your reading experience.\n\n# References\n[1] Niu, Yuwei, et al. \"Wise: A world knowledge-informed semantic evaluation for text-to-image generation.\" arXiv preprint arXiv:2503.07265 (2025)."}, {"review_id": "6EptUvgIqm", "reviewer": "Reviewer_5UPe", "summary": "This paper introduces **Mask-GRPO**, a novel approach that integrates online reinforcement learning (RL) into the underexplored paradigm of **Masked Generative Models (MGMs)** for text-to-image (T2I) generation. While prior RL-based methods have focused primarily on diffusion and autoregressive models, this work is the first to apply **Group Relative Policy Optimization (GRPO)** to MGMs. The authors reformulate the token unmasking process in MGMs as a multi-step Markov decision-making problem and propose two new definitions of the transition probability to better align with the unique structure of MGMs. To further enhance training efficiency and stability, they introduce several strategies including removing the Kullback–Leibler (KL) constraint, reducing the number of unmasking iterations, and filtering out low-quality samples based on reward variance. Experimental results on GenEval and MSCOCO benchmarks show that Mask-GRPO significantly improves over the base model, achieving 38% and 10% gains respectively, and demonstrates strong zero-shot generalization. These findings highlight the potential of applying online RL to MGMs, especially in settings with limited model size and data.", "questions": "1. **Clarification on Transition Probability Formulations (Eq. 14 vs. Eq. 15)**\nCould the authors provide further intuition and possibly a visual explanation for the two proposed transition probability definitions?\nIn particular, what are the implications of including the re-masked token terms in Eq. 14 versus excluding them in Eq. 15 in terms of bias/variance or convergence behavior?\nA more detailed comparison in both theory and experiment would help justify the final choice of `pθ₁`.\n\n2. **Robustness to Reward Models**\nThe current implementation heavily depends on CLIP for reward evaluation. Have the authors considered alternative reward models (e.g., human feedback, aesthetic scoring models, or task-specific metrics)?\nIt would be valuable to understand whether Mask-GRPO's performance holds under different reward functions or if it's highly tuned to CLIP.\nIncluding at least one additional reward model in the ablation would strengthen the generality claim.\n\n3. **Generalization to Other MGMs**\nThe paper uses Show-o as the sole base model. Have the authors tested Mask-GRPO on other MGM variants (e.g., MaskGIT or Meissonic)?\nResults on a second MGM would reinforce that the method is not overfitted to a specific architecture or training schedule.\n\n4. **Sample Efficiency and Compute Costs**\nThe method introduces strategies like iteration reduction and sample filtering to reduce overhead, but a more detailed analysis of the compute/resource trade-off would be helpful.\nFor example: how many GPU-hours were saved by unmasking reduction vs. how much performance was sacrificed?\n\n5. **Theoretical Justification or Guarantees**\nAlthough the GRPO framework is well-established, could the authors elaborate on why it is particularly well-suited for the masked generation setting?\nAre there theoretical intuitions or constraints under which GRPO is expected to perform better than PPO or DPO in MGMs?\n\n---", "limitations": "**No.**\n\nWhile the paper is technically thorough and presents strong empirical results, it does **not sufficiently discuss the limitations or potential societal impacts** of applying reinforcement learning to text-to-image generation using masked generative models. Some constructive suggestions for improvement:\n\nThe authors could explicitly address the following:\n\n   * The **reliance on CLIP** as a reward model, which may encode biases or misalignments with human judgment.\n   * The **scalability** of Mask-GRPO to larger models or datasets, particularly in terms of compute and memory cost.\n   * The potential **overfitting** or reward hacking issues in long training runs, which are common in RL-based systems.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### **Strengths**\n\n\n  The paper makes a novel and timely contribution by introducing the first reinforcement learning (RL) method tailored for **Masked Generative Models (MGMs)** in text-to-image generation. Most prior work in this area has been focused on diffusion and autoregressive models. The proposal to apply **Group Relative Policy Optimization (GRPO)** in this new setting is both original and well-motivated.\n\n\n  The methodology is sound and well-grounded. The authors carefully reformulate the unmasking process of MGMs as a multi-step Markov decision process and propose two principled definitions of transition probability, demonstrating a deep understanding of the underlying generative mechanism. Enhancements like KL removal, iteration reduction, and sample filtering are empirically validated and contribute to practical robustness.\n\n\n  The proposed Mask-GRPO achieves strong empirical performance, outperforming a strong MGM baseline (Show-o) by significant margins: 38% on GenEval and 10% on MSCOCO FID. These improvements are notable given the small model size (1.3B parameters), suggesting high sample efficiency and scalability potential. This work opens the door for further RL applications in efficient, non-autoregressive generative frameworks.\n\n\n  The paper is generally well-structured, with clear exposition of the motivations, methodology, and experimental design. Key ideas, such as the redefinition of transition probability and the rationale for each enhancement strategy, are communicated effectively with supporting figures and ablations.\n\n---\n\n### **Weaknesses**\n\n\n  While the main ideas are well-explained, some technical details—especially regarding the exact computation of transition probabilities and sampling logic in the proposed definitions—could benefit from further clarification or simplification. Equations (14) and (15), while correct, may be difficult to parse without strong background knowledge.\n\n\n  The paper lacks a deeper theoretical analysis of the convergence or stability properties of Mask-GRPO in the MGM setting. While empirical evidence supports its effectiveness, a theoretical justification for why the new transition probability definitions lead to better learning would further strengthen the work.\n\n\n  The evaluation is conducted on a single base model (Show-o) and specific benchmarks (GenEval and MSCOCO FID). It is unclear how generalizable the method is across different masked generative models or reward functions beyond CLIP-based scoring.\n\n\n  The paper relies heavily on CLIP as the reward model, but does not explore or compare alternative reward formulations. This limits understanding of how robust or sensitive the method is to different types of reward signals.\n\n---", "was_revised": true, "final_justification": "After carefully rereading the manuscript, I find that the authors’ rebuttal provides a clearer explanation than the paper itself. This leaves me with mixed impressions: while the method may be solid, the submitted manuscript is largely ineffective at conveying its ideas. Nonetheless, my overall rating is now somewhat more positive.", "rebuttal": "Dear Reviewer 5UPe,\n\nThank you for your time and thoughtful feedback on Mask-GRPO. We are pleased that you recognize the novelty and effectiveness of our method. Please find our responses below:\n\n# Q1: Clarification on Transition Probability Formulations\n> Could the authors provide further $\\cdots$ justify the final choice of $p_{\\theta 1}$.\n\nThanks for your valuable question. Due to the new regulation this year, we are not allowed to provide any visual contents in rebuttal. However, we are happy to elaborate a more in-depth discussion of $p_{\\theta 1}$ and $p_{\\theta 2}$, in Equation (14) and Equation (15), respectively. \n\nIn brief, $p_{\\theta 2}$ is a simplification of $p_{\\theta 1}$, focusing solely on the newly unmasked tokens and ignoring others. While $p_{\\theta 1}$ offers greater theoretical rigor (real transition probability) and lower bias in theory, we found them both effective in our experiments, resulting almost the same GenEval performance, as shown in Table 1 (in the paper).\n\nA slight advantage $p_{\\theta 1}$ has over $p_{\\theta 2}$ is its robustness to the learning rate. While $p_{\\theta 2}$ focuses solely on the newly unmasked tokens, it is more sensitive to the learning rate. The default learning rate we use is 3e-6 for $p_{\\theta 1}$, and 2e-6 for $p_{\\theta 2}$, respectively.  \n\nThanks for your question. We hope we explain well under the visual content limitation.\n\n# Q2: Robustness to Reward Models\n> The current implementation heavily depends on CLIP $\\cdots$ strengthen the generality claim.\n\nThanks for your valuable suggestion. We have conducted an additional experiment with two more reward models, ImageReward[1] and UnifiedReward[2]. We follow the same setting in default Mask-GRPO and the results on GenEval are shown below: \n\n**Table 8.** Performence of Various Reward Models.\n| Reward Model | Geneval Score |\n| ----- | ------ |\n| Baseline | 0.53 | \n| ClIP | 0.73 |\n| ImageReward | 0.70 |\n| UnifiedReward | 0.65 |\n\nAs shown, both ImageReward and UnifiedReward significantly outperform the baseline (by 32% and 23%, respectively), demonstrating the adaptability and effectiveness of Mask-GRPO across different reward models. Importantly, these results were achieved without any fine-tuning of hyperparameters, indicating potential for even higher performance with model-specific tuning.\n\nWe appreciate your suggestion and will include these results in the revised manuscript.\n\n# Q3: Generalization to Other MGMs \n> The paper uses Show-o  $\\cdots$ specific architecture or training schedule.\n\nThanks for your question. Actually, the decoding strategy of our base mdoel, Show-o, keeps the same with MaskGit. Moreover, they share a similary model architecture in terms of image generation. Therefore, we tend to believe that our Mask-GRPO can also work well in other MGMs like MaskGIT.\n\nFurthermore, we are trying to apply Mask-GRPO on Meissonic. However, due to limited time and computational resource, we have not finished it yet. We promise we will keep updating it. Thanks for your understanding.\n\nThanks for your question.\n\n# Q4: Sample Efficiency and Compute Costs\n> The method introduces strategies like iteration reduction $\\cdots$ how much performance was sacrificed?\n\nThanks for your valuable questions. To be brief, **the unmasking reduction strategy achieves convergence using only 20% of the samples, resulting in a 25% performance improvement on GenEval while reducing training costs by 60%–80%**; while sample filtering does not aim to reduce the training cost, instead it **stablizes the training and gains a improved performance, as shown in Figure 4 (in the paper)**    \n\nWe begin with a brief recap of the definition of our unmasking reduction strategy: it reduces the total number of unmasking iterations during training (in our experiment, from $T$ = 50 to $T$ = 10), while maintaining the full unmasking schedule during evaluation. Importantly, in GRPO-based training (our method), one sample does not correspond to a single prompt or image prompt or one image, but rather to a single unmasking iteration, since the loss is computed iteratively at each step (see Equation (10)). Therefore, by applying the unmasking reduction strategy, we reduce both the number of unmasking iterations and the sample count to just 20%. Despite using only 20% of the samples, this strategy delivers a 25% performance gain onGenEval. Moreover, it leads to a 60%~80% reduction in training cost. To be more specific, it reduces the training hours from 230 A100 GPU hours to 80 hours in our experiments.  \n\nFor sample filtering, we discard low-variance sample groups to better highlight relative advantages $A$. This **enhances learning signal and stability**, ultimately leading to improved performance.\n\nThanks for your question. \n\n# Q5: Theoretical Justification or Guarantees \n> Although the GRPO framework $\\cdots$ expected to perform better than PPO or DPO in MGMs?\n\nThanks for your insightful questions. The theoretical advantages of GRPO over PPO primarily stem from reduced bias and improved training stability, as discussed in [3,4]. These properties are particularly valuable in the masked generative modeling (MGM) setting, where multi-step iterative generation introduces long-term credit assignment challenges.\n\nWe appreciate your interest in the theoretical underpinnings and plan to explore more rigorous justifications in future work.\n\n# Reference\n[1] Xu, Jiazheng, et al. \"Imagereward: Learning and evaluating human preferences for text-to-image generation.\" Advances in Neural Information Processing Systems 36 (2023): 15903-15935.\n\n[2] Wang, Yibin, et al. \"Unified reward model for multimodal understanding and generation.\" arXiv preprint arXiv:2503.05236 (2025).\n\n[3] Yue, Yu, et al. \"Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.\" arXiv preprint arXiv:2504.05118 (2025).\n\n[4] Yu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" arXiv preprint arXiv:2503.14476 (2025)."}]

Author rebuttal:

Meta-review: Summary of Reviews:

The paper introduces Mask-GRPO, a reinforcement learning (RL) method tailored for masked generative models (MGMs) in text-to-image (T2I) generation. By framing the unmasking process as a multi-step Markov decision problem, the method applies Group Relative Policy Optimization (GRPO) with novel transition probability definitions. Mask-GRPO incorporates enhancements like KL constraint removal, iteration reduction, and sample filtering, achieving improvements on benchmarks like GenEval and MSCOCO-30K FID.

Strengths of this paper include:
- Novelty: First RL approach designed specifically for masked generative models, bridging the training-inference mismatch in MGMs.
- Methodology: Reformulates MGM unmasking as a Markov decision process with principled transition probability definitions.
- Empirical Performance: Outperforms the baseline (Show-o) with significant improvements (e.g., 38% on GenEval, 10% on MSCOCO FID) while maintaining efficiency with a small model size (1.3B parameters).
- Practical Enhancements: Strategies like KL removal and sample filtering contribute to the model's robustness and scalability.

Weaknesses raised by the reviewers:
- Limited Theoretical Analysis: The paper lacks deeper theoretical justification for the stability and convergence of Mask-GRPO.
- Evaluation Scope: Experiments are confined to a single base model (Show-o) and benchmarks (GenEval, MSCOCO FID), with no exploration of newer datasets or benchmarks (e.g., DPG-Bench, WISE).
- Dependence on CLIP: Relies heavily on CLIP as the reward function without exploring alternative reward models, limiting generalizability.
- Technical Complexity: Equations (e.g., 14, 15) and some methodology details (e.g., transition probabilities) are difficult to follow and could benefit from clearer explanations.
- Manuscript Quality: Typographical errors, unclear figure labels, and insufficient details in some sections detract from the paper’s presentation.

Conclusion:
Mask-GRPO is a novel and promising approach for applying RL to MGMs in text-to-image generation, with strong empirical results and practical contributions. Concerns from reviewers mainly included the limited generalization, reliance on CLIP, and lack of theoretical analysis reduce its overall impact. During the rebuttal, reviewers are mostly satisfied by the added clarifications and experiments from the authors. Therefore, publication is recommended had the authors included these revisions and clarifications in the final version as they promised.
Decision: Accept (poster)