Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO
Authors: Chengzhuo Tong
Ziyu Guo
Renrui Zhang
Wenyu Shan
Xinyu Wei
Zhenghao Xing
Hongsheng Li
Pheng-Ann Heng
Venue: NeurIPS 2025 poster ()
Keywords: Large Multi-modal Models
Image Generation
Reinforcement Learning
Chain-of-Thought (CoT) Reasoning
Primary area: deep_learning
TL;DR: We compare RL methods DPO & GRPO for image generation, showing their strengths & how rewards affect generalization. Explores scaling for better CoT-based synthesis.

Abstract: Recent advancements underscore the significant role of Reinforcement Learning (RL) in enhancing the Chain-of-Thought (CoT) reasoning capabilities of large language models (LLMs). Two prominent RL algorithms, Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO), are central to these developments, showcasing different pros and cons. Autoregressive image generation, also interpretable as a sequential CoT reasoning process, presents unique challenges distinct from LLM-based CoT reasoning. These encompass ensuring text-image consistency, improving image aesthetic quality, and designing sophisticated reward models, rather than relying on simpler rule-based rewards. While recent efforts have extended RL to this domain, these explorations typically lack an in-depth analysis of the domain-specific challenges and the characteristics of different RL strategies. To bridge this gap, we provide the first comprehensive investigation of the GRPO and DPO algorithms in autoregressive image generation, evaluating their ***in-domain*** performance and ***out-of-domain*** generalization, while scrutinizing the impact of ***different reward models*** on their respective capabilities. Our findings reveal that GRPO and DPO exhibit distinct advantages, and crucially, that reward models possessing stronger intrinsic generalization capabilities potentially enhance the generalization potential of the applied RL algorithms. Furthermore, we systematically explore ***three prevalent scaling strategies*** to enhance both their in-domain and out-of-domain proficiency, deriving unique insights into efficiently scaling performance for each paradigm. We hope our study paves a new path for inspiring future work on developing more effective RL algorithms to achieve robust CoT reasoning in the realm of autoregressive image generation.

Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO
Chengzhuo Tong" Ziyu Guo Renrui Zhang Wenyu Shan +3 Xinyu Wei
Zhenghao Xing', Hongsheng Li Pheng-Ann Heng
CUHK 'MIXR & "MMLab SPeking University *Shanghai Al Lab
'Equal Contribution
Project Leader
*Corresponding Author
Abstract
Recent advancements underscore the significant role of Reinforcement Learning (RL) in enhancing the of-ingof-Thhuugt (CoT) reasoning capabilities of large language models (LLMs). Two prominent RL algorithms, Direct Preference Op- timization (DPO) and Group Relative Policy Optimization (GRPO), are central these developments, showcasing different pros and cons. Autoregressive im- age generation, also interpretable as a sequential CoT reasoning process, presents unique challenges distinct from LLM-based CoT reasoning. These encompass ensuring text-image consistency, improving image aesthetic quality, and designing sophisticated reward models, rather than relying on simpler rule-based rewards. While recent efforts have extended RL to this domain, these explorations typically lack an in-depth analysis of the domain-specifi specific challenges and the characteristics of different RL strategies. To bridge this gap, we provide the first comprehensive investigation of the GRPO and DPO algorithms in autoregressive image generation, evaluating their in-domain performance and out-of-domain generalization, while scrutinizing the impact different reward models on their respective capabilities. Our findings reveal that GRPO and DPO exhibit distinct advantages, and crucially, that reward models possessing stronger intrinsic generalization capabilities poten- tially enhance the generalization potential of the applied RL algorithms. Further- more, we systematically explore three prevalent scaling strategies to enhance both their in-domain and out-of-domain proficiency, deriving unique insights into effi- ciently scaling performance for each paradigm. We hope our study paves a new path for inspiring future work developing more effective RL algorithms to achieve robust CoT reasoning in the realm of autoregressive image generation. Code is released https:// githhu..b.. com com/ZiyuGu099) Iaail//9mmmmmmmmemeeee Image- Generation-Col.
Introduction
Recent large language models (LLMs) [33, 34, 47, 59, 64] have demonstrated remarkable achieve- ments in diverse challenging tasks, such as mathematical problem-solving [4, 19, 32] and code generation [8, 5, 22]. This is driven by their emergent and robust reasoning capabilities through extended Chain-of- Thoughtss (CoT) [52, 24, 18, 65, 17, 66], as exemplified by models like OpenAl's ol [35], DeepSeck-R1 [16], and Kimi k1.5 [45]. These predominant advancements in reasoning abilities are primarily facilitated by reinforcement learning (RI.) [3, 61, 30] methods, applied during post-training, which elicits deliberate, stepwise reasoning process before reaching final answer.
Among the prominent RL algorithms employed for fine-tuning LLMs, two representative approaches stand out, i.e., Direct Preference Optimization (DPO) [39] and Group Relative Policy Optimization
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Paradigm Comparison
Impact of the Reward Model
Desaie Nevas Reward Mald
DPO's generalizatio
DPO ourperforms GRPO in in-domain evaluations
more sensitive to reward model choice than GRPO.
GRPO generalizes better than DPO
A better reward model Can potentially cohance generaiization.
mut-of-omminn.
GRIO
Bauale
GRPO
Effects of Scaling Strategies
GRPO Folling Tresd
ZITO Siilig Tend
Scaling sampling tends yield GRPO's best in-domain gains efficiently.
Iterative training tends maximize
DPO's in-domain gains but
compromises generaiization.
Scaling in-domain improves GRPO generalization but risks diminishing returns.
Moderate sampling enhances DPO; cacessive sampling induces bias.
Stalig Rate
Aucing
Dau Scaling
Heatte
Scaling in-domain boosts DPO'ss in- and out-of-domain performance.
ScullnggSooling
Figure I: Investigation for GRPO and DPO in Autoregressive Image Generation. We analyze the ad advantages of GRPO and DPO in both in-domain and out-of-domain scenarios (Top-left), the effect of different reward models (Top-right), and the influence of scaling strategies (Bottom), providing unique insights to this field.
(GRPO) [43]. DPO offers compelling advantages in reduced training time and computational cost, while achieving substantial performance gains, especially in tasks with simpler and shorter CoT rea- soning [37, [7]. Nevertheless, DPO's effectiveness diminishes in more complex reasoning scenarios due its reliance on static, pre-collected data, making increasingly prone noise and biases as task complexity rises [57]. In contrast, recent studies underscore GRPO's superior ability to tackle complicated challenges requiring intricate CoT reasoning [16]. It achieves this by iteratively refining policies using self- celf-generated data, effectively adapting to complex task distributions. Nevertheless, GRPO consistently incurs significantly higher computational costs and protracted training durations, as it samples the policy online and continuously during training.
In parallel, image generation [26, 12, 38, 40], of the most fundamental tasks in multi-modalige also requires substantial reasoning knowledge. Notably, autoregressive generation models [44, 55, 53] can be viewed as form of CoT reasoning similar that of LLMs, they sequentially predict visual tokens in step-by-step manner. One preliminary work, 'Image Generation with CoT" [18] has verified the feasibility of DPO in this domain, and follow-up studies [23, 48. 58] experiment with GRPO. However, these endeavors lack in-depth investigation, since the CoT in visual generation exhibits several critical distinctions from textual reasoning. First, image generation tasks often involve prompts describing diverse scenarios, ranging from intricate, detailed descriptions to concise, templated formats. This frequently exhibits substantial domain gap, with generative models typically excelling in one domain but may struggle in another. Consequently, evaluating the efficacy of RL from both in-domain and oft-of-domain perspectives is crucial. Second, distinct from the often verifiable, rule-based rewards of LLMs, image generation objectives, which rely on diverse criteria like iext-image alignment and human aesthetic preferences, necessitate a thorough exploration of how different reward models affect the performance of RI. algorithms. Third, while recent works, such as [18], have preliminarily adopted scaling strategies like itcrative-DPO [37] to improve performance, the impact of many widely used scaling strategies still remains insufficiently explored. Given these challenges, we raise the question: How does the performance of GRPO compare to that of DPO in autoregressive image generation, and which aspects may influence their optimal performance, e.g., reward models and scaling strategies?
To this end, we conduct systematic investigation comparatively evaluating GRPO and DPO regarding their capacity for COT reasoning autoregressive image generation. We adopt Janus-Pro [9] as our baseline autoregressive generative model, evaluating its in-domain performance on T21- CompBench [21] and out-of-domain generalization on GenEval [15]. Specifically, as illustrated in Figure 1, our investigation delves into three primary aspects concerning both GRPO and DPO:

---

In-Domain Performance vs Out-of- Generalization. Examining both in on-domain and out-of-domain scenarios enables comprehensive evaluation of a model's ' robustness in handling prompts of varying granularity. In our settings, we leverage preference data derived from a unified reward model, conducting rigorous comparative analysis.
Observation: 1)
The off-policy DPO method demonstrates superior performance on in- domain tasks compared to GRPO. 2) Conversely, GRPO exhibits stronger generalization capabilities, outperforming DPO on the dut-o--domain benchmark.
Impact of Different Reward Models. Reward models define the preference distribu- tion, with diverse models [54, 56, 51, 18] serving this role in text-to-image generation. Understanding how the choice reward model influences policy outcomes, particularly generalization, is crucial for preference-basee methods. To this end, we investigate the differential impact of employing various reward models on the generalization performance of GRPO and DPO, revealing how preference variations shape algorithmic capabilities.
Observation: 1) DPO exhibits greater sensitivity to reward model variations than GRPO, manifesting larger out-of-domen performance fluctuations. 2) A reward model with superior generalization can potentially improve the generalization performance of RL algorithms.
Investigation of Effective Scaling Strategies. Investigating how prevalent scaling strategies affect RL's in-domain and out-of-domain performance offers key insights into enhancing model adaptability across tasks. We examine three dimensions: scaling sampled images per prompt, scaling in-domain training data diversity and volume, and adopting an iterative training approach inspired by Iterative-DPO.
Observation: GRPO: 1) Scaling sampled images tends to yield more computationally efficient in-domain gains. 2) Moderate scaling of sampling size and in-domain data improve generalization, but excessive scaling risks overfitting. DPO: 1) Scaling iterative training tends to maximize in-domain performance but compromise generalization after multiple cycles. 2) Moderate sampling sharpens preference contrast, optimizing both in-domain and out-of-domain performance, while excessive sampling induces bias. 3) Scaling in- domain data enhances both in-domain and generalization performance by miligating biases introduced by the limited preference scope of small datasets.
In summary, our core contributions are as follows:
We present the first comprehensive empirical study comparing GRPO and DPO algorithms for autoregressive image generation, highlighting their respective strengths and providing unique insights into the future advancement this field.
By evaluating the inherent generalization capabilities of various reward models and their influence on both GRPO and DPO, we demonstrate that enhancing the generalization capacity of reward models potentially boosts the overall generalization performance of RL.
We systematically investigate scaling behaviors across multiple critical dimensions, includ- ing variations in the number of sampled images per prompt, the scale of the in-domain training data, and the deployment of iterative training paradigms, with the analysis of their differential impact on in-domain performance and out-of -fo-ai-domain generalization, yielding numerous valuable insights and highlighting promising avenues for future research.
Our Investigation
Group Relative Policy Optimization (GRPO) [43] and Direct Preference Optimization (DPO) [39] have demonstrated distinct advantages for fine-tuning large language models (LLMs). this study, following the step of 'Iaaee Generation with CoT" [18], we conduct a meticulous and systematic investigation aiming to evaluate the efficacy of GRPO and DPO in image generation and identify more impactful strategies for scaling performance.
3

---

Table I: In-Domain Performance of GRPO and DPO under Various Reward Models. We assess the performance on the T2l-CompBench benchmark [21]. comprehensively evaluating different
reward models including HPS [54], ImageReward [56], Unified Reward [51], Fine-tuned ORM [18], and 'Metric Reward'.
Reward Type
Average
Color
Attribute Binding
Shape
Texture
Spatial
Object Relationship
Non- Spa-Spatial
Complex
GRPO
DPO
GRPO
DPO
GRIO
DPO
GRPO
DPO
GRPO
DPO
GRPO
DPO
GRPO
DPO
Baseline
38,56
34.28
48.90
30.51
HPS
$0.49
53.90
77.39
85.25
53.59
64.72
71.54
76.08
31.14
25.29
31.10
31.17
39.20
40.89
ImageRwdd
49.76
55.67
76.57
86.18
52.58
64.92
70.71
76.20
28.
34.72
31.10
31.53
38.80
40.47
Unifed Rwd
48.01
53.91
74.34
82.88
50.28
61.74
68.66
76.64
27.
30.40
30.87
31.25
36,10
40.52
ORM
53.11
79.19
60.87
75.58
76.76
35.97
31.10
31.26
Metric Rwd
50.01
57.81
77.21
87.83
53.66
65.91
73.17
79.19
28.60 28.69
40.76
30.87
31.53
36,97
41.64
Table 2: Out-of-Domain Performance of GRPO and DPO under Various Reward Models.
We assessed the generalization performance of both algorithms on the out-of-domain GenEval dataset [15], with reward models aligning consistently with Table 1.
Reward Type
Overall
Color Altr
Counting
Position
Single Objoct
Twu Object
Colors
GRPO
DPO
GRPO
DPO
GRPO
DPO
GRPO
DPO
GRPO
DPO
GRPO
DPO
GRPO
DPO
Baseline
78.04
76.25
88.03
HPS
79.18
77.31
63.00
70.25
60.62
48.12
78.75
67.25
99.69
90.15
86.87
98.75
86.17
89.36
ImageRwd
79.26
77.67
59.50
69.25
62.50
43.44
$0.50
71.25
98.44
99.69
87.37
90.66
87.23
91.76
Unifsed Rwd
80.99
80.03
67.52
70.54
61.16
$5.69
82.02
75.21
99.58
99.00
88.40
91.50
86.96
88.21
78.86
65.75
79.50
75.24
88.03
2.1 Overview
Task Formulation. To enable the applicability of the prevailing two RL techniques, namely GRPO and DPO, we focus on the autoregressive image generation task, demonstrated by models such as LlamaGen [44], Show-o [55], and Janus-Pro [9]. These models employ a data representation and output paradigm analogous to that of LLMS and large multimodal models (LMMs). while attaining comparable performance to continuous diffusion models [20]. Specifically, they leverage quantized autoencoders [13] transform images into discrete tokens, enabling the seamless integration of loss mechanisms from both DPO and GRPO during the post-training phase.
Experimental Settings. We select Janus-Pro as our baseline model for this investigation, a latest autoregressive image generation model with advanced capabilities. To comprehensively evaluate the effectiveness of various RL strategies, we assess the text-to-image generation performance on both in-domain and -ft-oi-domain benchmarks. In-domain performance is evaluated using T2I- CompBench [21], which features long, detailed prompts designed to compose complex scenes with multiple objects, displaying various attributes and relationships. Out-of-domain generalization is examined with GenEval [15], utilizing short, templated prompts starting with "a photo of, to assess robustness to concise and standardized textual prompts. the subsequent sections, we explore three pivotal dimensions to investigate GRPO and DPO for implementing thain-of-thought reasoning in autoregressive image generation: in-domain vs. out-of-domain performance (Section 2.2), the impact of reward models (Section 2.3), and the effectiveness of scaling strategies (Section 2.4).
2.2 In-Domain Performance vs Out-of-Domain Generalization
Recently, the application of reinforcement learning (RL) has yielded substantial breakthroughs in lifting the reasoning capabilities of LLMs via Chain-of-Thought (CoT) techniques [52, 35, 11, 45, 2, 1]. Existing researches indicate that GRPO, the on-policy RL algorithm, can data-efficiently enhance performance on in-domain (ID) tasks without sacrificing capabilities on out-of-domain (OOD) tasks [28, 1, 14, 29]. Nevertheless, direct comparison of on-policy GRPO and off off-policy Deep Policy Optimization (DPO) regarding their ID and OOD performance under equivalent training data remains underexplored. Drawing inspiration from this, we train and evaluate these two algorithms within the domain of autoregressive image generation to elucidate their comparative strengths.
GRPO. GRPO improves upon Proximal Policy Optimization (PPO) by eliminating the learned value critic and estimating advantages directly through group-wise normalization of rewards across

---

Rt-Domoin
Out-off-Domoin
Jonus-Pro
wy DPO
av 68PO
Jonus-Pro
uu DRO
w/ SBPO
DPO outperforms GRPO in in- domain evaluations.
The smooth, glovey reforr of the lake reflested the fuory colors of wasse,, serone oasis of trampulbly and reflection.
Thee shorp, ongularr edge of the diamond refracted the light iuto dozsling disglay brilliance and darety, gomstone of sapanilleed beadly
A--oomain
Out of Domain
Janus-Pu
w/
Janus-Pro
GRPO generalizes better than DPO out-of- domain.
photo of an orange matorcydl pink
photo of a hackpack right of
Figure 2: Visualization Results of In-Domain vs. Out-of-Domain Performance Comparison.
multiple responses per query. Our empirical investigation initiates with GRPO training phase, employing group size of four completions per query with the hyperparameter for iteration times set to I, following the standard GRPO workflow. Building on the design reward models, as elaborated in 2.3, we focus here the curation of training data for GRPO. To facilitate fair comparison between GRPO and DPO, we ensured both methods utilized identical training data. As GRPO's s on-policy framework eliminates the need for auxiliary dataset construction, trained models using the official prompts from the "21-CompBench dataset, totaling 5.6k prompts.
In contrast to the GRPO on-policy RL paradigm, which solely requires training prompts, DPO functions as an off-po algorithm, necessitating both prompts and corresponding pairs of chosen and rejected images. Given that autoregressive image generation models are optimized using cross-entropy loss, we can directly adapt the maximum likelihood objective DPO to this setting. To clarify, we provide a detailed outline of the DPO methodology from the following two perspectives:
DPO.
The computational cost of DPO comprises three components: (i) generating training images based provided prompts, (ii) the scoring these images by a reward model, and (iii) the subsequent training process. To ensure a fair comparison between DPO and GRPO under comparable computational constraints, align the number of images generated per prompt in DPO with the group size in GRPO, while employing identical reward models maintain consistent learning preferences.
Maintaining Comparable Computational Cost:
DPO Ranking Data Curation: Leveraging the images generated by the model, we construct ranking pairs for each prompt by selecting the highest and lowest scoring images as the chosen and rejected images, respectively. This process yields a total of 5.6k ranking pairs for training, derived from the official prompts of the 121-CompBench dataset.
Experimental Analysis and Insights. As presented in Table and 2, we evaluate the in-domain and out-of-domain performance of GRPO and DPO. We provide qualitative results in Figure with additional visualization in the Supplementary. The key findings are summarized as follows:
DPO demonstrates stronger performance than GRPO in in-domain evaluations. As shown in Table 1, DPO consistently outperforms GRPO on T2I-CompBench across various reward models, with DPO's 's in-domain performance surpassing GRPO by average of 11.53%. Notably, when using T21-Comphr 221-CompBench's 's official evaluation tools as the reward signal, DPO attains peak enhancement of 7.8% over GRPO. This significantly highlights the advantages DPO over GRPO in terms of in-domain effectiveness and robustness.
GRPO exhibits superior generalization capabilities than DPO in out-of-domain scenarios. As illustrated in Table 2, GRPO consistently demonstrates enhanced generalization perfor- mance over DPO across various reward models the GenEval dataset, surpassing DPO by
5

---

HPS
Imagy Reward
Unified Reward
Dat Scaling
Itrratvve Scaling
Sampling Scalling
DPO 72l-Compbench
08PO TalCumphench
GunEval
dBRO GanEoal
733
100
742
GERO
3esto54
(1)
(h0
(1)
61)
Figure 3: (a) The Impact of Different Reward Models' Intrinsic Generalization Capability, We
evaluate the generalization performance of GRPO, DPO, and the intrinsic generalization performance (represented by best-of-4 strategy) of three reward models. (b-e) Effects of Three Scaling Strategies. We examine the effects of various scaling strategies, including sampling size, in-domain data diversity, and iterative training, on both in-domain and out-of-domain performance.
an average of 1.14%. Notably, when HPS serves as the reward model, GRPO achieves a peak improvement of 2.42% over DPO, suggesting its superior generalization capacity.
2.3 Impact of Different Reward Models
Recent advancements in LLMs have extensively investigated the influence of reward model variations on performance across diverse tasks, including reasoning [52, 50], safety [36, 67], and general conversation [46, 49]. In text-to-image generation, reward models have been developed capture human aesthetic and semantic preferences, thereby steering the generative process. However, in autoregressive image generation, the impact reward model-induced preferences on RL remains limited. A recent study [23] explores the influence of reward models on in-domain RL performance but still lacks exploration of the potential relationship between the intrinsic properties, especially the generalization capabilities, of reward models and RL generalization. To address this gap, we consider exploring the relationship between the generalization of RL. and the intrinsic generalization capabilities of reward models, yielding critical insights for the future development of reward models.
Reward Model Design. Unlike domains such as mathematics or programming, where reward signals typically derived from verification functions that ensure precise alignment with ground- truth solutions, evaluating the quality of generated images necessitates sophisticated learned reward models. The influence of these reward models, along with their inherent biases, the training process remains underexplored. To elucidate their impact, we examine three distinct types of reward models, each designed capture unique aspects of image quality and alignment with human preferences:
Human Preference Model. Human Preference Models, such as HPS [54] and ImageRe- ward [56], are constructed using Dision-languee models (VLMs) like CLIP or BLIP to evaluate images based on human aesthetic appeal and text-image alignment. Trained on datasets of human-annotated image rankings, these models provide a holistic assessment of visual quality, generating scores that reflect human-like preferences.
Visual Question Answering Model. Visual Question Answering Models, including Uni- ReedRe [5]]. Fine-tuned ORM [18] and PARM [18], leverage multimodal large language models (MLLMs) like LLaVA [27] to interpret visual inputs and perform scenario-based evaluations. Trained on diverse datasets comprising images and text, these models emphasize detailed reasoning, enabling precise scoring and evaluation of visual content.
Metric Reward. Metric Rewards utilize specialized, domain- specificc evaluation tools. In this study, for each training prompt, we identify its corresponding attribute in r21-CompBench and apply the associated evaluation protocol to score the generated images.
Intrinsic Generalization of Reward Models. In contrast to RL, we adopt Guo et al.'s scalable framework ([18)) for efficiently evaluating reward model (RM) capabilities during inference. This approach uses RMs as outcome reward models (ORMs) in a best-of-N strategy, assessing their capabilities via final scores. We extend this framework to evaluate RM generalization the GenEval dataset by deploying RMs as ORMs. As shown in Table a best-of-4 selection strategy yields the RM generalization ranking GenEval: Unified Reward Image Reward > HPS Reward.

---

DPP''s amseralzation
actmbiive rewardd indde! chotee chan CRPP.
plats of dinr
A plat of s donding opfigmmtte
ntt
Raunn
Unila Amand
Mord Rmvara
kevars
Unifue Busnd
Buaad
DPQ
betier
seeded potenssally entance gencinlizatiinn
CAPO
And and Moe
Af teko
Figure 4: Visualization Results of the Impact of Different Reward Models.
Table 3:
Comparison of Reward Model Generalization, We assess the generalization capabilities of reward models in best-of-4 selection strategy on the GenEval dataset [15].
Reward Type
Overall
Color Attr.
Counting
Position
Single Obj.
Two Obj
Colors
HPS
82.14
70.62
77.25
90.15
99.06
87.77
68.00
wwd
82.41
69.69
75.50
93.18
98.75
88.56
68.75
Unified Rwd
84.49
75.00
82.00
92.42
98.75
87.50
71.25
Experimental Analysis and Insights. Our experimental results visually summarized in Figure and the detailed numeric comparisons are reported Table 3. We provide qualitative results in Figure 4, with additional visualization the Supplementary. Then, draw two principal insights:
DPO's generalization performance exhibits heightened sensitivity to the choice of Reward Model compared to GRPO. As presented in Table 2, the range of GRPO's generalization performance the GenEval dataset is 1.81, whereas DPO's range is notably higher 2.72. Furthermore, GRPO's performance variance on GenEval is 0.5486, significantly lower than DPO's variance of 0.9547.. This suggests that DPO's generalization capabilities more susceptible to variations in training data preferences, indicating greater dependency on the specific characteristics of the chosen reward model.
A reward model with superior generalization can potentially improve the generalization performance of RL algorithms. As intuitively illustrated in Figure (a), the performance rankings of different reward models on the GenEval dataset, optimized using either GRPO or DPO, remain consistent. Crucially, these rankings align perfectly with our prior evaluations of the intrinsic generalization capabilities of these models. This indicates that the reward model's intrinsic capacity for generalization is pivotal factor that probably contributes to the overall generalization potential of the RL algorithm.
2.4 Investigation of Effective Scaling Strategies
Prior research has extensively explored methods to optimize the in-domain performance of RL algorithms for LLMs. Notable approaches include iterative DPO (DPO-Iter) [37] and techniques to enhance the efficacy of proximal policy optimization [57]. However, despite these advancements, common scaling behaviors improve the in-domain (ID) and out-of-domain (OOD) performance of both on-policy and off-policy RI algorithms in autoregressive image generation remain largely underexplored. To address this gap, this section investigates three critical scaling behaviors to enhance the performance of GRPO and DPO across ID and OOD datasets. Specifically, these factors include:

---

Table 4: Effect of Scaling Strategies on In-Domain Proficiency. This table presents the performance evaluation of GRPO and DPO on T21-CComBennch [21] under three distinct sealing strategies: sample scaling (scaling sampled images per prompt), data scaling, and iterative training. Specifically, configurations denoted as 'Base Size' (Data Scaling), 'Sampling 4' (Sample Scaling), and 'Base' (Iterative Training) correspond to the values in Table where the reward type is HPS for comparative analysis. All experiments consistently employ HPS as the reward model.
Scaling Puram.
Average
Color
Attribute Binding
Shape
Texture
Spatial
Object Relationship
Non-Spatial
Complex
GRPO
DPO
GRPO
DPO
GRPO
DPO
GRPO
DPO
GRPO
DPO
GRPO
DPO
GRPO
DPO
Baseline
38.55
63.30
34.28
48.90
20.23
30.51
34.12
Data Sooling
Base
50.49
77.39
71.54
25.29
31.10
31.17
39.20
Double Size
50.67
54.12
77.81
82.24
54.39
60.48
72.27
77.06
29.16
33.05
31.15
31.18
39.26
40.70
Triple Size
51.11
$4.30
77.73
83.77
56.31
62.35
72,66
77.73
29.21
29.81
31.18
31.28
39.54
40.84
Sumple Sculing
Sampling
50:19
53.90
77.39
31.10
31.17
Sampling
$1.06
55.17
77.79
83.31
55.47
63.86
72.63
75.50
29.99
36.44
31.19
31.05
39. 19.30
40.84
Sampling 12
$1.88
54.39
78. 18
82.37
96.81
63.17
73.72
77.82
31.51
31:57
31.21
31.14
39 68
40.28
Sampling 16
51.99
52.85
77.38
80.60
59.09
63.49
73.72
75.72
30.70
25.93
31.15
31,06
39.88
40.31
Iterative Training
Base
50:49
53.90
77.39
71.54
25.29
31.10
31.17
39.20
Herl
50.71
55.07
77.28
85.14
51.11
61.21
72.35
76. 15
29. 78
31.05
31.18
31.28
39.51
39.58
Ieer
$1.
55.19
78.02
84.66
53.73
64.44
72.86
76. 15
30.76
35.05
31.16
31.29
39 25
39.54
Ber3
$1.45
55.28
77.97
84.70
55.94
64.32
73.26
76.32
30.74
35.51
31.19
31.27
39.61
39.53
Table 5: Effect of Scaling Strategies on Out-of-Domail Generalization. This evaluation on the GenEval [15] dataset adopts the same scaling strategies as outlined in T2I- T2I-CompBench (see Table 4). The configurations, Base Size (Data Scaling), Sampling 4 (Sample Scaling), and Base (Iterative Training), correspond to the values in Table 2, where the reward type is HPS for comparison.
Scaling Puram.
Overall
Color Amr
Counting
Position
Single Object
Object
Colors
GRPO
DPO
GRPO
GRPO
GRPO
DPO
GRPO
DPO
GRPO
DPO
GRPO
Baseline
78.04
63.50
54.37
76.25
98.44
87.63
$3.03
Data Scoling
Base Size
79.18
77.31
63.00
70.25
60.62
48.12
78.75
67.25
$6.87
90.15
99.09
98.75
86.17
89.63
Double
$0.28
78.88
67.50
78.25
77.00
91.67
Triple Size
78.21
79.25
64.50
70.00
73.75
76.00
87.12
91.67
98.
99.06
87.50
Sample Soaling
Sampling 4
79.18
77.31
63.00
70.25
60.62
48 12
78.75
67.25
86, 87
90.15
99.69
98.75
86
89.63
Sampling
80.29
79.25
63.25
73.25
63.44
47.81
80.25
70.00
90.15
95.45
96.
98.75
87.77
90.16
Sampling 12
73.99
77.84
65.25
75.50
99.38
85.
Sampling 16
78.50
75.30
63.50
64.75
57.81
44.69
76.75
63.00
86.62
91.16
99.38
99.38
86.97
88.83
Nlvrative
Training
Base
79.18
77.31
63.00
70.25
60.62
48. 12
78.75
67.25
86, K7
90.15
99.69
98.75
86
89.36
79.35
77.63
67.00
69.00
54.37
69.00
90.91
98.75 75
87.50
78.79
78.16
64.75
71.00
57.19
76.00
97.81
87.23
Ite33
78.21
77.45
61.36
65.50
60.63
57.81
75.75
68.75
87.88
91.16
97.50
97.19
86.44
84.31
(1) scaling sampled images per prompt, (2) scaling the diversity of in-domain training data, and (3) implementing iterative training paradigms. The detailed application delineated as follows:
Scaling Sampled Images per Prompt: For GRPO, scaling the quantity of sampled images per prompt corresponds to expanding the group size of real-time samples utilized during the RL training phase. For DPO, where pairs are constructed by selecting the highest- and lowest-scoring images from a pre-generated set for given prompt, this strategy effectively amplifies the discriminative power of the preference pairs, facilitating more precise alignment with human preferences and the model's 's ability to distinguish subtle quality differences.
Scaling the Diversity and Quantity of In-Domain Training Data: To scale data while rigor- ously maintaining quality control, we develop a structured prompt generation pipeline lever- aging GPT-4o. Building upon the T21-CompBench, we generated set of category-specion prompts that was twice as large through carefully constrained API calls (see Supplementary for more implementation details). Our methodology incorporated two key principles:
1. Category-Aware Constraint Preservation: All generated prompts maintained strict adherence to their respective category's 's syntactic templates and semantic boundaries (c.g., 3D spatial prompts required exactly two objects and one spatial relation).
8

---

Eatt
Scople
Semple
Sar Bi
Acet
Corple
Soaple
Semgle
Mare yield GRPP''s sirdimaain grits efficiendly
The whihnng pattere shrig Menn mendivdd sikh diphic Ap of of mudd fouady
bubeee
The ovnn plans" Meghipeen jommled aold of waffe
Sen Sar
Sobbee Pot
Trpp
Seating to-slendian alse sopprumes
gerenalication bun ridl dtmoriiiingg
phill of safl end jubned pain
traliing 289''s li- domaiin grvits but rivks geseralliation.
J--onade
dndl aoollly boe giayiily
Biss
Semple
sompttin mnlonnces candeeiiee sampling cruoets oysraietnng
sul EUSONeT thr diningg of Astire
Deriche
Soe Suy
Soalle Ser
Tepp Ste
Fax Stoe
Troll Sir
Scaling in- doammin boosts and of dosaals perfortaatue
The of Enannig Intooend Pipraaly
dfliley sgen lnddy
Figure Visualization Results of Insights from Investigating Scaling Strategies.
2. Semantic Novelty Enforcement: We implemented specific generation constraints to prevent superficial variations, requiring GPT-4o produce genuinely novel compo- sitions rather than simple lexical substitutions. This yields semantically distinct yet plausible object pairings (e.g., "A person is exploring a forest and taking photos of the wildlife" vs. original "A child is playing with toy airplane in the buckyard") ", significantly expanding the conceptual coverage our training data.
Implementing Iterative Training Paradigms: The motivation for adopting iterative training paradigms lies in their ability progressively enhance model performance by leveraging updated reference policies, thereby reducing overfitting risks and improving generalization. Inspired by this, develop iterative variants of GRPO and DPO, termed GRPO-ICe Iter and DtO-Iter, respectively. For GRPO-Iter, we extend the standard GRPO framework with iterative cycles, updating the reference model with the policy model's parameters after each training round, which ensures the KI. penalty aligns with the current policy. For DPO-Iter, we followed iterative DPO protocols[37, 57], where preference pairs are re-sampled and re-evaluated after each training cycle, with the reference model updated accordingly.
Experimental Analysis and Insights. As shown in Table 4 and 5, we conduct a comprehensive investigation into the efficacy of three key scaling strategies. To enable fair comparison of GRPO and DPO, we standardize the experimental setting by adopting the HPS [54] as the reward model to compare the effectiveness of the three scaling strategies, with additional visualizations provided in the Supplementary due to space limitations. Our principal findings are summarized follows:

---

Scaling sampled images per prompt tends to yield more computationally efficient in- domain gains for GRPO: Qualitative visualizations in Figure 3 (c) for GRPO on T2I- CompBench demonstrate that scaling group size in autoregressive image generation out- performs both in-domain data scaling and iterative scaling. This enhanced performance driven by improved advantage estimation with larger group sizes, which stabilizes policy updates and strengthens in-domain optimization. Given that doubling each factor (group size, in-domain data scale, iterative training paradigms) incurs approximately comparable computational cost, prioritizing scaling sampling emerges as a highly efficient strategy.
Moderately scaling the sample size and increasing the diversity and quantity of in-domain data improves generalization for GRPO, but excessive scaling diminishes growth: In autoregressive image generation, moderate sampling size within the GRPO algorithm (e.g., to 4 or 8) and in-domain training data quantity (e.g., by doubling or tripling) progressively improves generalization, larger sample sizes are employed or when GRPO's s in-domain data scale is triple, gener- alization exhibits slight decline due to overfitting to in-domain characteristics, which underscores the critical need to balance in-domain optimization with robust generalization.
shown in Figure 3 (c) for GRPO on GenEval. However, when
Iterative training tends to maximize DPO's in-domain performance but risks generaliza- tion degradation: Iterative training substantially enhances the in-domain performance of DPO for autoregressive image generation, as evidenced by the Iterative Scaling curve in Figure: 3 (b) and (d). Notably, a single iteration of DPO-Iter outperforms the results obtained by tripling the training data, with additional iterations providing further incremental gains in in-domain metrics. However, generalization on the GenEval declines significantly after two iterations, likely due overfitting the training preference data, underscoring the trade-off between maximizing preference alignment and maintaining robust generalization.
Moderate sampling optimizes DPO's preference contrast for improved in-domain and generalization performance, while excessive sampling induces bias: In autoregressive image generation, scaling the sample size for DPO preference pair selection yields non- monotonic performance effects, as illustrated by the DPO sampling scaling curve in Figure 3 (b) and (d). Relative the baseline, sample sizes of 8, and 16 yield in-domain gains of 37.38%, 40.61%, 1, and 34.71% 1., alongside out-of- +1.52%, and -3.51% respectively. This suggests that scaling the sample size optimizes preference contrast while avoiding biases introduced by excessive scaling.
generalization changes of -0.93%,
Scaling in-domain data for DPO optimizes performance across both in-domain and out- of-domain by mitigating preference bias: Scaling in-domain data by factors of one, two, and three achieves relative in-domain improvements 37 37.40%, 37.97%, and 38.37% over the baseline, respectively. While single-scale training results in a 0.94% % decline on the oft-of-domain benchmark, scaling by factors of two and three produces gains of 1.08% and 1.55%. This highlights that carefully curating a diverse and representative set of preference pairs is critical to overcoming the constrained preference scope inherent in small datasets, thereby mitigating potential in domain and out-of-domain performance degradation.
3 Conclusion
In this paper, we conducted a rigorous experimental analysis demonstrating that DPO excels in in-domain tasks, while GRPO exhibits superior out-of-domain generalization. We further establish that the generalization capacity reward models potentially shapes both algorithms' generalization potential. Through systematic exploration of three scaling strategies, derive critical insights for achieving enhanced Chain-of- Chain-of-Thought reasoning in autoregressive image generation.
10

---

Acknowledgement
The work described in this paper supported in part by the Research Grants Council of the Hong Kong Special Administrative Region, China, under Project 14201321 and Project 14200824.
Overview of Appendix
Appendix A: Related work.
Appendix B: Implementation details of structured prompt generation pipeline.
Appendix C: Detailed record of computational time.
A
Related Work
Visual Generative Models. Visual generative models have advanced through two primary paradigms: autoregressive and diffusion approaches. Autoregressive methods. inspired by lan- guage modeling success [33, 34, 47, 59]. sequentially predict image tokens or pixels, seen in ViT-VQGAN [60] and VideoPoet [25]. Recent work like LlamaGen [44] demonstrates that pure autoregressive architectures can achieve state-of-the-arr generation, while Janus [9] introduces decou- pled visual encoding to unify multimodal understanding and generation. Meanwhile, diffusion models have emerged as powerful alternative, with continuous approaches [63] dominating text-to-image tasks and discrete variants like MaskGIT [7] operating on tokenized representations. Notably, Show-o adopts discrete diffusion through masked token prediction, achieving high-fidelity generation while maintaining training efficiency.
Reinforcement Learning (RL). Reinforcement I Learning (RI.) trains agents to maximize rewards through environment interactions, with methods split into on-policy (e.g., PPO [42], GRPO [43]) and off-policy (e.g., DPO (39]) approaches. On-policy methods like PPO use current policy data for stable but costly updates, employing techniques like GAE [41] for variance reduction, while GRPO replaces critics with group-wise reward comparisons. Off-policy methods like DPO reuse historical data for efficiency but risk distribution mismatch, directly optimizing preferences without reward modeling. The key difference lies in data usage: on-policy requires fresh data for stability, whereas off-policy trades some reliability for sample efficiency. Applied to language models via RLHF [6], these methods enhance alignment (c.g., RLOO [3]'s critic-free approach) and reasoning [10, 16, 45, 62] through MDP formulations, balancing computational cost and performance in tasks like mathematical reasoning. This demonstrates RL's adaptability across policy paradigms for improving language models.
B Implementation Details of Structured Prompt Generation Pipeline
As discussed in Sec. 2.4 of the main paper (Investigation of Effective Scaling Strategies), we enlarge T21-COMPBENH by generating additional set of category-specific prompts with GPT-4o. thereby doubling the size of the original benchmark. Specifically, for each of the eight categories, color, texture, shape, numeracy, spatial, 3D spatial, non-spatial, and complex, prompt. All meta-prompp are derived from a shared template, but include dategory-depencee constraints. An example for the color category is given after the following paragraph.
craft dedicated meta-
In practice, we iterate through every prompt in the color subset of T2I-COMPBENCH, replace the placeholder #Prompts From T2I-CompBench# with the current prompt, and feed the resulting meta-prompt to GPT--4o. We apply the same pipeline to the remaining seven categories. The complete collection of category-spectie meta-prompts will be released upon the paper's acceptance.
11

---

Color Category Example
am working on a reinforcement learning for image generation project, and need your assistance in generating additional prompts that focus on color-based descriptions.
Existing prompts: #Prompts From T2I-CompBench#
Task: Generate 2 additional prompts that maintain the eame syntactic structure while ensuring diversity.
Requirements:
Color Usage:
Each prompt must explicitly include at least two different colors.
The color words should be commonly used and perceptually distinct (e.g., "red" and "blue" are good, but "light red" and "dark red" are too similar). Allowed color descriptors: basic colors (e.g., red, blue, green, yellow, pink, purple, orange, brown, black, white, gray) and (e.g., "golden", "silver", "ivory"). Avoid uncommon or overly specific colors (e.g., "cerulean", "chartreuse"). Object Selection:
material-based variations
The first object should be tangible item with strong association color (e.g., clothing, furniture, makeup, vehicles, buildings). The second object (if applicable) should also be a realistic, color-relevant entity that fits within a scene.
Avoid repetition of objects already in the dataset (e.g., if "lipstick" and "blush" exist, do not Color and Object Compatibility: Ensure that the selected colors are realistically applicable to the given objects.
them again).
Examples of Good Color-object Pairings: sports car a black leather seat." " (both colors are reasonable for cars and seats)
Examples of Bad Color-object Pairings (to avoid)
"A purple banana and silver cloud." (unnatural color choices)
Diversity Constraints:
Do not generate prompts that are simple color swaps (e.g., "A red lipstick and pink blush" and "a pink lipstick and a red blush" are too similar).
Ensure semantic diversity by describing different types of objects and settings (e.g., fashion, interior design, nature, technology).
The sentence structure should mimic the provided examples but not identical. Output Format:
Return the response as Python list of strings in JSON-compatible format, e.g.:
{
"prompt1",
"prompt2"
Strictly use lovercase (no capitalization except for proper nouns) Now, generate two new prompts following these requirements.
Table 6: Comparison of DPO and GRPO Training Computational Costs (in GPU hours).
DPO
GRPO
Reward Type
Simple Image
Scoring
Training
Total
Total
HPS ImageReward
1.51
0.83
0.67
2.99
2.92
1,50
0.08
0.67 h
2.25
2.55 h
Unified Reward
1.51
1.80 h
0.67 h
3.97 h
4.03 h
C Detailed Record of Computational Time
To facilitate fair comparison between DPO and GRPO, as outlined Sections 2.2, we maintain comparable training computational costs, measured in terms of computational time. The computa- tional expense DPO consists of three main components: (i) generating training images based on provided prompts, (ii) scoring these images using reward model, and (ii) executing the subsequent
12

---

training phase. Detailed computational times for both GRPO and DPO are systematically recorded and presented in Table 6. Additionally, we assess and document the total training computational time for three key scaling strategies implemented for GRPO and DPO across different scaling ratios, as presented in Table 7. These computational time costs for both tables are evaluated using A100 GPUs, with Janus-Pre Pro [9] serving as the baseline.
Table 7: Total Computational Time for Scaling Strategies Across Varying Scaling Ratios.
Ratio
Ratio 2
Ratio
Scaling Strategy
DPO
GRPO
DPO
GRPO
DPO
GRPO
Data Scaling
2.99 h
2.92 h
5.97 h
5.84 h
9.01 h
8.76 h
Sampling Scaling
2.99 h
2.92 h
5.33
5.78 h
7.66 h
8.64
Iterative Scaling
2.99 h
2.92
5.99
5.84
8.98 h
8.76
References
[1] Claude 3.7 sonnet.
2025.
2025.
[2] Gemini 2.5 pro. http:///wgpppiiigggggoggo//ooooa
[3] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in Ilms. arXiv preprint arXiv:2402. 14740, 2024.
[4] Aida Amini, Saadia Gabriel, Peter Lin, Rik Hajishirzi. MathQA: Towards interpretable math word problem solving with operation- bbasd formalisms. arXiv preprint
Yejin Choi, and Hannaneh
2019.
[5] Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732. 2021.
[6] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05866 2022.
[7] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked generative image transformer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 13315-11325 2022.
[8] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov. Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, 2021.
[9] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen I Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Paous-Pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.178)) 2025.
[10] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models strong language models, arXiv preprint arXiv:2401.01: 0033355,, 2024.
13

---

[11] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, et al. DeepSeek- R1: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
[12] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[13] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12873-12883, 2021.
[14] Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-RI: Reinforeing video reasoning mllms, 2025.
[15] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132-521522 2023.
[16] Daya Guo. Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprini arXiv:2501.12948, 2025.
[17] Ziyu Guo, Ray Zhang, Hao Chen, Jialin Gao, Dongzhi Jiang, Jiaze Wang, and Pheng-Ann Heng. Sciverse: Unveiling the knowledge comprehension and visual reasoning of Imms on multi-modal scientific problems. arXiv preprint arXiv:2503.10627 2025.
[18] Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng. Can generate images with cot? let's verify and reinforce image generation step by step. arXiv preprint arXiv:2501.13926, 2025.
[19] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
[20] Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models, 2022.
[21] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2l-CompBench: A com- prehensive benchmark for open-world compositional ext-to-image generation. Advances in Neural Information Processing Systems, 36:78723-78744 2023.
[22] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and lon Stoica. LiveCodeBench: Holistic and contami- nation free evaluation of large language models for code. CoRR, 079977 2024.
[23] Dongzhi Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong. Hao Li, Le Zhuo, Shilin Yan, Pheng- Ann Heng, and Hongsheng Li. T2I-R1: Reinforcing image generation with collaborative semantic-level and token-level cot. arXiv preprint 1xXiv:2555.00700 2025.
[24] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-222133 2022.
[25] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming- Ming-Chang Chiu, et al. VideoPoet: A large language model for shro-shot video generation. arXiv preprint arXiv:2312. 2023.
[26] Black Forest Labs. Flux. http::///ggaao..co//aaaaaooootes
2024.
[27] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. OOeVA---OViiionn Easy visual task transfer, arXiv preprint arXiv:2408.03322 033266 2024.
14

---

[28] Xinhao Li, Ziang Yan, Desen Meng. Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. VideoChat-RI: Enhancing spatio-temporal perception via reinforce- ment fine-tuning, arXiv preprint arXiv:2504.0662 069588 2025.
[29] Xinhao Li, Ziang Yan, Desen Meng. Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao,
Yi Wang, and Limin Wang. VideoChat-RI: Enhancing spatio-tempore perception via reinforce- ment fine-tuning, 2025.
[30] Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization-based reasoning models. arXiv preprint arX1v:2503.22342, 2025.
[31] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-RFT: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785. 2025.
[32] MAA. American invitational mathematics examination - aime. In American Invitational Mathematics Examination -Atiiiioon-MAAAI 2024, February 2024.
[33] OpenAL. Chatgpt. https://chat. com, 2023.
[34] OpenAL. Hello gpt-4o. http:////pppeaa..coomnneeeeee/ee
20224
[35] OpenAl, Aaron Jaech, Adam Kalai, Adam Lerer, et al. OpenAl ol system card, 2024.
[36] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
[37] Richard Yuanzhe Pang, Weizhe Yuan. He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37:116617-116611 2024.
[38] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXtv:2307.01952, 2023.
[39] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-53741, 2023.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[41] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 2015.
[42] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634: 2017.
[43] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, et al. DeepSeekMath: Pushing the limits mathematical reasoning in open language models. arXiv preprint arXiv:2402.03332 2024.
[44] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525. 2024.
[45] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. Scaling reinforcement learning with 1lms. arXiv preprint arXiv:2501.12599, 2025.
15

---

[46] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, et al. LaMDA: Language models for dialog applications. 2022.
[47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[48] Junke Wang, Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang. SimpleAR: Pushing the frontier of autoregressive visual generation through pretraining. sft, and rl. arXiv preprint arXiv:2504. 11455,...145, 2025.
[49] Xiaolong Wang, Yile Wang, Yuanchi Zhang, Fuwen Luo, Peng Li, Maosong Sun, and Yang Liu. Reasoning in conversation: Solving subjective tasks through dialogue simulation for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Pupers), 2024.
[50] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistenee improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), 2023.
[51] Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified reward model for multimodal understanding and generation. arXiv preprint (rXxi:2503.022:33 2025.
[52] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24831 2022.
[53] Chengyue WU, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 13848 2024.
[54] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv: 09341, 2023.
[55] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
[56] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding. Jie Tang, and Yuxiao Dong. IewgeReward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 36:15903-15935, 2023.
[57] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for 11m alignment? a comprehensive study. arXiv preprint 2404. 10719, 2024.
[58] Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.0788 0008888888 2025.
[59] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv: 10671, 2024.
16

---

[60] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2 10.04627, 2021.
[61] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning 2025.
system scale. arXiv preprint ar arXi::2503.
[62] Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. ReST-MCTS*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735-647721 2024.
[63] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836-3847, 2023.
[64] Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. Llama-adapter Efficient fine-tuning of large language models with zero-initialized attention. In ICLR 2024, 2024.
[65] Renrui Zhang, Dongzhi Jiang, Yichi Zhang. Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Chang, Peng Gao, et al. Math Verse: Does your multi-modal Ilm truly see the diagrams in visual math problems? ECCV 2024, 2024.
[66] Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Shicheng Li, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, et al. Mavis: Mathematical visual instruction tuning with an automatic data engine. arXiv preprint Si0i:::207.08739 2024.
[67] Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Tang, and Minlie Huang. SafetyBench: Evaluating the safety of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2024.
17

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflect this paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
18

---

Justification:
We discuss the limitations of the work in Supplementary.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please see Section Experiments, and the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use publicly available datasets and will provide open access to the code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /ttpsscccppbblcc////eeccppcpppcp hwwbbliccguiidee guwdee/ /CodeSubmis ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
20

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see the Investigation section. Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We conduct experiments only once and report the accuracy of the best model, and it would be too computationally expensive to conduct the pre-training multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Supplementary.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
21

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics pettpa:/nneurppe: ctc.iis....c..aeeeeeoe http//ullcc/BBBtte csGuieellnnnn
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work focuses on academic, aublicl--vvailable datasets including T2I- CompBench and GenEval. This work is related to any private or personal data, and there's s no explicit negative social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
22

---

Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please see Section Investigation and Supplementary.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswithcome com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
23

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original,
non non- non-standard components. Please refer to our LLM policy (https : ptctps:/neer/// //euripi......// www.oooenenee////CnCC /202 /LLM) for what should or should not be described.
24


Reviews:
[{"review_id": "kQv2cuVZ7Y", "reviewer": "Reviewer_6KFq", "summary": "The authors benchmark Direct Preference Optimisation (DPO) against Group Relative Policy Optimisation (GRPO) on a 7 B‑parameter Janus‑Pro base model for autoregressive text‑to‑image generation. Experiments span two evaluation suites (T2I‑CompBench in‑domain, GenEval out‑of‑domain), four reward models, and three scaling axes (samples‑per‑prompt, data diversity, iterative fine‑tuning). Key findings:\n- DPO excels in‑domain quality, while\n- GRPO generalises better out‑of‑domain, and\n- Reward‑model generalisation strongly predicts policy generalisation.", "questions": "- Could a GRPO warmup followed by DPO refinement combine robustness with high in-distribution quality?\n- Do the findings remain consistent when comparing 1B and 7B Janus variants?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The evaluation is both broad and comprehensive, covering two benchmarks, four reward models, and three scaling dimensions. \n- The paper provides practical insights, offering clear guidance on compute and quality trade-offs. For example, identifying a moderate sampling size as ideal for GRPO.\n- The writing and structure are clear. Figures 1 to 4 effectively illustrate key trends\n\nWeaknesses:\n- The OOD generalization evaluation is limited: GenEval mainly uses single-object prompts, and tougher benchmarks like Pick-a-Pic or DrawBench are omitted. Including at least one compositional OOD benchmark would strengthen the assessment\n- Statistical rigor is lacking, with only single-run results, no standard deviations or confidence intervals, etc. It would be helpful if at least three seeds were reported.\n- Important baselines like PPO are missing. It would be good to include a 1-epoch PPO baseline should be included to provide context.", "was_revised": true, "final_justification": "Thanks to the authors for providing thorough additional experiments and clear explanations in the rebuttal. The updated benchmarks, statistical analyses, and cross-scale evaluations address most of my initial concerns and strengthen the validity of the claims. I am therefore increasing my rating.", "rebuttal": "We sincerely appreciate your valuable comments. We found them extremely helpful in improving our manuscript. We address each comment in detail, one by one below:\n\n---\n> #### **Q1: Limited Scope of Out-of-Distribution Generalization Evaluation.**\n\n> \nThank you for the insightful suggestion. In response, we have extended our evaluation to include **DrawBench**, a more challenging and compositional out-of-domain benchmark. Due to time constraints, we focused on representative experiments using **two** reward models. The results are as follows:\nAlgorithm|Reward Model|DrawBench Score\n|-|-|-|\nBaseline|N/A|69.13\nDPO|HPS|70.25\nDPO|Unified Reward|72.50\n**GRPO**|HPS|**71.25**\n**GRPO**|Unified Reward|**72.88**\n\nThese findings are consistent with our original conclusions: ***GRPO consistently demonstrates stronger generalization than DPO***, even on a more compositional and demanding benchmark. We believe this additional evidence strengthens the robustness of our claims, and we will include these results and a corresponding discussion in the final version.\n\n---\n> #### **Q2: Insufficient Statistical Rigor.**\n\n> \nThanks for your advice! To address this, we have re-executed a representative subset of our core experiments using three distinct random seeds. \nThe revised tables below present the mean and standard deviation. The findings indicate that performance variations are negligible, and our initial conclusions remain strongly supported. We will incorporate these statistically-validated results into the final manuscript.\n\n**Table 1: In-Domain Evaluation on T2I-CompBench**\n|Reward Model|Average||Color||Shape||Texture||Spatial||Non-Spatial||Complex||\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|Baseline|38.37±0.1|38.37±0.1|63.15±0.3|63.15±0.3|34.27±0.2|34.27±0.2|48.85±0.2|48.85±0.2|20.30±0.1|20.30±0.1|30.60±0.2|30.60±0.2|34.06±0.2|34.06±0.2|\n|HPS|50.45±0.3|53.92±0.4|77.39±0.5|85.24±0.5|53.62±0.3|64.62±0.5|71.39±0.4|75.87±0.4|30.12±0.2|25.32±0.2|31.04±0.2|31.18±0.2|39.18±0.2|40.87±0.2|\n|Unified Rwd|48.02±0.2|53.89±0.2|74.49±0.4|83.03±0.4|50.36±0.3|61.65±0.3|68.64±0.4|76.66±0.4|27.75±0.2|30.50±0.2|30.83±0.2|31.31±0.2|36.18±0.2|40.53±0.3|\n\n**Table 2: Out-of-Domain Generalization on GenEval**\n|Reward Model|Overall||Color Attr||Counting||Position||Two Object||Single Object||Colors||\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|Baseline|78.09±0.2|78.09±0.2|63.53±0.4|63.53±0.4|54.37±0.3|54.37±0.3|76.28±0.7|76.28±0.7|87.55±0.4|87.55±0.4|97.98±0.7|97.98±0.7|88.33±0.5|88.33±0.5|\n|HPS|79.18±0.4|77.30±0.4|62.88±0.3|70.21±0.4|60.59±0.3|48.05±0.4|78.70±0.5|67.12±0.3|86.80±0.4|90.04±0.6|99.66±0.4|99.00±0.6|86.36±0.5|89.37±0.6|\n|Unified Rwd|81.07±0.4|79.97±0.5|67.63±0.2|70.49±0.4|61.55±0.3|55.65±0.3|82.02±0.5|75.30±0.5|88.43±0.6|91.48±0.4|99.56±0.5|98.93±0.5|87.08±0.4|87.96±0.6|\n\n**Table 3: Impact of Scaling on In-Domain Proficiency**\n|Scaling Method|Average||Color||Shape||Texture||Spatial||Non-Spatial||Complex||\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|*Data Scaling*\n|Base Size|39.18±0.2|40.87±0.2|77.39±0.5|85.24±0.5|53.62±0.3|64.62±0.5|71.39±0.4|75.87±0.4|30.12±0.2|25.32±0.2|31.04±0.2|31.18±0.2|39.18±0.3|40.87±0.4|\n|Double Size|39.30±0.3|40.69±0.3|77.85±0.4|82.38±0.6|54.62±0.4|60.46±0.5|72.17±0.4|77.01±0.4|29.10±0.1|33.01±0.2|31.10±0.1|31.22±0.1|39.30±0.3|40.69±0.3|\n|*Sample Scaling*\n|Sampling 4|39.18±0.2|40.87±0.2|77.39±0.5|85.24±0.5|53.62±0.3|64.62±0.5|71.39±0.4|75.87±0.4|30.12±0.2|25.32±0.2|31.04±0.2|31.18±0.2|39.18±0.3|40.87±0.4|\n|Sampling 8|39.24±0.2|40.85±0.2|77.60±0.4|83.37±0.6|55.45±0.3|63.85±0.5|72.55±0.4|75.40±0.5|30.09±0.2|36.51±0.2|31.28±0.2|31.00±0.2|39.24±0.2|40.85±0.2|\n|*Iterative Training*\n|Base|39.18±0.2|40.87±0.2|77.39±0.5|85.24±0.5|53.62±0.3|64.62±0.5|71.39±0.4|75.87±0.4|30.12±0.2|25.32±0.2|31.04±0.2|31.18±0.2|39.18±0.3|40.87±0.4|\n|Iter1|39.47±0.2|39.52±0.4|77.28±0.4|85.30±0.5|54.11±0.3|64.22±0.4|72.43±0.4|76.31±0.4|29.85±0.1|34.06±0.1|31.14±0.2|31.25±0.2|39.47±0.1|39.52±0.3|\n\n ---\n> #### **Q3: Absence of PPO Baseline for Contextual Comparison.**\n\n> \nThank you for this valuable suggestion. While PPO is indeed a foundational RL algorithm, its application to autoregressive text-to-image generation remains a largely unexplored research direction, and its reliable implementation is impeded by significant technical constraints:\n\n***1) Intractable Value Estimation in Autoregressive Generation.***\n\nIn PPO, a learned critic must accurately predict the final reward, comprising global aesthetic quality and text alignment, based on **intermediate, partially generated image token sequences**. This constitutes a highly unstable and noisy task, as confirmed by emerging work [1,2] in related domains. \n\n***2) Variance-Induced Instability and Training Collapse.***\n\nDue to the difficulty of accurate value prediction, the value function incurs high variance. This undermines advantage estimation in PPO, leading to unstable policy updates, a known failure mode in actor‑critic setups when the critic is mis-trained [3,4]. \n\nBy contrast, preference‑based methods like DPO and GRPO bypass the need for a critic entirely, relying instead on pairwise comparisons or group‑based policy optimization. These methods have been shown to provide **greater stability and effectiveness in multimodal and autoregressive generation tasks**, including recent applications in diffusion‑based image generation and multimodal CoT reasoning.\n\nOn this basis, we focused our evaluation on DPO and GRPO. That said, we recognize the potential value of a simplified PPO baseline for contextual reference and will consider including such an experiment in future extensions of this work.\n\n[1] A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning.\n\n[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model.\n\n[3] On Proximal Policy Optimization's Heavy-tailed Gradients.\n\n[4] No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO.\n\n---\n> #### **Q4: Potential of GRPO Warm-up Followed by DPO Refinement.**\n\n>\nThanks for the insightful suggestion. To evaluate this hybrid strategy, we conducted a new experiment where GRPO was first applied to half of the T2I-CompBench training set, followed by DPO fine-tuning on the remaining half using the GRPO-trained checkpoint for initialization.\n\n| Method | In-Domain (ID) Performance | Out-of-Domain (OOD) Performance |\n| :--- | :---: | :---: |\n| **DPO** (Ours) | 53.90 | 77.31 |\n| **GRPO** (Ours) | 50.49 | 79.18 |\n| **GRPO→DPO** (New) | 51.56 | 78.83 |\n\nThe results confirm that this sequential approach produces a **balanced trade-off**: it improves DPO’s generalization and enhances GRPO’s in-domain performance, but does not surpass either method on their respective strengths. We will include this observation and its implications in the final version.\n\n---\n> #### **Q5: Consistency of Findings Across Janus 1B and 7B Variants.**\n\n>\nThanks for the suggestion. To assess the generalizability of our findings, we conducted a representative subset of experiments using the Janus 1B variant due to time constraints. Encouragingly, the results remain **highly consistent** with our main observations:\n\n***1)*** **DPO retains its advantage on in-domain tasks**, while **GRPO continues to generalize better out-of-domain.**\n\n***2)*** The **ranking of reward models** on GenEval is preserved, with Unified Reward outperforming HPS under both RL algorithms, as previously reported.\n\nThese findings reinforce ***the robustness of our conclusions across model scales***. We will include these additional results and analyses in the final version.\n\n**In-Domain Evaluation on T2I-CompBench**\n|Reward Model|Average||Color||Shape||Texture||Spatial||Non-Spatial||Complex||\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|Baseline|24.19|24.19|34.50|34.50|22.56|22.56|26.94|26.94|9.900|9.900|27.32|27.32|23.94|23.94|\n|HPS|46.84|51.85|76.12|81.01|45.32|58.97|66.85|72.41|26.87|28.64|30.56|31.10|35.33|38.98|\n|Unified Rwd|44.71|51.92|75.48|80.96|42.55|58.77|66.17|72.18|24.44|30.52|26.89|31.14|32.73|37.95|\n\n**Out-of-Domain Generalization on GenEval**\n|Reward Model|Overall||ColorAttr||Counting||Position||SingleObject||TwoObject||Colors||\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|Baseline|63.65|63.65|49.75|49.75|43.44|43.44|45.25|45.25|65.40|65.40|95.62|95.62|82.45|82.45|\n|HPS|74.90|73.86|55.00|61.25|53.75|42.81|72.00|65.75|84.02|87.37|98.12|98.75|86.44|87.23|\n|Unified Rwd|75.91|74.65|56.84|59.50|54.21|43.73|72.58|69.52|85.82|88.94|99.24|98.75|86.79|87.50|\n\n**Impact of Scaling on In-Domain Proficiency**\n|Scaling Method|Average||Color||Shape||Texture||Spatial||Non-Spatial||Complex||\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|*Data Scaling*\n|Base Size|46.84|51.85|76.12|81.01|45.32|58.97|66.85|72.41|26.87|28.64|30.56|31.10|35.33|38.98\n|Double Size|47.19|52.43|76.43|82.24|44.98|57.28|67.49|73.24|27.28|29.87|31.10|32.91|35.87|39.06\n|*Sample Scaling*\n|Sampling4|46.84|51.85|76.12|81.01|45.32|58.97|66.85|72.41|26.87|28.64|30.56|31.10|35.33|38.98\n|Sampling8|48.04|53.07|77.03|82.68|46.87|59.42|67.69|73.34|28.22|29.92|32.34|33.59|36.10|39.46\n|*Iterative Training*\n|Base|46.84|51.85|76.12|81.01|45.32|58.97|66.85|72.41|26.87|28.64|30.56|31.10|35.33|38.98\n|Iter1|46.96|52.49|75.44|82.19|45.86|59.02|67.14|73.28|26.52|28.77|31.17|32.22|35.63|39.45"}, {"review_id": "dKn3Jogo46", "reviewer": "Reviewer_FUrx", "summary": "This article for the first time conducts an analysis of the respective advantages and disadvantages of DPO and GRPO in the context of autoregressive generation. The analysis is carried out from three key perspectives: domain generalization capability, sensitivity to reward modeling, and the impact of scaling strategies. The main conclusions are as follows: GRPO demonstrates superior performance in terms of domain generalization; DPO shows greater sensitivity to the characteristics and quality of reward models; when it comes to image scaling strategies, GRPO benefits more significantly; and in scenarios involving multi-round iterative processes, DPO exhibits distinct advantages.", "questions": "Could the authors elaborate further on the commonalities and differences between GRPO and DPO in the context of the autoregressive image generation task, as compared to other autoregressive tasks?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "- Strengths  \n1. This paper represents the first exploration into the comparative strengths and weaknesses of GRPO and DPO in the context of autoregressive image generation.  \n\n2. The article demonstrates a logical structure, with paragraphs presented in a clear and coherent manner.  \n\n- Weaknesses  \n\n1. Many conclusions drawn in the paper regarding GRPO and DPO reflect well-established observations within the field, such as the inherent improvement in model generalization from eliminating the critic model. The work requires novel insights that distinguish it from existing research on autoregressive generation tasks.  \n\n2. The paper omits critical methodological details, notably the formulas integrating DPO with GRPO for autoregressive image generation, which should be incorporated.  \n\n3. The visual evidence presented, for example, in Figure 3, inadequately supports the paper’s claims. For instance, the left panel of Figure 3 appears to demonstrate superior performance by GRPO, contradicting the authors’ interpretation, and similar inconsistencies are evident in other visualization examples.  \n\n4. The table in paper needs to have the average value or bolding to present the results more intuitively.", "was_revised": true, "final_justification": "The authors addressed most of my concerns in rebuttal, and their discussion of DPO and GRPO brings new insights into the field of image generation. Therefore, I tend to accept.", "rebuttal": "We sincerely appreciate your valuable comments. We found them extremely helpful in improving our manuscript. We address each comment in detail, one by one below:\n\n---\n> #### **Q1: Novel Insights Beyond Known Properties of GRPO and DPO in Autoregressive Image Generation.**\n\n>\nThanks for your thoughtful question. While some high-level traits of GRPO and DPO are known, our work presents the ***first* systematic empirical study of these methods in the distinct setting of autoregressive text-to-image generation (T2I-AR)**, uncovering insights that do not directly follow from prior findings in text-based autoregressive tasks, due to the significant domain gaps between understanding and image generation settings.\n\n\n1. **Domain-Specific Challenges in T2I-AR**:\nUnlike text generation, T2I-AR models operate over sequences of quantized visual tokens from VQ-VAE spaces, where rewards are derived from subjective and non-symbolic measures like aesthetics and semantic alignment. These differences fundamentally alter how RL algorithms behave. Prior assumptions, such as the robustness of critic-free methods, had not been tested in this setting. Our study fills this gap by ***empirically evaluating GRPO and DPO under these domain-specific constraints***.\n\n\n2. **Distinct and Actionable Insights**: \nBeyond confirming known trends, we provide new findings tailored to T2I-AR:\n\n   ***1)*** We show that **DPO's generalization is substantially more sensitive to the choice of reward model than GRPO's** (Sec. 2.3, Table 2), a vulnerability less emphasized in textual domains.\n\n   ***2)*** Crucially, we establish that **the intrinsic generalization capability of a reward model correlates directly with the RL algorithm's out-of-domain performance** (Sec. 2.3, Fig. 2a), suggesting a novel and practical path for improving generalization by focusing on better reward model design.\n\n\nTogether, these insights advance our understanding of how RL paradigms interact with the unique structure of visual token spaces, reward feedback, and training dynamics in T2I-AR, contributions not addressed in previous work on autoregressive generation.\n\n---\n> #### **Q2: Missing Mathematical Details for DPO and GRPO in Autoregressive Image Generation.**\n\n> \nThank you for raising this important point. In the revision, we will include the core training objectives for both **GRPO and DPO** in the context of autoregressive image generation. Specifically:\n\n\nFor **GRPO**, we will present the advantage normalization and KL-regularized optimization: \n\n$$ A_i = \\frac{r_i - \\mathrm{mean}({r_i})}{\\mathrm{std}({r_i})} $$\n\n$$ \\mathcal{J}\\_{\\text{GRPO}} = \\mathbb{E}\\left[ \\frac{1}{G} \\\\sum_{i} \\sum_{t} \\left( \\min\\left( \\frac{\\pi_{\\theta}(o_{i,t})}{\\pi_{old}(o_{i,t})} A_i, \\mathrm{clip}\\left( \\frac{\\pi_{\\theta}}{\\pi_{old}}, 1 - \\epsilon, 1 + \\epsilon \\right) A_i \\right) - \\beta D_{KL}(\\pi_{\\theta} | \\pi_{ref}) \\right) \\right] \n$$\n\n\nFor **DPO**, we will formalize its preference-based loss without an explicit reward model:\n\n$$\nL_{\\text{DPO}} (\\pi_\\theta, \\pi_{ref}) = -\\mathbb{E} \\left[ \\log \\sigma \\left( \\beta \\log \\frac{\\pi_\\theta(o_w|q)}{\\pi_{ref}(o_w|q)} - \\beta \\log \\frac{\\pi_\\theta(o_l|q)}{\\pi_{ref}(o_l|q)} \\right) \\right]\n$$\n\n\nThese formulations reflect how both methods are applied to discrete token sequences in autoregressive image generation. We will ***incorporate them into Section 2.2 of the revised version, along with clear contextual explanations***.\n\n\n---\n> #### **Q3: Concerns About Inconsistencies in Visual Evidence.**\n\n> \nThank you for pointing this out. We acknowledge that some visual examples may not have been sufficiently representative, and we appreciate your careful observation.\n\nTo address this, we will revise the figures in the final version by replacing inconsistent examples with clearer, more representative samples that align with our quantitative findings. We also plan to expand the set of comparative visualizations to more clearly support each key claim. We will ensure that the visual evidence in the paper accurately and convincingly reflects the empirical results.\n\n---\n> #### **Q4: Clarifying Table Presentation for Better Readability.**\n\n> \nThank you for the suggestion. To clarify, the **Overall** column in Tables 2, 3, and 5 already represents the average across all attributes, though this was not made explicit. In the revised version, we will clearly note this and **bold the highest values** in key columns to improve clarity and ease of comparison."}, {"review_id": "KihClp8eBP", "reviewer": "Reviewer_JEST", "summary": "This paper presents the first comprehensive empirical study comparing Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO) for enhancing Chain-of-Thought (CoT) reasoning in autoregressive image generation. Using Janus-Pro as the baseline model, the authors evaluate in-domain (T2I-CompBench) and out-of-domain (GenEval) performance, scrutinize the impact of reward models, and analyze three scaling strategies: sampled images per prompt, in-domain data diversity/volume, and iterative training.", "questions": "1. The paper observes that GRPO (superior in OOD generalization) suffers from overfitting with aggressive data scaling , while DPO (inferior in OOD) consistently improves with scaling (Figure 2 (d) and (e)) Does this appear counterintuitive given GRPO's on-policy adaptability?\n\n2. Could the value of DPO with sampling 8 in Table 5 be a typo?\n\n3. Could you provide the human evaluation results (e.g., win rate) for GRPO and DPO?", "limitations": "See weakness and questions parts.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strength\n**Experimental rigor:** Comprehensive evaluation across ID/OOD benchmarks, multiple reward models, and scaling dimensions with clear ablation.\n\n**Practical insights:** The insightful findings (e.g., reward model generalization transfer, GRPO’s OOD stability, DPO’s iterative training trade-offs) benefit RL practitioners.\n\n### Weakness\n**Limited model scope:** Experiments confined to Janus-Pro; lack of validation on other autoregressive architectures (e.g., LlamaGen) weakens generalizability claims.\n\n**Statistical fragility:** Results based on single runs, risking unreported variance.", "was_revised": true, "final_justification": "The additional empirical study, which effectively addresses my initial concern regarding the limited model scope. The explanation about the on-policy versus off-policy dynamics also provides a helpful perspective on the counterintuitive overfitting problem, suggesting it may be a consequence of data coverage and diversity. Thus, I maintain my score as 5.", "rebuttal": "We sincerely appreciate your valuable comments. We found them extremely helpful in improving our manuscript. We address each comment in detail, one by one below:\n\n\n---\n> #### **Q1: Generalization on Other Autoregressive Architectures.**\n\n\n> \nThanks for your advice! We agree that validating our findings on additional autoregressive architectures is crucial for strengthening the generalizability of our claims. To this end, we conducted additional experiments using **Show-o**, a distinct autoregressive image generation model. While Show-o adopts a discrete diffusion approach, it still generates visual tokens in an autoregressive fashion, though using a parallel decoding strategy that produces multiple tokens per step. Despite this difference, the underlying generation remains token-by-token and sequential in nature. \n\nDue to time constraints, we focused on a representative subset of settings. The results consistently support our original findings:\n\n\n***1)*** DPO excels in in-domain tasks, while GRPO demonstrates stronger out-of-domain generalization.\n\n\n***2)*** The relative ranking of reward models on GenEval remains unchanged, with Unified Reward outperforming HPS under both RL algorithms.\n\n\nThese results confirm that our conclusions hold across architectures. We will include this extended analysis in the final version of the paper.\n\n\n**In-Domain Evaluation on T2I-CompBench**\n|Reward Model|Average||Color||Shape||Texture||Spatial||Non-Spatial||Complex||\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|Baseline|37.06||55.37||40.85||47.24||19.91||29.92||29.04||\n|HPS|48.61|51.77|67.83|74.68|63.72|76.95|69.05|73.58|29.78|25.04|30.36|30.59|33.29|34.90|\n|Unified Rwd|46.09|51.93|64.89|72.56|60.08|73.61|66.28|74.12|27.45|29.85|30.32|30.71|30.80|34.52|\n\n\n**Out-of-Domain Generalization on GenEval**\n|Reward Model|Overall||Color Attr||Counting||Position||Single Object||Two Object||Colors||\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|Baseline|53.07||28.18||49.26||10.88||95.35||52.13||82.28||\n|HPS|54.19|52.31|28.19|31.47|54.32|46.27|12.18|10.27|96.18|90.56|53.41|56.68|79.55|83.22|\n|Unified Rwd|55.12|54.23|29.93|32.08|55.46|50.17|13.02|11.28|97.07|96.48|54.27|55.26|81.02|82.14|\n\n\n**Impact of Scaling on In-Domain Proficiency**\n|Scaling Method|Average||Color||Shape||Texture||Spatial||Non-Spatial||Complex||\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|*Data Scaling*|||||||||||||||\n|Base Size|48.61|51.77|67.83|74.68|63.72|76.95|69.05|73.58|29.78|25.04|30.36|30.59|33.29|34.90|\n|Double Size|49.34|52.49|68.82|72.18|65.02|77.08|70.31|74.66|28.37|30.18|30.66|30.92|33.59|34.93|\n|*Sample Scaling*|||||||||||||||\n|Sampling 4|48.61|51.77|67.83|74.68|63.72|76.95|69.05|73.58|29.78|25.04|30.36|30.59|33.29|34.90|\n|Sampling 8|50.17|54.07|68.55|73.60|65.73|75.68|70.27|74.18|30.07|33.72|30.58|30.61|33.86|34.91|\n|*Iterative Training*|||||||||||||||\n|Base|48.61|51.77|67.83|74.68|63.72|76.95|69.05|73.58|29.78|25.04|30.36|30.59|33.29|34.90|\n|Iter 1|49.18|53.14|67.48|74.38|64.12|75.92|69.55|73.77|29.42|33.46|30.44|30.90|33.58|34.13|\n\n\n**Impact of Scaling on Out-of-Domain Generalization**\n|Scaling Method|Overall||Color Attr||Counting||Position||Single Object||Two Object||Colors||\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|*Data Scaling*|||||||||||||||\n|Base Size|54.19|52.31|28.19|31.47|54.32|46.27|12.18|10.27|96.18|90.56|53.41|56.68|79.55|83.22|\n|Double Size|55.76|53.41|29.71|32.05|56.93|49.18|12.05|14.52|97.03|93.07|52.42|58.51|80.47|81.53|\n|*Sample Scaling*|||||||||||||||\n|Sampling 4|54.19|52.31|28.19|31.47|54.32|46.27|12.18|10.27|96.18|90.56|53.41|56.68|79.55|83.22|\n|Sampling 8|56.18|54.05|29.20|31.94|57.08|47.84|13.37|13.19|98.02|95.40|53.75|58.28|81.49|84.12|\n|*Iterative Training*|||||||||||||||\n|Base|54.19|52.31|28.19|31.47|54.32|46.27|12.18|10.27|96.18|90.56|53.41|56.68|79.55|83.22|\n|Iter 1|55.29|53.07|30.04|30.51|53.06|45.74|11.99|11.46|97.29|91.45|54.27|56.37|80.83|82.55|\n\n\n---\n> #### **Q2: Insufficient Statistical Rigor.**\n\n\n>\nThanks for your advice! To address this, we have rerun a representative set of our key experiments with three different random seeds.\n\n\nThe updated tables below report the mean and standard deviation. The results show that performance fluctuations are minimal, and our original conclusions remain well-supported. We will integrate these statistically-validated results into the final version of the paper.\n\n\n**In-Domain Evaluation on T2I-CompBench**\n|Reward Model|Average||Color||Shape||Texture||Spatial||Non-Spatial||Complex||\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|Baseline|38.37±0.1|38.37±0.1|63.15±0.3|63.15±0.3|34.27±0.2|34.27±0.2|48.85±0.2|48.85±0.2|20.30±0.1|20.30±0.1|30.60±0.2|30.60±0.2|34.06±0.2|34.06±0.2|\n|HPS|50.45±0.3|53.92±0.4|77.39±0.5|85.24±0.5|53.62±0.3|64.62±0.5|71.39±0.4|75.87±0.4|30.12±0.2|25.32±0.2|31.04±0.2|31.18±0.2|39.18±0.2|40.87±0.2|\n|Unified Rwd|48.02±0.2|53.89±0.2|74.49±0.4|83.03±0.4|50.36±0.3|61.65±0.3|68.64±0.4|76.66±0.4|27.75±0.2|30.50±0.2|30.83±0.2|31.31±0.2|36.18±0.2|40.53±0.3|\n\n\n**Out-of-Domain Generalization on GenEval**\n|Reward Model|Overall||Color Attr||Counting||Position||Two Object||Single Object||Colors||\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|Baseline|78.09±0.2|78.09±0.2|63.53±0.4|63.53±0.4|54.37±0.3|54.37±0.3|76.28±0.7|76.28±0.7|87.55±0.4|87.55±0.4|97.98±0.7|97.98±0.7|88.33±0.5|88.33±0.5|\n|HPS|79.18±0.4|77.30±0.4|62.88±0.3|70.21±0.4|60.59±0.3|48.05±0.4|78.70±0.5|67.12±0.3|86.80±0.4|90.04±0.6|99.66±0.4|99.00±0.6|86.36±0.5|89.37±0.6|\n|Unified Rwd|81.07±0.4|79.97±0.5|67.63±0.2|70.49±0.4|61.55±0.3|55.65±0.3|82.02±0.5|75.30±0.5|88.43±0.6|91.48±0.4|99.56±0.5|98.93±0.5|87.08±0.4|87.96±0.6|\n\n\n**Impact of Scaling on In-Domain Proficiency**\n|Scaling Method|Average||Color||Shape||Texture||Spatial||Non-Spatial||Complex||\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|*Data Scaling*|||||||||||||||\n|Base Size|50.45±0.3|53.92±0.4|77.39±0.5|85.24±0.5|53.62±0.3|64.62±0.5|71.39±0.4|75.87±0.4|30.12±0.2|25.32±0.2|31.04±0.2|31.18±0.2|39.18±0.3|40.87±0.4|\n|Double Size|50.61±0.3|54.15±0.3|77.85±0.4|82.38±0.6|54.62±0.4|60.46±0.5|72.17±0.4|77.01±0.4|29.10±0.1|33.01±0.2|31.10±0.1|31.22±0.1|39.30±0.3|40.69±0.3|\n|*Sample Scaling*|||||||||||||||\n|Sampling 4|50.45±0.3|53.92±0.4|77.39±0.5|85.24±0.5|53.62±0.3|64.62±0.5|71.39±0.4|75.87±0.4|30.12±0.2|25.32±0.2|31.04±0.2|31.18±0.2|39.18±0.3|40.87±0.4|\n|Sampling 8|50.81±0.2|55.08±0.3|77.60±0.4|83.37±0.6|55.45±0.3|63.85±0.5|72.55±0.4|75.40±0.5|30.09±0.2|36.51±0.2|31.28±0.2|31.00±0.2|39.24±0.2|40.85±0.2|\n|*Iterative Training*|||||||||||||||\n|Base|50.45±0.3|53.92±0.4|77.39±0.5|85.24±0.5|53.62±0.3|64.62±0.5|71.39±0.4|75.87±0.4|30.12±0.2|25.32±0.2|31.04±0.2|31.18±0.2|39.18±0.3|40.87±0.4|\n|Iter1|50.78±0.3|55.17±0.3|77.28±0.4|85.30±0.5|54.11±0.3|64.22±0.4|72.43±0.4|76.31±0.4|29.85±0.1|34.06±0.1|31.14±0.2|31.25±0.2|39.47±0.1|39.52±0.3|\n\n\n---\n> #### **Q3: Counterintuitive Overfitting of GRPO Compared to DPO with Data Scaling.**\n\n>\nThank you for the insightful observation. While this trend may appear counterintuitive at first, we believe it reflects GRPO’s stronger on-policy optimization, which enables the model (Janus-Pro) to reach its performance ceiling on GenEval more quickly.\n\n\nData Scaling|GRPO (OOD)|DPO (OOD)\n---|---|---\nBase Size|79.18|77.31\nDouble Size|**80.28**|78.88\nTriple Size|78.21|**79.25**\n\n\n\n\nAs shown, GRPO peaks with the ***Double Size*** dataset; further scaling leads to overfitting to in-domain patterns, thereby degrading out-of-distribution performance. In contrast, DPO benefits more gradually from increased data, likely due to its off-policy nature and slower convergence, which delays performance saturation. We will clarify this interpretation in the revision.\n\n\n---\n> #### **Q4: Potential Typo Inquiry for DPO Value in Table 5.**\n\n\n>\nThanks for pointing out. We appreciate your careful review. This was indeed a typographical error. The DPO score in the **Overall** column for the *Sampling 8* setting in Table 5 should be 79.23, not 55.17. We will correct this in the revised version.\n\n\n---\n> #### **Q5: Human Evaluation Comparing GRPO and DPO.**\n\n\n>\nThank you for the suggestion. We conducted a human evaluation comparing GRPO and DPO models trained with both HPS and Unified Reward. For each setting (in-domain and out-of-domain), 100 prompts were randomly sampled, and annotators compared image pairs, one from each method, choosing a preferred output or marking a tie.\n\n\n\n\nTraining Reward | Scenario | GRPO Wins | DPO Wins | Ties\n| :--- | :--- | :---: | :---: | :---: |\nHPS | In-Domain (ID) | 31 | 44 | 25\nHPS | Out-of-Domain (OOD) | 49 | 34 | 17\nUnified Reward | In-Domain (ID) | 32 | 48 | 20\nUnified Reward | Out-of-Domain (OOD) | 47 | 37 | 16\n\n\nThese results from human evaluators align with our core findings: DPO performs better in-domain, while GRPO generalizes more effectively out-of-domain. We will include this evaluation in the revised version to strengthen the empirical support for our findings."}, {"review_id": "gyaYXECW3B", "reviewer": "Reviewer_zwxi", "summary": "The paper performs a thorough analysis testing the effects of applying DPO vs. GRPO RL strategies to autoregressive image generation.\nThey compare their effects for in-domain vs. out-of-domain tasks, using different scaling strategies and reward models with each.", "questions": "1. The paper states and uses the benchmark \"T2I-CompBench\" as the \"in-domain\" case, and the benchmark \"GenEval\" as the \"out-of-domain\" case, but there is no clear explanation for this choice. A closer examination reveals that some aspects of these benchmarks are quite similar (e.g., GenEval's \"Colors\" and \"Color Attr\" categories are similar to the \"Attribute Binding\" categories in T2I-CompBench, and as such, the DPO results for these categories are also higher than the GRPO results), which suggest a different separation to categories might be more appropriate. Why did you choose the current separation?\n\n2. Did the authors try to apply some of the experiments to other autoregressive models?\n\n3. Some citations currently refer to the arXiv preprints instead of the corresponding peer-reviewed publications. Please update these to cite the final published versions where available. Some examples:\n[3] back to basics – cite the ACL version, [4] MathQA – cite the NAACL version,\n[15] “Can we generate images with cot? let’s verify and reinforce image generation step by step.” – Cite the CVPR version, etc.\n\n4. More visual examples illustrating the strength of different strategies over the others could strengthen the paper's claims.", "limitations": "Limitations are discussed.\nI think an additional aspect to discuss is that adding the wrong strategy can sometimes even harm results, depending on the goal, e.g., using DPO for Counting/Position yields worse results than the baseline..", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* The paper presents a comprehensive study testing many different aspects of using RL strategies for autoregressive image generation.\n* The paper is well-written with clear research questions and goals.\n\nWeaknesses:\n* The paper focuses on a single autoregressive model (Janus-Pro), which does not guarantee immediate transferability of the conclusions to all autoregressive models.\n* The separation of benchmarks to \"in-domain\" vs. \"out-of-domain\" isn't clear. See first question below.\n* There are very few visual examples.", "was_revised": true, "final_justification": "I believe this is an important and interesting paper, and the authors addressed my concerns.", "rebuttal": "We sincerely appreciate your valuable comments. We found them extremely helpful in improving our manuscript. We address each comment in detail, one by one below:\n\n---\n> #### **Q1: Generalization on Other Autoregressive Architectures**\n\n> \nThanks for your advice! We agree that validating our findings on additional autoregressive architectures is crucial for strengthening the generalizability of our claims. To this end, we conducted additional experiments using **Show-o**, a distinct autoregressive image generation model. While Show-o adopts a discrete diffusion approach, it still generates visual tokens in an autoregressive fashion, though using a parallel decoding strategy that produces multiple tokens per step. Despite this difference, the underlying generation remains token-by-token and sequential in nature. \n\nDue to time constraints, we focused on a representative subset of settings. The results consistently support our original findings:\n\n\n***1)*** DPO excels in in-domain tasks, while GRPO demonstrates stronger out-of-domain generalization.\n\n\n***2)*** The relative ranking of reward models on GenEval remains unchanged, with Unified Reward outperforming HPS under both RL algorithms.\n\n\nThese results confirm that ***our conclusions hold across architectures***. We will include this extended analysis in the final version of the paper.\n\n\n**In-Domain Evaluation on T2I-CompBench**\n|Reward Model|Average||Color||Shape||Texture||Spatial||Non-Spatial||Complex||\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|Baseline|37.06||55.37||40.85||47.24||19.91||29.92||29.04||\n|HPS|48.61|51.77|67.83|74.68|63.72|76.95|69.05|73.58|29.78|25.04|30.36|30.59|33.29|34.90|\n|Unified Rwd|46.09|51.93|64.89|72.56|60.08|73.61|66.28|74.12|27.45|29.85|30.32|30.71|30.80|34.52|\n\n\n**Out-of-Domain Generalization on GenEval**\n|Reward Model|Overall||Color Attr||Counting||Position||Single Object||Two Object||Colors||\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|Baseline|53.07||28.18||49.26||10.88||95.35||52.13||82.28||\n|HPS|54.19|52.31|28.19|31.47|54.32|46.27|12.18|10.27|96.18|90.56|53.41|56.68|79.55|83.22|\n|Unified Rwd|55.12|54.23|29.93|32.08|55.46|50.17|13.02|11.28|97.07|96.48|54.27|55.26|81.02|82.14|\n\n\n**Impact of Scaling on In-Domain Proficiency**\n|Scaling Method|Average||Color||Shape||Texture||Spatial||Non-Spatial||Complex||\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|*Data Scaling*|||||||||||||||\n|Base Size|48.61|51.77|67.83|74.68|63.72|76.95|69.05|73.58|29.78|25.04|30.36|30.59|33.29|34.90|\n|Double Size|49.34|52.49|68.82|72.18|65.02|77.08|70.31|74.66|28.37|30.18|30.66|30.92|33.59|34.93|\n|*Sample Scaling*|||||||||||||||\n|Sampling 4|48.61|51.77|67.83|74.68|63.72|76.95|69.05|73.58|29.78|25.04|30.36|30.59|33.29|34.90|\n|Sampling 8|50.17|54.07|68.55|73.60|65.73|75.68|70.27|74.18|30.07|33.72|30.58|30.61|33.86|34.91|\n|*Iterative Training*|||||||||||||||\n|Base|48.61|51.77|67.83|74.68|63.72|76.95|69.05|73.58|29.78|25.04|30.36|30.59|33.29|34.90|\n|Iter 1|49.18|53.14|67.48|74.38|64.12|75.92|69.55|73.77|29.42|33.46|30.44|30.90|33.58|34.13|\n\n\n**Impact of Scaling on Out-of-Domain Generalization**\n|Scaling Method|Overall||Color Attr||Counting||Position||Single Object||Two Object||Colors||\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|GRPO|DPO|\n|*Data Scaling*|||||||||||||||\n|Base Size|54.19|52.31|28.19|31.47|54.32|46.27|12.18|10.27|96.18|90.56|53.41|56.68|79.55|83.22|\n|Double Size|55.76|53.41|29.71|32.05|56.93|49.18|12.05|14.52|97.03|93.07|52.42|58.51|80.47|81.53|\n|*Sample Scaling*|||||||||||||||\n|Sampling 4|54.19|52.31|28.19|31.47|54.32|46.27|12.18|10.27|96.18|90.56|53.41|56.68|79.55|83.22|\n|Sampling 8|56.18|54.05|29.20|31.94|57.08|47.84|13.37|13.19|98.02|95.40|53.75|58.28|81.49|84.12|\n|*Iterative Training*|||||||||||||||\n|Base|54.19|52.31|28.19|31.47|54.32|46.27|12.18|10.27|96.18|90.56|53.41|56.68|79.55|83.22|\n|Iter 1|55.29|53.07|30.04|30.51|53.06|45.74|11.99|11.46|97.29|91.45|54.27|56.37|80.83|82.55|\n\n\n---\n> #### **Q2: Clarification of In-Domain vs. Out-of-Domain Benchmark Choice.**\n\n>\nThank you for raising this important point. We agree that a clear justification is necessary, and we clarify our rationale below:\n\n\n***1) Prompt Complexity as the Core Source of Domain Shift.***\n\nOur in-domain/out-of-domain split is motivated primarily by the **differences in prompt complexity and structure** between the benchmarks, as outlined in Section 2.1. T2I-CompBench features **long, compositional prompts** designed to test fine-grained scene understanding and multi-attribute reasoning. In contrast, GenEval uses **short, template-based prompts** (e.g., “a photo of...”) aimed at assessing robustness to concise or minimal inputs. This creates a **substantial domain gap** in both linguistic form and generation difficulty. Supporting this, we observe that our baseline model (Janus-Pro) **outperforms SD3-Medium on GenEval** but **underperforms on T2I-CompBench**, indicating that proficiency on one benchmark does not transfer trivially to the other.\n\n\n***2) Additional Out-of-Domain Validation on DrawBench.***\n\nTo further strengthen our claims and address the reviewer’s concern regarding potential attribute overlaps between benchmarks, we conducted supplementary experiments on **DrawBench**, a widely adopted benchmark for evaluating out-of-distribution generalization. The results remain consistent with our main findings: **GRPO consistently outperforms DPO across multiple OOD-relevant categories**, reinforcing its advantage in generalization.\n\n\nAlgorithm|Reward Model|DrawBench Score\n|-|-|-|\nBaseline|N/A|69.13\nDPO|HPS|70.25\nDPO|Unified Reward|72.50\n**GRPO**|HPS|**71.25**\n**GRPO**|Unified Reward|**72.88**\n\n\nWe believe these points clarify the rationale behind our experimental design. We will incorporate this clarification and the new DrawBench results into the revised paper.\n\n\n---\n> #### **Q3: Expanding Visual Examples.**\n\n\n>\nThanks for your advice. It is necessary that including more visual examples can strengthen our claims. While the rebuttal format constraints prevent us from attaching new images or links, we will be sure to incorporate a more extensive set of visualizations in the camera-ready version of the paper.\n\n\n---\n> #### **Q4: Updating Citations to Peer-Reviewed Versions.**\n\n\n>\nThank you for the suggestion. We will carefully review all references and update any preprint citations to their corresponding peer-reviewed publications in the camera-ready version."}]

Author rebuttal:

Meta-review: This paper present a focused comparison of DPO vs. GRPO for autoregressive image generation, analyzing domain generalization, reward-model sensitivity, and scaling impacts. Particularly It benchmarked Direct Preference Optimisation (DPO) vs. Group Relative Policy Optimisation (GRPO) on a 7B Janus‑Pro autoregressive text‑to‑image model across two evaluation suites (T2I‑CompBench in‑domain, GenEval out‑of‑domain), four reward models, and three scaling axes (samples-per-prompt, data diversity, iterative fine‑tuning). It shows that DPO achieves better in‑domain image quality and GRPO generalizes better to out‑of‑domain tasks. How well a reward model generalizes is a strong predictor of the policy’s generalization.

After the rebuttal discussions, reviewers unanimously agree to accept the paper, so is the proposal.
Decision: Accept (poster)