Title: Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Multi-modal Models

Image Generation

Reinforcement Learning

Chain-of-Thought (CoT) Reasoning
Abstract: Recent advancements underscore the significant role of Reinforcement Learning (RL) in enhancing the Chain-of-Thought (CoT) reasoning capabilities of large language models (LLMs). Two prominent RL algorithms, Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO), are central to these developments, showcasing different pros and cons. Autoregressive image generation, also interpretable as a sequential CoT reasoning process, presents unique challenges distinct from LLM-based CoT reasoning. These encompass ensuring text-image consistency, improving image aesthetic quality, and designing sophisticated reward models, rather than relying on simpler rule-based rewards. While recent efforts have extended RL to this domain, these explorations typically lack an in-depth analysis of the domain-specific challenges and the characteristics of different RL strategies. To bridge this gap, we provide the first comprehensive investigation of the GRPO and DPO algorithms in autoregressive image generation, evaluating their ***in-domain*** performance and ***out-of-domain*** generalization, while scrutinizing the impact of ***different reward models*** on their respective capabilities. Our findings reveal that GRPO and DPO exhibit distinct advantages, and crucially, that reward models possessing stronger intrinsic generalization capabilities potentially enhance the generalization potential of the applied RL algorithms. Furthermore, we systematically explore ***three prevalent scaling strategies*** to enhance both their in-domain and out-of-domain proficiency, deriving unique insights into efficiently scaling performance for each paradigm. We hope our study paves a new path for inspiring future work on developing more effective RL algorithms to achieve robust CoT reasoning in the realm of autoregressive image generation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper performs a thorough analysis testing the effects of applying DPO vs. GRPO RL strategies to autoregressive image generation.\nThey compare their effects for in-domain vs. out-of-domain tasks, using different scaling strategies and reward models with each.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper presents a comprehensive study testing many different aspects of using RL strategies for autoregressive image generation.\n* The paper is well-written with clear research questions and goals.\n\nWeaknesses:\n* The paper focuses on a single autoregressive model (Janus-Pro), which does not guarantee immediate transferability of the conclusions to all autoregressive models.\n* The separation of benchmarks to \"in-domain\" vs. \"out-of-domain\" isn't clear. See first question below.\n* There are very few visual examples.\n\nQUESTIONS:\n1. The paper states and uses the benchmark \"T2I-CompBench\" as the \"in-domain\" case, and the benchmark \"GenEval\" as the \"out-of-domain\" case, but there is no clear explanation for this choice. A closer examination reveals that some aspects of these benchmarks are quite similar (e.g., GenEval's \"Colors\" and \"Color Attr\" categories are similar to the \"Attribute Binding\" categories in T2I-CompBench, and as such, the DPO results for these categories are also higher than the GRPO results), which suggest a different separation to categories might be more appropriate. Why did you choose the current separation?\n\n2. Did the authors try to apply some of the experiments to other autoregressive models?\n\n3. Some citations currently refer to the arXiv preprints instead of the corresponding peer-reviewed publications. Please update these to cite the final published versions where available. Some examples:\n[3] back to basics – cite the ACL version, [4] MathQA – cite the NAACL version,\n[15] “Can we generate images with cot? let’s verify and reinforce image generation step by step.” – Cite the CVPR version, etc.\n\n4. More visual examples illustrating the strength of different strategies over the others could strengthen the paper's claims.\n\nLIMITATIONS:\nLimitations are discussed.\nI think an additional aspect to discuss is that adding the wrong strategy can sometimes even harm results, depending on the goal, e.g., using DPO for Counting/Position yields worse results than the baseline..", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-25"}

comments: SUMMARY:
This paper presents the first comprehensive empirical study comparing Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO) for enhancing Chain-of-Thought (CoT) reasoning in autoregressive image generation. Using Janus-Pro as the baseline model, the authors evaluate in-domain (T2I-CompBench) and out-of-domain (GenEval) performance, scrutinize the impact of reward models, and analyze three scaling strategies: sampled images per prompt, in-domain data diversity/volume, and iterative training.

STRENGTHS AND WEAKNESSES:
### Strength
**Experimental rigor:** Comprehensive evaluation across ID/OOD benchmarks, multiple reward models, and scaling dimensions with clear ablation.

**Practical insights:** The insightful findings (e.g., reward model generalization transfer, GRPO’s OOD stability, DPO’s iterative training trade-offs) benefit RL practitioners.

### Weakness
**Limited model scope:** Experiments confined to Janus-Pro; lack of validation on other autoregressive architectures (e.g., LlamaGen) weakens generalizability claims.

**Statistical fragility:** Results based on single runs, risking unreported variance.

QUESTIONS:
1. The paper observes that GRPO (superior in OOD generalization) suffers from overfitting with aggressive data scaling , while DPO (inferior in OOD) consistently improves with scaling (Figure 2 (d) and (e)) Does this appear counterintuitive given GRPO's on-policy adaptability?

2. Could the value of DPO with sampling 8 in Table 5 be a typo?

3. Could you provide the human evaluation results (e.g., win rate) for GRPO and DPO?

LIMITATIONS:
See weakness and questions parts.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article for the first time conducts an analysis of the respective advantages and disadvantages of DPO and GRPO in the context of autoregressive generation. The analysis is carried out from three key perspectives: domain generalization capability, sensitivity to reward modeling, and the impact of scaling strategies. The main conclusions are as follows: GRPO demonstrates superior performance in terms of domain generalization; DPO shows greater sensitivity to the characteristics and quality of reward models; when it comes to image scaling strategies, GRPO benefits more significantly; and in scenarios involving multi-round iterative processes, DPO exhibits distinct advantages.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths  \n1. This paper represents the first exploration into the comparative strengths and weaknesses of GRPO and DPO in the context of autoregressive image generation.  \n\n2. The article demonstrates a logical structure, with paragraphs presented in a clear and coherent manner.  \n\n- Weaknesses  \n\n1. Many conclusions drawn in the paper regarding GRPO and DPO reflect well-established observations within the field, such as the inherent improvement in model generalization from eliminating the critic model. The work requires novel insights that distinguish it from existing research on autoregressive generation tasks.  \n\n2. The paper omits critical methodological details, notably the formulas integrating DPO with GRPO for autoregressive image generation, which should be incorporated.  \n\n3. The visual evidence presented, for example, in Figure 3, inadequately supports the paper’s claims. For instance, the left panel of Figure 3 appears to demonstrate superior performance by GRPO, contradicting the authors’ interpretation, and similar inconsistencies are evident in other visualization examples.  \n\n4. The table in paper needs to have the average value or bolding to present the results more intuitively.\n\nQUESTIONS:\nCould the authors elaborate further on the commonalities and differences between GRPO and DPO in the context of the autoregressive image generation task, as compared to other autoregressive tasks?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
The authors benchmark Direct Preference Optimisation (DPO) against Group Relative Policy Optimisation (GRPO) on a 7 B‑parameter Janus‑Pro base model for autoregressive text‑to‑image generation. Experiments span two evaluation suites (T2I‑CompBench in‑domain, GenEval out‑of‑domain), four reward models, and three scaling axes (samples‑per‑prompt, data diversity, iterative fine‑tuning). Key findings:
- DPO excels in‑domain quality, while
- GRPO generalises better out‑of‑domain, and
- Reward‑model generalisation strongly predicts policy generalisation.

STRENGTHS AND WEAKNESSES:
Strengths:
- The evaluation is both broad and comprehensive, covering two benchmarks, four reward models, and three scaling dimensions. 
- The paper provides practical insights, offering clear guidance on compute and quality trade-offs. For example, identifying a moderate sampling size as ideal for GRPO.
- The writing and structure are clear. Figures 1 to 4 effectively illustrate key trends

Weaknesses:
- The OOD generalization evaluation is limited: GenEval mainly uses single-object prompts, and tougher benchmarks like Pick-a-Pic or DrawBench are omitted. Including at least one compositional OOD benchmark would strengthen the assessment
- Statistical rigor is lacking, with only single-run results, no standard deviations or confidence intervals, etc. It would be helpful if at least three seeds were reported.
- Important baselines like PPO are missing. It would be good to include a 1-epoch PPO baseline should be included to provide context.

QUESTIONS:
- Could a GRPO warmup followed by DPO refinement combine robustness with high in-distribution quality?
- Do the findings remain consistent when comparing 1B and 7B Janus variants?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03