Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models
Authors: Jiajun Fan
Tong Wei
Chaoran Cheng
Yuxin Chen
Ge Liu
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement learning
generative models
adaptive regularization
advantage estimation
fine-tuning
flow matching
text-to-image generation
exploration-exploitation trade-off
Primary area: reinforcement_learning
TL;DR: We propose ADRPO, a method that dynamically adjusts divergence regularization strength based on advantage estimates, enabling more effective fine-tuning of generative models by automatically balancing exploration and exploitation at the sample level.

Abstract: Balancing exploration and exploitation during reinforcement learning fine-tuning of generative models presents a critical challenge, as existing approaches rely on fixed divergence regularization that creates an inherent dilemma: strong regularization preserves model capabilities but limits reward optimization, while weak regularization enables greater alignment but risks instability or reward hacking. We introduce Adaptive Divergence Regularized Policy Optimization (ADRPO), which automatically adjusts regularization strength based on advantage estimates—reducing regularization for high-value samples while applying stronger regularization to poor samples, enabling policies to navigate between exploration and aggressive exploitation according to data quality. Our implementation with Wasserstein-2 regularization for flow matching generative models achieves remarkable results on text-to-image generation, achieving better semantic alignment and diversity than offline methods like DPO and online methods with fixed regularization like ORW-CFM-W2. ADRPO enables a 2B parameter SD3 model to surpass much larger models with 4.8B and 12B parameters in attribute binding, semantic consistency, artistic style transfer, and compositional control while maintaining generation diversity. ADRPO generalizes to KL-regularized fine-tuning of both text-only LLMs and multi-modal reasoning models, enhancing existing online RL methods like GRPO while requiring no additional networks or complex architectural changes. In LLM fine-tuning, ADRPO demonstrates an emergent ability to escape local optima through active exploration, while in multi-modal audio reasoning, it outperforms GRPO through superior step-by-step reasoning, enabling a 7B model to outperform substantially larger commercial models including Gemini 2.5 Pro and GPT-4o Audio, offering an effective plug-and-play solution to the exploration-exploitation challenge across diverse generative architectures and modalities.

Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models
Jiajun Fan, Tong Wei, Chaoran Cheng, Yuxin Chen, Ge Liu University of Illinois Urbana-Champaign (jiajunf3, tweil1, chaoran7, nealchen, geliu)eillinois.om
Abstract
Balancing exploration and exploitation during reinforcement learning fine-tuning of generative models presents critical challenge, existing approaches rely fixed divergence regularization that creates an inherent dilemma: strong regularization preserves model capabilities but limits reward optimization, while weak regulariza- tion enables greater alignment but risks instability or reward hacking. We introduce Adaptive Divergence Regularized Policy Optimization (ADRPO), which auto- matically adjusts regularization strength based on advantage estimates-reduciin regularization for high-value samples while applying stronger regularization poor samples, enabling policies to navigate between exploration and aggressive exploitation according to data quality. Our implementation with Wasserstein-2 regularization for flow matching generative models achieves remarkable results on text-to-image generation, achieving better semantic alignment and diversity than offline methods like DPO and online methods with fixed regularization like ORW- CFM-W2. ADRPO enables 2B parameter SD3 model to surpass much larger models with 4.8B and 12B parameters in attribute binding, semantic consistency, artistic style transfer, and compositional control while maintaining generation di- versity. ADRPO generalizes to KI--regularized fine-tuning of both text-only L.MSs and multi-modal reasoning models, enhancing existing online RI. methods like GRPO while requiring no additional networks or complex architectural changes. In LLM fine-tuning, ADRPO demonstrates an emergent ability to escape local optima through active exploration, while in multi-modal audio reasoning, it outperforms GRPO through superior step-by-step reasoning, enabling a 7B model to outperform substantially larger commercial models including Gemini 2.5 Pro and GPT-4o Audio, offering an effective plug-and-yg and-play solution to the pellorationexplloiteee challenge across diverse generative architectures and modalities.
Introduction
Reinforcement learning fine-tuning has emerged powerful paradigm for aligning generative models with human preferences, driving remarkable improvements in capabilities from text gener- ation to image synthesis (2 4 34]. At the core of modern RLHF approaches lies fundamental challenge: effectively balancing divergence regularization against reward maximization during policy optimization. This balance is critical as it determines whether models retain the beneficial properties of their pre- pre-trained foundation while adapting to better satisfy human preferences [26] 46 2
The current standard practice employs divergence regularization with fixed coefficients constrain policy updates typically using Kullback-Leiin (KI.) 32 24 or Wasserstein-2 (W2) divergences [14]. However, this approach creates inherent dilemma that limits performance: strong regu- larization preserves model capabilities but hampers reward optimization, while weak regularization enables greater reward optimization but risks catastrophic forgetting, mode collapse, or reward
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

hacking 20 33]. This rrade-off is particularly pronounced in generative models where preserving diversity while improving quality represents critical balance B [5]. Existing approaches like PPO
31. GRPO R2, and DPO 226 37 employ fixed regularization coefficients that treat all data points equally, regardless of whether the policy should prioritize exploitation (when rewards are reliable) or exploration (when falling into suboptimal solutions). This one-size-fits-al approach fails to adapt to the varying exploration needs across the complex landscapes of generative model policy optimization.
To address these limitations, we propose Adaptive Divergence Regularized Policy Optimization (ADRPO), novel framework that dynamically adjusts regularization strength using advantage estimates [30 33). ADRPO employs advantage signals to fine-tune the exploration-exploine exploitation trade- off: high-advantage samples reduce regularization for aggressive optimization, while low-advantage samples increase it for stability. This sample-level adaptation integrates seamlessly into training, providing an efficient and automated approach. By aligning regularization with sample quality, ADRPO overcomes the shortcomings of prior methods, delivering superior alignment and generative performance across diverse tasks, as demonstrated in our experiments with text-to-image alignment and language model fine-tuning. In summary, our approach makes several important contributions:
1. General RL Framework with Adaptive Divergence Regularization. We introduce ADRPO as a general-purpose framework that dynamically adjusts regularization based on advantage estimates, integrating with existing RL fine-tuning methods without architec- tural changes. Our proposed methods address the euploration-exploitttiin dilemma while preventing reward hacking and model collapse.
2. Superior Text-to-Image Alignment with Smaller Model. We first propose a novel online RL method based on ADRPO, combining advantage-bast based policy optimization and adaptive W2 regularization for fine-tuning flow matching models. Our experiments of fine-tuning
SD3 demonstrate ADRPO's ' dominant Pareto frontier in the reward- -iveerssttt trade-off and
reward-divergence trade-off compared to DPO [26] and fxeddregulariizatoon approaches [4] (See Figs. and 3) Notably, our 2B parameter model outperforms larger 4.8B (38) and
12B parameter models across attribute binding, compositional control, and semantic consistency (See Figs. and Tab. D
3. Emergent Exploration in LLMs. We also apply our ADRPO to improve GRPO 32 for online fine-tuning of LLMs (See Figs. 4.. ADRPO not only improves alignment but exhibits an emergent ability to escape local optima by actively increasing exploration when needed-a a capability absent in rred-regularizztton methods like GRPO.
4. Cross-Domain Applicability. ADRPO provides a unified solution across continuous (flow matching with W2 regularization), discrete (LLMs with KL, divergence), and multi-modal reasoning generative paradigms (audio reasoning models), offering immediate practical benefits with minimal computational overhead (See Sec. and App. D.3)
2 Related Work
RL Fine-tuning for LLMs. Reinforcement learning has become the dominant approach for aligning large language models with human preferences. Pioneering work by 4 established the RLHF framework, which was later scaled by [24] to create models that better follow human instructions. The algorithmic landscape has evolved from PPO BI] to more efficient alternatives like GRPO B2] and offline approaches like DPO 26). These methods have significantly improved the reasoning abilities.
capabilities of models like DeepSeek-RI (16] and improved their cnstruction-follovini Despite their success, these approaches typically rely on fixed regularization parameters that treat all samples equally. regardless of whether they represent promising directions for optimization or uncertainty-ladee regions requiring more conservative updates (See Fig. 4).
RL Fine-tuning for Flow Matching Models. While RL fine-tuning is established for language models, its application to flow matching (FM) models 23 presents unique iitooratioo-eploiatatiii trade-off challenges due to their continuous- time nature and ODEE-bseed sampling. Recent approaches like Online Reward- -eighted Fine- MuniTT with Wasserstein regularization [1] and offline methods like diffusion-DPO
37 have made progress, but remain limited by fixed regularization schemes that cannot adapt to sample-specific characteristics. This fundamental limitation restricts their ability to optimally balance the critical exploration-exploitaiii trade-off necessary for effective fine-tuning of state-of-the thet image generation models like SD3 [13] (See Tab. and Figs.

---

Divergence Regularization in RL Fine-tuning, Divergence regularization plays crucial role in RL fine-tuning by preventing the policy from deviating too far from the initial model, thus preserving desirable properties while allowing for improvement. For language models, KI. divergence serves as the standard metric in methods like PPO A GRPO [2], and DPO [26], while flow models benefit from Wasserstein distances [14] that better handle continuous distributions. Despite their importance, existing approaches typically employ fixed regularization coefficients that fail to handle the varying significance of regularization across different samples and different learning stages. This limitation can lead to suboptimal trade-offs between preserving model capabilities and maximizing rewards (e.g.. GRPO in Fig. 4 risking model collapse [33 17 insufficient improvement. Our work addresses this gap through adaptive regularization based on advantage estimates, providing a novel approach to dynamically balancing exploration and exploitation during training.
3 Method
3.1 Problem Formulation
In this paper, we address the challenges of fine-tuning pre-trained generative models through online RL to improve their alignment with human preferences [4] 24). Given pre-trained reference policy Tef and its fine-tuned counterpart To parameterized by 0, our objective is to maximize the expected user-defined reward Eg [R(z, c)]. where R(x, c) quantifies human preference for generation x
p(c) (c.g.. CLIP Score (25) for T2I tasks). This context may be text prompt in LLMs [4] 42 I or an image description in text-to- text-to-image (T2I) models [13 38 43]. The
conditioned context
standard approach in RL fine-tuning formulates this constrained optimization problem:
J(0) = -x-re) R[x,, c)] B ( Tef
(1)
Here, p(c) is the sample distribution of prompts (e.g., uniform sampling in our paper), D(Tg, Te) represents a divergence measure between the fine-tuned and reference policies-typicalyy Kullback- Leibler divergence (KL) for discrete generative models B1 32] 26 or Wasserstein distance for
continuous distributions [36] I 14] The coefficient controls the trade-off between reward opti-
mization and preservation of the pre- pre-trained model's 's capabilities (e.g., diversity).
3.2
Adaptive Divergence Regularized Policy Optimization
Recent approaches to online RL fine-tuning of generative models have explored different divergence measures, including W2 regularization in flow matching models [4 and KL divergence in LLMs
However, these methods still rely on fixed regularization schemes that treat all samples equally, regardless of their potential for reward improvement or risk of degradation. This fundamental challenge of adaptive regularization eegulariztionnddnammaalll dynamically balancing exploration and exploitation (See Figs.
32
24
3 andd at the individual sample level-remains largely unaddressed in the literature.
3.2.1 Conventional RL Fine-tuning Approaches
The conventional RL objective in Equation I can be rewritten as combination of two loss terms:
PRIHF = (0) + 3 (0)
where CRL(O) is the policy optimization term such as policy gradient [31] reward- [4] and (D) = D(Te, Tet) is the divergence regularization term, such as KL divergence in LLMs 32 or W2 divergence in flow matching models [14]. In practice, this formulation has been
instantiated in various ways. For example, Group Relative Policy Optimization (GRPO) 32 employs KL-regularized policy gradient objective for LLMs:
(o) LPG(0) + 3- DKL(TTl Tret)
where LrG (f) represents a clipped policy gradient loss based on group- advantage estimation. Similarly, DRW-CFM W2 [4] applies a W2 regularization term for flow matching models:
LORW.CFM.22(0) Lorw + 3 E.,.. n [[vo(xr, 4 Vier 1, ()
where LORW E d.m, (x) c) * (x t, c) u, [2] is the reward weighted loss and vo and Vief the velocity fields of the finc-tuned and reference policies, respectively.
3

---

Critically, in all these approaches, the regularization strength 8 remains constant across all samples and training steps, failing to adapt to the varying quality of generated samples.
3.2.2 Our Approach: Advantage-Basel Adaptive Regularization
We introduce Adaptive Divergence Regularized Policy Optimization (ADRPO), a principled frame- work that dynamically adjusts regularization strength based on the estimated advantages of individual samples. The key insight in ADRPO is that the regularization coefficient should not be static, but should vary inversely with the sample's estimated advantage. Formally, we propose:
LADRFO (f) (0) + (30 A) LD(())
where A is an advantage estimate for the current sample and 3o is baseline regularization coefficient. This formulation creates an adaptive regularization coefficient Peo 80-- A that adapts based on the quality of each sample. This adaptive mechanism creates a natural balance: 1) Exploitation: in regions where the policy generates high-quality samples (high advantage), ADRPO allows for efficient exploitation by reducing divergence penalties; 2) Exploration: in uncertain low-quality regions (low advantage), it enforces stronger regularization to maintain stability and preserve the model's s original capabilities (See Figs. and
Based on Equ. ④ our ADRPO can be seamlessly integrated with various existing RI fine-tuning methods. For instance, when applied to GRPO for large language models (LL.Ms), the objective becomes LADRPO-GRPO (f) = LPG (0) + (3o - AgRPO) DKL (Tal Teet), where AGRPO is the advantage estimate from GRPO's group-based estimation procedure [32 [6].
3.3 ADRPO for Flow Matching Generative Models
We now demonstrate how our ADRPO framework can be effectively applied to fine-tuning flow matching models 23 36). particularly focusing text-to-image generation models like SD3 [13]
3.3.1 Flow Matching Preliminaries
Flow matching (FM) models define continuous-time transformation that maps a simple prior distribution p(=oo) (e.g.. Gaussian) to a complex target distribution via probability flow Pt- An FM model learns velocity field vo(x,, t,e) c) that approximates the true velocity field u, (2t lc). However, since u, (xx is often intractable [23]. Conditional Flow Matching (CFM) [36] proposes an equivalent yet tractable objective by conditioning the flow on target samples Jı while learning a conditional target velocity field (e.g., ue(za/z1, c) xj xo for linear interpolation path (23):
(CFM(() =
(n. (2) c) (1)
u, (2, c)
(5)
Given pre-trained reference model like SD3 13, flow matching fine-tuning aims to align generations with human preferences while preserving generative diversity. Traditional approaches, including supervised fine-tuning and offline RL methods like DPO 26 37 sample target states z Paata (2/|c)
from a fixed human-cus auman-curated dataset- a stable but limiting approach that restricts exploration of potentially better policy regions (See Figs. andd In contrast, our proposed ADRPO framework embraces an online RL paradigm, sampling target states from the fine- tuned policy itself: x1 ~ pa (xlc), with p - representing the policy at the previous iteration. This online sampling strategy enables the model to continuously improve upon its own generations and explore the policy space more effectively but is prone to collapse [4], while our adaptive regularization mechanism specifically addresses the inherent instability and exploration-exploiaee dilemma in online RL fine-tuning.
3.3.2 ADRPO with Wasserstein Regularization
key observation across RL fine-tuning methods [32 is that effective policy optimiza- tion requires differentially weighting samples based on quality (e.g.. upweighting probabilities of high-reward samples while downweighting poor ones). While traditional RL methods scale high-quality trajectories while weakening low-quality ones based on advantage estimates (3:555 hasn't 't been fully lever- aged in flow matching fine-tuma mun-ttning. Current approaches like weiggte flow matching [14]
updates by advantage estimates 33 30 this

---

can only down-weight poor samples without actively discouraging them, significantly reducing efficiency high-dimensional spaces (e.g,, image generation) where undesirable regions vastly outnumber desirable ones. We address this limitation by introducing advantage-based policy op- timization for flow matching models, creating bidirectional learning signals through advantage estimates rather than non-negative reward weights to both enhance high-quality generations and actively suppress poor ones. Specifically, we propose an advantage-weighter flow matching objective:
( E Ee-pte) e-- 10, (emppe (x..e) [A(21, c) [vo t, c) u, ?). where P/a the current fine-tuned policy and A(21, c) represents the estimated advantage for sample 21 given context e, vo (2 f, c) is the learned velocity field, Ft = (1 t) + tæi, and u, = x1 20 is the target
velocity for the straigh--line interpolation in FM models (23
This formulation creates fundamentally different learning dynamic compared swwar--weighting approaches. For samples with positive advantage (A > 0), the objective encourages matching the target velocity field, strengthening high-quality generations. Conversely, for samples with negative advantage (A < 0), the sign inversion reverses the gradient direction, actively pushing the model away from poor generations rather than merely down-weighting them. Mcanwhile, average-quality samples (where A x 0) contribute minimally to the gradient, naturally focusing computational resources on the most informative examples and facilitating efficient convergence (See Fig. 3
Advantage Estimation. For FM models, we compute the advantage as the difference between the reward of a sample and the expected reward under the current policy as A(z), c) R( Vc)), where R(xI,)) c) is the human preference reward for the generated sample 21 given context c, and V(c) is baseline value function estimated the average reward over a batch of samples for the same context, which is computationally efficient.
Adaptive Regularization. Based on Equ.
further propose to dynamically adjust the regular- ization strength based on the same advantage estimates. This creates a unified framework where the exploration-exploitatoon balance is automatically modulated at the individual sample level:
CADRPO-FM -=-o - m)) [A(x1, c) [vo(x,, - uzl3]
+(80 -((,, c)) B.t.z, [voo t, - (2,, (6)
The adaptive regularization coefficient A.n 8n A A(r1, c) establishes dynamic adaptation mecha- nism responsive to sample quality. For high-advantage samples (A > 0), regularization decreases proportionally. For low-advantage samples (A < 0), regularization strengthens proportionally, constraining updates to maintain proximity to the reference model. This bidirectional adaptation fun- damentally transforms the exploration-explooiatio landscape (Fig. 3 replacing fixed regularization with sample-wise W2 regularization that continuously adapt to the evolving policies.
Stabilization, Efficient Learning, To ensure stable training with our adaptive advantage-biig -ased approach, we use advantage clipping that constrains advantages to a reasonable range [Amin Amax] as Asiiped (z1, c) = clip(A(()) c), Amin: Amax). We also use LoRA [18 for efficient learning.
3.4 ADRPO for Fine-tuning LLMs
Applying our ADRPO framework Large Language Models (LLMs) can address the limitation of static regularization in conventional online RL methods by dynamically controlling the penalty for deviating from the pre- trained policy based sample advantage. High-advantage responses indicate promising directions warranting reduced regularization to encourage policy optimization, while tow-advantage responses signal areas avoid, requiring increased regularization to maintain proximity to the reliable pre-trained model and prevent undesirable outputs or instability. We integrate this principle with GRPO [32], modifying its objective by making the KL divergence regularization strength dependent on the advantage estimate (AGRPO) for each sample. The objective becomes:
C LADRFO-GRFO (0) = (PG(O) + (Bu - AGRPO) DKL (Tell
Here, (f) is the clipped policy gradient term (32] (i.e., min A wratio, A*clip(ratio, 1 1 1++)) and ratio
me)), DKL. is the KL divergence, and 80 baseline regularization. The term
Re.s (go AGRPO acts adaptive coefficient, decreasing for good samples (AGRPO > 0) to promote
5

---

Table 1: Comparison of text-to-image generation methods across different evaluation metrics. Best
scores are highlighted in blue second-best in green We report standard errors estimated over
random seeds. ClipDiversity measures the mean pairwise distance of CLIP embeddings 25
14
Task Metries
Image Quality
Human Preference
Method
ClipSooref
ClipDivesiiyy
X3
Acstheticf
BLIPScoref
InsgeRRwarrd
PicSeare1
Base Model
SD3 (28) 1
29.27=0.42
5.08=0.52
5 5.53+00009
0 +0,007
0.97+0.13
20.81+00099
Other Flow Mateeinng Models
FLUX (12B) 1
31.72=0.48
4.29::042
5,95=0.05
0.492=0.004
L1
21,83:0.11
SANA-11.5 (4)88)
X
32,18-0.36
4.31++0.0
5. 89-+0.12
0.526+000006
1.45+0.08
21,85+0.155
Fine-tuning Methods
SDI+RAFT 0
29.35=0.227
1,85
4.51220.01
0.512=0.001
0,2240.08
19.21
SD3+DPO HH
31.30
4.78:50.46
5.82.10.05
0.509:10.005
1,48.0.10
21.31110.10
SD3+ORW
44
31,42:00.39
3.86:40.37
5.29:40.5
0.542:00.006
1.2240.10
20.97:00.11
SD3+ADRPO (Ours)
32.97-00.46
5.13+00.47
6.27-+0.06
0.567+0.004
1.61++005
22.78+0.18
exploitation and increasing for poor samples (AGRPO 0) to enforce conservative exploration, allowing ADRPO-GRPO to achieve better exploration-exploittiii tradc-off (See Fig. 4
4 Experiment
4.1
Experimental Setup
For our experiments, we evaluated ADRPO across three distinct domains: flow matching models, LLMs, and multi-modal reasoning models. Fine-tuning Flow Matching Models. We implemented ADRPO on SD3 (2B parameters) using prompts from DrawBench (28] testing color attribute binding, compositional reasoning, object counting, spatial relationships, and text rendering, as well as artistic
style transfer prompts from RAFT (. Our method employed the advantage- based ADRPO loss
from Equation (6) with 8o = 1 and A maz Amaz
1, Amin -1, using CLIP score as rewards 25
We
compared against offline methods like DPO 37 online approaches like ORW-CFM-W2
14 and
substantially larger models including FLUX.1 Dev (12B) 43 and SANA-1.5 (4.8B)
38 Fine-
tuning LLMs. We fine-tuned Qwen2 4 and Qwen3
using RM-Gemma-2B [2
9 as the
httll////westteeeerrteeeee reward model on regularization (Equation 7 with 3g using KL-divergence compared against standard GRPO with fixed 3 0.04 32)
2k dataset 9
ADRPO was integrated with GRPO
0.04, Amin = -0.04, Amaz = 0.04,
Fine-tuning Multi-Modal Reasoning
Models. We fine-tuned rewards, evaluated the MMAU benchmark (29]. We used Bu 0.04, Amin -0.04, Amaz 0.04, comparing against GRPO bascline and commercial models including Gemini 2.5 Pro [5) and GPT-4o Audio 19]. All experiments employed advantage clipping for training stability and set 80 equal to the fixed 3 in baseline methods for fair comparison. See App. and for complete details.
5-55mmmmm--B [40 on the AVQA dataset [44) using verifialle and format
4.2 Main Results
Tab. I presents a comprehensive evaluation of text-to-image generation methods, demonstrating that our proposed ADRPO establishes superior Pareto frontier in all metrics and achieves the best reward-diversity trade-off. Unlike competing approaches such as DPO and ORW-CFM-W2 that make significant cempromises-improviin semantic alignment at the cost of diversity or vice versa-ADRPO achieves state-of-the-art performance in both dimensions simultancously through its dynamic regularization mechanism Equ. (6). Our adaptive approach intelligently modulates regularization strength based on sample-specific advantage estimates, enabling aggressive exploitation in high-reward regions while maintaining exploration elsewhere (See Figs. [ and 4 Perhaps most remarkably, our method enables a relatively modest 2B parameter SD3 model to outperform
substantially larger models including FLUX. 1-evv (12B) [43 and SANA- (4.8B) [3 across all
evaluation metrics, particularly in human preferences. This quantitative superiority is visually evident in our qualitative results Figs. and where -DRPO-geeerated images demonstrate exceptional
attribute binding, spatial understanding, text rendering, and artistic style transfer capabilities that

---

even larger models struggle to match. Together, these findings suggest that adaptive regularization offers more efficient path to performance improvement than simply scaling model parameters.
4.3
Qualitative Analysis
Jocques-Louts David style big ben
Edward Hopper style vase
Van Gogh style astronsut
green apple and block backpack.
503
(28)
1 (128)
SAMA-1.5
(4.58)
(28, (28,0urs)
Figure 1: Qualitative Comparison with Large FM Generative Models. Our ADRPO demonstrates superior performance in Artistic Style Rendering, Attribute Binding, Coloring and Counting.
Comparison with SOTA Large FM Models. Fig. shows our ADRPO fine-tuned SD3 model (2B parameters) significantly outperforming much larger models like FLUX.1 I Dev (12B) and SANA- 1.5 (4.8B). This challenges the conventional wisdom that parameter scaling is the primary path to performance improvements. Our method excels in areas where larger models struggle: for artistic style transfer ("Jaqques-Louis David style big ben"), complex compositions ("Van Gogh style astronaut"), and attribute binding (" ("green apple and black backpack"), ADRPO maintains both style accuracy and compositional integrity while larger models introduce visual artifacts despite their 2-6x parameter counts. These results demonstrate that adaptive regularization can enable smaller models match or exceed much larger models' capabilities. See App. D for more results.
A storefront with 'DaFfusion! written on
A sebra to the right of fire hydnent.
black calored banona
Edword Happer style vase
503 DPO
S03+
ORW-
GFM-wW2
AD8PO
Figure 2: Qualitative Comparison with Other RL Fine-tuning Methods. Our ADRPO demon- strates superior performance in Artistic Style Rendering, Text Rendering, Attribute Binding, Coloring, Counting and Position, We use a similar DPO method as described in g mune SD3 models.
7

---

Comparison with other RL Fine-tuning Methods. Fig. demonstrates ADRPO's 's clear superiority over existing reinforcement learning fine-tuning approaches. While DPO [7 preserves diversity at the cost of semantic alignment and ORW-CFM-W2 [14] improves alignment but sacrifices diversity, ADRPO achieves excellence in both dimensions through eavaattag-guudeen regularization. This is evident across text rendering ("Diffusion" storefront), attribute binding (zebra positioning), coloring (black banana), and style transfer tasks, where our method consistently delivers superior compositional accuracy. By dynamically modulating regularization strength- -nngeeinnnnnnnngggggg constraints for uncertain samples while allowing greater divergence for reliable ones-ADRPO effectively resolves the exploraton-exxpooitaion dilemma that static approaches cannot address.
4.4 Visualizing Exploration Tfa in Policy Optimization
Reward vs Diversity Trade-off off
Final Reward
Final Diversity
22.
50
322
32.0
4.8
21.5
4.5
31.2
90.6
31.5
302
23.
23.5
3.0
4.
42
4.6
4.0
28.0
Diversity
Reward vs W2 Divergence Trade-off
Divergence
02.5
Dopeggen sold
0.0030
525
0.0180
31.6
0.0125
51.2
0.0000
90.5
0.0075
902
235
0.00
0.009
0.212
0,0000
W2 Divergence
Muthod
Figure 3: Reward-DiversittyDDiverrs Trade-off. Left: policy optimization trajectories (using a same seed) of different methods throughout training, with transparency indicating progression from early (transparent) to convergent (solid) to final (star) checkpoints. Each point a learned policy from different iterations. Center and right: final reward and diversity/divergence across methods.
Reward Hacking Mitigation. Figs. B reveals distinct vulnerability patterns reward hacking across methods. While DPO maintains moderate diversity but plateaus in reward optimization, ORW- CFM-W2 aggressively pursues reward optimization but exhibits significant diversity collapse (right panel), resulting in template-like generations (See Fig. 2 Our ADRPO, through advantage-guide regularization, achieves the highest reward without sacrificing diversity-4 combination neither competing method attains. This translates to superior generations with precise attribute binding and high visual quality while maintaining creative flexibility. See App. D for whole learning curves.
cnatoration-Exploiataaie
Balance and Divergence Control. The trajectory visualization in Figs. landscape. The bottom
3 (left) captures each method's navigation of the ppploratioonexploiiatenoo plots further illustrate ADRPO's advantage in maintaining minimal W2 divergence while maximizing reward. While DPO makes modest improvements before plateauing and ORW-CFM-W2 follows an exploitation path that compromises diversity, ADRPO consistently expands the Pareto frontier. This superior balancing act stems from our adaptive mechanism making sample-specific regularization decisions, effectively resolving the dilemma that fixed-coefficient methods cannot address.
8

---

4.5
Application to Fine-tuning LLMs
Exploration vs Exploitation Trade-off
Final opy
14
12
Explelation [eoeltuttea/Boeee
1.0
0.8
0.5
35
0.0
Exploration (Entropy)
Exploration vs Exploltation Trade-off
Entropy
Ienerpaae Coamengen 6o0
Explettatio 25
0.0
Exploration (Entropy)
Figure 4: Ablation Studies and peplorattionExploitaee Trade-off in Fine-tuning LLMs. Left: policy optimization trajectories in reward-entropy space for ADRPO and GRPO (i.e., fixed 3 as 32 across Qwen2 (0.5B) [41 (top) and Qwen3 (0.6B) (42 (bottom) models, with transparency indicating progression from early to final checkpoints. Center and right: Final performance of different methods.
To demonstrate ADRPO's versatility beyond FM models, applied our approach LLM fine-tuning with the Qwen2 [1 and Qwen3 422 model families using RM-Gemma-2B (9 10 27 as the reward model, shown in Fig- 4 All methods are evaluated in W...Fllowwtss__eeeerrttttt
2k dataset 9
Superior Exploraation-Exploitatioo Control. The policy optimization trajectories (left panels) in Fig. reveal distinct patterns between methods. While GRPO maintains high entropy throughout training but struggles to find high-reward
substantial horizontal movement with limited vertical improvement -ARPP implements strategic exploration pattern that efficiently navigates landscape. For Qwen3 (bottom left), ADRPO exhibits a remarkable ability to first explore lower entropy regions, then actively increase entropy to escape local optima, before converging to final checkpoint with 5x higher reward than GRPO.
the iitooaation-eploitattii
Preventing Model Collapse. ADRPO demonstrates superior resistance to model collapse during extended training. GRPO's performance tends to plateau or deteriorate as training progresses (also Fig. D.2.1 with later checkpoints (darker red points) often showing lower rewards than carlier ones a common failure mode of fixedregularriztton methods. In contrast, ADRPO shows consistent improvement throughout training by dynamically adjusting regularization strength based advantage estimates, eliminating the need for careful early stopping to prevent performance regression.
Cross-Archieccture Generalizability. The consistent superior performance across both flow match- ing models and different L.M architectures confirms that ADRPO addresses fundamental limitations in reinforcement learning fine-tuning. By adapting regularization strength to sample-specific advan- tage estimates, our method provides generalizable solution to the iotoratioo-epplitattioon dilemma that effectively transfers between domains. See App. D for whole learning curves.

---

Table 2: Multi-modal audio reasoning results on MMAU benchmark 29
Method
Sound
Music (%)
Speech (%)
Total (%)
Qwen2.5-Omui (base)
72.37
64.37
69.07
68.6
GRPO
77.18
70.66
74.77
74.2
Gemini 2.5 Pro
75.08
71.47
71.6
GPT-4o Audio
64.56
56.29
66.67
62.5
ADRPO (Ours)
81,98
70.06
75.98
76.0
Table 3: Ablation on advantage clipping ranges.
Clipping Range
Auia
Awax
(%)
Music (%)
Speech (%)
1 x (recommended)
-0.04
0.04
81.98
70.06
75.98
76.0
2 X (g
-0.08
0.08
84.08
69.46
73.57
75.7
-0.02
0.02
71.26
74.47
GRPO (baseline)
77.18
70.66
74.77
74.2
4.6 Extension to Multi-Modal Audio Reasoning
To validate ADRPO's generalizability beyond text-to-image generation and text-only LLMs,
fine-tuned Siven2.5-0mni- [40] on audio reasoning tasks using the AVQA dataset [44], evaluated
on the MMAU benchmark [29]. This domain requires temporal sequence understanding and acoustic
pattern recognition-
distinct from text or visual generation.
Tab. presents our results comparing ADRPO against the base model, GRPO baseline, and larger commercial models. Our method achieves 76.0% accuracy, outperforming all baselines and enabling 7B model to surpass substantially larger systems including Gemini 2.5 Pro [5] and GPT-4o Audio
19 through more effective
ceruationnxxpliiaatton balance.
Hyperparameter Robustness. To assess ADRPO's practical applicability in new domains, we conducted ablation studies on advantage clipping ranges. Tab. [hows that all ADRPO configurations substantially outperform GRPO with stable performance across settings. The recommended So achieves the best overall balance, while 2 x Bo shows the trade-off between aggressive exploitation (higher sound accuracy) and generalization (lower music/speech accuracy). Performance variations across clipping ranges remain under 0.4%, confirming robustness.
These results, combined with our text-to-image experiments (Sec. and LLM fine-tuning (Sec. 4.3), demonstrate ADRPO's effectiveness across continuous generation, discrete sequences, and multi-modal reasoning. Qualitative analysis reveals that ADRPO enables superior step-by-step reasoning on complex temporal and contextual tasks compared to GRPO's premature convergence. See App. D.3 for detailed setup and qualitative examples.
5
Conclusion
The
dilemma represents a critical challenge in generative model RL fine- tuning that fixed regularization approaches fail address. To tackle this, we propose Adaptive Divergence Regularized Policy Optimization (ADRPO), which dynamically adjusts regularization strength based sample-specific advantage estimates-reducee while strengthening them for poor ones. Our experiments demonstrate ADRPO's effectiveness across diverse domains and modalities: in text-to-image generation. it outperforms other methods in alignment, quality, and diversity, enabling our 2B parameter SD3 model to surpass much larger models (4.8B and I2B) in various tasks; in LLM fine-tuning, it exhibits an emergent ability to escape local optima by actively increasing exploration; in multi- multi-modal audio reasoning, it outperforms GRPO through superior step-by-step reasoning. enabling a 7B model to outperform substantially larger commercial models including Gemini 2.5 Pro and GPT-4o Audio. ADRPO establishes a superior Pareto frontier in the reward-diversity trade-off across continuous generation, discrete sequences, and multi-modal reasoning tasks, confirming that sample-adaptive regularization offers plug-and-play solution that generalizes across generative architectures with minimal computational overhead. See App. A for more discussion.
constraints for high-value samples
10

---

References
[1] Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 214-223, PMLR, 2017.
[2] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds. Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204, 2.0562.. 2022.
[3] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[4] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4299 4307, 2017.
[5] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit S. Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan- Jiang Jiang. Krishna Haridasan, Ahmed Omran, Nikunj Saunshi, Dara Bahri, Gaurav Mishra, Eric Chu, Toby Boyd, Brad Hekman, Aaron Parisi, Chaoyi Zhang, Kornraphop Kawintiranon, Tania Bedrsx-Weiiss Oliver Wang, Ya Xu, Ollie Purkiss, Uri Mendlovic, Ilai Deutel, Nam Nguyen, Adam Langley, Flip Korn, Lucia Rossazza, Alexandre Ramé, Sagar Waghmare, Helen Miller, Nathan Byrd, Ashrith Sheshan, Raia Hadsell Sangnie Bhardwaj, Pawel Janus, Tero Rissa, Dan Horgan, Sharon Silver, Ayzaan Wahid, Sergey Brin, Yves Raimond, Klemen Kloboves, Cindy Wang, Nitesh Bharadwaj Gundavarapu, Ilia Shumailov, Bo Wang, Mantas Pajarskas, Joe Heyward, Martin Nikoltchev, Maciej Kula, Hao Zhou, Zachary Garrett, Sushant Kafle, Sercan Arik, Ankita Goel, Mingyao Yang, Jiho Park, Koji Kojima, Parsa Mahmoudich, Koray Kavukcuoglu, Grace Chen, Doug Fritz, Anton Bulyenov, Sudeshna Roy, Dimitris Paparas, Hadar Shemtov, Bo-Juen Chen, Robin Strudel, David Reitter, Aurko Roy, Andrey Vlasov, Changwan Ryu, Chas Leichner, Haichuan Yang. Zelda Mariet, Denis Vnukov, Tim Sohn, Amy Stuart, Wei Liang. Minmin Chen, Praynaa Rawlani, Christy Koh, JD Co-Reyes, Guangda Lai, Praseem Banzal, Dimitrios Vytiniotis, Jieru Mei, and Mu Cai. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. CoRR, abs/2507.0622 2025.
[6] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023.
[7] Luigi Daniele and Suphavadeeprasit. Amplify-instruet. Synthetically generated diverse multi- turn conversations for efficient I1m training. arXiv preprint arXiv:(coming soon), 2023.
[8] Carles Domingo-Eninn Ennich, Michal Drozdzal, Brian Karrer, and Ricky T. Q. Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. In The Thirleenth International Conference on Learning Representations, 2025.
[9] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767 2023.
11

---

[10] Hanze Dong, Wei Xiong, Bo Pang. Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang. Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprini urXiv:2405.07865; 2024.
[11]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshomn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary. Dhruy Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Ject Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang. Jiawen Liu, Jie Wang. Jiccao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca. Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafell, Kevin Stone, and et al. The llama 3 herd of models. CoRR, abs/2407.2177 2024.
[12]
Maksim Dzabraev, Alexander Kunitsyn, and Andrei Ivaniuta. VIRM: vision-languag models act as reward models for image captioning. CoRR, abs/2404.01911, 2024.
[13] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21 21-27, 2024. OpenReview.net, 2024.
[14) Jiajun Fan, Shuaike Shen, Chaoran Cheng, Yuxin Chen, Chumeng Liang, and Ge Liu. Online reighe-weighted fine-tuning of flow matching with wasserstein regularization. In The Thirleenth International Conference on Learning Representations, 2025.
[15] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. DPOK: reinforcement learning for fine-tuning text-to-image diffusion models. CoRR, abs/2305. 16381, 2023.
[16] Daya Guo. Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprini arXiv:2501.11244 2025.
[17] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[18) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[19] Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker- Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, An- drew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben
12

---

Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll I.. Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, and Dane Sherburn. Gpt-4o system card. CoRR, abs/2410.21276. 2024.
[20] James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan. Tiago Ramalho, Agnieszka Graassa-BBrwwing Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. CoRR, abs/1612.00796, 2016.
[21] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023.
[22] Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. htttps:/httts:/// 2023.
[23] Yaron Lipman, Ricky T. Q. Chen, Heli Ben- Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[24] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 77763.. PMLR, 2021.
(26) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information P Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
[27] Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Deattenstien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David
13

---

Weinberger, Dimple Vijaykumar. Dominika Rogozinska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljaz, Jocelyn Becker, Joe Fernandez. Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju-yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjösund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, and Lilly McNealus. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.00 2024.
[28]
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[29] S. Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. MMAU: A massive multi-task audio understanding and reasoning benchmark. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
[30] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[31] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 0177777007777, 2017.
[32] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li,
Y. Wu, and Daya Guo. Deepscekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024.
[33] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross J. Anderson, and Yarin Gal. Al models collapse when trained on recursively generated data. Nat., (8822))7775555,,, 2024.
[34] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12 12, 2020, virtual, 2020.
[35] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Trans. Neural Networks, 95): 1998.
[36] Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. Trans. Mach. Learn. Res., 2024, 2024.
[37] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In IEEF/CVF Conference Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 9228-8238. IEEE, 2024.
[38] Enze Xie, Junsong Chen, Yuyang Zhao, Jincheng Yu, Ligeng Zhu, Yujun Lin, Zhekai Zhang, Muyang Li, Junyu Chen, Han Cai, Bingchen Liu, Daquan Zhou, and Song Han. SANA 1.5: Efficient scaling of training-time and inference-time compute in linear diffusion transformer. CoRR, abs/2501. 18427, 2025.
14

---

[39] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[40] Jin Xu, Zhifang Guo, Jinzheng Hc, Hangrui Hu, Ting Hc. Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report. CoRR, abs/2503.202155 2025.
[41] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang. Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang. Jianhong Tu, Jianwei Zhang, Jiamxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He. Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni. Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng. Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. CoRR, abs/2407.1061 10671, 2024.
[42] An Yang. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang. Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang Ren, Xinlong Yang. Yong Li, Zhiying Xu, and Zipeng Zhang. Qwen2.5-1m technical report. CoRR, abs/2501. 115383, 2025.
[43] Chenglin Yang. Celong Liu, Xueqing Deng, Dongwon Kim, Xing Mei, Xiaohui Shen, and Liang-Chieh Chen. 1.58-bit FLUX. CoRR, abs/2412.18655. 2024.
[44] Pinci Yang. Xin Wang. Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. AVQA: A dataset for audio-visual question answering on videos. In João Magalhães, Alberto Del Bimbo, Shin'ich Satoh, Nicu Sebe, Xavier Alameda-Pineda. Qin Jin, Vincent Oria, and Laura Toni, editors, MM 22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 14, 2022, pages 3480-3491. ACM, 2022.
[45] Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng. Zhiyuan Liu, and Maosong Sun. Advancing Ilm reasoning generalists with preference trees, 2024.
[46] Daniel M. Ziegler, Nisan Stiennon, StiffrnnnJeeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, abs/ Abs//199,,, 2019.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction clearly articulate the key contributions, specifi- cally: (1) introducing ADRPO as general framework for adjusting regularization based
on advantage estimates in Sec. 3 (2) demonstrating superior text-to-image alignment with ADRPO enabling 2B parameter model to outperform larger models and other RL methods (See Fig. [and Fig. 2. (3) showing emergent exploration behavior in LLMS (See Fig. 4), (4) validating effectiveness on multi-modal audio reasoning where ADRPO outperforms GRPO and enables a 7B model to surpass larger commercial systems (See Sec. [46 Tab. 2 and (5) establishing cross-domain applicability across flow matching models, text-only LLMs, and multi-modal reasoning models. These claims are fully supported by the experimental results in Sec. 4 and additional results in App. D
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We include a detailed limitations section App. A. that discusses com- putational overhead, scaling to larger models, and potential improvements to advantage estimation techniques. We acknowledge that our experiments were primarily conducted models of moderate scale (SD3-2B [13,, Qwen2-0.5B [1], Qwen3-0.6B [42], and Qwen2.5-0mmi- [40]) due to computational constraints and discuss how the approach might behave much larger foundation models.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
16

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper focuses on an empirical approach to reinforcement learning fine- tuning rather than providing formal theoretical results or proofs. We present algorithm
formulations (see Equs. 4 6 and and empirical validations of the proposed ADRPO
method through extensive experiments in Sec.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide comprehensive details on the experimental setup in Sec. 4.1 and App- B including model architectures, datasets, reward models, and evaluation metrics. Our algorithm pseudocode in App- A (Algorithm[] further enhances reproducibility by detailing the implementation of ADRPO for SD3 fine-tuning. We also discuss LLM/FM fine-tuning details in App. C Cnd App.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
17

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use publicly available models (SD3 13 Qwen2 41 Owen3 42].
Owen2.5-Omni [40] and datasets (DrawBench (28), RLHFlow O AVQA 44), MMAU
[29]) as noted in App. B Our implementation details and algorithm pseudocode in App.
and App. B provide sufficient information for reproduction, and we plan to release our codes soon after publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
Sublic/guidese http://uideee/eeebiieeeeeeeeeeee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
sliips.cc/public//eee hwtp:///giieeeeeee http://///iiieee/odeuuubiiiii ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify training and test details in Sec. 4.1 and App- B including prompt datasets (DrawBench for text-to-image, RLHFlow for ILMss, hyperparameters, and optimization approaches (LoRA adaptation for SD3, full parameter mun-ttning for Qwen).
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Tab. reports standard errors estimated over runs with different random seeds for all evaluation metrics. This is clearly stated in the table caption ("We report standard error estimated over runs of different random seeds") and the results consistently show ADRPO outperforming other methods beyond statistical error margins.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In App. B we specify that all experiments were conducted NVIDIA A6000 (48GB) GPUs, with LoRA [18] used for SD3 fine-tuning to reduce memory requirements. We also note the approximate time requirements: LLM experiments (Qwen2/3) completed within 12-24 hours per model configuration, SD3 fine-tuning required 2-3 days, and multi- modal audio reasoning fine-tuning (0wen2.5-0mmi-777 required 10 GPUs running for 3-4 days due to the larger model scale and multi-modal processing complexity.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
19

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurips:
Answer: [Yes]
Justification: Our research conforms to the NeurlPS Code of Ethics. We have ensured transparency in our methodology, cited all sources appropriately in our bibliography, and discussed both benefits and potential limitations of our approach in App. A App. A.1
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In App. A.2 we discuss both positive impacts (parameter efficiency leading to reduced computational resources and costs, making advanced generative capabilities more accessible) and potential implications of our work. We highlight that "our finding that a relatively small model (2B parameters) can outperform substantially larger models (4.8B and 12B parameters) when fine-tuned with ADRPO has important implications for personal access high-quality generative AI."
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
20

---

Answer: [NA]
Justification: Our paper doesn't 't explicitly release models or datasets that pose high risk for misuse. We focus on improving fine-tuning methodology for existing models (Sec. rather than releasing new assets that would require specific safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite the original sources for all models and datasets used, including SD3 13,. Qwen2 AM) Owen3 [42] Qwen2.5-0mni [40], DrawBench (28) AVQA [4)
MMAU 29 SA5A-1.5 38 FLUX. I-Dev 43). and RLHFlow [ as referenced throughout
our paper (see Sections
and App.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pwppppapppthaaooo thcode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pane se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: NA]
Justification:
Our paper doesn't introduce new datasets, code packages, model releases; it presents a new methodology (ADRPO) for fine-tuning existing models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
21

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research doesn't 't involve crowdsoureing or direct human subjects. We use existing public datasets and evaluation metrics rather than collecting new human preference data conducting human evaluations.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper doesn't 't involve research with human subjects that would require IRB approval equivalent.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: While we fine-tune LLMs (Qwen2, Qwen3), LLMSs are not used as components in our research methodology itself; they are the subject of study rather than tools used to develop the core method.
Guidelines:
22

---

The answer NA means that the core method development in this research does not involve LLMs as any important. original, non-standard components.
Please refer to our LLM policy (https://neurip: /ttp://pp//cc/oofffeeeee//////// for what should or should not be described.
23

---

A Discussion
In reinforcement learning fine-tuning of generative models, the exploration-exploitaiio trade-off represents a critical challenge: too much exploitation leads to reward hacking and diversity collapse, while excessive exploration prevents effective alignment. This dilemma is particularly pronounced in online RL fine-tuning where models continuously learn from their own generations. Existing methods rely on fixed regularization coefficients that treat all samples equally, regardless of their reward potential or uncertainty, creating an inherent tension between preserving model capabilities and optimizing for reward.
Key Insight 1: ADRPO solves the fundamental iegloration-explotttaie dilemma in generative model fine-tuning through a principled adaptive mechanism where regularization strength automatically scales with sample-specific advantage estimates.
In this paper, we introduced Adaptive Divergence Regularized Policy Optimization (ADRPO), novel approach that fundamentally reimagines how divergence regularization is applied during gen- erative model fine-tuning. Unlike existing approaches that employ fixed regularization coefficients, ADRPO dynamically modulates regularization strength based on sample-speciff advantage estimates, effectively turning the static trade-off parameter into adaptive function. Through comprehensive experimentation, we have demonstrated that this simple yet powerful modification successfully over- comes the inherent limitations of reuulregularization methods across both text-to-image generation and language model domains, establishing a new paradigm for reinforcement learning fine-tuning of generative models.
Key Finding 1: ADRPO establishes a dominant Pareto frontier in the reward-diversine trade- off, achieving state-of-tie of-the-art performance in both dimensions simultaneously where previous methods could only optimize one at the expense of the other (Fig.
Our experimental results reveal critical insights about the nature of reinforcement learning fine- tuning. The reward-diversity trade-off, long considered an unavoidable compromise in generative model alignment, can be effectively navigated through our adaptive regularization framework. As shown in Fig. 3 (left), DPO preserves moderate diversity but plateaus in reward optimization, while ORW-CFM-WS W2 improves alignment but suffers significant diversity collapse. In contrast, ADRPO achieves dominant performance in both dimensions simultaneously (Fig. B center and right panels), establishing a new state-of-the-art that was previously considered unattainable. This is achieved through our bidirectional adaptation mechanism, where regularization strength decreases for high- advantage samples enable aggressive optimization while increasing for low-advantage samples to maintain stability and diversity.
Key Finding 2: ADRPO enables remarkable parameter efficiency, allowing 2B parameter model to consistently outperform substantially larger models (4.8B and 12B), demonstrating that optimization strategy can be more consequential than model scale (Tab. I].
The parameter efficiency enabled by our approach challenges fundamental assumptions about scaling laws in generative AI. As demonstrated in Tab, our 2B parameter SD3 model fine-tuned with ADRPO consistently outperformed substantially larger models including FLUX. (12B) and SANA-L.5 (4.8B) across all evaluation metrics-from ClipScore to human preference metrics like ImwgeReward and PicScore. This finding suggests that optimization strategy can be more consequen- tial than parameter count for generative quality, with profound implications for both research focus and practical deployment. By enabling smaller models to match or exceed the capabilities of models 2-6x their size, ADRPO offers path to personal access to high-qualit generative AI while significantly reducing computational and environmental costs. The qualitative results in Fig. [ further support this finding, showing superior attribute binding and style transfer capabilities compared to larger models.
Key Finding 3: ADRPO exhibits emergent ability to intelligently navigate the exploration- exploitation landscape, actively increasing exploration to escape local optima- sophisticated capability that emerges naturally from our advantage- gugdded mechanism (Fig.
Perhaps the most surprising property of ADRPO is emergent ability to escape local optima through strategic exploration. As visualized in Fig. 4l our LLM experiments with Qwen3 revealed that
24

---

ADRPO implements a sophisticated optimization trajectory absent in fxxedregularization methods. While GRPO maintained high entropy throughout training but struggled to find high-reward regions (Fig. 4 bottom left, red points), ADRPO exhibited a remarkable three-phase pattern: first exploring lower entropy regions, then actively increasing entropy to escape local optima, before finally converg- ing to high-reward solution (Fig. 4 bottom left, blue trajectory). This behavior-reminiss sophisticated simulated annealing schedules-emerge
of organically from our advantage-guided adap- tive mechanism without explicit programming. resulting in final checkpoints with 5x higher reward than GRPO (Fig. E bottom center). This finding suggests that advantage-guin regularization may unlock entirely new regions of policy space previously inaccessible to exed-regularization methods.
A.1 Limitations
While ADRPO demonstrates significant improvements over existing approaches, several limitations should be acknowledged. Our experiments, though comprehensive, were primarily conducted on models of moderate scale (SD3 2B, Qwen2-0.5B, and Qwen3-0.6B) due to computational constraints. An important avenue for future work is extending these findings to much larger foundation models, where the exploration-exploitttoo dilemma becomes even more critical. Particularly, models with parameters in the hundreds of billions are known to be more susceptible training collapse during online RL fine-tuning, potentially making adaptive regularization even more crucial at scale.
The advantage-bers brase formulation introduces slight computational overhead compared to purely reward-weid weighted methods like DRW-CFM-W- [1] and RAFT 9.. Though we mitigate this through efficient batch-based normalization techniques similar to those used in GRPO [32), further optimiza- tion could reduce this overhead. Our current implementation of advantage estimation using batch statisties works well in practice but could be improved with more sophisticated value approximation methods, especially for complex reward landscapes with high variance.
A.2 Broader Impact
In general, ADRPO represents a significant advance in reinforcement learning fine-tuning methodol- ogy with potential impacts extending beyond the specific models tested. The ability to efficiently navigate the exploration explovation-exploittiinn trade-off through adaptive regularization addresses fundamen- tal challenge in the field, potentially influencing how the broader research community approaches model alignment.
Our finding that relatively small model (2B parameters) can outperform substantially larger models (4.8B and 12B parameters) when fine-tuned with ADRPO has important implications for personal access high-quality generative AI. This parameter efficiency could significantly reduce the compu- tational resources required for state-of-the-arr performance, making advanced generative capabilities more accessible to researchers and organizations with limited resources while reducing the environ- mental footprint of both training and deploying such systems.
The emergent ability to escape local optima demonstrated in LLM experiments suggests that advantage-guide guided adaptive regularization may unlock previously unattainable regions of policy space. This capability could inspire new approaches optimization in high-dimensional spaces where fixed regularization schemes tend to converge prematurely to a sub-optimal.
Our work also introduces unified framework that bridges continuous and discrete generative paradigms, offering a consistent solution to the apploration-cxploittaii dilemma across domains. This cross-domain applicability could foster greater knowledge transfer between previously disparate research communitics working on different types of generative models.
25

---

B Experimental Details
In this section, we provide comprehensive details of our experimental setup for both text-to-image alignment and language model fine-tuning tasks. To maintain clarity, we present these domains separately.
B.1 Flow Matching Model Fine-tuning Tasks
B.1.1 Baseline Methods
Base Model We adopt Stable Diffusion 3 (SD3) 13 as our base model 2B parameter architec- ture that combines Multimodal Diffusion Transformer (MMDiT) backbone with rectified flow training objective. SD3 represents the latest evolution of latent diffusion models, introducing a joint text-image Transformer that enables rich bidirectional attention between prompt tokens and image latents. Unlike earlier score-based approaches, SD3 is trained via conditional flow matching under rectified trajectory, where the model learns to predict direct velocity fields between noise and data, improving sample efficiency and semantic alignment. It leverages multiple frozen text encoders (CLIP and T5) and improved autoencoding for high-resolution image synthesis, while achieving state-of-the-art performance combination of high quality, controllability, and architectural flexibility makes SD3 a robust and representative base model for studying the effects of reinforcement learning fine-tuning, such as ADRPO.
prompt fidelity, compositional reasoning, and text rendering. This
Larger-Scale Flow Matching Models To comprehensively evaluate ADRPO against parameter scaling approaches, we selected two state-of-the-arr text-to-image diffusion models that leverage flow-matching training objectives and significantly larger parameter counts than our base model. These models serve as strong upper baselines in terms of both capacity and generation quality:
FLUX.I-Dev [43, with 12B parameters, represents the seg-pperformanc frontier of open- source flow-matching architectures. It employs rectified flow training objective based on Wasserstein-2 optimal transport, enabling more stable and efficient training compared to traditional score-matching methods. FLUX integrates multimodal diffusion transformer (MMDiT) with powerful prompt conditioning mechanisms, achieving pear-photorealistic output, superior compositional fidelity, and high stylistic diversity. It is widely regarded as one of the most capable open models in terms of prompt adherence, fine-grained semantic alignment, and artistic control.
2. SANA-1.5 38 with 4.8B parameters, serves strong intermediate- scale baseline. It introduces an efficient diffusion transformer architecture that combines linear attention mechanisms and highly compressed autoencoder (32x downsampling), enabling high- resolution generation at lower computational cost. SANA adopts a decoder-style language model for text conditioning and achieves state-of-the-art results the GenEval benchmark for prompt-image alignment. Despite its moderate size, SANA. 1.5 offers a competitive trade-off between generation quality, efficiency, and controllability
Both models exemplify the benefits of scaling parameter count and architectural sophistication within the flow-matching paradigm. By comparing against them, we isolate the advantages of ADRPO in improving alignment and control without relying solely on larger models. This allows to highlight ADRPO's efficiency and generalization capabilities, especially when applied to smaller models such as our 2B-parameter SD3 baseline.
For fine-tuning method comparisons, we included approaches representing the most representative spectrum of current techniques:
RL Baseline Methods
implements a reward-ranked fine-tune fine-tuning approach that selects high-quality outputs based reward scores, providing online RL baseline. This approach has demonstrated considerable effec- tiveness in improving generative models but lacks the adaptive divergence regularization mechanisms essential for preserving model capabilities during policy optimization.
RAFT 9
DPO 37 adapts the Direct Preference Optimization method to flow matching models, providing an established offline RL baseline. We apply diffusion-DPet to flow matching following methodologies
26

---

established in recent literature 8 This approach offers stable optimization and effective diversity preservation through its implicit regularization properties. though may be limited in its ability to explore the full policy space due to its offline nature. Given DPO's widespread adoption and demonstrated success across various fine-tuning tasks, it serves as our primary offline RI. fine-tuning baseline.
ORW-CFM-W2 [4] represents the current state-of-the-arr in online RL fine-tuning for flow matching models, employing fixed Wasserstein-2 regularization combined with reward weighting. As the first online RL fine-tuning method developed specifically for flow matching models, achieves leading performance in this domain through its W2 regularized online RL framework. This method provides crucial benchmark against which evaluate our ADRPO approach, as it represents the online SOTA method with fixed Wasserstein-2 regularization, allowing us to directly highlight the effectiveness of our proposed adaptive divergence regularization mechanism.
B.1.2 Reward Models and Evaluation
For text-to-image generation, we implemented a comprehensive reward system combining multiple complementary models to evaluate different aspects of generation quality:
1. Reward Model. We used CLIP Score 25 to compute cosine similarity between text prompts and generated images (ClipScore) as our reward model for all text-to-image align- ment task.
2. Quality Assessment. We employed aesthetic predictor to evaluate visual appeal from ImageReward [39]]
3. Human Preference Models. We incorporated ImageReward trained on direct human judgments, and PickScore [21], developed through large-scale mack-one-from-forr prefer- ence data, to align our generations with human aesthetic preferences. We complemented this with BLiP-based [12] evaluation to mitigate architecture-specfie biases.
4. Diversity Evaluation. For diversity evaluation, we developed ClipDiversity, which measures the average pairwise distance between CLIP embeddings of multiple generated images of current FM model.
B.1.3 Prompt Datasets
Our text-to-image experiments utilized a diverse collection of prompts selected to evaluate different capabilities. DrawBench (2 provided our primary test set, covering attribute binding, spatial relationships, counting accuracy, and text rendering. We extended this with artistic style prompts from RAFT [9] (c.g.. "Van Gogh style astronaut") and custom compositional prompts testing multi- object relationships (e.g., "A green apple and a black backpack").
B.2
LLM Fine-tuning Tasks
B.2.1
Baseline Methods
Base Models For our language model experiments, we employed Qwen2 [1 and Qwen3 42 models base architectures, representing recent advancements in autoregressive LLM models. These models demonstrate strong foundational capabilities and serve as robust pre-trained reference model for fine-tuning experiments. Specifically, Qwen2 and Qwen3 incorporate key architectural enhance- ments such as Grouped Query Attention (GQA), RoPE positional encoding, and long-context support (up 128K for Qwen3), which contribute to efficient inference and robust context handling. More- over, despite their relatively small parameter sizes (0.5B and 0.6B respectively), these models exhibit competitive performance across a range of reasoning, code generation, and language understanding benchmarks. Their strong pretraining on diverse multilingual and domain-spectin corpora including high- sugh-tuality instructional data, code, and math ensures excellent generalization. Qwen3 further introduces hybrid prompting paradigm that enables dynamic switching between direct answering and step-by-step reasoning, significantly enhancing the model's flexibility and interpretability during instruction-fol -olleioinnoolowiingg tasks. These strengths make Qwen2 and Qwen3 especially well-suited for fine-tuning via reinforcement learning from human feedback (RLHF), where high-quality priors and reasoning ability essential for aligning model behavior with human preferences.
27

---

with fixed KL regularization (3 0.04 (32) as it represents the current state-of-the-art online RL fine-tuning for L.Mss. GRPO improves upon earlier methods like PPO [31] through group-level advantage estimation and more efficient policy optimization. However, it crucially still relies on static regularization that treats all samples equally regardless of their quality or uncertainty. This limitation makes GRPO an ideal candidate for demonstrating the advantages of our adaptive regularization approach, as both methods share the same underlying optimization framework but differ specifically in their treatment of regularization (also can be served as our ablation studies).
RL Baseline Methods For RL fine-tuning comparison, we selected GRPO 32
B.2.2 Reward Model and Evaluation
For LLM fine-tuning, we used RM-Gemma-2B 27 9]. a reward model built upon the Gemma-2B language model and fine-tuned using a diverse collection of human preference datasets. RM-Gemma- 2B maps imput completions scalar reward values, which serve proxy signals for alignment with human preferences. The model is trained using pairwise comparison data spanning a wide range of tasks-including helpfulness, harmlessness, factuality, and cassoning-throuue style objective that encourages higher scores for preferred responses. This formulation enables the reward model to capture nuanced quality differences across candidate outputs. To support more stable and informed policy updates, we further incorporated entropy-based bbsed regularization to evaluate and balance the exploration-exploiaaiie ensures that the optimization process not only aligns outputs with human values but also maintains diversity and adaptability in model behavior.
Bradley-Terry
dynamics of the fine-tuned policies. This combined approach
B.2.3 Prompt Datasets
For the large language model fine-tuning, we have used the RLHFlow/test geerreattonnnnniinnnnnoo 2k dataset [0] containing 2,000 diverse prompts compiled from high-quality eestruction-folloiing datasets, and we randomly choose 10% as test prompts. This diverse prompt set allowed comprehensive evaluation across multiple dimensions, including factual accuracy, reasoning capabilities, and response quality. Specifically, the prompts were drawn from a combination of several representative and complementary
sources: Feerbbae 6), Capybara 77. UltraInteract [45]. and OpenOrea 22
UltraFeedback provides
single-turn instruction-responee pairs with rich feed- back annotations generated by GPT-4, including multi cillii-dimenional numerical scores (e.g.. helpfulness. correctness, conciseness) and textual critiques. These annotations support fine-grained evaluation and reward modeling.
Capybara contributes multi-turn dialogues generated through the Amplify-Instruct pipeline, which enriches single-turn seed prompts into deep, logically consistent conversations. emphasizes diverse topics, natural phrasing, and contextual reasoning, making it valuable for evaluating sustained dialogue coherence.
UltraInteract focuses on complex tasks involving step-by-step reasoning, such as math, coding, and logic problems. Each example includes multi-step trajectories with intermediate model outputs, environment feedback, and correctness signals, enabling assessment of models' planning and iterative refinement abilities.
OpenOrca offers a large-scale collection of instruction response pairs distilled from GPT-4 and GPT-3.5 using the FLAN dataset collection. Its responses often include chain-of-thougle style rationales, making a useful benchmark for evaluating models' reasoning depth and informativeness.
By combining prompts from these datasets, the test set enables comprehensive evaluation of a model's capabilities across wide range of real-world tasks and dialogue scenarios, from single-turn factual queries to multi-tuma multi-step reasoning challenges.
B.3
Multi-Modal Audio Reasoning Tasks
B.3.1
Baseline Methods
Base Model For our multi-modal reasoning experiments, we employed O0wen2.5-0mni- [40] (bf16 precision), slateoof-the-al multi-modal model capable of processing both audio and text inputs.
28

---

This model represents significant advancement in audio- understanding, incorporating dual-tower architecture that separately encodes audio and text modalities before fusing them through cross-attention mechanisms. Despite its 7B parameter size, Qwen2.5- demonstrates competitive performance on various audio understanding benchmarks through pre-training on diverse audio-text paired data including speech, music, and environmental sounds. The model's architecture enables fine-grained temporal reasoning and acoustic pattern recognition, making it well-suited for complex audio-visual question answering tasks requiring multi-step reasoning and contextual understanding.
Baseline Methods For RL fine-tuning comparison, we selected GRPO 32 with fixed KL reg- ularization (3 = 0.04) as our primary baseline, consistent with our L.M experiments. GRPO's group-level advantage estimation and efficient policy optimization make it strong baseline for multi- modal reasoning tasks. Additionally, we compared against commercial systems including Gemini 2.5 Pro [5] and GPT-4o Audio [9] to evaluate our method's competitiveness against substantially larger proprietary models. This comparison demonstrates whether adaptive regularization can enable smaller open-source models to match or exceed the capabilities of large-scale commercial systems.
B.3.2 Reward Model and Evaluation
For multi-modal audio reasoning fine-tuning, we employed dual reward mechanism combining accuracy rewards and format rewards, following the approach in DeepSeek-R1 (16). Accuracy rewards are based on answer correctness, computed by exact matching between the model's predicted answer and the ground-truth answer from the dataset. Format rewards ensure that responses follow the expected structure with explicit thinking traces before final answers, encouraging the model to generate effective reasoning processes. This combined reward structure promotes both correctness and interpretable reasoning patterns.
We evaluated performance on the MMAU benchmark 29 a comprehensive multi-task audio un- derstanding benchmark testing reasoning across three primary categories: sound understanding (environmental sounds, acoustic events), music understanding (genre, instruments, tempo), and speech understanding (speaker characteristics, linguistic content, prosody). The benchmark's s 1k test- mini split provides diverse evaluation across these modalities, enabling assessment of generalization and reasoning capabilities.
B.3.3 Dataset
which provides audio- visual question answering tasks requiring temporal reasoning and contextual understanding. The dataset contains diverse question types spanning spatial localization, temporal ordering, counting, and causal reasoning about audio-visur events. Each example consists of an audio clip (we extract them from the video), natural language question, and multiple-choic answers, requiring models to integrate acoustic information with semantic understanding to arrive at correct conclusions. The training split contains various difficulty levels, from simple sound source identification to complex multi-step reasoning about temporal sequences and causal relationships. This diversity makes AVQA particularly suitable for evaluating whether adaptive regularization can enhance multi-modal reasoning capabilities compared to fixed regularization approaches. By fine-tuning AVQA and evaluating on the independent MMAU benchmark, we assess both in-domain reasoning improvement and cross-domain generalization to unseen audio understanding tasks.
For
audio reasoning fine-tuning, we used the AVQA dataset 44
B.4 Computation Resources
All experiments were conducted NVIDIA A6000 (48GB) GPUs. For SD3 fine-tuning tasks 13 we employed parameter-efficient LoRA [8] adaptation to reduce memory requirements and training time. while still achieving excellent results. For the relatively smaller Qwen2-0.5B [4] and Qwen3-0.6k [2 language models, we performed direct full- hall-paraeeter fine- mee-tunig without LoRA. For Owen2.5-0mmi-771 [40), we utilized at least 10 A6000 GPUs to accommodate the model's multi-modal architecture and larger parameter size.
Our experimental setup utilized publicly available open-source reward/evaluation models and datasets across all domains, ensuring reprodueibility and alignment with established benchmarks. The compu- tation requirements varied significantly between tasks: LLM fine-tuning experiments (Qwen2/3) were
29

---

relatively efficient, typically completing within 12-24 hours per model configuration; SD3 fine-tuning tasks were more computationally intensive, requiring approximately 2-3 days; and multi-modal audio reasoning fine-tuning (Qwen2.5-0mni-7B) required 3-4 days due to the larger model scale and multi-modal processing complexity.
30

---

C Algorithm Pseudocode
We first detail our algorithm pseudocode in Algorithm for fine-tuning flow matching models (we use linear interpolation path as an example). Noting that, we can sample from current learned velocity field (2 t, c) via solving: 21 2a + So vo c)dt, wherein Zo p(xo) and p(zo) is a standard gaussian distribution [23 13 As for our method for fine-tuning LLM models, we can simply add an extra adtantage-weighted KI. divergence into the original GRPO training loss Equ. (7) therefore it is easy to be implemented.
Algorithm 1 Adaptive Divergence Regularized Policy Optimization (ADRPO) for SD3 Fine-tuning
Require:
Pre-trained flow matching model Trer (SD3), baseline regularization coefficient So, advan- tage clipping range [Amin: Amex]. learning rate 77
1: Initialize fine-tuned policy nh with pre-trained parameters (or LoRA adaptation)
2: for training iteration n =1,2.... do
3: Sample a batch of text prompts 1 p(c)
4: Sample target states {>i ~ Tg - (z|e) from current policy Online sampling strategy
5:
for each prompt and its generated image xi do
6:
Compute reward R( () using CLIP Score
7:
Sample intermediate time step i; ~ (0. Compute intermediate state x
8:
tx)=g + tta
Straight- interpolation
9:
Compute target velocity ui 2 - 2 end for Compute baseline value V(G) = 1 Σ R(x ci) for each context
11:
12:
Compute advantage A)) R( V(e))
Apply advantage clipping: (1) = clip) cllp( (z); Amiu; Amax)
Compute adaptive regularization coefficient Bror :0 Actipned (x)
15:
Update model parameters using the ADRPO loss LADRPO-PM from Equation (6:
16:
- 0 - CADROO-MMMM
17:
end for
18: return Fine-tuned policy m
31

---

D Additional Experimental Results
D.1 Flow Matching Model Fine-tuning Tasks
D.1.1
Additional Qualitative Results
A stack of cubes, A sitting on red cube. middle, sitting on cube. The cube on the bottom.
top,
storefront with "Diffusion' written
zebra the right fire hydrant,
black colored banona
SDB (28)
FLUX. 1
Diffusion
Dev (128)
SANA-1.5
(4.88)
S03+ ADRPO (28, (urs)
Figure 5: Additional Qualitative Comparison with Large FM Generative Models. Our ADRPO demonstrates superior performance Attribute Binding, Coloring, Counting, Text Rendering and Position.
32

---

storefront with 'Diffusion' written on it.
green apple black backpack
banana on left of apple.
yellow book and red vase.
SD3
Difusion
SD3+ DPO
SD3+
CFM-W2
OF-UON
SIP
SD3+ RAFT
SD3+ ADRPO (Ours)
Figure 6: Additional Qualitative Comparison with Other RL Fine-tuning Methods. Our ADRPO demonstrates superior performance in Text Rendering, Attribute Binding, Coloring, Counting and Position.
33

---

D.1.2 Learning Curves
Reward vs Diversity Trade-off
Reward Training Curve
Diversity Training Curve
[mnal
32.5
32.0
31.5
Reward
01.0
4.75
30.5
30.0
235
29.5
4.00
60
60
Traning
Thahingg Progress
ADRPO (Our)
Pre-trainee
Reward W2 Divergence Trade-off
Reward Training Curve
W2 Divergence Training Curve
Corerrprn sold
52.5
3200
32.0
31.
0.015
Raward
AAA
21.0
31.0
0.010
30.0
30.0
0.008
29.5
0.010
0.000
29.0
0.000
Divergance
Traniin Progresu
Trainin Progress (%)
Method
AUREPO (Duns)
DPG
CHRW-O M-A2
Predbainec 5135
Figure 7: Learning Curves of Fine-tuning SD3. Left: Complete policy optimization trajectories across three different methods throughout training using a same seed (for fairness). Transparency indicates progression from early (transparent) stages through convergent (solid) final (star) checkpoints, with each point representing a learned policy from different iterations. Center and right: Learning curves of RL agents.
34

---

D.2 LLM Fine-tuning Tasks
D.2.1 Learning Curves
Exploration vs Exploitation Trade-off
Exploitation (Reward) Training Curve
Exploration (Entropy) Training Curve
M
1.75
Mu
5.0
Expletaaion (Roward)
1.90
1.50
Exploaation (ggcaatton[/2eeeee (2Eeneet)
4.6
4.0
1.00
1.00
M
0.78
0.75
3.6
3.0
0.25
0:5
2.5
35
5.0
40
Exploration (Entropy)
Thaiiing Proggean (1)
Teatning Phogresx, (44)
Method
AORPO
GRPO
Prodiained Owanz
Exploration vs Exploitation Trade-off
Exploitation (Reward) Training
Exploration (Entropy) Training
Exploitation (Roward)
2.0
2.0
1.5
Expetatioo
1.5
1.0
1.0
0.
100
100
Exploration (Entropy)
Progrwes
Tranning
Method
Pre-traine Qward
Figure 8: Learning Curves of LLM Fine-tuning Experiments (100 iterations, early stop).
35

---

D.2.2
Reward and KL Divergence Trade-off
Reward vs KL Trade-off
Reward Training Curve
Divergence Training Curve
2.0
2.0
M
1.75
1.75
Mu
Reward
1.25
25
0.75
0.75
0.50
0.30
100
KL Divergence
Trasing Progress
Progreon (14)
Method
Reward KL Trade-off
Reward Training Curve
KL Divergence Training Curve
2.00
1.75
1,50
Reward
0.75
0.60
0.5
0.5
0.2
0.00
0.0
0.5
2.0
20
40
100
20
100
KL Divergence
Progrees
Touning Progress (15)
Method
ADRPO
GRPO
Pro Pre-tralned Owend
Figure 9: Reward Divergence Trade-off of LLM Fine-tuning Experiments (100 iterations, no early stop).
36

---

D.3
Multi-Modal Audio Reasoning Tasks
To further demonstrate ADRPO's versatility beyond text-to-imee -i--ge generation and text-only LLMs, we conducted additional experiments fine-tuning the multi-modal reasoning model Qwen2.5-0mni-77 40 on audio understanding tasks. This evaluation showcases ADRPO's effectiveness a completely different domain requiring complex multi-modal reasoning capabilities.
D.3.1 Experimental Setup
We fine-tuned Qwen2.5-0mmi-77 (bfl6 precision) on the AVQA dataset which provides audio- visual question answering tasks requiring temporal reasoning and contextual understanding, The model was optimized using verifiable rewards based on answer correctness and format rewards for proper response structure. We evaluated performance on the MMAU benchmark (1k test-mini split) 29 a comprehensive multi-task audio understanding benchmark that tests reasoning across sound, music, and speech understanding.
D.3.2
Quantitative Results
Tab. presents our results on the MMAU benchmark (Ik test-mini split), comparing ADRPO against the base model, baseline RL method (GRPO), and strong commercial models. Our method achieves substantial improvements across all audio reasoning categories, with particularly strong gains in sound understanding (+9.61 % over base model) and speech understanding (+6.91% % over base model).
Table 4:
Performance comparison on MMAU benchmark for audio reasoning tasks (1k test oist-mini split) 29 ADRPO significantly outperforms both baseline RL methods and larger commercial models across all categories.
Method
Sound (%)
Music (%)
Speech (%)
Total Accuracy (%)
Qwen2.5 5-mn2.5-mmni--- (base)
72.37
64.37
69.07
68.6
GRPO (baseline RL)
77.18
70.66
74.77
74.2
Gemini 2.5 Pro
75.08
68.26
71.47
71.6
GPT-4o Audio
64.56
56.29
66.67
62.5
ADRPO (Ours)
81.98
70.06
75.98
76.0
Notably, our ADRPO fine-tuned 7B model outperforms substantially larger commercial models in- cluding Gemini 2.5 Pro and GPT-4o Audio, demonstrating the effectiveness of adaptive regularization across different model scales and architectures. The consistent improvements over GRPO (+1.8% total accuracy) validate that our advantage- based adaptive mechanism provides benefits beyond text generation domains.
D.3.3 Qualitative Analysis
Beyond quantitative metrics, we analyzed the reasoning processes of ADRPO and GRPO to under- stand how adaptive regularization improves multi-modas understanding. Tab. presents representative examples where ADRPO demonstrates superior step-by-step reasoning compared to fixed regulariza- tion methods.
These examples reveal that ADRPO enables more accurate temporal sequence understanding, better source identification through attention acoustic characteristics, and superior contextual reasoning that avoids conflating multiple environmental cues. The fixed regularization GRPO leads to premature convergence incorrect answers, while ADRPO's adaptive mechanism allows the model to better explore the reasoning space and arrive at correct conclusions.
D.3.4 Ablation Advantage Clipping
To understand the robustness of ADRPO hyperparameter choices in this new domain, conducted ablation studies on different advantage clipping ranges. Tab, shows results with different clipping configurations.
The results demonstrate that ADRPO achieves consistent improvements over GRPO across all clipping configurations, with performance variations of less than 0.4% between different settings.
37

---

Table 5: Qualitative comparison of reasoning traces between ADRPO and GRPO on challenging audio reasoning tasks. ADRPO demonstrates more accurate temporal analysis and contextual understanding.
Question
ADRPO Response
GRPO Response
Temporal
Se-
The sequence of sounds includes footsteps, dog barking, a camera shutter clicking, and tapping glass. The last sound is a distinct 'tpppin glass', which is clear and unique sound compared to the others.
The last sound in the sequence is a camera shutter click, which distinct sound from the other options provided.
quence: What was the last sound in the sequence?
Choices: footsteps, barking, cam- era gra_shutter__ee sllcktterrclccknng, tapping_on. ong_sss
Answer: era shutter ceciiter_rliiigggt x
cam-
Answer: tapping on glagglsss /
Source Identifi- cation: Based on the given audio,
The audio suggests continuous, steady flow of water, which is char- acteristic of a bathtub. The other options (river, fountain, rain) typi- cally have more varied or intermit- tent sounds.
The sound is consistent with a steady flow of water, likely from a man-made source like a bathtub or fountain rather than natural sources like river or rain. Answer: Fountain X
identify the source of the flowing water.
Choices: Bathtub, River, Fountain,
Answer: Bathtub
Rain
Contextual Rea- soning: Consider- ing the information the audio, what type of weather condition can be inferred?
The presence of wind noise sug- gests a windy condition. The ab- sounds indi-
The presence of wind noise and the sound of rain hitting a surface sug- gest windy and rainy weather con- dition. Answer: Rainy
sence of rain cates it's not rainy or snowy. Answer: Windy /
Choices: Windy, Calm, Rainy, Snowy
Table 6: Ablation study advantage clipping ranges for audio reasoning tasks. All ADRPO configurations substantially outperform GRPO baseline, with stable performance across different clipping settings.
Clipping Range
Amin
Amux
Sound
Music (%)
Speech (%)
Total (%)
x Bu (recommended)
-0.04
0.04
81.98
70.06
75.98
76.0
2 Ag
-0.08
0.08
84.08
69.46
73.57
75.7
0.5 x Ao
-0.02
0.02
82.58
71.26
74.47
76.1
GRPO (baseline)
77.18
70.66
74.77
74.2
This stability confirms that the fundamental advantage lies in having adaptive regularization versus fixed regularization, rather than in precise hyperparameter tuming. The 2 x ôy setting achieves the highest sound understanding accuracy (84.08%) but shows reduced performance tasks, suggesting that excessive negative regularization can lead to overly aggressive exploitation that hurts generalization. Our recommended x 3o setting provides the best balance across all modalities.
music and speech
These multi-modal audio reasoning experiments provide several key insights: (1) ADRPO's ef- fectiveness extends beyond visual and text generation to complex multi-modal reasoning tasks; (2) the advantage-based adaptive mechanism enables superior step-by-step reasoning and reduces failure modes compared to fixed regularization; (3) ADRPO demonstrates robust performance across different hyperparameter settings, making practical for new domains; and (4) our method en- ables smaller models (7B) to outperform much larger commercial models through more effective balance.
38


Reviews:
[{"review_id": "aUkTy2TJB9", "reviewer": "Reviewer_Kr8c", "summary": "The paper introduces Adaptive Divergence Regularized Policy Optimization (ADRPO), a method for reinforcement learning fine-tuning of generative models with adaptive regularization, allowing for balanced exploitation-exploration tradeoffs, learning from useful examples while not being destructive to generative model capabilities.\n\nThe method replaces traditional fixed-weight W2 or KL-divergence regularization with one weighted dynamically with an Advantage estimate for the current sample. The technique is applied to both RL fine-tuning of flow matching, and RL fine-tuning of LLMs, beating non-adaptive baselines on both, with both quantitative and qualitative metrics.\n\nThe exploration-exploitation tradeoff is also quantified with measures of sample diversity and model divergence, demonstrating that the ADRPO technique leads to an optimal combination of both diversity, low model divergence and final reward.", "questions": "* Regarding small improvements: Did you produce a breakdown of whether it is possible for the baselines to achieve comparable results with more compute expenditure? And if so, with how much more expenditure?\n* Could you provide more details on how does the procedure behave when the regularization coefficient is allowed to be negative vs when it is clipped?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The proposed technique is very simple, but reliably improves the performance of RL finetuning techniques on the exploration-exploitation tradeoff frontier.\n* The approach is tested on both continuous generative modeling, for flow matching based image generation, and discrete generative modeling, in LLM finetuning, showcasing that the underlying insight is not model or modality dependent, but is generally applicable to generative models.\n\nWeaknesses:\n* Quantitative and qualitative comparisons do not straightforwardly appear as a fundamental improvement over the state of the art: benchmark improvements are relatively small, and image generation quality does not seem to conclusively be much beyond other methods.", "was_revised": true, "final_justification": "The authors addressed my concerns in the rebuttal, I thus raised my score to a 5.", "rebuttal": "Thank you for the thoughtful and constructive review. Your feedback has been invaluable in strengthening our paper's clarity and impact. We greatly appreciate the recognition of our method's simplicity and reliability in improving exploration-exploitation tradeoffs, as well as its general applicability across both continuous and discrete generative modeling tasks. We appreciate the recognition of our approach's ability to achieve balanced exploitation-exploration tradeoffs while preserving generative model capabilities, and that our method consistently outperforms non-adaptive baselines across both flow matching and LLM fine-tuning with superior quantitative and qualitative metrics. Below, we address each of your concerns systematically and will incorporate these clarifications and analysis into our final revision.\n\n> **Q1: Quantitative and qualitative comparisons do not straightforwardly appear as a fundamental improvement over the state of the art: benchmark improvements are relatively small, and image generation quality does not seem to conclusively be much beyond other methods.**\n\n**A1**: Thank you for this insightful comment. We would like to clarify several key aspects of our significant improvements:\n\n**Quantitative Improvement**: As shown in Table 1, when using the same base model (SD3), our benchmark performance on preference scores (ClipScore, BLIPScore, ImageReward, PicScore), Image Quality and Diversity consistently surpasses all other RL post-training methods. More remarkably, compared to much larger models like FLUX.1-Dev (12B) and SANA-1.5 (4.8B), our significantly smaller 2B model outperforms them across all evaluation metrics. These improvements are substantial because we achieve gains across all dimensions simultaneously rather than sacrificing one metric for another, avoiding the trade-offs that plague existing methods.\n\n**Qualitative Improvement**: The visual evidence in Figures 1 and 5 demonstrates clear semantic alignment and generation quality improvement. In Figure 1, for complex compositional tasks like \"A green apple and a black backpack,\" FLUX.1-Dev (12B) incorrectly generates an apple larger than the backpack, while the SD3 base model misunderstands the quantity, generating two apples instead of one. SANA-1.5 (4.8B) produces common sense errors like generating an astronaut with three arms for \"Van Gogh style astronaut.\" Particularly evident in Figure 5, the spatial relationship task \"A zebra to the right of a fire hydrant\" reveals systematic failures across all baseline models: FLUX.1-Dev (12B) completely fails to generate the zebra, SANA-1.5 (4.8B) misplaces the spatial relationship with the zebra (6 legs) positioned incorrectly, while the SD3 base model generates the zebra and fire hydrant but fails to establish the correct positional relationship. Our method correctly handles these challenging cases involving attribute binding, spatial relationships, and compositional reasoning.\n\n**RL Method Comparison**: In Figures 2 and 6, baseline RL methods consistently demonstrate failure patterns. DPO maintains diversity but demonstrates poor semantic alignment - generating zebras with unnatural images (extra legs), failing text rendering (\"Diffusion\" storefronts), and incorrect object quantities (\"A green apple and a black backpack\" produces multiple apples in Figure 6). ORW-CFM-W2 exhibits similar failures with malformed zebras, text rendering issues, and compositional errors (e.g., an apple as large as the backpack in Figure 6). Our ADRPO method successfully avoids these fundamental errors across all tasks, demonstrating superior control over naturalness, text rendering, object counting, positioning, and color attribution while maintaining generation diversity.\n\n**Additional Evidence of Substantial Improvements in Multi-Modal Reasoning Tasks**: To further validate our method's effectiveness, we conducted additional experiments fine-tuning the multi-modal reasoning model Qwen2.5-Omni-7B [1] (bf16 precision) on the AVQA dataset [2] with verifiable rewards and format rewards, evaluated on the MMAU benchmark [3]. On the MMAU benchmark [3], ADRPO significantly outperforms both baseline RL methods and strong commercial models:\n\n| Method | Sound (%) | Music (%) | Speech (%) | Total Accuracy (%) |\n|:--------:|:---------:|:---------:|:----------:|:------------------:|\n| Qwen2.5-Omni-7B (base model) | 72.37 | 64.37 | 69.07 | 68.6 |\n| **ADRPO (Ours)** | **81.98** | **70.06** | **75.98** | **76.0** |\n| GRPO (baseline RL) | 77.18 | 70.66 | 74.77 | 74.2 |\n| Gemini 2.5 Pro | 75.08 | 68.26 | 71.47 | 71.6 |\n| GPT-4o Audio | 64.56 | 56.29 | 66.67 | 62.5 |\n\nBeyond quantitative improvements, the qualitative analysis reveals superior reasoning capabilities. For instance, in complex audio reasoning tasks (more results will be included in our final revision):\n\n**Sample ID:** ff7bff97-342e-4285-bbb9-15841364b072\n- **Question**: \"Based on the given audio, identify the source of the flowing water. Choices: Bathtub, River, Fountain, Rain\"\n- **ADRPO**: \"The audio suggests a continuous, steady flow of water, which is characteristic of a bathtub. The other options (river, fountain, rain) typically have more varied or intermittent sounds.\" → **Bathtub** ✓\n- **GRPO**: \"The sound is consistent with a steady flow of water, likely from a man-made source like a bathtub or fountain rather than natural sources like a river or rain.\" → **Fountain** ✗\n\n\n**Sample ID:** b7701ab1-c37e-49f2-8ad9-7177fe0465e9\n- **Question**: \"What was the last sound in the sequence? Choices: footsteps, dog_barking, camera_shutter_clicking, tapping_on_glass\"\n- **ADRPO**: \"The sequence of sounds includes footsteps, a dog barking, a camera shutter clicking, and a tapping on glass. The last sound is a distinct 'tapping on glass', which is a clear and unique sound compared to the others.\" → **tapping_on_glass** ✓\n- **GRPO**: \"The last sound in the sequence is a camera shutter click, which is a distinct sound from the other options provided.\" → **camera_shutter_clicking** ✗\n\n\nThis demonstrates that our adaptive regularization mechanism enables superior step-by-step reasoning and reduces failure modes compared to fixed regularization RL methods across diverse modalities, confirming the broad applicability and effectiveness of our approach beyond visual generation tasks.\n    \n\n> **Q2: Regarding small improvements: Did you produce a breakdown of whether it is possible for the baselines to achieve comparable results with more compute expenditure? And if so, with how much more expenditure?**\n\n**A2:** Thank you for this important question. As demonstrated in our learning curves in Figures 7, 8, and 9, the improvements from **our ADRPO are not achievable through additional compute** for baseline methods because our method fundamentally changes the learning dynamics, exploration-exploitation trade-off and final convergence points.\n\nIn Figure 7, both DPO and ORW-CFM-W2 exhibit plateau behavior in their reward curves during training, indicating they have converged to stable but suboptimal solutions. Even with further increases in training time, these methods cannot reach ADRPO's performance levels due to their inherently limited exploration-exploitation trade-offs. **The final convergence points clearly demonstrate that our method achieves a superior reward-diversity Pareto frontier compared to baseline methods.**\n\nMore critically, in Figures 8 and 9 for LLM fine-tuning, baseline methods like GRPO actually suffer from model training collapse during extended training, where performance degrades rather than improves with longer training time. This demonstrates that additional compute is not only ineffective but potentially harmful for fixed regularization methods. In contrast, ADRPO's adaptive regularization mechanism enables it to discover better regions in the solution space that fixed methods cannot reach regardless of training duration (See Figure 4), fundamentally achieving better exploration-exploitation trade-off rather than simply requiring more computational resources.\n \n\n> **Q3: Could you provide more details on how does the procedure behave when the regularization coefficient is allowed to be negative vs when it is clipped?**\n\n**A3**: Thank you for this insightful suggestion. Our main contribution is introducing advantage-based adaptive regularization, which has demonstrated substantial improvements over fixed regularization across various tasks in our paper.\n\nWe use advantage clipping to ensure $β_{tot} = β_0 - A$ remains positive and stable. To address your concern about clipping behavior, we provide ablation studies on fine-tuning Qwen2.5-Omni-7B with different clipping ranges:\n\n| Clipping Range | Amin | Amax | Sound (%) | Music (%) | Speech (%) | Total Accuracy (%) |\n|:----------------:|:------:|:------:|:-----------:|:-----------:|:------------:|:-------------------:|\n| 1×β0 (recommended) | -0.04 | 0.04 | 81.98 | 70.06 | 75.98 | **76.0** |\n| 2×β0 | -0.08 | 0.08 | 84.08 | 69.46 | 73.57 | 75.7 |\n| 0.5×β0 | -0.02 | 0.02 | 82.58 | 71.26 | 74.47 | 76.1 |\n| GRPO (baseline) | - | - | 77.18 | 70.66 | 74.77 | 74.2 |\n\n**When allowing negative regularization** (2×β0 setting), the coefficient can become negative ($β_{tot} < 0$), reversing the regularization effect and encouraging greedy exploitation. While this achieves higher performance on specific tasks (84.08% on sound), it reduces generalization across other modalities (73.57% vs. 75.98% on speech). This demonstrates that excessive negative regularization leads to aggressive optimization at the expense of generalization. We recommend maintaining positive coefficients through appropriate clipping (1×β0) for better balance and more robust performance.\n\n[1] Xu, Jin, et al. \"Qwen2.5-omni technical report.\" 2025.\n\n[2] Yang, Pinci, et al. \"Avqa: A dataset for audio-visual question answering on videos.\" 2022. \n\n[3] Sakshi, S., et al. \"Mmau: A massive multi-task audio understanding and reasoning benchmark.\" 2024."}, {"review_id": "bE0qkGHeBV", "reviewer": "Reviewer_4P4K", "summary": "The paper suggest to use adaptive regularization scheme for policy optimization enabling to balance between exploration and exploitation. The scheme called ADRPO is based on the advantage estimation - reducing regularization for high-value samples while applying stronger regularization to poor samples. The method is applied for online fine-tuning of LLMs (to extend DeepSeekMath’s GRPO idea), text to image alignment etc. The authors also suggest an advantage-weighted flow matching objective strengthening high-quality generations and pushing the model\naway from poor generations. Numerical experiments (LLM fine-tuning and text to image generation) support suggested method.", "questions": "- I would suggest to add more explanation of advantage function estimation in particular for the section 3.4. ADRPO for Fine-tuning LLMs. At least summarise main ideas from the paper [29]. \n- I would also suggest to add more experiments replacing Qwen by other comparable models. \n- Is it possible to overcome the problem of choosing baseline parameter $\\beta_0$?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper is well written in easy to understand. It illustrates how different components work together. \n - Interesting idea of adaptive regularisation based on the advantage estimation. ADRPO automatically increases regularization for low-advantage (less promising) samples and decreases it for high-advantage (promising) samples, providing fine-grained control over the policy updates. \n- ADRPO generalizes to both flow matching models (using Wasserstein-2 divergence) and LLMs (using KL divergence)\n - The experimental results look promising. \n\nWeaknesses\n\n- Additional computational costs due to advantage estimation. The paper would benefit from a more quantitative analysis of this overhead and practical guidance on how it scales with model size.\n- While the empirical results are strong, the paper would be improved by discussing theoretical properties of the proposed adaptation.", "was_revised": true, "final_justification": "I thank the authors for their answers. I keep my score and stay positive.", "rebuttal": "We sincerely thank you for your comprehensive and insightful review. Your feedback significantly enhances our work's presentation and clarity. We are grateful for your recognition of our adaptive regularization concept and its clear exposition, as well as the recognition of ADRPO's generalizability across different model architectures and the promising experimental outcomes. Your suggestions for   computational overhead and advantage estimation discussion are particularly valuable. We address your specific questions and concerns below and will incorporate these clarifications and analysis into our final revision.\n\n> **Q1: Additional computational costs due to advantage estimation. The paper would benefit from a more quantitative analysis of this overhead and practical guidance on how it scales with model size.**\n\n**A1:** Thank you for this insightful feedback. Our advantage estimation approach follows the group relative advantage estimation method used in GRPO, which does not require introducing additional critic models or value networks. For methods already using advantage estimation (like LLM fine-tuning with GRPO), ADRPO does not introduce too much additional computational overheads (i.e., plug and play). For tasks requiring new advantage estimation (like SD3 fine-tuning), we do not introduce substantial computational burden because we avoid additional critic networks. The main computational bottlenecks in these tasks are data rollout and reward model inferences, which exist across RL baseline methods, so our approach does not add significant extra computational cost. It is worth noting that group relative  advantage estimation has been widely used for being computation-efficient [1-2] and having excellent scalability properties [2].  \n\n> **Q2: While the empirical results are strong, the paper would be improved by discussing theoretical properties of the proposed adaptation.**\n\n**A2:** Thank you for this insightful comments. The primary goal of our work is to propose a practically effective solution to improve existing RL fine-tuning methods by addressing key issues including exploration-exploitation trade-offs, reward hacking, and model collapse. Our extensive experiments demonstrate that the introduction of advantage-based dynamic regularization is indeed beneficial across diverse domains including LLMs (Figure 4), flow matching generative models (Table 1, Figures 1-3), and multi-modal reasoning models (audio reasoning tasks, See A4), validating its robustness and general applicability. \n\nOur adaptive mechanism $\\mathcal{L}_{RL} + (\\beta_0 - A) \\cdot \\mathcal{L}_D$ as formulated in Equation (4) provides intuitive understanding for exploration-exploitation balance: positive advantages (good samples) reduce regularization to encourage exploitation, while negative advantages (poor samples) increase regularization to maintain stability. The advantage clipping ensures bounded adaptation, preventing instability while maintaining meaningful regularization adjustment. \n    \nWe agree that more theoretical analysis is an area where more exciting and novel work can be made, and we would be interested in exploring the theoretical properties of dynamic regularization, particularly convergence guarantees and optimality conditions, in future work following your suggestion.\n\n> **Q3: I would suggest to add more explanation of advantage function estimation in particular for the section 3.4. ADRPO for Fine-tuning LLMs. At least summarise main ideas from the paper [29].**\n\n**A3**: Thank you for the constructive suggestion. We would like to clarify that paper [29] discusses Generalized Advantage Estimation (GAE), which normally relies on a critic model to estimate value functions. In contrast, in Section 3.4, we adopt the group-relative advantage estimation method from GRPO, which is fundamentally different from GAE. Specifically, GRPO does not require a critic model and instead computes advantages directly from outcome rewards by comparing samples within a batch or group. This approach is simpler and more stable, making it particularly well-suited for fine-tuning LLMs where training a separate critic can be challenging and computationally inefficient. We included the description of advantage estimation for the flow-matching version in Section 3.3, which follows similar principles to the LLM version.\n\nFollowing your suggestion, we will include more detailed descriptions of our advantage estimation in the final revision to enhance the clarity of our paper.\n\n> **Q4: I would also suggest to add more experiments replacing Qwen by other comparable models.**\n\n**A4**: We appreciate this insightful suggestion. In addition to Qwen models, **we have also provided extensive experiments on SD3 models for text-to-image generation** (Table 1, Figures 1-3) and demonstrated ADRPO's effectiveness across both architectures. These two model architectures are fundamentally different - SD3 uses flow matching for continuous generation (e.g., image) while Qwen uses autoregressive generation for discrete sequences (e.g., text) - yet both achieve significant improvements over baseline methods, sufficiently demonstrating the universal applicability of our approach. Besides, we chose Qwen models because they are widely used in RL post-training and represent state-of-the-art performance across various tasks [2-4], making them ideal choices for validating our method's effectiveness.\n\nTo further address the concern, we conducted additional experiments fine-tuning the multi-modal reasoning model Qwen2.5-Omni-7B [5] (bf16 precision) on the AVQA dataset [6] with verifiable rewards and format rewards, evaluated on the MMAU benchmark [7]. Our results demonstrate that ADRPO significantly outperforms both baseline RL methods and strong commercial models:\n\n| Method | Sound (%) | Music (%) | Speech (%) | Total Accuracy (%) |\n|:--------:|:---------:|:---------:|:----------:|:------------------:|\n| Qwen2.5-Omni-7B (base model) | 72.37 | 64.37 | 69.07 | 68.6 |\n| **ADRPO (Ours)** | **81.98** | **70.06** | **75.98** | **76.0** |\n| GRPO (baseline RL) | 77.18 | 70.66 | 74.77 | 74.2 |\n| Gemini 2.5 Pro | 75.08 | 68.26 | 71.47 | 71.6 |\n| GPT-4o Audio | 64.56 | 56.29 | 66.67 | 62.5 |\n\nThis demonstrates that our adaptive regularization approach achieves consistent improvements across three distinct model families: **flow matching models** (SD3), **autoregressive LLMs** (Qwen2/3), and **multi-modal reasoning models** (Qwen2.5-Omni), confirming the broad applicability across diverse architectures and modalities.\n\n> **Q5: Is it possible to overcome the problem of choosing baseline parameter $β_0$?**\n\n**A5**: Thank you for this important question. Our method actually simplifies the hyperparameter selection process compared to existing approaches. Rather than requiring extensive tuning of $β_0$, we demonstrate that ADRPO works effectively by simply adopting the same  values used in existing fixed regularization methods. For example, in our experiments (Section 4.1), we set $β_0=0.04$ for LLM fine-tuning (matching GRPO's fixed $β$) and $β_0=1$ for SD3 fine-tuning (matching baseline flow matching methods), without any additional hyperparameter search.\n\nThis design choice serves two purposes: first, it ensures fair comparison by using identical baseline regularization strengths; second, it makes ADRPO a plug-and-play enhancement to existing methods. Users can directly replace fixed regularization with our adaptive approach using their current fixed regularization weight as $β_0$, immediately benefiting from superior exploration-exploitation trade-offs without additional tuning overhead.\n\nThe substantial performance improvements we observe across all tasks—despite using the same  $β_0$ values as baseline methods—demonstrate that our core contribution lies in the adaptive mechanism itself rather than in hyperparameter optimization. Our advantage-based regularization adaptation automatically handles the exploration-exploitation balance that fixed methods cannot achieve.\n\n\n[1] Alonso, Noguer I. \"The Mathematics of Group Relative Policy Optimization: A Multi-Agent Reinforcement Learning Approach.\" 2025.\n    \n[2] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" 2025.\n\n[3] M. Chen, et al. ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning. 2025.\n\n[4] Y. Fu, et al. RLAE: Reinforcement Learning-Assisted Ensemble for LLMs. 2025.\n\n[5] Xu, Jin, et al. \"Qwen2.5-omni technical report.\" 2025.\n\n[6] Yang, Pinci, et al. \"Avqa: A dataset for audio-visual question answering on videos.\" ACM MM. 2022.\n\n[7] Sakshi, S., et al. \"Mmau: A massive multi-task audio understanding and reasoning benchmark.\" 2024."}, {"review_id": "ihA8pHkhpm", "reviewer": "Reviewer_sGRU", "summary": "The paper introduces Adaptive Divergence Regularized Policy Optimization (ADRPO), a plug-and-play framework that dynamically adjusts the weights of divergence regularization in RLHF of generative models. Instead of using a fixed KL or Wasserstein-2 penalty, ADRPO modulates the coefficient on a per-sample basis, reducing regularization for high-advantage samples to encourage exploitation, and strengthening it for low-advantage samples to preserve stability and diversity. The authors implement ADRPO with W-2 regularization for text-to-image flow-matching models (SD3) and with KL regularization for large language models (LLMs, via GRPO). ADRPO enables a 2B SD3 model to outperform larger 4.8 B and 12 B models across alignment, style transfer, and compositional control, and outperform other RL fine-tuning methods. In LLM fine-tuning (Qwen2/3), ADRPO achieves higher reward-entropy trade-offs than fixed-$\\beta$ GRPO.", "questions": "see weaknesses. and\n1. In my opinion, $L_{D}$ can be combined with $L_{PG} (L_{RL})$to form $A * min(ratio - D_{kl}, clip(ratio, 1+ \\epsilon, 1-\\epsilon) - D_{kl})$ or $A* (\\lVert v_{\\theta} - u \\rVert^2 - \\lVert v_\\theta - v_{ref}\\rVert^2)$, and plus a fixed KL penalty, could some adjustment to the ratio achieve a similar effect as described in the paper?\n2. In fact, [1] mentions a similar idea of an Adaptive KL Penalty Coefficient. Discussing the differences between the approach in this paper and that in [1] would be beneficial for highlighting the originality of the current work.\n\n[1] Schulman, John, et al. \"Proximal policy optimization algorithms.\" arXiv preprint arXiv:1707.06347 (2017).", "limitations": "yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strengths\n1. The proposed method appears simple yet effective, making it easy to follow.\n2. ADRPO’s sample-wise adaptation of divergence strength directly addresses the core dilemma in RL fine-tuning, yielding more stable and effective policy updates than fixed-$\\beta$ methods.\n3. The paper presents convincing results of fine-tuning SD3 using ADRPO, outperforming larger models. Additionally, compared to other RL fine-tuning methods in SD3, ADRPO achieves better performance in both reward and divergence metrics.\n4. ADRPO integrates seamlessly into existing RLHF/RL-from-reward frameworks for both continuous (flow matching) and discrete (LLMs) generative paradigms, without additional networks or architectural changes.\n\nweaknesses\n1. The paper claims that the method is applicable to both diffusion (flow matching) models and LLMs. However, the experimental section only presents results for text-to-image tasks, lacking quantitative or qualitative experiments involving LLMs.\n2. ADRPO introduces several new hyperparameters ($\\beta_0, A_{max}, A_{min}$). A ablation over these settings (and the effect of different clipping ranges) would help practitioners tune ADRPO in new domains.", "was_revised": true, "final_justification": "I thank the authors for addressing my concerns. I have no further questions, and recommend a borderline accept.", "rebuttal": "We deeply appreciate your thorough and constructive review. Your feedback provides crucial insights that will strengthen both the technical presentation and practical applicability of our work. We are particularly grateful for your recognition of ADRPO's simplicity and effectiveness, its seamless integration into existing frameworks, and the convincing experimental results demonstrating our method's ability to enable smaller models to outperform larger counterparts. Your detailed questions about technical connections and suggestions for additional ablations are extremely helpful. Below we systematically address each of your concerns and will incorporate these clarifications and analysis into our final revision.\n\n\n> **Q1: The paper claims that the method is applicable to both diffusion (flow matching) models and LLMs. However, the experimental section only presents results for text-to-image tasks, lacking quantitative or qualitative experiments involving LLMs.**\n\n**A1**: Thank you for the insightful feedback. We want to clarify that our paper includes comprehensive LLM fine-tuning experiments in Section 4.5 and Figure 4, demonstrating ADRPO applied to Qwen2 and Qwen3 models with detailed quantitative results including reward-entropy trade-offs and complete learning curves in Figure 8 showing ADRPO's superior exploration-exploitation dynamics.\n\n**Extra Multi-Modal Reasoning LLM Experimental Results**: To further validate our method's effectiveness across diverse LLM applications, we provide additional experimental results on fine-tuning Qwen2.5-Omni-7B [1] (bf16 precision) for multi-modal audio reasoning tasks using AVQA training data [2] with verifiable rewards and format rewards. On the MMAU benchmark [3] (a complex multi-modal reasoning benchmark), ADRPO demonstrates substantial improvements in audio reasoning tasks:\n\n| Method | Sound (%) | Music (%) | Speech (%) | Total Accuracy (%) |\n|--------|-----------|-----------|------------|-------------------|\n| Qwen2.5-Omni-7B (Base) | 72.37 | 64.37 | 69.07 | 68.6 |\n| **ADRPO (Ours)** | **81.98** | **70.06** | **75.98** | **76.0** |\n| GRPO (baseline RL) | 77.18 | 70.66 | 74.77 | 74.2 |\n| Gemini 2.5 Pro | 75.08 | 68.26 | 71.47 | 71.6 |\n| GPT-4o Audio | 64.56 | 56.29 | 66.67 | 62.5 |\n\nNotably, our ADRPO fine-tuned 7B model significantly outperforms much larger commercial models including Gemini 2.5 Pro and GPT-4o Audio, demonstrating the effectiveness of our adaptive regularization approach across different model scales and architectures.\n\n**Qualitative Analysis - Superior Reasoning Capabilities**: Beyond quantitative improvements, we provide detailed qualitative analysis comparing thinking traces between ADRPO and GRPO, revealing that our method enables superior step-by-step reasoning capabilities (more results will be included in our final revision):\n\n\n**Sample ID:** b7701ab1-c37e-49f2-8ad9-7177fe0465e9\n- **Question**: \"What was the last sound in the sequence? Choices: footsteps, dog_barking, camera_shutter_clicking, tapping_on_glass\"\n- **ADRPO**: \"The sequence of sounds includes footsteps, a dog barking, a camera shutter clicking, and a tapping on glass. The last sound is a distinct 'tapping on glass', which is a clear and unique sound compared to the others.\" → **tapping_on_glass** ✓\n- **GRPO**: \"The last sound in the sequence is a camera shutter click, which is a distinct sound from the other options provided.\" → **camera_shutter_clicking** ✗\n\n\n**Cross-Architecture Validation**: These results, combined with our original experiments on autoregressive LLMs (Qwen2/3) and flow matching models (SD3), demonstrate that ADRPO provides consistent improvements across **three distinct model families**: (1) **flow matching models** for continuous generation, (2) **autoregressive LLMs** for discrete sequence generation, and (3) **multi-modal reasoning models** for complex audio-text understanding. This comprehensive validation confirms the broad applicability and effectiveness of our adaptive regularization approach across diverse architectures, modalities, and reasoning tasks.\n\n\n> **Q2: ADRPO introduces several new hyperparameters $\\left(\\beta_0, A_{\\text {max }}, A_{\\text {min }}\\right)$. A ablation over these settings (and the effect of different clipping ranges) would help practitioners tune ADRPO in new domains.**\n\n**A2**: We appreciate this feedback and understand practitioners' concerns about hyperparameter tuning in new domains. To directly address the concern, we conducted comprehensive ablation studies in a completely new domain - multi-modal audio reasoning - to demonstrate ADRPO's practical transferability and provide concrete tuning guidance.\n\n**Simple Guidelines for Practitioners**: ADRPO introduces three hyperparameters that can be set with minimal tuning effort:\n1. **β₀ (baseline regularization)**: Set equal to the fixed regularization coefficient used in existing methods (e.g., β₀=0.04 for GRPO, β₀=1 for SD3 fine-tuning)\n2. **A_min, A_max (advantage clipping)**: Use symmetric ranges around zero: A_min = -β₀, A_max = β₀\n\nThis design makes ADRPO a plug-and-play enhancement requiring no additional hyperparameter search beyond existing methods.\n\n**New Domain Validation**: To validate these guidelines beyond our main experiments, we applied ADRPO to fine-tune the multi-modal reasoning model Qwen2.5-Omni-7B [1] on the AVQA dataset [2] using verifiable and format rewards, with evaluation on the MMAU benchmark [3]. We tested various clipping configurations:\n\n| Clipping Range | A_min | A_max | Sound (%) | Music (%) | Speech (%) | Total Accuracy (%) |\n|----------------|-------|-------|-----------|-----------|------------|-------------------|\n| 1×β₀ (recommended) | -0.04 | 0.04 | 81.98 | 70.06 | 75.98 | **76.0** |\n| 2×β₀ | -0.08 | 0.08 | 84.08 | 69.46 | 73.57 | 75.7 |\n| 0.5×β₀ | -0.02 | 0.02 | 82.58 | 71.26 | 74.47 | 76.1 |\n| GRPO (baseline) | - | - | 77.18 | 70.66 | 74.77 | 74.2 |\n\n**Key Findings for New Domain Adoption**: This validation in an entirely different domain confirms that practitioners can safely apply our recommended 1×β₀ setting when transferring ADRPO to unseen domains. All ADRPO configurations significantly outperform the GRPO baseline (74.2%), with remarkably stable performance variations of less than 0.4% across different clipping ranges. This demonstrates that the fundamental advantage lies in having adaptive regularization versus fixed regularization, rather than in precise hyperparameter tuning. The advantage-based mechanism automatically adjusts regularization strength based on sample quality, making ADRPO inherently robust across domains and reducing the hyperparameter tuning burden for practitioners.\n\n> **Q3: In my opinion, $L_D$ can be combined with $L_{PG}(L_{RL})$ to form $A * \\min(\\text{ratio} - D_{kl}, \\text{clip}(\\text{ratio}, 1+\\epsilon, 1-\\epsilon) - D_{kl})$ or $A * (||v_\\theta - u||^2 - ||v_\\theta - v_{\\text{ref}}||^2)$, and plus a fixed KL penalty, could some adjustment to the ratio achieve a similar effect as described in the paper?**\n\n**A3**: Thank you for this insightful suggestion. The suggested formulations represent a creative way to absorb adaptive divergence terms into the RL terms, which is an interesting  perspective.\n\nWhile these formulations might potentially achieve similar adaptive effects, we found our current decoupled formulation $\\mathcal{L}_{RL} + (\\beta_0 - A) \\cdot \\mathcal{L}_D$ (Equation 4) offers several key advantages: **clear interpretability** where the regularization strength $(\\beta_0 - A)$ directly shows how advantage-based adaptation works, and **plug-and-play integration** that enables direct enhancement of existing methods like GRPO by simply replacing fixed $\\beta$ with $(\\beta_0 - A)$ without modifying core policy gradient computations. Our design prioritizes simplicity and practical applicability, allowing practitioners to easily adopt it in their current methods.\n\n> **Q4: In fact, [1] mentions a similar idea of an Adaptive KL Penalty Coefficient. Discussing the differences between the approach in this paper and that in [1] would be beneficial for highlighting the originality of the current work.**\n\n**A4**: Thank you for raising this important point. While PPO [1] does include an \"adaptive KL penalty\" variant, our approaches are fundamentally different in both mechanism and objective. **PPO operates at a global population level with rule-based adjustment to achieve a fixed target KL divergence $d_{\\text{targ}}$ (a fixed hyperparameter throughout training), while our ADRPO performs sample-level adaptation to address the exploration-exploitation trade-off**.\n\nPPO's adaptive mechanism adopts simple heuristic rules (if KL < $d_{\\text{targ}}$/1.5, then β ← β/2; if KL > $d_{\\text{targ}}$ × 1.5, then β ← β × 2) to achieve a fixed target KL value $d_{\\text{targ}}$ and applies the same β uniformly to all samples within each training iteration, regardless of individual sample quality. Critically, PPO's authors found this adaptive KL penalty approach performed significantly worse than their clipped objective (0.74 vs 0.82 in their experiments) and explicitly stated \"the KL penalty performed worse than the clipped surrogate objective.\"\n\nIn contrast, our ADRPO uses $\\mathcal{L}^{ADRPO}(\\theta) = \\mathcal{L}^{RL}(\\theta) + (\\beta_0 - A) \\cdot \\mathcal{L}^D(\\theta)$ to automatically differentiate between high-quality and low-quality samples, applying individualized regularization based on advantage estimates. This enables dynamic balancing of aggressive optimization for promising samples and conservative updates for inferior samples, achieving good performance across multiple domains as demonstrated in our experiments.\n\n[1] Xu, Jin, et al. \"Qwen2.5-omni technical report.\" 2025.\n\n[2] Yang, Pinci, et al. \"Avqa: A dataset for audio-visual question answering on videos.\" ACM MM. 2022.\n\n[3] Sakshi, S., et al. \"Mmau: A massive multi-task audio understanding and reasoning benchmark.\" 2024."}, {"review_id": "U69zhw8ArT", "reviewer": "Reviewer_oj6J", "summary": "This paper proposes an adaptive divergence regularization technique that effectively improves reward fine-tuning results by introducing per-sample divergence control, which provides more flexibility to the optimization process. Essentially, it strengthens divergence regularization for samples of lower quality (in terms of the target reward) and relaxes for those of higher quality. This method works well for both text-to-image generation (continuous domain FM velocity-based divergence regularization) and LLM (classic KL divergence on discrete domain).", "questions": "+ Wouldn't the reward hacking be worse if the proposed approach relaxes divergence control for samples that are highly aligned with the underlying reward? For instance, say the target reward model gives a high reward to certain visual artifacts, and stronger divergence control helps to mitigate this.", "limitations": "See \"weakness\" and \"questions\"", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ Good motivation for an important problem in reward fine-tuning generation models (the exploitation vs exploration dilemma).\n+ Results look convincing (especially Fig. 3 & 4).\n\nWeakness:\n+ I would like to see more visual results, including the failure cases of the proposed method, to understand the exploitation vs exploration dilemma better and how the method improves the generation quality on the hard cases.\n+ Besides ClipDiversity and W2 divergence as the y-axis (say the x-axis shows reward scores), can the authors also show the plot with FIDs (in the style of Fig. 8 from the appendix of [1])? It better reflects image fidelity. As discussed in [1], the exploitation vs exploration dilemma in fine-tuning image generation models often comes down to reward scores vs. image fidelity. Maybe the authors can add a discussion on this matter. \n\n[1] Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward, CVPR 2025", "was_revised": true, "final_justification": "I keep my original score.", "rebuttal": "We are extremely grateful for your valuable and detailed review. Your feedback is instrumental in refining our work's impact and presentation. We especially appreciate your recognition of our method's strong motivation for addressing the exploitation-exploration dilemma in reward fine-tuning and your assessment of our convincing results. The suggestions for additional visual analyses, FID, and discussions on reward hacking provide valuable directions for strengthening our impact and clarity. We address your insightful questions below and will incorporate these clarifications and analysis into our final revision.\n\n> **Q1: I would like to see more visual results, including the failure cases of the proposed method, to understand the exploitation vs exploration dilemma better and how the method improves the generation quality on the hard cases.**\n\n**A1**: Thank you for this valuable feedback. We believe that ADRPO's improvements in generation quality and alignment can be best demonstrated by comparing our method to the failure modes of exploration-heavy/exploitation-heavy methods. We have provided extensive qualitative analysis in Figures 2 and 6, which clearly demonstrate these trade-offs and failure modes compared to our ADRPO.\n\n**Failures of Exploration-Heavy Methods (Poor Semantic Alignment)**: DPO maintains good generation diversity but consistently fails at semantic alignment. In Figure 2, DPO fails to render text properly (\"Diffusion\" appears incorrectly), generates anatomically incorrect zebras with malformed legs, and produces black bananas that lack realistic texture. In Figure 6, DPO demonstrates severe compositional failures—generating multiple apples instead of \"A green apple\" and misplacing objects in spatial relationship tasks.\n\n**Failures of Exploitation-Heavy Methods (Diversity and Quality Collapse)**: ORW-CFM-W2 aggressively pursues reward optimization but suffers from significant quality degradation. In Figure 2, it generates zebras with incorrect anatomy (extra limbs), fails at text rendering with blurred \"Diffusion\" text, and produces unnatural black bananas with poor texture quality. In Figure 6, exploitation-focused method RAFT exhibits similar failures with poor text rendering, overexposed images, and distorted bananas, struggling with compositional accuracy and visual naturalness across multiple prompts.\n\n**ADRPO's Superior Balance**: Our method successfully resolves these fundamental trade-offs by achieving both excellent semantic alignment and generation diversity simultaneously. In Figure 2, ADRPO correctly renders \"Diffusion\" text, generates anatomically accurate zebras, and produces realistic black bananas with proper texture. Figure 6 demonstrates ADRPO's superior performance on compositional challenges—correctly generating single objects, maintaining proper spatial relationships, and achieving accurate attribute binding while preserving visual quality.\n\nThis comprehensive comparison demonstrates that ADRPO's adaptive regularization mechanism enables optimal exploration-exploitation balance, avoiding both semantic alignment failures of exploration-focused methods and the quality degradation issues that plague exploitation-focused methods.\n\n**Performance on Hard Reasoning Tasks**: Beyond visual generation, our additional experiments on fine-tuning Qwen2.5-Omni-7B [1] (bf16 precision) with AVQA dataset [2] demonstrate ADRPO's superior capabilities on complex reasoning scenarios that require multi-modal understanding and multi-step analysis. On the challenging MMAU benchmark [3], ADRPO achieves 76% total accuracy vs 74.2% for GRPO, with particularly strong improvements in complex audio reasoning (more will be included in our final revision).\n\n**Qualitative Analysis of Hard Reasoning Cases**:\n\n**Complex Temporal Analysis**:\n\n**Sample ID:** b7701ab1-c37e-49f2-8ad9-7177fe0465e9\n- **Question**: \"What was the last sound in the sequence? Choices: footsteps, dog_barking, camera_shutter_clicking, tapping_on_glass\"\n- **ADRPO**: \"The sequence of sounds includes footsteps, a dog barking, a camera shutter clicking, and a tapping on glass. The last sound is a distinct 'tapping on glass', which is a clear and unique sound compared to the others.\" → **tapping_on_glass** ✓\n- **GRPO**: \"The last sound in the sequence is a camera shutter click, which is a distinct sound from the other options provided.\" → **camera_shutter_clicking** ✗\n\n**Complex Contextual Reasoning**:\n\n**Sample ID:** 6d1ab354-944d-4155-a4ec-c851fbcb7c93\n- **Question**: \"Considering the information in the audio, what type of weather condition can be inferred? Choices: Windy, Calm, Rainy, Snowy\"\n- **ADRPO**: \"The presence of wind noise suggests a windy condition. The absence of rain or snow sounds indicates it's not rainy or snowy.\" → **Windy** ✓\n- **GRPO**: \"The presence of wind noise and the sound of rain hitting a surface suggest a windy and rainy weather condition.\" → **Rainy** ✗\n\nThese challenging cases demonstrate that ADRPO enables superior step-by-step reasoning with more accurate temporal analysis, contextual understanding, and multi-modal integration, while GRPO consistently fails on complex reasoning tasks that require nuanced audio comprehension and logical inference.\n\n> **Q2: Besides ClipDiversity and W2 divergence as the y-axis (say the x-axis shows reward scores), can the authors also show the plot with FIDs (in the style of Fig. 8 from the appendix of [1])? It better reflects image fidelity. As discussed in [1], the exploitation vs exploration dilemma in fine-tuning image generation models often comes down to reward scores vs. image fidelity. Maybe the authors can add a discussion on this matter.**\n\n**A2**: We appreciate this suggestion about including FID metrics. However, it is worth noting that FID typically requires a ground-truth dataset for comparison, which is not available for the base models (SD3, SANA, etc.) and baselines we're comparing as their training data is usually different and hidden. During training, we only have prompts and train based on data generated by the model itself, rather than from ground-truth image datasets. Many other RL post-training works (such as ORW-CFM-W2 [4], DDPO [5] and DPOK [6]) also do not include FID metrics in their evaluations. \n\n**Alternative Quality Assessment**: While we cannot provide FID, the image quality metrics in Table 1 and visual results in Figures 2 and 6 address the fidelity concerns you raise. Our comprehensive evaluation includes multiple quality dimensions: Aesthetic scores (5.53→6.27), BLIPScore improvements (0.501→0.567), and human preference metrics (ImageReward: 0.97→1.61, PicScore: 20.81→22.78). These metrics, combined with our qualitative results, demonstrate that ADRPO achieves superior generation quality and semantic alignment compared to baseline methods and even much larger models (FLUX.1-Dev 12B, SANA-1.5 4.8B).\n\n**Reward, Quality and Diversity Balance**: Our training curves in Figure 3 demonstrate that ADRPO achieves better exploration-exploitation trade-offs by finding superior convergence points that maintain both high reward and diversity, suggesting improved fidelity-reward balance compared to methods that suffer from diversity collapse (ORW-CFM-W2) or reward plateau (DPO). The visual evidence in Figures 2 and 6 confirms this balance - our method avoids the overexposure artifacts and quality degradation seen in exploitation-heavy methods while maintaining more precise semantic alignment compared to exploration-heavy approaches.\n\n> **Q3: Wouldn't the reward hacking be worse if the proposed approach relaxes divergence control for samples that are highly aligned with the underlying reward? For instance, say the target reward model gives a high reward to certain visual artifacts, and stronger divergence control helps to mitigate this.**\n\n**A3**: Thank you for this insightful question. However, our adaptive mechanism actually provides better protection against reward hacking than fixed regularization methods through several key safeguards.\n\n**Advantage-Based Adaptive Regularization**: We use advantage estimates rather than raw rewards to adjust regularization strength. Our advantage estimation provides much more stable and normalized values compared to raw rewards, which can have high variance and extreme values. This advantage-based approach prevents drastic regularization changes that could occur with raw reward-based adjustment, maintaining more stable training dynamics.\n\n**Advantage Clipping Protection**: As detailed in Section 4.1, we employ advantage clipping $A_{clipped}= \\text{clip}(A, A_{min}, A_{max})$ that **constrains regularization adjustments within reasonable bounds**. This prevents extreme regularization reduction even for samples with very high advantages while maintaining training stability.\n\n**Empirical Evidence Against Reward Hacking**: Figure 3 demonstrates that ADRPO achieves high reward performance **without the diversity collapse** characteristic of reward hacking seen in ORW-CFM-W2. Our visual results in Figures 2 and 6 consistently avoid the overexposure artifacts, unnatural distortions, and quality degradation that typically characterize reward hacking in exploitation-heavy methods. Instead, ADRPO maintains both high semantic alignment and natural visual quality, suggesting effective mitigation of reward hacking while achieving genuine improvements in generation quality.\n\n[1] Xu, Jin, et al. \"Qwen2.5-omni technical report.\" 2025.\n\n[2] Yang, P., et al. \"Avqa: A dataset for audio-visual question answering on videos.\" 2022.\n\n[3] Sakshi, S., et al. \"Mmau: A massive multi-task audio understanding and reasoning benchmark.\" 2024.\n\n[4] Fan, J., et al. \"Online reward-weighted fine-tuning of flow matching with wasserstein regularization.\" 2025.\n\n[5] Black, K., et al. \"Training diffusion models with reinforcement learning.\" 2023.\n\n[6] Fan, Y., et al. \"Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.\" 2023."}]

Author rebuttal:

Meta-review: The authors propose an adaptive regularization approach based on advantage functions for fine-tuning generative models, including LLMs and diffusion models. Many reviewers agree that the paper demonstrates strong empirical performance. However, reviewers raised concerns about whether the proposed method is truly well-founded, and the overhead to estimate the advantages functions.
Decision: Accept (poster)