Title: Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement learning

generative models

adaptive regularization

advantage estimation

fine-tuning

flow matching

text-to-image generation

exploration-exploitation trade-off
Abstract: Balancing exploration and exploitation during reinforcement learning fine-tuning of generative models presents a critical challenge, as existing approaches rely on fixed divergence regularization that creates an inherent dilemma: strong regularization preserves model capabilities but limits reward optimization, while weak regularization enables greater alignment but risks instability or reward hacking. We introduce Adaptive Divergence Regularized Policy Optimization (ADRPO), which automatically adjusts regularization strength based on advantage estimates—reducing regularization for high-value samples while applying stronger regularization to poor samples, enabling policies to navigate between exploration and aggressive exploitation according to data quality. Our implementation with Wasserstein-2 regularization for flow matching generative models achieves remarkable results on text-to-image generation, achieving better semantic alignment and diversity than offline methods like DPO and online methods with fixed regularization like ORW-CFM-W2. ADRPO enables a 2B parameter SD3 model to surpass much larger models with 4.8B and 12B parameters in attribute binding, semantic consistency, artistic style transfer, and compositional control while maintaining generation diversity. ADRPO generalizes to KL-regularized fine-tuning of both text-only LLMs and multi-modal reasoning models, enhancing existing online RL methods like GRPO while requiring no additional networks or complex architectural changes. In LLM fine-tuning, ADRPO demonstrates an emergent ability to escape local optima through active exploration, while in multi-modal audio reasoning, it outperforms GRPO through superior step-by-step reasoning, enabling a 7B model to outperform substantially larger commercial models including Gemini 2.5 Pro and GPT-4o Audio, offering an effective plug-and-play solution to the exploration-exploitation challenge across diverse generative architectures and modalities.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an adaptive divergence regularization technique that effectively improves reward fine-tuning results by introducing per-sample divergence control, which provides more flexibility to the optimization process. Essentially, it strengthens divergence regularization for samples of lower quality (in terms of the target reward) and relaxes for those of higher quality. This method works well for both text-to-image generation (continuous domain FM velocity-based divergence regularization) and LLM (classic KL divergence on discrete domain).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ Good motivation for an important problem in reward fine-tuning generation models (the exploitation vs exploration dilemma).\n+ Results look convincing (especially Fig. 3 & 4).\n\nWeakness:\n+ I would like to see more visual results, including the failure cases of the proposed method, to understand the exploitation vs exploration dilemma better and how the method improves the generation quality on the hard cases.\n+ Besides ClipDiversity and W2 divergence as the y-axis (say the x-axis shows reward scores), can the authors also show the plot with FIDs (in the style of Fig. 8 from the appendix of [1])? It better reflects image fidelity. As discussed in [1], the exploitation vs exploration dilemma in fine-tuning image generation models often comes down to reward scores vs. image fidelity. Maybe the authors can add a discussion on this matter. \n\n[1] Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward, CVPR 2025\n\nQUESTIONS:\n+ Wouldn't the reward hacking be worse if the proposed approach relaxes divergence control for samples that are highly aligned with the underlying reward? For instance, say the target reward model gives a high reward to certain visual artifacts, and stronger divergence control helps to mitigate this.\n\nLIMITATIONS:\nSee \"weakness\" and \"questions\"", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Adaptive Divergence Regularized Policy Optimization (ADRPO), a plug-and-play framework that dynamically adjusts the weights of divergence regularization in RLHF of generative models. Instead of using a fixed KL or Wasserstein-2 penalty, ADRPO modulates the coefficient on a per-sample basis, reducing regularization for high-advantage samples to encourage exploitation, and strengthening it for low-advantage samples to preserve stability and diversity. The authors implement ADRPO with W-2 regularization for text-to-image flow-matching models (SD3) and with KL regularization for large language models (LLMs, via GRPO). ADRPO enables a 2B SD3 model to outperform larger 4.8 B and 12 B models across alignment, style transfer, and compositional control, and outperform other RL fine-tuning methods. In LLM fine-tuning (Qwen2/3), ADRPO achieves higher reward-entropy trade-offs than fixed-$\\beta$ GRPO.\n\nSTRENGTHS AND WEAKNESSES:\nstrengths\n1. The proposed method appears simple yet effective, making it easy to follow.\n2. ADRPO’s sample-wise adaptation of divergence strength directly addresses the core dilemma in RL fine-tuning, yielding more stable and effective policy updates than fixed-$\\beta$ methods.\n3. The paper presents convincing results of fine-tuning SD3 using ADRPO, outperforming larger models. Additionally, compared to other RL fine-tuning methods in SD3, ADRPO achieves better performance in both reward and divergence metrics.\n4. ADRPO integrates seamlessly into existing RLHF/RL-from-reward frameworks for both continuous (flow matching) and discrete (LLMs) generative paradigms, without additional networks or architectural changes.\n\nweaknesses\n1. The paper claims that the method is applicable to both diffusion (flow matching) models and LLMs. However, the experimental section only presents results for text-to-image tasks, lacking quantitative or qualitative experiments involving LLMs.\n2. ADRPO introduces several new hyperparameters ($\\beta_0, A_{max}, A_{min}$). A ablation over these settings (and the effect of different clipping ranges) would help practitioners tune ADRPO in new domains.\n\nQUESTIONS:\nsee weaknesses. and\n1. In my opinion, $L_{D}$ can be combined with $L_{PG} (L_{RL})$to form $A * min(ratio - D_{kl}, clip(ratio, 1+ \\epsilon, 1-\\epsilon) - D_{kl})$ or $A* (\\lVert v_{\\theta} - u \\rVert^2 - \\lVert v_\\theta - v_{ref}\\rVert^2)$, and plus a fixed KL penalty, could some adjustment to the ratio achieve a similar effect as described in the paper?\n2. In fact, [1] mentions a similar idea of an Adaptive KL Penalty Coefficient. Discussing the differences between the approach in this paper and that in [1] would be beneficial for highlighting the originality of the current work.\n\n[1] Schulman, John, et al. \"Proximal policy optimization algorithms.\" arXiv preprint arXiv:1707.06347 (2017).\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper suggest to use adaptive regularization scheme for policy optimization enabling to balance between exploration and exploitation. The scheme called ADRPO is based on the advantage estimation - reducing regularization for high-value samples while applying stronger regularization to poor samples. The method is applied for online fine-tuning of LLMs (to extend DeepSeekMath’s GRPO idea), text to image alignment etc. The authors also suggest an advantage-weighted flow matching objective strengthening high-quality generations and pushing the model\naway from poor generations. Numerical experiments (LLM fine-tuning and text to image generation) support suggested method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper is well written in easy to understand. It illustrates how different components work together. \n - Interesting idea of adaptive regularisation based on the advantage estimation. ADRPO automatically increases regularization for low-advantage (less promising) samples and decreases it for high-advantage (promising) samples, providing fine-grained control over the policy updates. \n- ADRPO generalizes to both flow matching models (using Wasserstein-2 divergence) and LLMs (using KL divergence)\n - The experimental results look promising. \n\nWeaknesses\n\n- Additional computational costs due to advantage estimation. The paper would benefit from a more quantitative analysis of this overhead and practical guidance on how it scales with model size.\n- While the empirical results are strong, the paper would be improved by discussing theoretical properties of the proposed adaptation.\n\nQUESTIONS:\n- I would suggest to add more explanation of advantage function estimation in particular for the section 3.4. ADRPO for Fine-tuning LLMs. At least summarise main ideas from the paper [29]. \n- I would also suggest to add more experiments replacing Qwen by other comparable models. \n- Is it possible to overcome the problem of choosing baseline parameter $\\beta_0$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Adaptive Divergence Regularized Policy Optimization (ADRPO), a method for reinforcement learning fine-tuning of generative models with adaptive regularization, allowing for balanced exploitation-exploration tradeoffs, learning from useful examples while not being destructive to generative model capabilities.\n\nThe method replaces traditional fixed-weight W2 or KL-divergence regularization with one weighted dynamically with an Advantage estimate for the current sample. The technique is applied to both RL fine-tuning of flow matching, and RL fine-tuning of LLMs, beating non-adaptive baselines on both, with both quantitative and qualitative metrics.\n\nThe exploration-exploitation tradeoff is also quantified with measures of sample diversity and model divergence, demonstrating that the ADRPO technique leads to an optimal combination of both diversity, low model divergence and final reward.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The proposed technique is very simple, but reliably improves the performance of RL finetuning techniques on the exploration-exploitation tradeoff frontier.\n* The approach is tested on both continuous generative modeling, for flow matching based image generation, and discrete generative modeling, in LLM finetuning, showcasing that the underlying insight is not model or modality dependent, but is generally applicable to generative models.\n\nWeaknesses:\n* Quantitative and qualitative comparisons do not straightforwardly appear as a fundamental improvement over the state of the art: benchmark improvements are relatively small, and image generation quality does not seem to conclusively be much beyond other methods.\n\nQUESTIONS:\n* Regarding small improvements: Did you produce a breakdown of whether it is possible for the baselines to achieve comparable results with more compute expenditure? And if so, with how much more expenditure?\n* Could you provide more details on how does the procedure behave when the regularization coefficient is allowed to be negative vs when it is clipped?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}