Title: Uni-RL: Unifying Online and Offline RL via Implicit Value Regularization
Venue: NeurIPS 2025
Accepted: yes
Keywords: off-policy RL

offline RL

online RL

offline-to-online RL
Abstract: The practical use of reinforcement learning (RL) requires handling diverse settings, including online, offline, and offline-to-online learning. Instead of developing separate algorithms for each setting, we propose Uni-RL, a unified model-free RL framework that addresses all these scenarios within a single formulation. Uni-RL builds on the Implicit Value Regularization (IVR) framework and generalizes its dataset behavior constraint to the constraint w.r.t a reference policy, yielding an unified value learning objective for general settings. The reference policy is chosen to be the target policy in the online setting and the behavior policy in the offline setting. Using an iteratively refined behavior policy solves the over-constrained problem of directly applying IVR in the online setting, it provides an implicit trust-region style update through the value function while being off-policy. Uni-RL also introduces an unified policy extraction objective that estimates in-sample policy gradient using only actions from the reference policy. This supports various policy classes and theoretically guaranntees less value estimation error and larger performance improvement over the reference policy. We evaluate Uni-RL on a range of standard RL benchmarks across online, offline, and offline-to-online settings. In online RL, Uni-RL achieves higher sample efficiency than both off-policy methods without trust-region updates and on-policy methods with trust-region updates. In offline RL, Uni-RL retains the benefits of in-sample learning while outperforming IVR through better policy extraction. In offline-to-online RL, Uni-RL beats both constraint-based methods and unconstrained approaches by effectively balancing stability and adaptability.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes Uni-IVR, a single model-free algorithm that adapts to three settings --- offline, offline-to-online, and online learning. The key idea builds on the Implicit Value Regularization (IVR) framework and utilizes different reference policies depending on the settings: the behavior policy in the offline setting and a target policy in the online setting. Additionally, authors focus on providing a policy extraction scheme to effectively derive the best policy from the learned value function via Uni-IVR. The In-sample Policy Gradient (InPG) solves a constrained QP that determines weights using zero-order gradients, remains mode-seeking to maximize Q-values, and aligns a weighted BC loss with the true PG direction. In evaluation, Uni-IVR achieves higher sample efficiency and outperforms several SOTA algorithms in three settings.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n  1. While previous works in RL mainly focus on one of three settings, this work designs a scalable RL framework that unifies online, offline, and offline-to-online settings.\n  2. InPG combines mode-seeking behavior with in-support actions, avoiding the over-estimation of Q-value while outperforming existing weighted-BC or MaxQ+BC. Also, Theorems 1-3 guarantee that the policy extraction with InPG improves performance and readily scales to generative models like diffusion and flow matching models.\n  3. When comparing several algorithms in each RL setting, Uni-IVR achieves SOTA performance and sample efficiency in diverse datasets.\n\n\n- Weaknesses\n   1. Uni-IVR is closely related to IVR (Xu et al., 2023). The main difference in offline RL settings is the addition of the InPG method. In Table 1, I think IVR's performance is comparable to that of Uni-IVR (Gaussian). Additionally, baselines (BCQ, TD3+BC, CQL, and IQL) are not SOTA algorithms in offline RL. To make the empirical evaluation more convincing, Uni-IVR should be compared with more recent methods, such as ReBRAC [1] and QDQ [2].\n   2. Lemma 1 states that the optimal value functions $Q^{\\*}$ and $V^{\\*}$  can be solved by Equations 3 and 4, respectively. In Equation 5, however, the authors replace the maximisation with an expectation over actions sampled from $\\bar{\\pi}$. Does this modified operator still converge to $V^{*}$?\n   3. In Figure 5, learning curves are plotted after 250k offline pertaining. Yet this work claims that Uni-IVR (i) alleviates Q-value overestimation and (ii) avoids the performance drop typically seen at the start of offline-to-online finetuning. To substantiate these points, wouldn’t it be essential to plot the offline performance at the end of pre-training, alongside the subsequent finetuning curves?\n\n[1] Tarasov, Denis, et al. \"Revisiting the minimalist approach to offline reinforcement learning.\" Advances in Neural Information Processing Systems 36 (2023): 11592-11620.\n\n[2] Zhang, Jing, et al. \"Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model.\" Advances in Neural Information Processing Systems 37 (2024): 54421-54462.\n\nQUESTIONS:\n1. The related-work section cites recent algorithms such as Uni-O4 and DICE-based methods, yet Figure 5 benchmarks Uni-IVR only against Cal-QL, AWAC, RLPD, and IQL—methods published no later than 2023. To the best of my knowledge, BOORL [3], WSRL [4], and  O2SAC [5] are SOTA algorithms. Could you clarify whether Uni-IVR is expected to deliver superior sample efficiency, early-phase stability, and final performance when evaluated against these baselines?\n\n2. Tables 4 and 5 list different hyperparameters for the Antmaze tasks: $\\alpha$ = 0.1 or 2.0 in the pure-offline experiments, but $\\alpha=0.5$ in the offline-to-online setting. \tDoes this difference imply that a policy trained with the offline hyperparameter cannot be carried over directly into the online fine-tuning phase without re-tuning $\\alpha$?\n\n3. Lemma 1 appears to reproduce a result that was already proved in the original IVR paper (Xu et al., 2023). Please add an explicit citation and indicate the exact place in that paper where the derivation can be found.\n\n[3] Hu, Hao, et al. \"Bayesian design principles for offline-to-online reinforcement learning.\" arXiv preprint arXiv:2405.20984 (2024).\n\n[4] Zhou, Zhiyuan, et al. \"Efficient online reinforcement learning fine-tuning need not retain offline data.\" arXiv preprint arXiv:2412.07762 (2024).\n\n[5] Luo, Qin-Wen, et al. \"Optimistic critic reconstruction and constrained fine-tuning for general offline-to-online RL.\" Advances in Neural Information Processing Systems 37 (2024): 108167-108207.\n\nLIMITATIONS:\nIn the Limitations section, the authors state only that Uni-IVR is restricted to the model-free setting. A more comprehensive ablation study is warranted. For instance, they could remove InPG (replacing it with vanilla BC or MaxQ + BC) and demonstrate robustness across a range of hyperparameter settings.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes **Uni-IVR**, an extension of IVR that achieves strong performance across offline, off-to-on, and online reinforcement learning (RL) settings, without requiring substantial modifications between these modes. The main contributions can be summarized in two parts: **iterative refinement** and **in-sample policy gradient (InPG)**.\n\nThe **iterative refinement** component uses the target policy instead of the behavior policy to optimize the value function $V$ in online RL and in the online phase of off-to-on RL. The **InPG** component introduces a weight optimization scheme for weighted behavioral cloning (BC), aiming to align the Q-function maximization gradient with the weighted BC gradient.\n\nEmpirically, Uni-IVR demonstrates strong performance across all evaluated domains, with high sample efficiency and low sensitivity to hyperparameter tuning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. InPG offers an intuitive and novel formulation of weighted behavioral cloning (BC).\n2. The empirical results demonstrate strong performance, especially considering that Uni-IVR is applicable across diverse domains and operates as an in-distribution algorithm.\n3. Uni-IVR requires minimal hyperparameter tuning and performs well with configurations close to standard defaults.\n4. The theoretical support is thorough and well-presented.\n\n**Weaknesses**\n\n1. Some configurations and notations are ambiguous, including:\n   - The method used to combine policies during iterative refinement. It is unclear whether this is done via parameter mixing or probability mixing.\n   - The specific type of $f$-divergence employed. While IVR distinguishes between different divergences, the presented formulation appears similar to sparse Q-learning (SQL), which could benefit from clarification.\n   - The type of noise scheduling used in diffusion-based cases. While VP scheduling is expected, cosine scheduling is also commonly applied and should be explicitly stated if used.\n\n2. The iterative refinement component appears to be a straightforward application of the target policy method, which is already widely used in online RL algorithms such as TD3, as well as in some offline RL approaches. If the combination method corresponds to parameter mixing, the contribution may be limited to employing a larger mixing coefficient than the commonly used value of 0.005, which raises questions about the novelty of this aspect.\n\n3. The paper does not include ablation studies, which makes it difficult to assess the individual contributions of each component.\n\n4. If the second point holds, the overall algorithmic contribution may be somewhat limited.\n\nQUESTIONS:\n1. Could you provide ablation results for some of the key hyperparameters or for different choices of $f$-divergence used in the algorithm?\n\n2. What specific method of policy mixing is used in the iterative refinement process, parameter mixing or probability mixing? If it is parameter mixing, could you clarify in what way this contributes to the novelty of the method?\n\n3. Given that IVR utilizes both KL divergence and $\\chi^2$-divergence in separate algorithmic formulations, it would be helpful to clarify which divergence is used in Uni-IVR.\n\n4. The use of 500K steps is an interesting design choice. However, it raises the question of whether training continues to improve beyond that point. Could you provide additional results or learning curves to illustrate this?\n\n5. Would it be possible to include results using exactly the same hyperparameter settings as in the original IVR paper for the locomotion tasks in offline RL? This could help isolate the contribution of the proposed modifications.\n\nLIMITATIONS:\nThe authors are encouraged to explicitly discuss the limitations of their proposed algorithm, either in terms of its general applicability or in relation to the specific domains used in the experimental evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Uni-IVR, a single reinforcement learning framework designed to operate across online, offline, and offline-to-online settings. The method builds upon the Implicit Value Regularization (IVR) framework by generalizing its dataset-based constraint to a more flexible 'reference policy' constraint, which adapts to each learning paradigm. The paper also introduces a novel policy extraction method, In-sample Policy Gradient (InPG), to improve policy updates.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper tackles the important and ambitious goal of unifying RL across different learning paradigms (online, offline, offline-to-online) with a single, coherent algorithm.\n- The approach is a well-motivated extension of the established IVR framework, providing a solid theoretical foundation for its value learning component.\n- The experimental results are comprehensive and consistently demonstrate that Uni-IVR is competitive with or superior to a range of baseline methods across all three settings.\n\n**Weaknesses:**\n- The core value learning objective is a direct and incremental extension of the IVR framework. Moreover, the paper only compare the improvement in Value learning objective in one task with no name (Figure 2).\n- The novel policy extraction method InPG lacks a comprehensive evaluation, only demonstrated primarily on a single environment (Figure 3), which raises concerns about the generalizability of the results. Moreover, the absence of supplementary source code makes it difficult to check the effective of this novel method.\n- Lack of description about how to build and train $w$ in the implementation. Moreover, computational cost analysis for the InPG module are missing.\n- The paper proposes two alternative objectives that enable the use of Flow-Matching and Diffusion models instead of traditional Gaussian policies. However, if I understand correctly, these objectives treat all actions from the dataset uniformly and do not incorporate information from the Q and V functions.\n- in Figure 2, the authors provide Q-curves to illustrate higher performance; however, Q-learning methods (IQ-learn [1] for example) can produce high Q-values without corresponding performance gains. There are no explanation for these Q value results.\n\n\n**Minors:**\n- Figure 2 uses terms like MaxQ and Weighted BC without prior definition.\n- Two provided learning rates in Table 2 are for offline.\n\n[1] Garg, Divyansh, et al. \"Iq-learn: Inverse soft-q learning for imitation.\" Advances in Neural Information Processing Systems 34 (2021): 4028-4039.\n\nQUESTIONS:\n1. Could you provide a full comparison of environmental returns for Uni-IVR vs. IVR across all online environments?\n2. To validate the contribution of InPG, could you provide a comprehensive comparison of policy extraction methods (the proposed InPG vs. baselines like weighted BC and MaxQ) across all tasks in all three settings (online, offline, and offline-to-online)? \n3. There is a significant discrepancy in the λ hyperparameter values between online (0.1–0.2) and offline-to-online (0.01) settings, as shown in Tables 3 and 5. This raises questions about the sensitivity of the method to this hyperparameter. Could you provide an ablation study that shows how performance varies with λ in each setting? \n4. you explain how Uni-IVR leverages Q and V to guide training in these types of generative models and achieve the results shown in Table 1?\n5. To support the claim of scalability, could you please add the results for a Flow-matching policy to Table 1, as was done for the Diffusion policy? \n6. Could you provide the missing implementation details for InPG, including the architecture of the network used to learn the weights $w$, its optimization schedule, and an analysis of its training time compared to Weighted BC and MaxQ?\n7. why Q-values in Figure 2 are used as indicator of performance?\n8. In line 298-299, author stated that “DICE-based methods only work for the offline setting”. Can you justify the reason why they can not be applied by the reference policy?\n\nLIMITATIONS:\nThe authors did not explicitly mention the societal impact of their work, but I do not perceive any negative societal implications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Uni-IVR, a unified model-free Reinforcement Learning (RL) framework. Uni-IVR  builds upon the Implicit Value Regularization (IVR) framework, and is designed to address online, offline, and offline-to-online settings. It resolves the issue of over-conservatism when directly applying IVR in online settings, and provides an implicit trust-region-style update via the value function. Furthermore, Uni-IVR introduces a unified policy extraction objective that theoretically guarantees lower value estimation error and greater performance improvement relative to the reference policy. Evaluations on 6 RL benchmarks and 23 environments demonstrate that Uni-IVR achieves competitive or superior performance compared to state-of-the-art baselines across all three settings.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. Uni-IVR provides a unified model-free RL framework that integrates online, offline, and offline-to-online learning settings, demonstrating strong versatility.\n2. The paper offers theoretical guarantees, including lower value estimation error and greater performance improvement for its unified policy extraction objective relative to the reference policy. It also provides theoretical insights, indicating that InPG (In-sample Policy Gradient) optimizes the constrained learning objective, can better utilize the Q-function, and can achieve a greater performance improvement than $\\pi_{IVR}^*$.\n3. Uni-IVR achieves higher sample efficiency in online RL compared to both off-policy methods without trust-region updates and on-policy methods with trust-region updates.\n\n## Weaknesses\n1. The algorithms compared in the online RL experiments are outdated (the latest is from 2018); it should consider comparing with more advanced algorithms.\n2. The \"iteratively refined behavior policy\" used to address the over-conservatism issue in online IVR is a soft update (Equation 5). While more lightweight than dataset filtering, the choice of hyperparameter λ might affect performance.\n\nQUESTIONS:\n-\n\nLIMITATIONS:\n1. The paper does not provide detailed information on the computational resources used for the experiments.\n2. Although the paper considers various learning scenarios such as online, offline, and offline-to-online, it is limited to model-free RL. Verification of the proposed method in other settings, such as model-based RL, could further demonstrate its generality.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}