Uni-RL: Unifying Online and Offline RL via Implicit Value Regularization
Authors: Haoran Xu
Liyuan Mao
Hui Jin
Weinan Zhang
Xianyuan Zhan
Amy Zhang
Venue: NeurIPS 2025 poster ()
Keywords: off-policy RL
offline RL
online RL
offline-to-online RL
Primary area: reinforcement_learning
TL;DR: A unified and scalable RL framework applicable to online, offline, and offline-to-online settings.

Abstract: The practical use of reinforcement learning (RL) requires handling diverse settings, including online, offline, and offline-to-online learning. Instead of developing separate algorithms for each setting, we propose Uni-RL, a unified model-free RL framework that addresses all these scenarios within a single formulation. Uni-RL builds on the Implicit Value Regularization (IVR) framework and generalizes its dataset behavior constraint to the constraint w.r.t a reference policy, yielding an unified value learning objective for general settings. The reference policy is chosen to be the target policy in the online setting and the behavior policy in the offline setting. Using an iteratively refined behavior policy solves the over-constrained problem of directly applying IVR in the online setting, it provides an implicit trust-region style update through the value function while being off-policy. Uni-RL also introduces an unified policy extraction objective that estimates in-sample policy gradient using only actions from the reference policy. This supports various policy classes and theoretically guaranntees less value estimation error and larger performance improvement over the reference policy. We evaluate Uni-RL on a range of standard RL benchmarks across online, offline, and offline-to-online settings. In online RL, Uni-RL achieves higher sample efficiency than both off-policy methods without trust-region updates and on-policy methods with trust-region updates. In offline RL, Uni-RL retains the benefits of in-sample learning while outperforming IVR through better policy extraction. In offline-to-online RL, Uni-RL beats both constraint-based methods and unconstrained approaches by effectively balancing stability and adaptability.

Uni-RL: Unifying Online and Offline RL via Implicit Value Regularization
Haoran Xu
Liyuan M
Hui Jin
Weinan Zhang
Xianyuan Zhan
Amy Zhang!
University of Texas at Austin
Shanghai Jiao Tong University
3 Nanyang Technological University TTinngua University
Abstract
The practical implementations of reinforcement learning (RL) often face diverse settings, such online, offline, and coffine-to-onlinng learning. Instead of develop- ing separate algorithms for each setting, we propose Uni-RL, unified model-free RL framework that addresses all these scenarios within single formulation. Uni- RI. builds on the Implicit Value Regularization (IVR) framework (Xu et al., 2023) and generalizes its dataset behavior constraint to the constraint w.r.t. a reference policy, yielding a unified value learning objective for general settings. The refer- ence policy is chosen be the target policy in the online setting and the behavior policy in the offline setting. Using an iteratively refined behavior policy solves the over-conservative issue of directly applying IVR in the online setting, it provides an implicit trust-region style update through the value function while being off- policy. Uni-RL also introduces unified policy extraction objective that estimates in-sample policy gradient using only actions from the reference policy. This not only supports various policy classes, but also theoretically guarantees less value estimation error and larger performance improvement over the reference policy. We evaluate Uni-RL range of standard RL benchmarks across online, offline, and offlnne-to-onliee settings. In online RI., Uni-RL achieves higher sample efficiency than both off-policy methods without trust-region updates and on-policy methods with trust-region updates. In offline RL, Uni-RL retains the benefits of in-sample learning while outprroommng IVR through better policy extraction. In offline- to on-online RL, Uni-RL beats both constraint baased methods and unconstrained approaches by effectively balancing stability and adaptability.
Code: https: / / tittt.u.b... www.coom//aaaeeeeee wtt.rryaaahhuurr UnniRRL
Online RL
Offine RL
Offiinetto-oolin RL
Gym, PyBuller, DMControl
D4RL MuloCo and Antmaze
D4RL Antmaze, Kitchen and Adroit
1.0
1.8
Expert Normalized
Expert- Normalized
SAC-Nommlized
1.5
0.8
1.2
0.6
0.9
0.4
Diftusion
83
UNIVR
МРО
SAC
TD3
PPO
0.2
UNIVE
IDQL
JVR
IQL.
0.2
RLPD
CalQLL
JQL
AWAC
Figure 1: Summary of results. Aggregate mean performance across six common RL benchmarks and 23 environments with diverse characteristics (e.g., observation and action spaces, task types, and offline data compositions). Error bars indicate the 95% stratified bootstrap confidence interval. UNIVR achieves competitive performance compared state-of-the-art baselines online, offline, and offline-to-online settings.
"Equal contribution. Correspondence haoran xatrrenxxxtexxasss.e edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Reinforcement learning (RL) has achieved impressive results across a wide range of domains, from playing complex games like Go (Mnih et al., 2013; Silver et al., 2017) to robotic manipulation tasks (Levine et al., 2016). However, the practical deployment of RL demands adaptability to diverse learning scenarios, such as online, offline, and offline-to-onii paradigm, agents interact continuously with the environment, gaining experience and improving policy performance via tril--ndderror approaches such as volicy-gradient methods (Schulman et al., 2015, 2017) and actor-critic algorithms (Lillicrap et al., 2016; Fujimoto et al., 2018; Haarnoja et al., 2018). However, in safety-critical or high-cost domains, such healthcare (Gottesman al., 2018) and industrial control (Zhan et al., 2022, 2025), direct online exploration can be prohibitively risky or expensive, necessitating offline RL approaches that leverage pre-collected datasets without further environment interaction. Recent advances in offline RL have introduced conservative algorithms (Ku- mar al., 2020; Fujimoto et al., 2019) and regularization-basee 2023) designed specifically to mitigate extrapolation errors arising from dataset distribution shifts. Moreover, many practical applications adopt an offline-to-onllee pipeline, initially leveraging offline datasets for safe and effective policy initialization. followed by online fine-tuning to explore more high-quality data to overcome the suboptimality of offline data (Nair et al., 2020; Lee et al., 2021b).
settings. In the classical online RL
methods (Wu et al., 2019; Xu et al.,
Existing RI. algorithms are typically specialized for just one setting. For example, online algorithms (e.g., PPO (Schulman et al., 2017), SAC (Haarnoja et al., 2018)) struggle with offline data due to distribution shift. Offline methods (e.g., BCQ (Fujimoto et al., 2019), CQL (Kumar et al., 2020), IVR (Xu et al., 2023)) tend be overly conservative or generalize poorly when given online access. Offline-to-online methods (e.g., Off2On (Lee et al., 202 Cal-QL (Nakamoto et al., 2023)) often require special design choices, making them difficult to generalize to other settings. Designing different algorithms for each setting separately greatly limits the scalability and widespread adoption of RL in real-world applications. Therefore, a critical research question emerges:
Can we design scalable RL. frameeokorkk that unifies online, offline, and offline-to--nne to-nline settings?
A Unified Reinforcement Learning Framework. A desirable unified framework should fulfill several crucial properties: (1) Adaptability, seamlessly adapt to different settings, including online, offline, and offline-to-online, without changing the learning objective; (2) Sample efficiency, exhibit high sample efficiency, minimizing required interactions with the environment; and (3) Scalability, scale with growing volumes of data and learn effectively in large and complex environments. We consider the constraint optimization problem with a reference policy p.:
Uni-RL
π
arg max E (s.a)aa [r(),a) s.t. Eg [Dy (1 (T(+)))]l(((ss) c, (I)
where (plla) = Ex~gllll If( ((())) I is the f f-divergence (Boyd et al., 2004). By converting the con- straint problem into an unconstrained, regularized one, we get learning objective that imposes a policy-level value regularization by adding the f-divergence regularization term the reward (Geist et al., 2019). This learning objective generalizes the Implicit Value Regularization (IVR) frame- work (Xu al., 2023) from the dataset behavior constraint to the constraint w.r.t. the reference policy μ, yielding Unified RL (Uni-RL), unified framework for general settings: For offline RL, μ is set to be the behavior policy TD of the dataset. For online RL, we set μ to be the target policy which is periodically or softly updated towards the current policy TD the offline pre-training stage while changing to म in the online stage. This framework fully off-policy: offline RI can be considered as a one-step version of online RI. (Brandfonbrener et al., 2021), and offline-to-online RL uses offline data to provide a good initialization for online RL.
For offline-to-online RL, μu is set to be
One issue with Eq. (1) is the reliance on the probability distributions of π and μ. which limits the scalability of using advanced generative models such as diffusion and flow matching models (Song et al., 2020; Lipman et al., 2022) as the policy. However, we show later that by using techniques from duality (Xu et al., 2023; Sikchi et al., 2023), we can get the closed-form solution of the optimal policy n², and the value function of π" can be learned "in-sample" (using samples from the reference policy). By doing so, Uni-RL provides an implicit trust-region style update through the value function, and this implicit update results in an off-policy learning scheme that previous trust-region based methods could not achieve (Schulman et al., 2015, 2017). The implicit trust-region update not only increases sample-efficieng: but also solves the over-conservative issue when directly applying IVR in the

---

online setting. Furthermore, it enables smooth dataset constraint relaxation in offline-to-nling RL, preventing early performance drops (Li et al., 2023). The in-sample learning scheme in Uni-RL also isolates the process of value learning and policy extraction, offering better learning stability. We further introduce a unified policy extraction objective that estimates in-sample policy gradient using only actions from the reference policy. This design scales successfully to more powerful policy classes beyond Gaussian distributions, and theoretically guarantees less value estimation error and larger performance improvement over the reference policy.
We evaluate Uni-RL on 6 widely used RL benchmarks and 23 environments across online, offline, and offline-to-one to-online settings, achieving superior or competitive performance against state-of-the art domain- domain-specifie baselines, In online RL, Uni-RL achieves higher sample efficiency than both off-policy methods without trust-region updates and on-policy methods with trust-region updates. In offline RI., Uni-RI. retains the benefits of in-sample learning while outperforming IVR through better policy extraction. In affline-to-onlng RI., Uni-RI. beats both constraint- based methods and unconstrained approaches by effectively balancing stability and adaptability.
2 Preliminaries
Reinforcement Learning We consider the RL problem presented as Markov Decision Process (MDP) (Sutton et al., 1998), which is specified by a tuple M = A, P, ,,,,, r, 7)- Here S and A are state and action space, P((s'ss, a) and do denote transition dynamics and initial state distribution, r(s, a) and y represent reward function and discount factor, respectively. The goal of RI. to find a policy œ (als) which maximizes expected return J(r) = F. (E> 76 r(s,, a.)]. Offline RL considers the setting where interaction with the environment is prohibited, and one needs to learn the optimal x from static replay buffer D {s;, aq s i=1-
f We also refer to D as the online replay buffer that is updated by filling in new transitions in the online or offline-to-online setting.
Value functions and visitation distributions Let V : :S S R and Q SS R be the state and state-action value function of π, where (T) E, 1E T az)]s0 s] and Q" a) E, IS oo 7"r(st, a3) = 8, a0 a]. The visitation distribution d" is defined as d= (s. = (1 7) Er= 7 Pr (st = s, = a so do. Vt, a π (st), $(41 P (st, at)), which measures how likely π is to encounter (s, when interacting with the environment, averaging over time via 7 discounting. Let V*, V* and d' denote the value functions and visitation distribution corresponding to the regularized optimal policy π². We denote the empirical visitation distribution of 'D as dd and the empirical behavior policy of 'D TD. which represents the conditional distribution p(als) observed in the dataset. Let T be the Bellman operator with policy æ such that (T Q)(s, a) r(s, a) + ". [Q(s', a²)] and (T* == Eann [r(s, a) + +B [V(s)]]..
3 Uni-RL: Unified RL via Implicit Value Regularization
We give a detailed introduction of Uni-RL in this section. We begin by recalling how the dual form of Eq. (1) provides implicit value regularization, which unified, in-sample value learning objective. We then show a naïve extension of IVR to the online setting suffers from suboptimality, and address it with an iteratively improved behavior policy, resulting in an implicit trust-region style update. Finally, we propose a unified policy extraction method that is scalable and versatile to use across different policy classes. We theoretically prove that it enjoys lower value estimation error and larger performance improvement over the reference policy than previous policy extraction methods.
3.1 Towards Unified Value Learning
Implicit Value Regularization.
Note that the Lagrangian relaxation of Eq. (1) is equal to
(((r(s)) (r(s,, a. 1=0
p(a, )
IVR arg max E.
where g(x) = /xx)/ is differentiable and satisfies g(1) = O. Eq. (2) can be thought of as solving a
behavior-regularized MDP problem with a modified reward function (Vieillard al., 2020; Xu et al.,
3

---

2023). In this cehavior-regularized MDP, the Bellman operator is changed to T7 such that (7F) = r(s, a) + +Bg s.a [V(s))
T(als)
(s)) = Besa Q(s.a) a)
((a
Compared with the original Bellman operator T", 77' is actually applying a value regularization to the Q-function. This transfers the greedy- policy n²" softened max (depending on f) over the reference policy ya, which enables a scalable in-sample learning scheme".
Lemma 1 (Results in IVR). Using duality, the optimal value function Q" and V can be solved by min EanDamy ((5)) + a. [Nv((Q(s, a) Vo))
min E ((sas)) + (11) Q(s,
(4)
where livR exp(1) if Dy is the KL divergence and 1(r > 22² x + 1) if D, is the x² divergence. Note that Eq. (3) only uses samples from the reference policy distribution, without needing the knowledge of r[ass) and u(als). This makes the usage of advanced generative models possible in the online setting, and avoids the need to use additional models to fit a dataset behavior distribution in the offline setting. We can recover IVR by setting μ be TD, where offine actions serve as existing samples from TD.
Does IVR work in the online setting? We first examine whether IVR can be naturally extended to the online setting. In online IVR, the offline dataset will the replay buffer collected so far. Unlike the offline setting, the replay buffer is collected by the learned policy that is periodically updated, which means it may contain a large portion of subopti- mal or random data previously collected. Learning from highly suboptimal dataset is known to be hard offline RL due to the tendency to anchor the learned policy to the dataset behavior policy caused by regularization (Hong et al., 2023; Xu et al., 2025). In online IVR, the optimality of the regularized policy also highly depends on the re- play buffer, and we need to break the regularization barrier to get the optimal value function and policy.
IVR ( mp)
IVR (μ== f
Weighted BC
MaxQ
2000
2000
2010
2000
1000
1000
Weighted BC
MaxQ
value
200
200-
0.3M 0.24 0.39 0AM 0.5 Gradient Steps
0.15 0.2 0.3% 0.4M 2.3M
Gredient Steps
Figure 2: Using an iteratively refined behavior pol- icy improves online IVR across different policy extraction methods.
One way to do so is by filtering the offline dataset iteratively according to the learned optimal policy at each iteration, and gradually optimizing towards the optimal policy using the filtered dataset, as done in Xu et al. (2025). However, this dataset filtering process is costly in the online setting. We thus propose a more lightweight option, where instead of iteratively refining the dataset, we maintain an iteratively refined reference policy म. which is updated towards the current policy π and improves beyond TD. This yields the learning objective in the online or offlnn--o-online setting.
min En V
(s) + fivR (Q(s,a) - ((s)) /a) with 7 + Au + (1- (1 (5)
where À denotes the soft updating frequency, and if X previous policy T at iteration A: Using से as the reference policy gives a trust-region style update with an improved μ4, solving the over-constrained problem of directly applying IVR in the online setting. Figure 2 illustrates this, Uni-RL achieves higher Q values and better sample efficiency than IVR across different policy extraction methods. Since the trust-region update is implicitly imposed through the value learning, the algorithm remains off-policy. This improves sample-effiiincy over previous trust-region methods, which are all on-policy (Schulman et al., 2015, 2017), and exhibits less value overestimation and more stability compared to existing off-policy methods (Fujimoto et al., 2018; Haarnoja et al., 2018; Abdolmaleki et al., (b).. Furthermore, it enables smooth dataset constraint relaxation in offline-to-online RL, preventing the finetuned policy from early performance dropping (Nair et al., 2020; Nakamoto et al., 2023; Li et al., 2023).
this amounts assign के with the
2 Here in sample means samples from the reference policy μ.

---

3.2 Towards Unified Policy Extraction
After introducing the unified value learning objective and demonstrating how Uni-RL addresses issues with IVR in the online setting, we now focus on providing a unified policy extraction scheme to effectively extract the best policy from the value functions learned with Uni-RL
Note that in IVR (Eq. (2)), we have a closed-form solution for the ratio of the optimal regularized policy VंvR to the reference policy μ (Xu et al., 2023), which can be expressed
WivR (s, a) VivR (als) max (0. (f.( V
(6)
({ass) s)
Previous works try extract this policy by using either Forward KL Reverse KL divergence. (1) DKL (*ivel(=) + weighted BC
π arg max EewDDarc [ivR (s, (8 logg r(a|s)]-
Using Forward KI. divergence tends to be mode-covering, resulting in weighted behavior cloning style loss where action is sampled from the reference policy. Although it queries the Q function with only in-sample actions (since Q is trained using actions from μ)), it prefers to cover all modes of the target distribution, including those with low probability mass (Park al., 2024; Xu al., 2025).
(2) DKL (π][ + MaxQ+BC
n arg max BawD,am log wivR (s, a) + log u(als) - log ( (T]ss)].
Using Reverse KL divergence gives a MaxQ+BC style loss (Tomar al., 2020; Fujimoto and Gu, 2021; Mao et al., 2024b). Reverse KL divergence is mode-seeking, however, actions are sampled from the policy æ. which are potentially out-of-distribution and causes over-estimation errors to the Q function. Futhermore, this loss needs an explicit u(a[s), which might not be available in the offline setting, and Er [log (a|s) can be hard to estimate when ı parameterized with generative models such as diffusion or flow matching models (Kong et al., 2023).
In summary, weighted BC has lower overestimation error and scales well across different settings and policy classes, while remaining mode-covering, In contrast, MaxQ+BC is mode-secking but suffers from inaccurate gradient estimation and limited scalability. Can we design policy extraction method that shares the best of both worlds?
Policy extraction via In-sample Policy Gradient. Our key intuition is that still try maintain the weighted BC style loss due to its stability and scalability, but we want to incorporate the first-order gradient information from policy gradient into the zero-order gradient induced by the weighted BC loss. To achieve that, we introduce In-sample Policy Gradient (InPG) where we ignore the original IVR weights wive (s, a) and instead think of those weights learnable variables. In-sample PG projects the MaxQ gradient into the weighted BC gradient by learning the new optimal weights "hPG (s, a) such that
WinPe (s, a) arg min BawD[(Ger GBC(W))² 2220
arg
where GPG Vo Eawn, [Q(s, and Gne(w) = Ea-p [w[s, Vo log mo(a|s)].
GBe(w) is the gradient vector of using weighted BC, by changing the weight value on different (s, a). and we want to find a non-negative w metric to make the resulting vector stay close possible to the MaxQ gradient vector. Note that Eq. a constrained quadratic programming (QP) problem (Nocedal and Wright, 2006). Rather than solving exactly using QP solvers (Stellato et al., 2020), parameterize optimize it by minimizing the L2 loss in (7) via gradient descent. We clip the w at 0 during policy extraction ensure weight non-negativity.
modify the direction of this vector
neural network and
1nPG arg max B.-DDar W (s, a) log [[ass)]
(8)
Using "ing (8, a) as the BC weight enables better utilization of the Q function since using wivR (s, a) cannot guarantee the policy is maximizing the Q function. It also makes the new weighted BC loss
5

---

more mode-seeking, enabling policy extraction under support constraint since the BC actions are always within the support of the reference policy.
Theoretically, we find InPG could potentially enable a larger performance improvement than TiVR by increasing [rr. In fact, InPG is optimizing towards the constraint learning objective from (q.(1) while oivR is the solution of Eq. (2), the relaxed Lagrangian of Eq. (I).
Theorem 1. Define Wmin min, w(x), Wmax maxx w(x) and Z = Ev [ E [Wmin; Wmax)
the f-divergence between FiPG and is bounded by
f( 121 S DD(mmralll)) 5( Ummxx
Wmin
Z Theorem 2. There exists range of Wmin and Wmax that satisfies J (TTAPG) 2 (TÍvR)
We give the proof of these two theorems in Appendix A. These two theorems reveal that using in- in-sample policy gradient is actually finding the best policy (i.e., maximizes the Q function) within a fixed constraint (depends on Wmin and Wmax) rather than treating that constraint more aligned with the original optimization problem Eq. (1). This gives more freedom and allows for better policy extraction. The following theorem further demonstrates the necessity of applying the in-sample projection to the policy gradient, since directly using policy gradient has no improvement guarantee over the reference policy (also known to suffer from overestimation error in offline RL).
penalty, which is
Theorem Define मं. arg max, Ee-D,a (s a)], there is no guarantee that J (TTa 2 J(k).
We use the mazzee dataset in D4RL (Fu et al., 2020) as an example to demonstrate the effect of using InPG. The Maze2D domain is a navigation task in which a 2D agent must reach a fixed goal location as quickly as
Weighted BC
InPG
Veleciy
possible; the action corresponds to the velocity along the (1, v) axes. We visualize actions sampled from policies learned with IVR (wivR(s, a)) and Uni-RL (Winea (8, a)) in Figure 3. As shown, IVR tends to produce actions that are centered around the offline action distribution due to its mode-covering nature. In contrast, Uni-RL generates better actions that are near the boundaries of the action space by effectively using the value function. Intuitively, weighted behavior cloning can also exhibit mode-seeking behavior if appropriate weights are assigned to different actions. Notably, in-sample policy gradient does not require sensitive hyperparameter tuning like MaxQ+BC (Tarasov et al., 2024a; Park al., 2025); it only requires setting a lower and upper weight threshold and we use the same value across all experiments.
X Velccity
Velecty
Figure 3: InPG discovers better actions from (grey points) than weighted BC.
Scalable to advanced generative models. In- sample policy gradient can easily scale to the usage of generative models like diffusion and flow matching models. Maximizing the log- likelihood objective in InPG is equivalent to min- imizing the matching loss of different generative
Algorithm 1 Unified Implicit Value Regularization
Require: D, a.
1: Initialize Q, V, n, क
2: > Unified value and policy learning
3: procedure UNI-RL (T, 14, D)
4: Sample transitions (s, a, a, r') D
models (Song et al., 2021). For flow matching models, we have the following matching loss
5: Update V by (3) with ~ p,. Q and D
6: Update Q by Eq.(4) with and
Litow( (0) (Lipman et al., 2022).
7: Update π by Eq,(7,8) with a ~ μ4, Q and D
an D.a [llva(t, s,,)) (x (I1==1))11,
8: > Offline training (offline and cfffine-to-onliee
9: for t 1,2, . do
(0.10)
10: UNI-RL (R, D)
(nin((((011)
11: Online finetuning (online and offline-to-onlin)
where vo 8, is state-and-time dependent
12: for f = 1,2, ,N do
vector field with parameter 0. For the use of dif-
13: Explore using and append (s, (s,,,r,))) to D
fusion models, we have the following matching
14:
UNI-RL (T, t,, D)
loss Diffusiio (0) based on Ho et al. (2020).
15: Update n by Eq.
EewD,a--p.ee (.., [|le Ile-aa Eg (Vāça vI āye, s,t) [12].
(9)
In summary, InPG provides unified policy extraction method based on the weighted BC objective, which uses zero-order gradient while being mode-seeking to maximize Q values. InPG avoids value overestimation errors and scales to implicit behavior cloning through generative modeling, bringing better generalization. The full Uni-RI. algorithm combines the unified policy extraction module with the unified value learning module. The pscudocode of Uni-RL is provided in Algorithm 1.

---

4 Prior Work
Online RL. Model-free online RL algorithms can be categorized into on-policy and off-policy methods. While on-policy methods only use experience from the current policy to perform up- dates (Kakade, 2001; Mnih et al., 2016), off-policy methods, in general, can utilize experience from any arbitrary policy (Watkins and Dayan, 1992; Haarnoja et al., 2018; Fujimoto et al., 2018). To stabilize training, previous methods (Schulman et al., 2015, 2017) add an explicit trust-region policy constraint that limits the derivation from the previous policy. However, this results in on-policy learning. while Uni-RL uses an implicit value regularization that is off-policy, Some works also use weighted BC style loss (Abdolmaleki al., 2018b,a, Oh al., 2018; Tomar al., 2020) in the policy extraction step. However, they don't impose regularization in the value learning. Their value learning and policy learning are coupled, causing the overestimation issue in the offiine setting. Note that Uni-RL decouples the value and policy learning owing to the implicit value regularization.
Offline RL. To tackle the distributional shift problem, most model-free offline RL methods augment existing off-policy methods with a dataset behavior regularization term. One class of methods imposes behavior regularization explicitly as a divergence penalty (Wu et al., 2019; Kumar et al., 2019; Fujimoto and Gu, 2021) or intervene in the value learning to encourage staying near the behavioral distribution and being pessimistic about unknown state-action pairs (Nachum et al., 2019; Kumar et al., 2020; Kostrikov et al., 202 Xu et al., 2022; Wu et al., 2021; An et al., 2021). The other class of methods implicitly impose the behavior regularization through weighted behavior cloning (Kostrikov et al., 10; Xu et al., 2023), which filters useful actions to perform behavior cloning based on how advantageous they are. Compared with the first class, Uni-RL is inherently more stable due to its imitation-style policy update. Compared with the second class, Uni-RL is better at utilizing the value function to avoid suboptimal policy extraction.
Offline-to-online RL. Offine-to-online RL aims to overcome the suboptimality of pure offine learning by collecting more high-quality data with online fine-tuning (Lee et al., 2022; Zhang et al., 2023; Li al., 2023). To prevent performance drop caused by distribution shift from the offline to online stage, previous methods impose regularization to stay close to the offline data or the offline pretrained policy during online fine-tuning Nair et al. (2020); Kostrikov et al. (2021b); Lee al. (2022); Nakamoto et al. (2023); Zhao et al. (2022). However, these methods are wver-conservative since the offline data could be highly suboptimal. Uni-RL solves this issue by enforcing an iteratively refined behavior policy initialized from an
policy, allowing us to gradually release the dataset constraint to continue improving throughout training. There are also some works that try to avoid the performance drop issue by ignoring the offline stage and learning from scratch using offline data (Song et al., 2022; Ball et al., 2023). However, these methods are less sample efficient since they lack the usage of pretrained value functions to perform effective exploration.
Other prior works related to unified RL. There are several other works that try to provide unification for general RL settings. Uni-O4 (Lei et al., 2023) provides unification of offline RL and offline-to-online RL based on on-policy policy gradient, but suffers from sample inefficiency compared to Uni-RL, which is off-policy. Policy Agnostic RL (Mark et al., 2024) provides unification for policy extraction methods, whereas Uni-RL also achieves principled unification of value learning for general RL settings. Note that although some pise-based methods (Lee et al., 202 Sikchi et al.. 2023; Mao et al., 2024a) have a similar optimization objective with Uni-RL, they can't 't be unified both the online and offline settings. Specifically, if we apply DICE-based methods to the online setting, sampling from the stationary distribution d' d' a) of the reference policy is needed. This is intractable in the online setting, making DICE-basee methods only work for the offline setting.
5 Experiments
Online RL. We first test Uni-RL in the online setting. We choose environments from OpenAI Gym (Brockman et al., 2016), DeepMind Control Suite (Tassa et al., 2018), and PyBullet (Coumans and Bai, 2016), representing a large and diverse set of domains based on Box2D (Catto, 201 1), MuloCo (Todorov et al., 2012) and Bullet (Coumans et al., 2010) physics engines. To demonstrate the effectiveness of our method, we compare Uni-RL with several state-of-the-art model-free online RL algorithms including TD3 (Fujimoto et al., 2019), SAC (Haarnoja et al., 2018), PPO (Schulman
7

---

Uni-IVR
MPO
PPO
SAC
TD3
AntBullecEnv-vo
BipedalWalkerry
finger-turn hard
HaitCheetah-ws
3,000
1,000
750
2,000
200
Score
1,000-
0-
0-
0.1M
0.2M
0.3M 0.4M 0.5M
0.1M D. 0.2M D.IM 0.4M 0.5M
0.1M 0.2M O, 0. 0.4M 0.5M
0.1M 0. 0.29 D.IM 0.4M 0.5M
Hoppervv3
Humanaid v3
quadruped walk
Walker2diva
4,000
6, 0,00
3,000
750
Score
4, 0.000
1,000
2,000
0.1M 0.2M 0.5M
0.3M 0.4M 0.5M Steps
0.1M
0.3M 0.4M Steps
Steps
Steps
Figure 4: Results in online RL. Learning curves are plotted across 5 seeds with smoothing window of 5000. Shading represents one standard deviation.
Table 1: Results in offline RL. Scores are averaged over the final 10 evaluations across 5 seeds with standard deviation reported, we highlight the best score in integer-level.
10%BC
TD3-BC
CQL
IQL
Dilfusion QL
IDQL
Uni-RL (Gaussian)
(Diffusion)
Uni-RL
hallchectal-mm
42.5
48.3
44.0
0088
47.4 10.2
48.3:0.2
51.1 1:.5
$1.0
49.4:0.2
58.0 1:.6
hopper-m
=2.1
66.3
15.7
75.51334
90.5 1:4.6
1111
101.3
walker2d-m
75.0
83.7
72.5 =0.8
72.5 +8.7
84.2:t4.6
87.0 +0.9
82.5
89.3=0.3
92.3 =0.1
halfcheetah-mmrr
40.6
44.6
45.5 0.5
44.2 1.2
44.8-10.7
47,8 1.0.3
45.9
45.3-003
48.4 10.9
hopper-m-r
75.9
60.9
95.0 =6.4
95.2 1:866
99.71:3.3
101.3 10.6
92.1
101.1 22.7
101.3 12.1
walker2d-m-r
62.5
81.8
77.2=5.5
76.1 +7.3
81,2++3.8
11.
85.1
86 36.6=1111
90.8+
halfehertallmm-e
92.9
90.7
90.7=4433
86.74:3.
94.0=+04
96.8 +0.3
95.9
94.2:0066
97.3 00.6
hopper-m-e
110.9
98.0
105.4+688
101.5 +7.3
111.8 422.2
111.1 +-..3
108.6
111.0 0.6
111.2 +0.3
walker2d-m-e
109.0
110.1
109.6:20.7
110.61::00
110.0:0.88
110.1 =0.3
112.7
110.8:1022
114.1:0055
antmaze-u
62.8
78.6
84.8 L..3
85.5 11.99
92.2.114
93.4 1.34
94.0
94.1.11.6
98.1 11.8
antmaze-ud
43.4
66.7
80.4
11.4
untmazo-m-p
65.2
72.2 #5.3
80.21:3.7
86.0=2.
13.1
antmaze-m-d
14.0=11.7
+:.0
79.1+-4.2
78.6 10.3
84.8
82.7=3344
89.7 +288
ap
38.41123
39.6 1.4.5
53.2
46.4 1.8.3
63.5
59.912.9
68.6
13.66
antmaze-l.d
6.0
0.0
31.6
47.5
14..4
52.3-15.2
56.6 1:7.6
67.9
60.2 3.8
69.0 14,55
et al., 2017) and MPO (Abdolmaleki et al., 2018b), All baselines use a Gaussian policy, so we also Gaussian policy with Uni-RL for fair comparison. The solid curve represents the average return, and the transparent shaded region represents the standard deviation. Each experiment was conducted over 5e5 training steps. According the learning curves in Figure 2, Uni-RL achieves state-of-the-ad performance and sample efficiency compared to the other four bascline algorithms, especially challenging tasks in DMControl (e.g., finger-ture hard and quadruped
Offline RL. In the offline setting, we evaluate Uni-RL on the D4RL benchmark (Fu et al., 2020) and compare it with several state-of-the-art algorithms. For the evaluation tasks, we select MuJoCo locomotion tasks and AntMaze navigation tasks which require both locomotion and navigation. While MuJoCo tasks are popular in offline RL, AntMaze tasks are more challenging due to their stronger need for selecting optimal parts of different trajectories to perform stitching. For baseline algorithms, we selected saate-of-th-ase methods not only from traditional methods that use a Gaussian policy but also methods that use diffusion models.
baselines include 10%BC (Chen et al., 2021) !), BCQ (Fujimoto et al., 2018), TD3+BC (Fujimoto and Gu, 2021). CQL (Kumar et al., 2020), IQL (Kostrikov et al., 2021a) and IVR (Xu al., 2023). Diffusion-policy. www---oolllcysssseeeee based baselines include Diffusion-Qu -L (Wang et al., 2023) and IDQL Glansen-Estruce et al., 2023).
caaussian-policy--ee
We implement Uni-RL with both Gaussian policy and Diffusion policy as in Eq. The results in Table show that Uni-RL with Gaussian policy already matches or outperforms most baseline algorithms, especially MuJoCo medium, medium-replay datasets, and AntMaze datasets. The use

---

Uni IVR
Cal QL
AWAC
RLPD
IQL
pan-cloned-v1
dcor-doned-v3
hammer-clonddv4
discate-donediv
100
120
100
Normaliad Score
30
50
60-
40
40
20
20
0.3M
0.3M
5,3
3,5M
33
Step
Step
antmaace modiiu play
antmaze medium diverse
antmace large play
antmaaee large diverse
150
Normlized Seare
80
50
50
60-
10-
40
40
0.3M
0.25
0.3N
0.4K
0.5M
0
0.3M
0.2M
0.3M
0.4N
0.3M
0:3M
0.2M 0:5M
0.4M
0.3M
0
0:3M
2:24
0.30
2:4M
3.3M
Steps
Steps
Steps
Steps
Figure 5: Results in offline-to-online RL. Learning Curves is plotted after 250k offline pretraining (5 seeds). Shading represents one standard deviation.
of diffusion models in Uni-RI. further enhances its performance on tasks with multi-modal datasets, demonstrating the scalability of Uni-RL. Additionally, the consistently better performance of Uni-RL over IVR demonstrates the benefit of policy extraction using in-sample policy gradient.
Offline-to-online RL. In the offline-to-online setting, we conduct extensive experiments on AntMaze and Adroit tasks with D4RL datasets to demonstrate the stable, optimal policy learn- ing and adaptability of Uni-RL We compare Uni-RL with the following baselines: (i) AWAC (Nair al., 2020): an offline-to-online method that learns the finetuning policy using AWR-style (Peng et al., 2019) policy loss. (ii) IQL (Kostrikov et al., 2021b): a SOTA offline RL approach based on weighted BC that can directly transfer to online finetuning. (iii) Cal-QL (Nakamoto et al., 2023): a SOTA offline-to-onling approach specially designed based on CQL offline training. (iv) RLPD (Ball et al., 2023); a method that uses offline data to accelerate online training, it ignores the offline pretraining stage and learns from scratch. Note that we remove the high aopdate-ooddtte trick in RLPD (i.e., adding layer normalization and ensembles to the Q-function) for fair comparasion.
Figure 5 shows that existing constraint-basel approaches (IQL, AWAC) in most cases only marginally improve the offline pretrained policy, due the over-conservatism introduced by the constraint w.r.t. the offline dataset. This is especially pronounced when the offline dataset or pretrained policy is subbpi-luboptimal such as in Adroit and Antmaze tasks. In contrast, Uni-RL enjoys stable initial finetuning and superior final performance owing to the iteratively refined policy regularization. Cal-QL limited to CQL, making it hard yield reasonable performance when the tasks are too difficult for CQL to obtain good results (e.g., Adroit and Antmaze- tasks). While RLPD achieves appealing results in some tasks, the sample efficiency greatly limited in tasks with diverse or good offline data (e.g., Antmaze tasks).
6 Limitations and Future Work
In this paper, we propose Uni-RL, scalable framework that unifies different reinforcement learning settings, including online RL, offline RL, and offline-to-online RL. Uni-RL builds on the Implicit Value Regularization framework but generalizes the offline data constraint to a reference policy constraint, resulting in unified value learning and policy extraction objectives based on in-sample learning. Uni-RL is simple, effective, and scalable, and achieves superior performance across diverse RL settings. Uni-RL only considers the model-free setting and future directions include incorporating Uni-RL with model-based RL methods, and extending it to the L.M+RL setting. We believe that Uni-RL represents concrete step toward building general and scalable off off-policy RL algorithms.

---

Acknowledgement
We thank members from MIDI lab for discussions on the method and feedback on the early draft of the paper. This work is partially supported by NSF 2340651, NSF 2402650, DARPA HR001 12490431, and ARO -911NT-24--0019,,, This research used the computational cluster resource provided by the Texas Advanced Computing Center at UT Austin.
References
Abbas Abdolmaleki. Jost Tobias Springenberg, Jonas Degrave, Steven Bohez, Yuval Tassa, Belov, Nicolas Heess, and Martin Riedmiller. Relative entropy regularized policy iteration. arXiv preprint arXiv: 1812.02256. 20.
Abbas Abdolmaleki, Jost Tobias Springenberg. Yuval Tassa, Rémi Munos, Nicolas Heess, and Martin A. Riedmiller. Maximum a posteriori policy optimisation. In Proc. of ICLR, 2018b.
Anurag Ajay. Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional decision decision-making? making? arXiv preprint vrXiv:2211.56577 2022.
generative modeling you need
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-base offline reinforcement learning with diversified q-ensemble. Proc. of NewrlPS, 2021.
Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine Efficient online reinforcement learning with offline data, In International Conference on Machine Learning, pages 1577-1594, PMLR, 2023.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
David Brandfonbrener, William Whitney, Rajesh Ranganath, and Joan Bruna. Offline without off- pooippliy evaluation. Proc. of NewrIPS, 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. ArXiv preprint, 2016.
Erin Catto. Box2d: A 2d physics engine for games. URL: http://www. box2d. org, 2011.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Proc. of NewrIPS, 2021.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning, 2016.
Erwin Coumans et al. Bullet physics engine. Open Source Software: http://bulletphysiis. org, 133)844, 2010.
Justin Fu, Aviral Kumar. Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforeement learning. ArXiv preprins, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. ArXiv preprini, 2021.
Scott Fujimoto, Herke van Hoof, David Meger. Addressing function approximation error actor-critic methods. In Proc. of ICML, pages 1582-1591, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proc. of ICML, pages 2052-2062, 2019.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In Proc. of ICMI,, pages 21 60-0166,, 2019.
Omer Gottesman, Fredrik Johansson, Joshua Meier, Jack Dent, Donghun Lec, Srivatsan Srinivasan, Liny- ing Zhang, Yi Ding, David Wihl, Xuefeng Peng, et al. Evaluating reinforeement learning algorithms in observational health settings. arXiv preprint frXiv:1805. 1805.12980,, 2018
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off Offlpo maximum entropy deep reinforcement learning with stochastic actor. In Proc. of ICML, pages 1856-1865. 2018.
10

---

Philippe Hansen-Estruch, Ilya Kostrikov. Michael Janner. Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning actor-critic method with diffusion policies. arXiv preprint wrXiv:2304.10573 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems. 33:6840 6851, 2020.
Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momehev, Danila Sinopalnikov, Piotr Stanczyy,, Sabela Ramos, Anton Raichuk, Damien Vincent, Léonard Hussenot, Robert Dadashi, Gabriel Dulac-Arnold, Manu Orsini, Alexis Jacq, Johan Ferret, Nino Vieillard, Seyed Kamyar Seyed Ghasemipour, Sertan Girgin, Olivier Pielquin, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang. Kate Baumli, Sarah Henderson, Abe Friesen, Ruba Haroun, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Paine, Srivatsan Srinivasan, Andrew Cowie, Ziyu Wang. Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv: 00979, 2020.
Zhang-Wa Wei Hong, Pulkit Agrawal, Rémi Tachet des Combes, and Romain Laroche. Harnessing mixed offline reinforcement learning datasets via trajectory weighting. arXiv preprint arXiv:2306. / 13085, 2023.
Sham M. Kakade. A natural policy gradient. In Proc. of NeurlPS, pages 1531-1538, 2001.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of ICLR, 2015.
Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. nntematiin-theorelie diffusion. arXiv preprint 2.Xii:2302..3379,, 2023.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. Proc. of ICML, pages 5774-5783, 2021a.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. ArXiv preprint, 2021b.
Aviral Kumar. Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Proc. NeurlPS, pages 11761-11771,
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Proc. of NeurlPS, 2020.
Jongmin Lee, Wonscok Jeon, Byungjun Lec, Joelle Pincau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In Proc. of ICML, pages 6120-6130, 20
Seunghyun Lee, Younggyo Sea, Kimin Lee, P. Abbeel, and Jinwoo Shin. Offinneto-onllne reinforcement learning via balanced replay and pessimistic g-ensemble. Conference on Robot Learning, 2021b.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702-1712. PMLR, 2022.
Kun Lei, Zhengmao He, Chenhao Lu, Kaizhe Hu, Yang Gao, and Huazhe Xu. Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization. arXiv preprint arXiv:231 1.03351, 2023.
Sergey Levine, Chelsea Fian, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal Machine Learning Research, 2016.
Jianxiong Li. Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. Proto: Iterative policy regularized offline-to-online reinforcement learning. arXiv preprint arXiv:2305 2023.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement lcarning. Proc. of ICLR, 2016.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:22
2022.
Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhan. Odice: Revealing the mystery of distribution correction estimation via orthogonal-gradiinn update. arXiv preprint arXiv:2402.00:: 2024a.
Liyuan Mao, Haoran Xu, Weinan Zhang, Xianyuan Zhan, and Amy Zhang. Diffusion-die dice: In-sample diffusion guidance for offline reinforcement learning. arXiv preprint arXiv:2407.20109, 2024b.
11

---

Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy agnostic rl: Offline and online rl fine-tuning of any class and backbone. arXiv preprint arXiv:2412. 06685, 2024.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Toannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXiv:1312.5602, 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PmLR, 2016.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algacdice: Policy gradient from arbitrary experience. ArXiv preprint, 2019.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. ArXiv preprint. 2020.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36:62244-62269, 2023.
Jorge Nocedal and Stephen Wright. Numerical Optimization. Springer Science & Business Media, 2nd edition, 2006. ISBN 97803.7303031.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International conference on machine learning, pages 3878-3887. PMLR, 2018.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offiin rl? arXiv preprint 24X.v:2406.099329 2024.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning, arXiv preprint arXiv:2502.025338 2025.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weigghed regression: Simple and scalable off-po reinforcement learning. ArXiv preprint, 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proc. of ICML. pages 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv:1007.06347, 2017.
Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Nickum. Dual rl: Unification and new methods for reinforcement and imitation learning. arXiv preprint arXiv:2302.088660 2023.
David Silver, Julian Schrittwieser. Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez. Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 2017.
Yang Song. Jascha Sohl-Dickstedi Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint a.X1::201.11456. 2020.
Yang Song. Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415- 1428, 2021.
Yuda Song, Yifei Zhou, Ayush Sekhari, Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make efficient. arXiv preprint arX1v:2210.06718, 2022.
Bartolomeo Stellato, Goran Banjac, Paul Goulart, Alberto Bemporad, and Stephen Boyd. Osqp: An operator splitting solver for quadratic programs. Mathematical Programming Computation, 12:637-677 2020. doi: 10.1007//12532--266 00179-2.
Richard S Sutton, Andrew G Barto, et Introduction reinforcement learning. MIT press Cambridge, 1998.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36. 2024a.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research- oriented deep offline reinforcement learning library. Advances in Neural Information Processing Systems, 36. 2024b.
12

---

Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez. Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefraneq, al. Deepmind control suite. arXiv preprint 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Proc. of IROS, 2012.
Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. arXiv preprint arXiv:2005. 09814, 2020.
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage average: an analysis kl regularization reinforcement learning. Advances in Neural Information Processing Systems, 33:12163-12174 2020.
Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offline multi-agent reinforcement learning with implicit global-to-loca value regularization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 1992.
Yifan Wu, George Tucker, and Ofir Nachum, Behavior regularized offline reinforcement learning. ArXiv preprint. 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In Proc. of ICML, pages 11319-11326 2021.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8753-8760, 2022.
Haoran Xu. Li Jiang, Jianxiong Li, Zhuoran Yang. Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303.158/ 2023.
Haoran Xu, Shuozhe Li. Harshit Sikchi, Scott Nickum, and Amy Zhang. An optimal discriminator weighted imitation perspective reinforcement learning. arXiv preprint arXiv:2504. 13368, 2025.
Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4680-466 4688, 2022.
Xianyuan Zhan, Xiangyu Zhu, Peng Cheng. Xiao Hu. Ziteng He, Hanfei Geng, Jichao Leng, Huiwen Zheng, Chenhui Liu, Tianshun Hong, et al. Data center cooling system optimization using offline reinforcement learning. International Conference on Learning Representations, 2025.
Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offfine-to ciline-to-onling reinforcement learning. arXiv preprint arXiv:2302. 2023.
Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behavior cloning regularization for stable offfine-to-onee to-online reinforcement learning. arXiv preprint arXiv:2210, 20l.::2222........ 13846, 2022.
13

---

A Proof
Theorem 1. Define Wmin = ming w(x), Wmax max, w(x) and Z Exa [w(x)] E [Wminn Wmnxx)
the f-divergence between Fiao and is bounded by
Wmin
- ( S S( Wmax
f(
Z
2
Proof. Remember that TPPG has the following expression:
u(aals)u"(s,00
iPG (a|s)
Ev' (s
The f-divergence between и and yu is defined as:
(=(x))((x)
(Tine(a)) Em(ess m1x))
(T(nlllAA))
Σ m(a)
(e(x;)
w (2((x)
Eanny [s (s(ut) (w"(x)) ]
Σ m(r)s
Given that (x)) is bounded as:
Wmin I w(xx S Wmaxx Va,
we obtain the following bounds:
Wmin
Earnyy f w((z) S max
Z
Thus, the f f-divergence between TiPG and fu is bounded.
Theorem 2. There exists Wmin and Wmax that satisfies J (TinPG) 2 (TivE).
Proof. Remember that IVR considers a new MDP where the reward is augmented with behavior regularization term (r(s, a) r(s, a) ag(x(a)s) [[a[s))). the (TT) consider here is equal
to the value V* And an ideal algorithm wants to make sure that V*(s) 2 V*(s), ~ D, ie., safe policy improvement over the behavior policy over the replay buffer.
According to IVR,
(s) Eeva ((s.aa) a.g
(T(a(s))
p(aas)
Since TiPG is the optimal solution of Eq. (7), i.e., the optimal behavior cloning weight that maximizes the Q function, we have
J (TTPP) = EEwD.aniine
an ((s, -ag
( p(als)
(Tiro(als))
)
TineG (als))))))) (laass)
> By-Da (Q(s -aBgDDan
9
mive(a)s)
)
भÎve (rive(als)
-EowDaarriaall
a(s,a)- agg
+ aB a.B
u(als)
u(als)
(als)
-aBanD,arti
p(aass)
mivR(al)))
)
Tapa Taner(al/0) (a|s) )}
=((iiR)) + &.BynDaatin
a.. D.awsiier
p(aas)
p(aas)) is)
14

---

To make J (mPG)) 2 J (TivR), one safe condition to satisfy is.
(al))
T "np (a|s)
EgND [Dy (T (-s))(l()))((( Be iia
9
p(als)
TivR (((v((()))
Bgg
g
p((ass)
ENDD ( (-s))((l))ll(((10)
Note that in Theorem we have f < (TirG (4) $ fu /Z) Let f( [Wmax/2 v E, (--ss] will satisfy this condition. In conclusion, the range of Wmin and Wmax to satisfy J inTG ()) J (TivR: p)) is
Wmas
S S (
Note that some weaker conditions could also satisfy J (Tira)) 2 J (vv)). For example, we could choose WiPG be close to wivR such that D (Tipa (-Is)]la(-(ss) Dy (-1s ((p(s)) Also, in the worst case, setting WaaG a) = wivR(e (s, a) we have J (Tira) = (Tiv))
Theorem 3. Define TPG = arg max; Eaw an (Q, (8,a)) a)}. there is no guarantee that (TTG) 2 (nx..
Proof. Since TP is the policy that maximizes EEND, a)), we have
)
(Fpo(a)s))
((TTa) EDNDDanrio
[(s..a) - ag u(als)
Tro(als) p(aas)
2 EErD.ara [Q(s,a)] -aEsnD,anaa
[((als)
rFo(als)
]
=EnDDana a(),a)= ag
((aas))
9
u(als)
(als)
Dy (Tpal (T (ss)))((((((s) ).
J(4) aBEnDanTii
J(n) ABEND
p(als)
This inequality leverages the fact that g(1) = 1 f11 0. Since Dy ( 2 0, there is no guarantee that J(Tpc) 2 J(.).
B Experimental Details
B.1 Online RL experimental details
Environments, Tasks, and Datasets We evaluate Uni-RL on 8 standard continuous control environments from OpenAl Gym, DeepMind Control Suite (DM Control), and PyBullet. These tasks vary in agent morphology, dimensionality, and dynamics complexity, ensuring a comprehensive evaluation of our method.
AntBulle MarlleeEEne Env-vo: It has a 28-dimensional state space consisting of joint positions, velocities. and body orientation features, and an 8-dimensional action space where each action dimension lies in [-1, IJ. The agent is quadruped ant-like robot simulated using the PyBullet physics engine, The task requires learning stable and efficient forward locomotion under noisy and contact-rich dynamics.
BipedalWalker-vss This task has a 24-dimensional state space including hull angle, angular velocity, leg joint positions, and contact sensor readings, and a 4-dimensional continuous action space. The system represents planar biped that must walk across varying terrain. The challenge lies in foot placement, balance control, and adapting to sparse footholds.
f finger-turn -had: The environment has a 12-dimensional state space and a 6-dimensional action space. It features robotic finger that must rotate an object specific target orientation. The dynamics are sensitive, the rewards are sparse, and the task demands fine motor control and long-horizon reasoning. This environment is from the DeepMind Control Suite.
15

---

F
Figure 6: Tasks used in online RL.
HalfCheetah-v33
The task includes a 17 17-dimensional state space representing joint positions and velocities, and 6-dimensional action space. The system models a planar cheetah-like robot that learns to run forward as fast as possible. It is a widely used benchmark for evaluating the stability and efficiency of learned locomotion policies.
Hopper-v3: The environment has --dimensional state space and 3-dimensional action space. It consists of a one-legged robot that must learn to hop forward without falling. The task is sensitive small disturbances and evaluates learning in unstable, underactuated systems.
Humanoid-v3: This is high-dimensional task with 376-dimensional state space and 17- dimensional action space. The agent is a humanoid robot with 21 actuated joints and must learn to walk upright. The complexity of the dynamics and control makes it one of the most challenging continuous control benchmarks.
guadruped-walle The task includes a 78-dimensional state space representing joint and body kinematics and a 12-dimensional action space. The system models quadrupedal robot using the DeepMind Control Suite. The goal is to learn stable walking behavior in a realistic 3D setting with proprioceptive sensing.
Walker2d-vžs v3: The environment has a 17-dimensional state space and 6-dimensional action space. It simulates planar bipedal robot that must learn to walk forward using two legs with multiple joints. This task provides balanced challenge, commonly used to evaluate both stability and learning efficiency.
Methods and Hyperparameters In all tasks, we computed the average mean returns over 10 evaluations every 103 training steps, across different seeds.
For the network, we use 3-layer MLP with 256 hidden units and Adam optimizer (Kingma and Ba, 2015) with learning rate of x 10-3 for both policy and value functions in all tasks. We also use target network with soft update weight 5 x 10-3 for Q-function. We clip the output of the weight function by max(ut (I:), 0) to ensure non-negative BC weight. We use 0-1 I normalization to the weight in each batch and then clip it to [Wmin. Wmax] where we set Wmin to 0.1 and Wmax to 0.9 through all the datasets.
We implemented Uni-RL using PyTorch and ran it on all datasets. We followed the same reporting methods as mentioned earlier. In online experiments, from ACME (Hoffman et al., 2020)3. We use the reported hyperparameters in each paper.
baselines using the implementation
In Uni-RL, we have two hyperparameters frequency We search a over [0.1, 0.5, 1.0, 2.0) and over [0.01, 0.05, 0.1, 0.2] The best value of and X for all environments are listed in Table 3.
regularization weight and reference policy updating
B.2 Offline and Cofine--T-Online RL experimental details
Environments, Tasks, and Datasets In offline and offline-to-online, Uni-RL evaluated on different kinds of datasets from various environments.
hnttps://githbb cactiaub..omm////////////ee/ comp/gooll-ddddepnnna//ceme
16

---

For MuloCo environments, we have the following datasets.
halfcheetah/hoppeee httphetttat/hopperraaeeeeee (medium): Collected by a policy with moderate perfor- mance, typically reaching around one-third of expert returns. These datasets represent structured but suboptimal behavior.
(mediun-replayy: Contains the replay buffer of the mediocre SAC policy. It includes a wide range of off-policy transitions, many of which are suboptimal or noisy.
halfcheetah/ htwphhewwhhhhhhppeeeeeeeaap pooperrwalkerdd-iie m mmedium-expert;; A 50-50 mixture of medium and expert trajectories. These datasets are designed to test whether algorithms can leverage near- optimal data when it partially present.
The AntMaze environments involve quadruped ant navigating through 2D maze using sparse goal-based rewards. The agent has 29-dimensional state space and an 8-dimensional action space, corresponding to joint positions, velocities, and target location encoding. The tasks are particularly challenging due to long-horizon planning and sparse supervision.
antmaze-u (umaze): small maze where the agent must reach a fixed goal location using sparse rewards. The environment is relatively easy due to short trajectories.
antmaze-u-d (umaze-diverse): Similar to umaze, but with broader trajectory diversity collected from random exploration.
antmaze-m m-p (medium- :eedium-play): medium-sized maze where data is collected via play policy. The task is harder due to longer horizons and sparse goal rewards.
untmaze-m-6 (medium-diverse): Features more diverse and noisy behavior than medium-pl play, increasing exploration coverage but decreasing consistency.
antmaze- 1-p (large-play): A large maze with random play data. The agent must navigate long distances, making the task especially difficult under sparse reward signals.
antmaze-1-4 1-d (large-diverse): Similar to sarge-play: but with broader and more varied be- havior. It is one of the most challenging offline datasets due to the size of the environment and variability in data.
The Adroit environments are nigh-dimensional dexterous manipulation tasks based on 24-DoF Shadow Hand. Each environment has 100-dimensional state space that includes joint angles, velocities, and object pose information, and a 24-dimensional continuous action space controlling finger joints. The cloned datasets generated by behavior cloning from expert demonstrations and represent a moderate level task success.
pen-clon man-cloned-vil The task requires rotating a pen to a target orientation using a 24-DoF anthro- pomorphic hand. The dataset is generated by cloning expert demonstrations. It is highly sensitive to precision and coordination.
door-cloned-vl. dexterous hand must unlock and open a door by grasping and rotating the handle. This task involves contact-rich control and precise force application, with cloned demonstrations as the data source.
fammer-cloned-ee The agent must use hammer to drive a nail into a board. This involves both grasping and tool use, making it one of the most complex manipulation tasks in D4RI..
The goal is to pick up a ball and move it to a target location using precise grasping and positioning. The cloned dataset reflects human-like strategies but difficult to exploit due to sparse rewards and nigh-dimensional dynamics.
clloate--coneed
Methods and Hyperparameters In Mujoco locomotion tasks, we computed the average mean returns over 10 evaluations every 5- 103 training steps, across 5 different seeds. For Antmaze and Kitchen tasks, we calculated the average over 50 evaluations every 2 10 training steps, also across 5 seeds. We measure binary task success rates (in percentage) for AntMaze and normalized returns for Adroit, following the original evaluation scheme (Fu et al., 2020). Following previous research, we standardized the returns by dividing the difference in returns between the best and worst trajectories in MuJoCo tasks. In AntMaze tasks, we subtracted from the rewards.
17

---

+
Figure Tasks used in offline and offline-to-online RL.
Table 2: Hyperparameters for Uni-RL
Hyperparameter
Value
Learning rate
0.0002 (offline), 0.001 (offline)
Optimizer
Adam
Gradient steps
500000
Minibatch size
256
MLP dimensions
1256,266 256, 256]
Target network smoothing coefficient
0.005
Discount factor
0.99
Diffusion steps (if used)
10
Regularization weight
Tables 3 and
7 updating frequency
Tables 3 and
For the network, we use 4-layer MLP with 256 hidden units and Adam optimizer (Kingma and Ba, 2015) with learning rate of x 10-4 for both policy and value functions in all tasks. We also use target network with soft update weight x 10-3 for Q-function. For Uni-RL with Diffusion models the policy, the score network in diffusion-bar based RL algorithms (Ajay et al., 2022; Mao et al., 2024b). We clip the output of the weight function by max(w(x), 0) ensure non-negative BC weight. We use 0-1 normalization to the weight in each batch and then clip it to [Umin, Wmax] where we set Wmin to 0.1 and Wmax to 0.9 through all the datasets.
used is based
U-net architecture, which is fairly common
We implemented Uni-RL using PyTorch and ran it on all datascts. We followed the same reporting methods mentioned carlier. In offline experiments, baseline results for other methods were directly sourced from their respective papers. In offline-to-online experiments, we run baselines using the pytorch implementation from CORL (Tarasov et al., 2024by Since CORL doesn't have the implementation of RLPD, we re-implement it in the codebase. We use the reported hyperparameters in each paper.
In Uni-RL, we have two hyperparameters: regularization weight a and reference policy updating frequency X. We search a over 10.1, 0.5, 1.0, 2.0) and À over [0.01, 0.05, 0.1, 0.2] The best value of a and for all datasets are listed in Table and Table
"nttps://github.comtt cot.aabb..ccoccootttee//////I
18

---

Table 3: Uni-RL hyperparameters in online RL..
Env
a
AntBulletEnv-vO
Bipeda/Walker-v3
1.0
0.1
finger-turn-hane
0.5
0.1
quadruped-wale
0.5
0.1
HallCheetah-v3
Hopper-v3
1.0
0.2
Humanoid-v3
1.0
0.2
Walker2d-v3
1.0
0.2
Table 4: Uni-RL hyperparameters in offline RL.
Env
salfcheetah-medium--2
1.0
copper-medium-22
1.0
walker2d-medium-n22
1.0
hulfchectal-mmcdiim--r--------
1.0
hopper-medduum--eellay-y2
walkerr2d-mcdiuu--rrppp--222
1.0
halfchhetal-meediim-errettzz-
1.0
hopper-meediuu--xxpert-22
walkerrdd-mcdduum-xpptttv
1.0
antmaze-umaze-v2
0.1
autmaazeuumaze-diveree-->
2.0
antmaaz-meedium--layy
0.1
aatmazz-nediuun-ddiees---
0.1
antmaze-large-play-22
0.1
utmaze-large-diverse.
0.1
Table 5: Uni-RL hyperparameters in offline-to-online RL.
Env
X
pen-cloned-vl
1.0
0.01
door-cloned-vi
2.0
0.01
hammer-clomed-vi
2.0
0.01
relocate-clomed-vl
2.0
0.01
antmaze-medium-play-a
0.5
0.01
natmazzemeddum-ddiveree...
0.5
0.01
antmaze-large--lay-22
0.5
0.01
Humnaze-lageediverss
0.5
0.01
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose scalable RL framework that unifies online, offline, and offline- to-online RL.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See the Limitations and Future Work section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
See Appendix A for full proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix for full experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [No]
Justification:
Code will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
wwwsssssssoooooooiooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips.cooo.. wttpccccpppccceeeeccec.cccccccct /ttie//eiiideeooeeuutiiiiiiii citommissioon wwwssnsnllllooooeeeoe.ooo oniooicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix for full experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report error bars in all figures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix for full experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: / http///eerrp/.ee/ee cc.pp.c....o.ooo cttpppulcc/ttiicccteeeelleeeelll
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: The paper follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix for details.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
23

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: For this paper, safeguards needed.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the data and simulator environments used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi wnhersswithodeee thcode. com/ datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
24

---

Answer: [Yes]
Justification:
Code will be released upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
25

---

Justification:
This paper only uses LLM for polishing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
: Please refer to LLM policy (https://neurl)) http:://wennuce//eee/eee httpr//s....//ooeeeeeee/e/////// for what should or should not be described.
26


Reviews:
[{"review_id": "llmZQFDBuE", "reviewer": "Reviewer_kppM", "summary": "This paper proposes Uni-IVR, a unified model-free Reinforcement Learning (RL) framework. Uni-IVR  builds upon the Implicit Value Regularization (IVR) framework, and is designed to address online, offline, and offline-to-online settings. It resolves the issue of over-conservatism when directly applying IVR in online settings, and provides an implicit trust-region-style update via the value function. Furthermore, Uni-IVR introduces a unified policy extraction objective that theoretically guarantees lower value estimation error and greater performance improvement relative to the reference policy. Evaluations on 6 RL benchmarks and 23 environments demonstrate that Uni-IVR achieves competitive or superior performance compared to state-of-the-art baselines across all three settings.", "questions": "-", "limitations": "1. The paper does not provide detailed information on the computational resources used for the experiments.\n2. Although the paper considers various learning scenarios such as online, offline, and offline-to-online, it is limited to model-free RL. Verification of the proposed method in other settings, such as model-based RL, could further demonstrate its generality.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n1. Uni-IVR provides a unified model-free RL framework that integrates online, offline, and offline-to-online learning settings, demonstrating strong versatility.\n2. The paper offers theoretical guarantees, including lower value estimation error and greater performance improvement for its unified policy extraction objective relative to the reference policy. It also provides theoretical insights, indicating that InPG (In-sample Policy Gradient) optimizes the constrained learning objective, can better utilize the Q-function, and can achieve a greater performance improvement than $\\pi_{IVR}^*$.\n3. Uni-IVR achieves higher sample efficiency in online RL compared to both off-policy methods without trust-region updates and on-policy methods with trust-region updates.\n\n## Weaknesses\n1. The algorithms compared in the online RL experiments are outdated (the latest is from 2018); it should consider comparing with more advanced algorithms.\n2. The \"iteratively refined behavior policy\" used to address the over-conservatism issue in online IVR is a soft update (Equation 5). While more lightweight than dataset filtering, the choice of hyperparameter λ might affect performance.", "was_revised": true, "final_justification": "The paper makes a contribution to the field of reinforcement learning, with good writing, methodology, and experiments. The authors have also provided timely and thorough responses to the reviewers’ comments. I recommend accepting this paper.", "rebuttal": "We thank the reviewer for the effort engaged in the review phase and the constructive comments. Regarding the concerns, we provide the detailed responses separately as follows.\n\n>The algorithms compared in the online RL experiments are outdated (the latest is from 2018); it should consider comparing with more advanced algorithms.\n\nThank you for your suggestions. The main contribution of our paper is the proposal of a unified framework for off-policy RL. To highlight Uni-IVR’s scalability across different settings, we select representative baselines from online, offline, and offline-to-online RL. To further demonstrate Uni-IVR’s competitiveness in the online setting, we compare it with BAC [1], which, to the best of our knowledge, is the strongest model-free, off-policy online RL algorithm in recent years. Using the official implementation of BAC, our results show that Uni-IVR consistently outperforms it across all tasks.\n\n\n|  Algorithm      | Uni-IVR | BAC |\n| --------------- | ---- | --- |\n| Hopper-v3  | **3683**±15  | 3450±10    |\n| HalfCheetah-v3  | **14305**±25 | 13110±22  |\n| Humanoid-v3  | **5820**±27 | 5040±10    |\n| Walker2d-v3  | **5940**±16 | 5540±31    |\n| finger-turn_hard  | **625**±80 | 327±49    |\n| quadruped-walk  | **924**±14 | 457±22    |\n\n\n\n\n\n>The \"iteratively refined behavior policy\" used to address the over-conservatism issue in online IVR is a soft update (Equation 5). While more lightweight than dataset filtering, the choice of hyperparameter λ might affect performance.\n\nThank you for raising this point. We provide full ablation results for different values of $\\lambda$ across all environments (with * marking the $\\lambda$ values we selected and their corresponding results reported in the paper). The optimal range of $\\lambda$ **varies across settings**, as the desired contribution of the previous policy differs. In the online setting, a smaller $\\lambda$ is preferred because earlier policies are generally suboptimal. In contrast, in the offline-to-online setting, a larger $\\lambda$ is beneficial due to the availability of a strong initial policy.\n\nNonetheless, within the same setting, the performance is **relatively robust** to the choice of $\\lambda$, as shown in the table below. We note that we did not explore a broader range of values, which might yield even better results.\n\n\n|   Online     | $\\lambda=0.01$ | $\\lambda=0.05$ | $\\lambda=0.1$ | $\\lambda=0.2$ |\n| --------------- | ---- | --- | --- |--- |\n| Hopper-v3  | 2674±24  | 3034±18  | 3628±15  | 3683±15 (\\*) | \n| HalfCheetah-v3  | 11785±19 | 13943±17  | 14331±23  | 14305±25 (\\*) | \n| Humanoid-v3  | 4124±27 | 5423±19  | 5689±22  | 5820±27 (\\*) | \n| Walker2d-v3  | 5332±23 | 5734±15  | 5966±20  | 5940±16 (\\*) | \n| finger-turn_hard  | 336±56 | 595±64  | 625±80 (\\*) | 612±54 | \n| quadruped-walk  | 746±16 | 902±12  | 924±14 (\\*) | 954±17 | \n| AntBulletEnv-v0  | 2393±13 | 3683±23  | 2892±12 (\\*) | 2692±19 | \n| BipedalWalker-v3  | 167±16 | 271±11  | 301±16 (\\*) | 236±18 | \n\n|   Offline-to-Online     | $\\lambda=0.01$ | $\\lambda=0.05$ | $\\lambda=0.1$ | $\\lambda=0.2$ |\n| --------------- | ---- | --- | --- |--- |\n| pen-cloned-v1  | 88.5±0.4  (\\*)  |86.6±0.5  | 83.5±0.6  | 82.7±1.6 | \n| door-cloned-v1  | 93.3±1.9  (\\*) | 90.3±1.2  | 91.6±1.4  | 87.9±0.6 | \n| hammer-cloned-v1  | 92.6±0.7  (\\*) | 93.2±0.6  | 90.1±1.5  | 88.1±0.4| \n| relocate-cloned-v1  | 38.5±0.3  (\\*) | 35.6±0.6  | 36.2±0.7  | 32.5±0.2| \n| antmaze-medium-play-v2  | 99.7±0.6  (\\*) | 97.1±0.3  | 93.3±1.2| 90.7±1.4 | \n| antmaze-medium-diverse-v2  | 99.1±0.6  (\\*) | 97.1±0.3| 92.4±0.9  |92.3±1.0| \n| antmaze-large-play-v2  | 92.5±1.3 (\\*) | 91.0±1.6  | 87.6±0.6 | 85.1±0.9 | \n| antmaze-large-diverse-v2  | 99.1±1.6  (\\*) | 92.7±1.2  | 90.2±1.5 | 89.7±0.7 | \n\n\n>The paper does not provide detailed information on the computational resources used for the experiments.\n\nThanks for pointing out that, we use NVIDIA A40 for all experiments.\n\n\n>Although the paper considers various learning scenarios such as online, offline, and offline-to-online, it is limited to model-free RL. Verification of the proposed method in other settings, such as model-based RL, could further demonstrate its generality.\n\nThanks for pointing out that, the value and policy learning in Uni-IVR can be naturally extended to dyna-style model-based RL algorithms like MBPO [2] where the standard Q-value update rule is replaced by ours. We leave it as future work.\n\n[1] [Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy  Actor-Critic](https://arxiv.org/pdf/2306.02865), Ji et al, ICML 2024.\n\n[2] [When to Trust Your Model: Model-Based Policy Optimization](https://arxiv.org/pdf/1906.08253), Janner et al, NeurIPS 2019."}, {"review_id": "pzAEyrx5xf", "reviewer": "Reviewer_HJMu", "summary": "This paper proposes Uni-IVR, a single reinforcement learning framework designed to operate across online, offline, and offline-to-online settings. The method builds upon the Implicit Value Regularization (IVR) framework by generalizing its dataset-based constraint to a more flexible 'reference policy' constraint, which adapts to each learning paradigm. The paper also introduces a novel policy extraction method, In-sample Policy Gradient (InPG), to improve policy updates.", "questions": "1. Could you provide a full comparison of environmental returns for Uni-IVR vs. IVR across all online environments?\n2. To validate the contribution of InPG, could you provide a comprehensive comparison of policy extraction methods (the proposed InPG vs. baselines like weighted BC and MaxQ) across all tasks in all three settings (online, offline, and offline-to-online)? \n3. There is a significant discrepancy in the λ hyperparameter values between online (0.1–0.2) and offline-to-online (0.01) settings, as shown in Tables 3 and 5. This raises questions about the sensitivity of the method to this hyperparameter. Could you provide an ablation study that shows how performance varies with λ in each setting? \n4. you explain how Uni-IVR leverages Q and V to guide training in these types of generative models and achieve the results shown in Table 1?\n5. To support the claim of scalability, could you please add the results for a Flow-matching policy to Table 1, as was done for the Diffusion policy? \n6. Could you provide the missing implementation details for InPG, including the architecture of the network used to learn the weights $w$, its optimization schedule, and an analysis of its training time compared to Weighted BC and MaxQ?\n7. why Q-values in Figure 2 are used as indicator of performance?\n8. In line 298-299, author stated that “DICE-based methods only work for the offline setting”. Can you justify the reason why they can not be applied by the reference policy?", "limitations": "The authors did not explicitly mention the societal impact of their work, but I do not perceive any negative societal implications.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- The paper tackles the important and ambitious goal of unifying RL across different learning paradigms (online, offline, offline-to-online) with a single, coherent algorithm.\n- The approach is a well-motivated extension of the established IVR framework, providing a solid theoretical foundation for its value learning component.\n- The experimental results are comprehensive and consistently demonstrate that Uni-IVR is competitive with or superior to a range of baseline methods across all three settings.\n\n**Weaknesses:**\n- The core value learning objective is a direct and incremental extension of the IVR framework. Moreover, the paper only compare the improvement in Value learning objective in one task with no name (Figure 2).\n- The novel policy extraction method InPG lacks a comprehensive evaluation, only demonstrated primarily on a single environment (Figure 3), which raises concerns about the generalizability of the results. Moreover, the absence of supplementary source code makes it difficult to check the effective of this novel method.\n- Lack of description about how to build and train $w$ in the implementation. Moreover, computational cost analysis for the InPG module are missing.\n- The paper proposes two alternative objectives that enable the use of Flow-Matching and Diffusion models instead of traditional Gaussian policies. However, if I understand correctly, these objectives treat all actions from the dataset uniformly and do not incorporate information from the Q and V functions.\n- in Figure 2, the authors provide Q-curves to illustrate higher performance; however, Q-learning methods (IQ-learn [1] for example) can produce high Q-values without corresponding performance gains. There are no explanation for these Q value results.\n\n\n**Minors:**\n- Figure 2 uses terms like MaxQ and Weighted BC without prior definition.\n- Two provided learning rates in Table 2 are for offline.\n\n[1] Garg, Divyansh, et al. \"Iq-learn: Inverse soft-q learning for imitation.\" Advances in Neural Information Processing Systems 34 (2021): 4028-4039.", "was_revised": true, "final_justification": "The authors have addressed nearly all of my concerns; however, I still have some reservations regarding the implementation and reproducibility. The rating would increase if the source code were made available in the supplementary material.", "rebuttal": "We thank the reviewer for their time, effort, and constructive comments on our paper. We believe there may be some misunderstandings, which we address in detail below:\n\n\n>The core value learning objective is a direct and incremental extension of the IVR framework. Moreover, the paper only compare the improvement in Value learning objective in one task with no name (Figure 2).\n\nNote that although the value learning objective appears similar to the IVR objective, extending IVR to the online setting is non-trivial. The key contribution of this paper lies in **identifying the ineffectiveness of online IVR and proposing a lightweight solution that that not only addresses this issue but also generalize to other settings**. This fix naturally leads to a unified framework applicable across various RL settings.\n\nFigure 2 presents results on Hopper-v3. We apologize for omitting this information in the paper and will add it in the revised version. The observed behavior in Hopper-v3 is representative of all online environments; thus, we selected it as an illustrative example. For completeness, we provide the full performance comparison between Uni-IVR and IVR below. Note that IVR here uses the same policy extraction method (InPG) as Uni-IVR to ensure a fair comparison.\n\n| Online setting       | Uni-IVR | IVR |\n| --------------- | ---- | --- |\n| Hopper-v3  | 3683±15  | 2034±8  | \n| HalfCheetah-v3  | 14305±25 | 6743±28  | \n| Humanoid-v3  | 5820±27 | 3423±15  | \n| Walker2d-v3  | 5940±16 | 2334±19  |\n| finger-turn_hard  | 625±80 | 195±15  | \n| quadruped-walk  | 924±14 | 302±23  |\n| AntBulletEnv-v0  |  2892±12 | 2083±11 | \n| BipedalWalker-v3  | 301±16 | 71±24  |\n\n>The novel policy extraction method InPG lacks a comprehensive evaluation, only demonstrated primarily on a single environment (Figure 3), which raises concerns about the generalizability of the results. Moreover, the absence of supplementary source code makes it difficult to check the effective of this novel method.\n\nWe respectfully disagree with the reviewer. As mentioned, Figure 3 is intended solely as an illustrative example to convey the motivation and intuition behind using InPG. The actual impact of InPG can be inferred from Table 1 in the paper by comparing Uni-IVR and IVR in the offline setting, where the value learning components are identical. For clarity, we include the relevant results here for easier reference.\n\n|  Offline      | Uni-IVR | IVR | Improvement |\n| --------------- | ---- | --- |--- |\n| halfcheetah-m  | 49.4±0.2  | 48.3±0.2  | +1.1  |\n| hopper-m  | 102.1 ±0.1 | 75.5±3.4  | +26.6  |\n| walker2d-m  | 89.3±0.3 | 84.2±4.6  | +5.1  |\n| halfcheetah-m-r  | 45.3±0.3 | 44.8±0.7  | +0.7  |\n| hopper-m-r  | 101.1 ±1.7 | 99.7±0.3  | +0.4  |\n| walker2d-m-r  | 86.6±1.1 | 81.2±3.8  | +5.4  |\n| halfcheetah-m-e  |  94.2±0.6 | 94.0±0.4 | +0.2  |\n| hopper-m-e  | 111.0±0.6 | 111.0±0.2  | =  |\n| walker2d-m-e  | 110.8±0.2 | 110.0±0.8  | +0.8  |\n| antmaze-u  | 94.1±1.6 | 92.2±1.4  | +1.9  |\n| antmaze-u-d  | 80.4±2.3 | 74.0±2.3  | +6.4  |\n| antmaze-m-p  | 86.0±2.6 | 80.2±3.7  | +5.8  |\n| antmaze-m-d  | 82.7±3.4 | 79.1±4.2  | +3.6  |\n| antmaze-l-p  | 59.9±2.9 | 53.2±4.8  | +6.7  |\n| antmaze-l-d  | 60.2±3.8 | 52.3±5.2  | +7.9  |\n\nIn the online and offline-to-online setting, to validate the contribution of InPG, we provide the comparision of using InPG vs MaxQ and weighted BC as follows.\n\n|  Online      | Uni-IVR (InPG) | Uni-IVR (weighted BC) | Uni-IVR (MaxQ) |\n| --------------- | ---- | --- |--- | \n| Hopper-v3  | 3683±15  | 3234±34  | 3157±26  | \n| HalfCheetah-v3  | 14305±25 | 11334±12  | 12436±15  | \n| Humanoid-v3  | 5820±27 | 4845±15  | 4934±13  | \n| Walker2d-v3  | 5940±16 | 5134±8  |5310±34  | \n| finger-turn_hard  | 625±80 | 434±23  | 445±32  | \n| quadruped-walk  | 924±14 | 832±15  |781±24  | \n| AntBulletEnv-v0  |  2892±12 | 2245±17 | 2467±23  | \n| BipedalWalker-v3  | 301±16 | 296±24  |189±12  | \n\n\n|  Offline-to-online      | Uni-IVR (InPG) | Uni-IVR (weighted BC) | Uni-IVR (MaxQ) |\n| --------------- | ---- | --- |--- |\n| pen-cloned-v1  | 88.5±0.4  |80.6±0.5  | 65.5±0.6  | \n| door-cloned-v1  | 93.3±1.9  | 80.3±1.2  | 11.6±1.4  | \n| hammer-cloned-v1  | 92.6±0.7   | 83.2±0.6  | 30.1±1.5  | \n| relocate-cloned-v1  | 38.5±0.3  | 25.6±0.6  | 6.2±0.7  | \n| antmaze-medium-play-v2  | 99.7±0.6   | 84.1±0.3  | 83.3±1.2| \n| antmaze-medium-diverse-v2  | 99.1±0.6   | 87.1±0.3| 72.4±0.9 |\n| antmaze-large-play-v2  | 92.5±1.3| 81.0±1.6  | 67.6±0.6 | \n| antmaze-large-diverse-v2  | 99.1±1.6   | 72.7±1.2  | 70.2±1.5 |\n\n>Lack of description about how to build and train w in the implementation. Moreover, computational cost analysis for the InPG module are missing.\n\nThank you for pointing that out. The w network is initialized as a value function and trained using the loss defined in Equation (7). The learning rate and network architecture are provided in the appendix. For clarity, we include a code snippet below demonstrating the training of the w network, which only requires **less than 20** lines beyond the original IVR implementation.\n\n```\n# grad of q to pi\npi, _ = self.actor.sample(observations, deterministic=False)\nq_pi = self.q(observations, pi)\nq_grad_pi = torch.autograd.grad(\n    q_pi.mean(),\n    self.actor.parameters(),\n    retain_graph=True\n)\nq_grad_pi_flat = torch.cat([grad.view(-1) for grad in q_grad_pi], dim=0)\nw = self.w(observations, actions)\nbc_losses = self.actor.evaluate(observations, actions)\npolicy_loss = torch.mean(w * bc_losses)\n# grad of wbc to pi\nwbc_grad_pi = torch.autograd.grad(\n    policy_loss,\n    self.actor.parameters(),\n    retain_graph=True,\n    create_graph=True,\n)\nwbc_grad_pi_flat = torch.cat([grad.view(-1) for grad in wbc_grad_pi], dim=0)\n# w loss\nw_loss = F.mse_loss(q_grad_pi_flat, wbc_grad_pi_flat)\n\n```\n\nThe add of the InPG module doesn't add too much computational cost, as shown below.\n\n|  Offline Setting      | IVR | Uni-IVR\n| --------------- | ----| ----| \n|Run Time (training+evaluation) | ~5h| ~6h| \n\n\n>The paper proposes two alternative objectives that enable the use of Flow-Matching and Diffusion models instead of traditional Gaussian policies. However, if I understand correctly, these objectives treat all actions from the dataset uniformly and do not incorporate information from the Q and V functions.\n\nThere seems to be a misunderstanding here. Note that our proposed two alternative objectives are intended to replace the log-likelihood maximization objective used for training Gaussian policies, as mentioned in lines 240–241. In other words, we are replacing the standard objective $\\max E_D(\\log \\pi(a|s))$ with our alternative formulations. However, the learning of the w function remains essential for performing weighted behavior cloning. We will clarify this more explicitly in the revised version of the paper.\n\n\n>in Figure 2, the authors provide Q-curves to illustrate higher performance; however, Q-learning methods (IQ-learn [1] for example) can produce high Q-values without corresponding performance gains. There are no explanation for these Q value results.\n\nNote that we provide the corresponding evaluation scores in the first row of Figure 2, and the Q-value is included only as an additional metric for reference. As shown in Figure 2, the actual performance aligns well with the Q-values.\nIn fact, Q-value overestimation is a much more serious issue in the offline setting. In the online setting, the Q-value can still serve as a useful performance indicator—though it may exhibit slight overestimation due to TD learning, it is not nearly as severe as in the offline case.\n\n>In line 298-299, author stated that “DICE-based methods only work for the offline setting”. Can you justify the reason why they can not be applied by the reference policy?\n\nThanks for pointing out that, the reason is that if we apply DICE-based methods to the online setting, sampling from the stationary distribution $d^{\\mu}(s, a)$ of the reference policy is needed.\n\nMore specifically, this is the dice learning objective:\n$$\n\\pi^{*} = \\arg \\max_{\\pi} E_{(s, a) \\sim d^{\\pi}}[r(s, a)] - \\alpha D_f(d^{\\pi}||d^{\\mu}).\n$$\nSolve by duality we get\n$$\n\\min_{V}  (1 - \\gamma)E_{s \\sim d_{0}}\\left[ V(s) \\right] + \\alpha E_{(s, a) \\sim d^{\\mu}} \\big[ f_{dual}\\big(\\left[T_r V(s, a) - V(s)\\right] / \\alpha\\big) \\big],\n$$\nSamples from $d^{\\mu}(s, a)$ are intractable in the online setting where $\\mu$ is the previous policy.\n\n>This raises questions about the sensitivity of the method to this hyperparameter. Could you provide an ablation study that shows how performance varies with λ in each setting?\n\nDue to space limit, please see the response to Reviewer kppM for ablation study of different $\\lambda$.\n\n\n>To support the claim of scalability, could you please add the results for a Flow-matching policy to Table 1, as was done for the Diffusion policy?\n\n|  Offline setting      | Uni-IVR (diffusion) | IVR (flow-matching) | \n| --------------- | ---- | --- |\n| halfcheetah-m  | 58.0 ±0.6  | 56.5 ±1.4  | \n| hopper-m  | 102.5 ±0.2 | 100.6 ±2.2  | \n| walker2d-m  | 92.3 ±0.1 | 91.8 ±1.4  | \n| halfcheetah-m-r  | 48.4 ±0.9 | 49.7 ±0.4  | \n| hopper-m-r  | 101.3 ±2.1 | 101.8 ±1.1  | \n| walker2d-m-r  | 90.8±1.6 | 92.8±1.3 | \n| halfcheetah-m-e  |  97.3 ±0.6 | 97.4 ±0.6 | \n| hopper-m-e  | 111.2 ±0.3 | 112.6 ±0.4  | \n| walker2d-m-e  | 114.1±0.5 | 113.7±0.4  | \n| antmaze-u  | 98.1 ±1.8 | 97.3 ±1.0  | \n| antmaze-u-d  | 82.0 ±1.4 | 83.5 ±1.8  | \n| antmaze-m-p  | 88.5 ±3.1 | 89.6 ±4.5  | \n| antmaze-m-d  | 89.7 ±2.8 | 86.7 ±1.1  | \n| antmaze-l-p  | 68.6 ±3.6 | 72.4 ±2.2  | \n| antmaze-l-d  | 69.0 ±4.5 | 74.2 ±3.8  | \n\n\nWe believe our clarifications to your responses naturally answered your questions listed. Please let us know if any further questions remain. We hope the reviewer can reassess our work with these clarifications."}, {"review_id": "pTcNJn105N", "reviewer": "Reviewer_MCWA", "summary": "This paper proposes **Uni-IVR**, an extension of IVR that achieves strong performance across offline, off-to-on, and online reinforcement learning (RL) settings, without requiring substantial modifications between these modes. The main contributions can be summarized in two parts: **iterative refinement** and **in-sample policy gradient (InPG)**.\n\nThe **iterative refinement** component uses the target policy instead of the behavior policy to optimize the value function $V$ in online RL and in the online phase of off-to-on RL. The **InPG** component introduces a weight optimization scheme for weighted behavioral cloning (BC), aiming to align the Q-function maximization gradient with the weighted BC gradient.\n\nEmpirically, Uni-IVR demonstrates strong performance across all evaluated domains, with high sample efficiency and low sensitivity to hyperparameter tuning.", "questions": "1. Could you provide ablation results for some of the key hyperparameters or for different choices of $f$-divergence used in the algorithm?\n\n2. What specific method of policy mixing is used in the iterative refinement process, parameter mixing or probability mixing? If it is parameter mixing, could you clarify in what way this contributes to the novelty of the method?\n\n3. Given that IVR utilizes both KL divergence and $\\chi^2$-divergence in separate algorithmic formulations, it would be helpful to clarify which divergence is used in Uni-IVR.\n\n4. The use of 500K steps is an interesting design choice. However, it raises the question of whether training continues to improve beyond that point. Could you provide additional results or learning curves to illustrate this?\n\n5. Would it be possible to include results using exactly the same hyperparameter settings as in the original IVR paper for the locomotion tasks in offline RL? This could help isolate the contribution of the proposed modifications.", "limitations": "The authors are encouraged to explicitly discuss the limitations of their proposed algorithm, either in terms of its general applicability or in relation to the specific domains used in the experimental evaluation.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. InPG offers an intuitive and novel formulation of weighted behavioral cloning (BC).\n2. The empirical results demonstrate strong performance, especially considering that Uni-IVR is applicable across diverse domains and operates as an in-distribution algorithm.\n3. Uni-IVR requires minimal hyperparameter tuning and performs well with configurations close to standard defaults.\n4. The theoretical support is thorough and well-presented.\n\n**Weaknesses**\n\n1. Some configurations and notations are ambiguous, including:\n   - The method used to combine policies during iterative refinement. It is unclear whether this is done via parameter mixing or probability mixing.\n   - The specific type of $f$-divergence employed. While IVR distinguishes between different divergences, the presented formulation appears similar to sparse Q-learning (SQL), which could benefit from clarification.\n   - The type of noise scheduling used in diffusion-based cases. While VP scheduling is expected, cosine scheduling is also commonly applied and should be explicitly stated if used.\n\n2. The iterative refinement component appears to be a straightforward application of the target policy method, which is already widely used in online RL algorithms such as TD3, as well as in some offline RL approaches. If the combination method corresponds to parameter mixing, the contribution may be limited to employing a larger mixing coefficient than the commonly used value of 0.005, which raises questions about the novelty of this aspect.\n\n3. The paper does not include ablation studies, which makes it difficult to assess the individual contributions of each component.\n\n4. If the second point holds, the overall algorithmic contribution may be somewhat limited.", "was_revised": true, "final_justification": "The authors have adequately addressed the concerns I raised during the review period.", "rebuttal": "We thank the reviewer for their thoughful review of our work. We address the reviewers questions and concerns below:\n\n\n>Given that IVR utilizes both KL divergence and chi-divergence in separate algorithmic formulations, it would be helpful to clarify which divergence is used in Uni-IVR.\n\nThanks for pointing out that, we use $\\chi^2$-divergence throught the paper because SQL, which uses $\\chi^2$-divergence, produces better and more stable results than EQL, which uses KL divergence. We will add this clarification in the revised paper.\n\n\n>Could you provide ablation results for some of the key hyperparameters in the algorithm?\n\nDue to space limit, please see the response to Reviewer HJMu for a comprehensive ablation study including different $\\lambda$, using InPG vs weighted BC vs MaxQ, using Uni-IVR vs IVR.\n\n\n>What specific method of policy mixing is used in the iterative refinement process, parameter mixing or probability mixing? If it is parameter mixing, could you clarify in what way this contributes to the novelty of the method?\n\nWe are using parameter mixing but do not claim the specific method of policy mixing as a contribution. Instead, the key contribution of this paper is **identifying the ineffectiveness of online IVR and proposing a lightweight solution that that not only addresses this issue but also generalize to other settings**. This solution further serves as a general framework applicable across various RL settings, which is a **non-trivial extension from the original offline IVR that no previous work has achieved**.\n\n\n\n\n>The use of 500K steps is an interesting design choice. However, it raises the question of whether training continues to improve beyond that point. Could you provide additional results or learning curves to illustrate this?\n\n\nNote that the key metric for evaluating online RL algorithms is the sample efficiency, i.e., achieve higher performance given limited interaction steps. We addiontially provide the results of training  1000K steps below to further show the performance. It can be seen that the improvement is limited with additional 500k steps, indicating that the learning curve is almost converged (we will add the learning curve to the revised paper). \n\n\n|   Online     | Uni-IVR (500k) | Uni-IVR (1000k) | \n| --------------- | ---- | --- | \n| Hopper-v3  | 3683±15  | 3691±4  | \n| HalfCheetah-v3  | 14305±25 | 15344±3  | \n| Humanoid-v3  | 5820±27 | 6034±6  | \n| Walker2d-v3  | 5940±16 | 6123±4  |\n| finger-turn_hard  | 625±80 | 654±22  | \n| quadruped-walk  | 924±14 | 945±12  | \n| AntBulletEnv-v0  |  2892±12 | 3021±9 | \n| BipedalWalker-v3  | 301±16 | 303±4  |\n\n\n>Would it be possible to include results using exactly the same hyperparameter settings as in the original IVR paper for the locomotion tasks in offline RL? This could help isolate the contribution of the proposed modifications.\n\nNote that we didn't change the hyperparameter settings of IVT when training Uni-IVR. The full comparision of Uni-IVR and IVR is shown below to justify the benefit of using InPG policy extraction.\n\n|  Offline setting      | Uni-IVR | IVR | Improvement |\n| --------------- | ---- | --- |--- | \n| halfcheetah-m  | 49.4±0.2  | 48.3±0.2  | +1.1  |\n| hopper-m  | 102.1 ±0.1 | 75.5±3.4  |  +26.6  |\n| walker2d-m  | 89.3±0.3 | 84.2±4.6  |  +5.1  |\n| halfcheetah-m-r  | 45.3±0.3 | 44.8±0.7  |  +0.7  |\n| hopper-m-r  | 101.1 ±1.7 | 99.7±0.3  |  +0.4  |\n| walker2d-m-r  | 86.6±1.1 | 81.2±3.8  |  +5.4  |\n| halfcheetah-m-e  |  94.2±0.6 | 94.0±0.4 | +0.2  |\n| hopper-m-e  | 111.0±0.6 | 111.0±0.2  |  =  |\n| walker2d-m-e  | 110.8±0.2 | 110.0±0.8  |  +0.8  |\n| antmaze-u  | 94.1±1.6 | 92.2±1.4  |  +1.9  |\n| antmaze-u-d  | 80.4±2.3 | 74.0±2.3  | +6.4  |\n| antmaze-m-p  | 86.0±2.6 | 80.2±3.7  |  +5.8  |\n| antmaze-m-d  | 82.7±3.4 | 79.1±4.2  |  +3.6  |\n| antmaze-l-p  | 59.9±2.9 | 53.2±4.8  |+6.7  |\n| antmaze-l-d  | 60.2±3.8 | 52.3±5.2  |  +7.9  |\n\n\nPlease let us know if any further questions remain. We hope the reviewer can reassess our work with these clarifications."}, {"review_id": "f9t4inJ4WA", "reviewer": "Reviewer_wWsA", "summary": "This work proposes Uni-IVR, a single model-free algorithm that adapts to three settings --- offline, offline-to-online, and online learning. The key idea builds on the Implicit Value Regularization (IVR) framework and utilizes different reference policies depending on the settings: the behavior policy in the offline setting and a target policy in the online setting. Additionally, authors focus on providing a policy extraction scheme to effectively derive the best policy from the learned value function via Uni-IVR. The In-sample Policy Gradient (InPG) solves a constrained QP that determines weights using zero-order gradients, remains mode-seeking to maximize Q-values, and aligns a weighted BC loss with the true PG direction. In evaluation, Uni-IVR achieves higher sample efficiency and outperforms several SOTA algorithms in three settings.", "questions": "1. The related-work section cites recent algorithms such as Uni-O4 and DICE-based methods, yet Figure 5 benchmarks Uni-IVR only against Cal-QL, AWAC, RLPD, and IQL—methods published no later than 2023. To the best of my knowledge, BOORL [3], WSRL [4], and  O2SAC [5] are SOTA algorithms. Could you clarify whether Uni-IVR is expected to deliver superior sample efficiency, early-phase stability, and final performance when evaluated against these baselines?\n\n2. Tables 4 and 5 list different hyperparameters for the Antmaze tasks: $\\alpha$ = 0.1 or 2.0 in the pure-offline experiments, but $\\alpha=0.5$ in the offline-to-online setting. \tDoes this difference imply that a policy trained with the offline hyperparameter cannot be carried over directly into the online fine-tuning phase without re-tuning $\\alpha$?\n\n3. Lemma 1 appears to reproduce a result that was already proved in the original IVR paper (Xu et al., 2023). Please add an explicit citation and indicate the exact place in that paper where the derivation can be found.\n\n[3] Hu, Hao, et al. \"Bayesian design principles for offline-to-online reinforcement learning.\" arXiv preprint arXiv:2405.20984 (2024).\n\n[4] Zhou, Zhiyuan, et al. \"Efficient online reinforcement learning fine-tuning need not retain offline data.\" arXiv preprint arXiv:2412.07762 (2024).\n\n[5] Luo, Qin-Wen, et al. \"Optimistic critic reconstruction and constrained fine-tuning for general offline-to-online RL.\" Advances in Neural Information Processing Systems 37 (2024): 108167-108207.", "limitations": "In the Limitations section, the authors state only that Uni-IVR is restricted to the model-free setting. A more comprehensive ablation study is warranted. For instance, they could remove InPG (replacing it with vanilla BC or MaxQ + BC) and demonstrate robustness across a range of hyperparameter settings.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- Strengths\n  1. While previous works in RL mainly focus on one of three settings, this work designs a scalable RL framework that unifies online, offline, and offline-to-online settings.\n  2. InPG combines mode-seeking behavior with in-support actions, avoiding the over-estimation of Q-value while outperforming existing weighted-BC or MaxQ+BC. Also, Theorems 1-3 guarantee that the policy extraction with InPG improves performance and readily scales to generative models like diffusion and flow matching models.\n  3. When comparing several algorithms in each RL setting, Uni-IVR achieves SOTA performance and sample efficiency in diverse datasets.\n\n\n- Weaknesses\n   1. Uni-IVR is closely related to IVR (Xu et al., 2023). The main difference in offline RL settings is the addition of the InPG method. In Table 1, I think IVR's performance is comparable to that of Uni-IVR (Gaussian). Additionally, baselines (BCQ, TD3+BC, CQL, and IQL) are not SOTA algorithms in offline RL. To make the empirical evaluation more convincing, Uni-IVR should be compared with more recent methods, such as ReBRAC [1] and QDQ [2].\n   2. Lemma 1 states that the optimal value functions $Q^{\\*}$ and $V^{\\*}$  can be solved by Equations 3 and 4, respectively. In Equation 5, however, the authors replace the maximisation with an expectation over actions sampled from $\\bar{\\pi}$. Does this modified operator still converge to $V^{*}$?\n   3. In Figure 5, learning curves are plotted after 250k offline pertaining. Yet this work claims that Uni-IVR (i) alleviates Q-value overestimation and (ii) avoids the performance drop typically seen at the start of offline-to-online finetuning. To substantiate these points, wouldn’t it be essential to plot the offline performance at the end of pre-training, alongside the subsequent finetuning curves?\n\n[1] Tarasov, Denis, et al. \"Revisiting the minimalist approach to offline reinforcement learning.\" Advances in Neural Information Processing Systems 36 (2023): 11592-11620.\n\n[2] Zhang, Jing, et al. \"Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model.\" Advances in Neural Information Processing Systems 37 (2024): 54421-54462.", "was_revised": true, "final_justification": "The authors address the reviewer's concerns and provide more sufficient experiments.", "rebuttal": "We thank the reviewer for their thoughful review of our work. We address the reviewers questions and concerns below:\n\n\n>The related-work section cites recent algorithms such as Uni-O4 and DICE-based methods, yet Figure 5 benchmarks Uni-IVR only against Cal-QL, AWAC, RLPD, and IQL—methods published no later than 2023. To the best of my knowledge, BOORL [3], WSRL [4], and O2SAC [5] are SOTA algorithms. Could you clarify whether Uni-IVR is expected to deliver superior sample efficiency, early-phase stability, and final performance when evaluated against these baselines?.\n\nTo further compare the performance of Uni-IVR, we evaluate it against BOORL, WSRL, and O2SAC on AntMaze tasks, as BOORL and O2SAC did not conduct experiments on the Adroit datasets. Uni-IVR continues to demonstrate strong performance compared to these baselines.\n\n|   Offline-to-Online (200k online steps)    | Uni-IVR | BOORL | WSRL | O2SAC |\n| --------------- | ---- | --- | --- |--- |\n| antmaze-medium-play-v2  | 85.7  | **91.9**  | -| 78.8 | \n| antmaze-medium-diverse-v2  | **99.1**   | 84.1| -  |70.6| \n| antmaze-large-play-v2  | 85.5 | 55.9  | **96** | 44.2 | \n| antmaze-large-diverse-v2  | **98.1**  | 69.1  | 95 | 40.2 | \n\n>Tables 4 and 5 list different hyperparameters for the Antmaze tasks: \nalpha = 0.1 or 2.0 in the pure-offline experiments, but alpha=0.5 in the offline-to-online setting. Does this difference imply that a policy trained with the offline hyperparameter cannot be carried over directly into the online fine-tuning phase without re-tuning alpha?\n\nThere is no guarantee that the best hyperparameters for the offline setting will be optimal for the off-to-online setting. Empirically, we find that using a less optimistic $\\alpha$ leads to more stable value learning, which in turn benefits online fine-tuning.\n\n\n>Lemma 1 appears to reproduce a result that was already proved in the original IVR paper (Xu et al., 2023). Please add an explicit citation and indicate the exact place in that paper where the derivation can be found.\n\nThank you for pointing that out. We would like to clarify that Lemma 1 is a result from the IVR paper and is not claimed as a novel contribution in our work (as indicated by labeling it as a lemma and explicitly stating it is from IVR). It is included solely for completeness and consistency in the presentation. The detailed derivation is provided in Appendix C in the IVR paper.\n\n>In the Limitations section, the authors state only that Uni-IVR is restricted to the model-free setting. A more comprehensive ablation study is warranted. For instance, they could remove InPG (replacing it with vanilla BC or MaxQ + BC) and demonstrate robustness across a range of hyperparameter settings.\n\nDue to space limit, please see the response to Reviewer HJMu for a comprehensive ablation study including different $\\lambda$, using InPG vs weighted BC vs MaxQ, using Uni-IVR vs IVR.\n\n\n>Uni-IVR is closely related to IVR (Xu et al., 2023). The main difference in offline RL settings is the addition of the InPG method. In Table 1, I think IVR's performance is comparable to that of Uni-IVR (Gaussian). Additionally, baselines (BCQ, TD3+BC, CQL, and IQL) are not SOTA algorithms in offline RL. To make the empirical evaluation more convincing, Uni-IVR should be compared with more recent methods, such as ReBRAC [1] and QDQ [2].\n\nWe compare Uni-IVR with ReBRAC and QDQ, and find that Uni-IVR achieves comparable performance to these advanced methods. However, we would like to emphasize that the key contribution of our work is **not** to outperform methods specifically designed for the offline setting. Rather, our goal is to propose a **general and unified framework** applicable across a wide range of RL settings, including but not limited to the offline scenario.\n\n\n|  Offline setting      | Uni-IVR | ReBRAC | QDQ |\n| --------------- | ---- | --- |--- | \n| halfcheetah-m  | 49.4  | 65.6  | 74.1  |\n| hopper-m  | 102.1  | 102  |  99.0  |\n| walker2d-m  | 89.3 | 82.5  |  86.9 |\n| halfcheetah-m-r  | 45.3 | 51  |  63.7  |\n| hopper-m-r  | 101.1  | 98.1  |  102.4  |\n| walker2d-m-r  | 86.6 | 77.3  |  93.2  |\n| halfcheetah-m-e  |  94.2 | 101.1 | 99.3  |\n| hopper-m-e  | 111.0 | 107  |  113.5  |\n| walker2d-m-e  | 110.8 | 111.6  |  115.9  |\n| antmaze-u  | 94.1 | 97.8  |  98.6  |\n| antmaze-u-d  | 80.4 | 88.3  | 67.8  |\n| antmaze-m-p  | 86.0 | 84.0  |  81.5  |\n| antmaze-m-d  | 82.7 | 76.3  |  85.4  |\n| antmaze-l-p  | 59.9 | 60.4  |35.6  |\n| antmaze-l-d  | 60.2 | 54.4 |  31.2  |\n\n\n>Lemma 1 states that the optimal value functions \nQ* and V* can be solved by Equations 3 and 4, respectively. In Equation 5, however, the authors replace the maximisation with an expectation over actions sampled from \\hat{\\pi}. Does this modified operator still converge to V*?\n\nNote that Lemma 1 shows that updating according to Equations (3) and (4) yields the optimal policy $\\pi^{\\ast}$ and value function $V^{\\ast}$ that maximize the expected return **under the constraint induced by $\\mu$**. In the online setting, we replace $\\mu$ with a weighted mixture of $\\pi^{\\ast}$ and $\\mu$, which ensures continual improvement of the reference policy. This allows $\\pi^{\\ast}$ to progressively improve toward the global optimum.\n\n\n>In Figure 5, learning curves are plotted after 250k offline pertaining. Yet this work claims that Uni-IVR (i) alleviates Q-value overestimation and (ii) avoids the performance drop typically seen at the start of offline-to-online finetuning. To substantiate these points, wouldn’t it be essential to plot the offline performance at the end of pre-training, alongside the subsequent finetuning curves?\n\nThe offline performance at the end of pre-training is reflected in the initial point of the online learning curve (which is non-zero). Our method improves upon this initial performance without dropping down during online fine-tuning, supporting claim (2). Additionally, the high sample efficiency observed during online learning further supports claim (1).\n\n\nPlease let us know if any further questions remain. We hope the reviewer can reassess our work with these clarifications."}]

Author rebuttal:

Meta-review: This paper introduces Uni-IVR, a framework that aims to use a single formulation for various reinforcement learning setups, such as online RL, offline RL, and offline-to-online RL. Reviewers highlighted multiple strengths of the paper: the paper tackles an important problem based on a well-presented theoretical analysis and provides strong empirical results. I agree with these points. Overall, this paper is technically solid and achieved the ambitious goal of being competitive on various setups with a unified setup. 

After carefully reading the paper, reviews, and discussion, I recommend accepting the paper. Initially there were some concerns raised by reviewers such as the lack of comparison to recent baselines or hyperparameters, but the authors’ rebuttal successfully addressed these concerns by providing additional experimental results. While only one reviewer ended up in a non-borderline acceptance recommendation, all the reviewers have no remaining concerns as the result of active discussion with authors. It is encouraged that the authors release open-source implementation by following the suggestion of Reviewer HJMu, incorporate all the points raised by reviewers, and provide confidence intervals as pointed out by Reviewer wWsA.
Decision: Accept (poster)