Efficient Off-Policy Learning for High-Dimensional Action Spaces
Authors: Fabian Otto
Philipp Becker
Vien Anh Ngo
Gerhard Neumann
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning
trust region
Primary area: reinforcement learning

Abstract: Existing off-policy reinforcement learning algorithms often rely on an explicit state-action-value function representation, which can be problematic in high-dimensional action spaces due to the curse of dimensionality.
This reliance results in data inefficiency as maintaining a state-action-value function in such spaces is challenging. 
We present an efficient approach that utilizes only a state-value function as the critic for off-policy deep reinforcement learning.
This approach, which we refer to as Vlearn, effectively circumvents the limitations of existing methods by eliminating the necessity for an explicit state-action-value function. 
To this end, we leverage a weighted importance sampling loss for learning deep value functions from off-policy data. 
While this is common for linear methods, it has not been combined with deep value function networks. 
This transfer to deep methods is not straightforward and requires novel design choices such as robust policy updates, twin value function networks to avoid an optimization bias, and importance weight clipping.
We also present a novel analysis of the variance of our estimate compared to commonly used importance sampling estimators such as V-trace. 
Our approach improves sample complexity as well as final performance and ensures consistent and robust performance across various benchmark tasks.
Eliminating the state-action-value function in Vlearn facilitates a streamlined learning process, yielding high-return agents.

Published as a conference paper at ICLR 2025
EFFICIENT OFF-POLICY LEARNING FOR HIGH-DIMENSIONAL ACTION SPACES
Fabian Otto
Philipp Becker
Microsoft Research
Karlsruhe Institute of Technology
wtw.::ttttttttoootoooooooooooooo croso wt.rsooo.coom rabian.ctton
Ngo Anh Vien Bosch Center for Artificial Intelligence
Gerhard Neumann
Karlsruhe Institute of Technology
ABSTRACT
Existing off-policy reinforcement learning algorithms often rely an explicit state- action-value function representation, which can be problematic in high-dimensional action spaces due to the curse of dimensionality. This reliance results in data ineffi- ciency as maintaining a state-action-val value function in such spaces is challenging. We present efficient approach that utilizes only state-value function the critic for off- off-policy deep reinforcement learning. This approach, which we refer Vlearn, effectively circumvents the limitations of existing methods by elim- inating the necessity for an explicit state-action-value function. To this end, we leverage weighted importance sampling loss for learning deep value functions from off-policy data. While this is common for linear methods, it has not been combined with deep value function networks. This transfer to deep methods is not straightforward and requires novel design choices such as robust policy up- dates, twin value function networks avoid an optimization bias, and importance weight clipping. We also present novel analysis of the variance of our estimate compared to commonly used importance sampling estimators such as V-trace. Our approach improves sample complexity as well as final performance and ensures consistent and robust performance across various benchmark tasks. Eliminating the iitte-cctionvallut function in Vlearn facilitates streamlined learning process, yielding high-return agents.
INTRODUCTION
Reinforcement learning (RL) has emerged as a powerful paradigm for training intelligent agents through interaction with their environment (Mnih et al., 2013; Silver et al., 2016). On-policy and off-policy are the two primary approaches within model-free RI. On-policy methods rely on newly generated (quasi-)onling samples in each iteration (Schulman et al., 2017; Otto et al., 2021), whereas off-policy methods leverage replay buffer populated by behavior policy (Abdolmaleki al., 2018; Haarnoja et al., 2018; Fujimoto al., 2018). Although on-policy methods can compensate for off-policy data to some extent via importance sampling (Espeholt et al., 2018), they cannot exploit it fully. To harness the full potential of off off-policy data, off-policy methods traditionally focus learning state-at ciiteeaction-ooe action-value functions (Q-functions) as critics (Degris et al., 2012).
In this work, present Vlearn off-policy policy gradient method that exclusively utilizes V- functions. With Vlearn we do not seek to replace existing methods, but rather to offer a new perspective and practical insights for deep off off-policy RL by demonstrating the general feasibility and benefits of such a setting. Unlike commonly used Q-function based methods, this approach does require learning over the joint state-action space, making it less susceptible the curse of dimensionality and more effective at updating the critic regardless of the actions taken when acting the environment. Consequently, Vlearn naturally demonstrates better suitability for high- dimensional action spaces. While existing V-function methods, such as V-trace (Espeholt et al.,
Work partially done Bosch Center for Artificial Intelligence and University of Tübingen

---

Published as a conference paper at ICLR 2025
Vleam
V-trace (1 (I-Step)
Bellman Target
Target Critic
p
1.0
0.5
0.1
p
0.0
10
Criti Loss
-10
-5
0
5
10
10
-5
5
10
10
-5
0
5
10
10
-5
0
5
10
(8x)
(o(*t)
Vo(ss)
Vo(st)
Figure 1: To provide intuition on the differences between Vleam and V-trace, we consider the following example for different importance ratios p. Suppose for state s, the Bellman target is r(x, a) + +VV = 4, the target critic predicts V(((z) = -6 and we plot the loss for values of Ve(s) for Vleam and V-trace. For on-policy samples (p 1.0), both losses the same. However, for increasingly off-policy samples (p + 0), we see how V-trace increasingly relies on the target critic, shifting the optimal value towards Vlearn, the other hand, simply reduces the scale of the loss and thus the importance of the sample, making Vlearn more robust to errors in the target critic.
2018), attempt increase the usability of stale data in on-policy methods, find that they are less effective in a completely off-policy context. These methods typically focus on reweighting Bellman targets, but prior research (Mahmood et al., 2014) in the linear domain indicates that applying importance sampling to the entire Bellman error is more ad advantageous. Inspired by these findings, we demonstrate that this objective can be recovered in deep RL setting as an upper bound on the original naive Bellman error. This upper bound can be derived using Jensen's 's inequality and retains the same minimum. This loss shifts the importance weights from the Bellman targets to the objective, simplifying V-function updates and enhancing the stability of learning V-function from off- off-policy data, which posed significant challenges in earlier methods. Our analysis the bandit setting shows that this Bellman error estimator has lower variance compared to the estimators from common deep RL importance sampling schemes like V-trace (Espeholt et al., 2018).
Crucially, propose practical algorithm that leverages this objective and improves the stability of policy learning by using the trust region update from Otto et al. (2021) and other commonly used techniques from off-policy deep RL, such twin critics and delayed policy updates (Fujimoto et al., 2018), tanh squashing (Haarnoja et al., 2018), and advantage normalization (Schulman et al., 2017).
Our experiments demonstrate the relevance and importance of these design choices in achieving high performance. Notably, by removing the dependence on the Q-function our method is particularly well suited for environments with complex action spaces, such the challenging MyoSuite (Vittorio et al., 2022) and dog locomotion tasks in DeepMind control (DMC) (Tassa et al., 2018 8), which most standard off aff-policy actor-critic methods cannot solve.
RELATED WORK
To enhance sample efficiency in RL, off-policy algorithms aim leverage historical data stored in a replay buffer (Lin, 1992). While deep D-Learning-basee methods (Mnih et al., 2013; Van Hasselt al., 2016; Wang et al., 2016; Hessel al., 2018) excel in efficiently learning from discrete action spaces, they are often not directly applicable to continuous problems. For such problems, off- off-policy actor-critic methods optimize a policy network that generates continuous actions based on the state- action-value function estimator (Degris et al., 2012; Zhang et al., 2019). This approach accommodates both deterministic (Lillicrap et al., 2015; Fujimoto et al., 2018) and probabilistic policies (Haarnoja et al., 2018; Abdolmaleki et al., 2018).
In the on-policy setting, trust-region methods (Schulman et al., 2017; 2015; Otto et al., 2021) have proven effective in stabilizing the policy gradient. However, these methods have not received the same level of attention in the off- policy setting. Past efforts have focused on training the value function with off- toolcy data (Gu et al., 2016) extended trust region policy optimization (Schulman et al., 2015) to the off -olico setting (Nachum et al., 2018; Meng et al., 2022). While Wang et al. (2017), for instance, employs a more standard approach by combining off-policy trust regions with additional
2

---

Published as a conference paper at ICLR 2025
advancements, such as Retrace (Munos et al., 2016) and truncated importance sampling, it is still not able to compete with modern actor-critic approaches. Peng et al. (2019), on the other hand, provide a different perspective and split policy and falue-function learning into separate supervised learning steps. Generally, standard off-policy trust region methods often fall behind modern actor-critic approaches (Haarnoja et al., 2018; Fujimoto al., 2018) and cannot achieve competitive performance. However, maximum aposteriori policy optimisation (MPO) (Abdolmaleki et al., 2018) can be seen as a non-classical trust region method. Its EM-based formulation offers more flexibility, with trust regions akin to optimizing a parametric E-step without an explicit M-step.
While on on-policy trust region methods estimate the less complex state-value function, most off-pollcy methods rely on estimating state- iitte-action-vallee functions to learn from replay buffer data (Haarnoja et al., 2018; Fujimoto et al., 2018; Abdolmaleki et al., 2018). Methods exclusively utilizing state-value functions must employ importance sampling to address the distributional discrepancy between target and behavior policies. However, this has been primarily studied the context of asynchronous on-policy settings, where the policy distributions differ only slightly (Espeholt et al., 2018; Luo et al., 2020). In particular, V-trace (Espeholt et al., 2018) aims to compensate for limited distributional shift due asynchronous computations and is not designed for a full off off-policy setting, such in this work. Further, such an off-policy correction is computationally expensive as it relies on storing and processing trajectories. Moreover, the truncation technique used in importance weight calculations introduces a bias (Espeholt et al., 2018). Consequently, the implementation of trajectory-based Bellman target estimators would exacerbate the already pronounced issue of bias propagation.
In the linear case, the effect of varying the placement of the importance weight in the off-policy case has already been studied. Applying importance weighting the entire Bellman error, rather than just the Bellman target, yields better performance and may even lead to reduced variance. This approach naturally used in methods such as off off-policy TD() (Sutton & Barto, 2018), and further supported by Mahmood et al. (2014): Dann et al. (2014); Graves & Ghiassian (2022) on its effectiveness in the linear setting. However, translating these findings to deep RI. presents a challenge, as insights from linear models are often not directly applicable (Mnih et al., 2013; Lillicrap et al., 2015).
3 DEEP OFF-POLICY LEARNING FROM STATE- VALUE FUNCTIONS
We seek to determine an optimal policy ((a s) within the framework of a Markov decision process (MDP), which is defined by a tuple (S,A, T, r, 7). Here, both the state space S and action space A are continuous. The transition density function T S x A x S + IR+ is defined as a function mapping from the current state 8; S and action a; E A to the probability density of transitioning to the subsequent state St+1 E S. The initial state distribution's 's density is denoted as Po : S + R+ The reward attained from interactions with the environment is determined by the function r S x A + R, and the parameter 7 E [0, 1) represents the discount factor applied to future rewards. Our primary objective to maximize the expected cumulative discounted reward G, = E, .p.. k=t x frss, ax)]. Most popular off policy actor-critic methods (Haarnoja et al., 2018; Fujimoto et al., 2018; Abdolmaleki et al., 2018) now aim to find policy that maximizes the cumulative discounted reward by making use of a learnable state-action value estimate Q7 (s.a) Es [Gr|s; = ag = a). To train this estimator, they rely on a dataset D = {((r,, Qz, Ft,, $141)t=1...1 and a behavioral policy (-(s)) responsible for generating this dataset. Typically, D takes the form of a replay buffer (Lin, 1992), with the corresponding behavior policy ny being a mixture of the historical policies used to populate the buffer. Training the state aatteacttion value function then usually involves temporal difference learning (Sutton, 1988; Watkins & Dayan, 1992), with updates grounded in the Bellman equation (Bellman, 1957). The objective commonly optimized is
E(o [Q8 (st, (r(se, a.) 7BB.
Qg(se41 ()-)))) (D)
where Qø(s, a) is a frozen target network. Furthermore, to mitigate overestimation bias, most methods employ two state-action value functions (Fujimoto et al., 2018). To maintain the stability of this approach, the target network's weights are updated via polyak averaging 0 +- 70 + (1 7)0.
3.1 IMPORTANCE SAMPLING ESTIMATORS FOR THE BELLMAN ERROR
Instead of using the the date-value function V (8t). This base estimator can be trained by minimizing the following loss
state state-action function Q7 (s,, a,), an alternative approach is to work solely with
3

---

Published as a conference paper at ICLR 2025
function using importance sampling
( (VT(0)) D.e.
(Tac(1))) Tb(a)((s)
27
Traase (0) = BR
V
(2)
where V == rss,, az) + (8+(81+)) with target network weights 0. Directly optimizing Equation 1 or Equation can be difficult due to the double sampling problem (Zhu & Ying, 2020). Approximating the inner expectation with one Monte Carlo sample yields high variance. Getting more reliable estimate requires multiple samples, which implies either multiple action executions per state or occurrences of the same state in the replay buffer, an unrealistic assumption for most RL problems. key distinction from the Q-function based objective (Equation 1) is the introduction of the importance weight, which compensates for the discrepancy between the behavior distribution ()5([s) and the current policy л (-1s). Unlike the Q-function that updates estimates solely for the selected action, the V-function does not depend on specific actions and implicitly updates estimates for all actions. The difference between the current and the dete-gengg policy must therefore be taken into account.
Equation 2 closely related to the 1-step V-trace estimate (Espeholt et al., 2018). It can be seen as naive version of V-trace because large difference between the target and behavior policies may result in importance weights approaching either zero or infinity, consequently impacting the Bellman optimization target for the state-value function VØT (st). While truncated importance weights can prevent excessively large values for the Bellman target, the importance ratios and Bellman target may
still approach values close zero. V-trace circumvents this issue by effectively interpolating between the Bellman target and the value function estimate in the 1-step case
Lvaance (0) = EAND [VV (8t)
[(1 [11 (o) +(V)))
(3)
where Pt = ( [x)s///)/////)))) / (ar (8)) (. are the truncated importance weights with a user-specified upper bound (typically Ep = 1). This objective can then also be extended for off-policy corrections of n-step returns (Espeholt et al., 2018). Yet, this formulation has two main drawbacks. Since executing multiple actions for the same state is impractical, or even impossible, V-trace approximates the inner expectation with just one action sample, resulting in potentially undesirable high variance estimates. Moreover, while the V-trace reformulation avoids optimizing the value estimate toward zero for small importance ratios, the interpolation now optimizes it increasingly toward the current (target) value function (see Figure 1). This interpolation also leads to shift of the optimum, which means for samples with small importance ratios the value function is barely making any learning progress and maintains its current estimate. It is important to note that only the optimum changes; the scale of the loss function, and thus the scale of the gradient, remains the same for all importance ratios. In the original work, this issue is less pronounced, as V-trace provides off-policy corrections for asynchronous distributed workers of an on-policy method. In this setup, the policy distributions are expected to stay relatively close, mitigating the impact of any potential divergence. However, in complete off-policy setting, such as in our case, the samples the replay buffer and the current policy deviate significantly faster and stronger from each other. The size of the replay buffer, learning speed, or entropy of different policies can all influence the importance ratio.
3.2 OFF-POLICY STATE- VALUE FUNCTION LEARNING
To address the above-mentioned issues of V-trace, we propose a more effective approach to training the value function by following Mahmood et al. (2014); Dann al. (2014) and shifting the importance ratio in front of the loss function by extending their results to the general function approximation case. We show that both loss functions, i.e., Equation and Equation 4 share the same optimum.
Theorem Consider the following loss minimized with respect to the V-function's 's parameters 0
(aa(s))
Lwis(O) = E ( (-1ss,aa))
(V
(4)
This objective serves upper bound to the importance Bellman loss (Equation 2). Furthermore, this upper bound is consistent; that is, a value function minimizing Equation 4 also minimizes Equation? 2.
4

---

Published as a conference paper at ICLR 2025
The first statement's 's proof relies on Jensen's Inequality, while the proof of the second statement is an extension of Neumann & Peters (2008). The complete proofs are provided in Appendix A. and Appendix A.2, respectively.
This bound closely resembles the standard state-action loss functions used in other off- policy methods but is based on the action-independen state-value function. It just introduces importance weights in front the loss to account for the mismatch between behavior and policy distribution. Importantly, evaluating Equation 4 becomes straightforward using the provided replay buffer D.
Connection to Linear Off-Policy Methods. Interestingly, optimizing this upper bound recovers the linear ooo-pc TD(0) update (Sutton Barto. 2018, Chapter 11.1) for the non-linear setting. While the intuitive objectives for linear off-policy TD(0) might appear to be either Equation 2 or Equation 3, the actual update results from optimizing Equation 4. Mahmood al. (2014); Dann et al. (2014) further demonstrated that Equation 2 and Equation 4 can be connected to ordinary and weighted importance sampling (WIS) in the linear setting, respectively. These findings can be applied for linear value function estimation methods such LSTD (Bradtke & Barto, 1996) by employing the linear version of the loss from Equation 4, thereby obtaining similar theoretical and empirical benefits as those seen with WIS. These previous works in the linear domain concluded that weighting only the Bellman targets is less effective than applying importance weights to the entire loss function, which might even reduce variance (Graves & Ghiassian, 2022). Consequently, linear off-policy TD(0) naturally optimizes a superior objective without explicitly aiming to do so. Despite this evidence, deep learning approaches have predominantly continued to apply importance weights solely to the Bellman targets (Munos et al., 2016; Espeholt et al., 2018), as naively using the WIS loss with non-linear function approximations results in poor performance. Following the standard pattern for transferring ideas from tabular or linear function approximation settings to deep RL (Mnih al., 2013; Lillicrap et al., 2015), set of non-trivial design choices and extensions are required to achieve scalable and performant WIS-based deep RI approach. We will discuss these design choices in Section 3.3 and their effectiveness in our experiments.
Variance Analysis of Importance Sampling Estimators of the Bellman Error. When using the WIS loss, we can draw Monte Carlo samples from the joint state-action distribution. This mitigates the issue of having single action per state, which can be limiting for Equation since requires separate estimates for both state and action distributions. Consequently, the WIS loss helps to reduce variance compared the original and V-trace objectives. Moreover, one of the challenges with the use of importance sampling is that its estimator can exhibit high variance when the target and behavior policies significantly diverge. However, evaluating the variance of the importance sampling estimator is generally intractable (Munos et al., 2016; Koller & Friedman, 2009). Therefore, we consider a stateless MDP, i.e. multi armed bandit problem, simplified scenario. The critic's ss empirical losses for the base (Equation 2), WIS (Equation 3), and V-trace objective (Equation 4) can then be simplified for N samples as
Σ Par(ca)
1 Σ
Va)V V - ((1 Pa) V + Par(a))) 2
Lause
V
Li =
N
N
aTiTS
Σ Pa (v- r(a)
Lwis
N
am
where we denote weights Pa = T(a) (())/m/(a). Taking the derivatives of the above losses with respect to the estimator V and solving the derivative of zero for V yields the following three estimators
Vonse Eanny Par(a)
Eoney par(a) Earny Pa
E Par(a) V
Vyarace
N
Exrns Po.
This simplified scenario illustrates the intuition that the WIS loss yields the self-normalized im- portance weighting estimator, while V-trace results in a squared self-normalized estimator and the base objective yields standard importance weighting estimator. It is well established that the self-normalized estimator is more robust in the presence of extreme importance weights and often yields lower variance than the standard importance weighting estimator in general machine learning (Koller& & Friedman, 2009) and in RL settings (Swaminathan & Joachims, 2015; Futoma et al., 2020). However, the squared estimator from V-trace yields higher variances, discussed in Appendix A.3.
5

---

Published as a conference paper at ICLR 2025
Ant-v4
Humanoid-vv v4
6,000
8,000
Reward
4,000
Reward
6,000
Viearn
MPO
PPO
SAC
2,000
TRPL
V-trace
2,000
Steps (x 10%)
Steps (x10%)
dog-walk
dog-trot
dog-run
1,000
1,000
1,000
Reward
Reward
Reward
2 Env. Steps (x 10")
5
5
Env. Steps (x 10")
Env. Steps (x100))
Figure 2: Mean over seeds and 95% bootstrapped confidence intervals for the nigh-dimensional Gymnasium tasks, the 18-dimensional DMC dog tasks. Vlearn consistently achieves a better asymp- totic performance for tasks. For the dog tasks SAC and MPO even struggle to learn a consistent policy. Compared to V-trace our method is significantly more stable and achieves better final performance.
Furthermore, unlike V-trace, each sample optimizes towards the Bellman target but has an impact on the total loss per step depending on its importance weight, as shown in Figure 1. The smaller importance weights mainly scale the gradient without causing shift in optimizing optimum, such as the current (target) value function. This approach makes learning the state-value function in an off-policy setting more stable and efficient.
different
3.3 DEEP OFT-POLICY POLICY LEARNING WITH STATE- VALUE FUNCTIONS
In order to leverage the previous findings about learning state-value function, we need to ensure, we can also learn policy. Conventional policy gradient techniques frequently use the gradient of the likelihood ratio and an importance sampling estimator to optimize the estimated Q-function. In particular, more effective approach involves optimizing the advantage function, denoted as A" a) Qa (s,a) (*)) Adding the V-function baseline yields unbiased gradient estimator with reduced variance. The optimization can then be formulated as follows
(a(s) AT (s( Ta(as))
max (TT,, = max E (s.0)~
(5)
In the on-policy case, the advantage values commonly estimated via Monte Carlo approaches, such as general advantage estimation (Schulman et al., 2016). Yet, our setting, we cannot reliably compute the Monte Carlo estimate and do have Q-function estimate (Degris et al., 2012). Further, the direct use of the V-function to improve the policy is not feasible. Nevertheless, using replay buffer enables strategy akin to the standard policy gradient. This approach enables to utilize an off-policg estimate of the V-function, resulting in a significant improvement in sample efficiency. The advantage estimate computed using the one-step return of our off- off-policy evaluated value function A (s at) = 7V²" (8++1) (s+). Equation 5 can then be optimized by any policy gradient algorithm, e.g., proximal policy optimization (PPO) (Schulman et al., 2017) or trust region projection layer (TRPL) (Otto et al.. 2021).
In this work, we use TRPL (Otto et al., 2021) as it has been shown to stabilize learning even for complex and high-dimensional action spaces (Otto et al., 2022; 2023). Unlike PPO, it provides mathematically sound and scalable approach to enforce trust regions exactly per state. Moreover, TRPL allows us the constraint policy also during the value function update, which now requires importance sampling. PPO has only the clipped objective for the policy update, and using clipped value function has been shown to potentially degrade performance (Engstrom et al., 2020).

---

Published as a conference paper at ICLR 2025
TRPL efficiently enforces a trust region for each input state of the policy using differentiable convex optimization layers (Agrawal et al.. 2019), providing more stability and control during training and at the same time reducing the dependency implementation choices (Engstrom et al., 2020). Intuitively, the layer ensures the predicted Gaussian distribution from the policy network always satisfies the trust region constraint. This way, the objective from Equation 5 can directly be optimized as the trust region always holds. The layer receives the network's 's initial prediction for the mean I and covariance of a Gaussian distribution and projects them into the trust region when either exceeds their respective bounds. This is done individually for each state provided to the network. The resulting Gaussian policy distribution, characterized by the projected parameters fi and Σ, is then used for subsequent computations, e.g, sampling and loss computation. Formally, as part of the forward pass, the layer solves the following two optimization problems for each state
arg min dlmean (its (((5)) S.I. dmean (fr,, Moold S cp P.
arg min dewv E., I(s) 8.1. deav (Σe, Enta(s)) S SE E,
where ft, and Eg are the optimization variables for input state s. The trust region bounds Eu. eg are for the mean and covariance of the Gaussian distribution, respectively. For the dissimilarities between means dmean and covariances dewv, we use the decomposed KL--ivvrgence. How to receive the gradients for the backward pass via implicit differentiation is described in Otto et al. (2021).
As the original V-trace algorithm is primarily used for off- off-policy corrections in distributed on on-policy learning methods, it typically does not incorporate a target network. In contrast, it is common practice for off-polic methods (Haarnoja et al., 2018; Fujimoto et al., 2018; Abdolmaleki et al., 2018) to do so. This approach was also employed by V-trace's 's predecessor, Retrace (Munos et al., 2016), hence we adopt it for our use case as well. Additionally, we make use of further common improvements of actor-critic and policy gradient methods, including twin critics and delayed policy updates (Fujimoto et al., 2018), tanh squashing (Haarnoja et al., 2018), as well as advantage normalization (Schulman et al., 2017; Otto et al., 2021). While the overestimation bias is not direct problem when using just state-value functions, we found twin critics beneficial in practice. We assume the twin network acts as a small ensemble that provides a form of regularization. Given there is no significant drawback and the widespread adoption of the twin network approach in other baseline methods, we chose to maintain in our case as well. Finally, similar to prior works (Munos et al., 2016; Espeholt al., 2018) aim to reduce the variance by replacing the standard importance sampling ratio with truncated importance sampling min( (T (ax) /t)/Tb( (T((a))))) <p)
Section 4.2 demonstrates the importance of employing principled trust region approach and the necessity of these design choices for increased final performance. We call the final practical algorithm Vlearn, referring to its ability to learn from state-value functions. Pseudo-Code is given in Appendix B.
3.4 CONSIDERATION REGARDING OLD AND BEHAVIOR POLICY
For Vlearn, we keep track of three different policies: The current policy Ta. which is optimized, the old policy Vold- which is used as a reference for the trust region, and the behavioral policy m.. which is needed for off-policy correction using importance sampling.
In particular, in an off-policy setting, the behavior policy m, is a mixture of all past policies that contributed to the replay buffer. However, storing and/or evaluating this would be expensive, so we assume that each sample belongs to only one mixture component, specifically the policy that originally created the action. Since we rely on importance sampling for the off-policy correction, we can simply store the loo--pprobayilityy for each action as part of the replay buffer to represent T6 during training at minimal additional cost. This ppobbabity may then be used for importance sampling without the necessity of further computations.
Similar to most on-policy trust region methods, Told can simply be copy of the main policy network from the previous iteration. Choosing Tr for the trust region would be detrimental to performance because it would slow down the policy network update or, in the worst case, force the policy back to a much older and worse policy distribution. Conversely, we cannot use the copied Told as a behavior policy because the actual behavior policy m, can be arbitrarily far away, especially for older samples.
7

---

Published as a conference paper at ICLR 2025
Viearn
SAC
PPO
TRPL
V-trace
Pose
Pose
Hard
Reach
Reach
Hard
KeyTurn
MyoSuite MyoHand
Success
0.8
Succ0se
0.4
Key Turn
Hard
ObjHoll
ObjHoold Hard
PenTwirl
PenTwirl- Hard
0.2
Success
0.5
Env. Steps (x 10%)
Figure 3: Performance on the 39-dimensional MyoHand from MyoSuite. Left. IQM and 95% bootstrapped confidence intervals for the aggregated performance over all MyoHand tasks. Right. Mean over 10 seeds and 95% hootstrapped confidence intervals for the individual performances of all MyoHand tasks. While Vlearn does not outperform baselines on all tasks, it performs well across all tasks and achieves the highest aggregated performance.
EXPERIMENTS
For our experiments, we evaluate Vlearn on various digensiimnsi continuous control tasks from Gymnasium (Towers al., 2023), DMC (Tunyasuvunakool al., 2020) and MyoSuite (Vittorio et al., 2022). As baselines, we trained the standard off-policy methods SAC (Haarnoja et al., 2018) and MPO (Abdolmaleki al., 2018) as well as PPO (Schulman et al., 2017) and the on-policy version of TRPL (Otto et al., 2021). In addition, by replacing our lower bound objective from Equation 4 with the objective in Equation 3. All other components remain the same in Vlear.. This comparison aims highlight the difference in the placement of the importance ratios. To ensure a fair assessment, we want to eliminate any external factors that could influence the results and thus do not use n-step returns for both V-trace and Vleam.
compare method to V-trace (Espeholt et al., 2018)
All methods are evaluated for 10 different seeds each, and their performance is aggregated using mean values and 95% bootstrapped confidence intervals for individual tasks described in Agarwal et al. (2021). We maintain uniformity in the architecture of the policy and critic networks for all methods, incorporating layer normalization (Ba al., 2016) before the first hidden layer. Hyperparameters are kept constant and only adjusted appropriately for the higher dimensional dog and MyoSuite tasks. Detailed hyperparameter information for all methods can be found in Appendix D.
4.1 HIGH-DIMENSIIOAL CONTROL TASKS
As illustrated in Figure 2, for the two high-dimensional Gymnasium tasks, Ant-v4 and Humanoid-v4, Vlearn outperforms all other baselines. Across both environments, Vlearn demonstrates superior convergence speed and asymptotic performance. While the improvement is more subtle for Ant-v4 with its 8-dimensional action space, Vlearn achieves 25% increase over SAC in the larger 17- dimensional action space of Humanoid-v4, Here, focusing solely on learning state-value function proves to be less complex and more effective approach than attempting learn the full state-action value function. MPO is unable to achieve competitive performance for either Ant-v4 or Humanoid-v4,
The DMC (Tunyasuvunakool et al., 2020) dog tasks pose the most challenging locomotion problems in our evaluation, with a 38-dimensional action space modeling realistic pharaoh dog. Consistent with the above Gymnasium tasks, Vlearn exhibits superior performance on these high-dimensional dog locomotion tasks. Although we found that SAC benefits from layer normalization for these tasks, it struggles to learn well-performing policies for all three movement types. While SAC improves for "casier" motions, its convergence remains highly unstable, and its final performance often falls below that of on- on-policy methods. Similar to the higher dimensional Gymnasium tasks, MPO is also unable solve the dog tasks. In contrast, Vlearn learns reliably for all three distinct dog movement types.
8

---

Published as a conference paper at ICLR 2025
2c3
1e4
5ed
Vlearn
V-trace
PPO loss
1e5
2.GeG
le6
No Twin
Cy
Humanoid-vv Buffer size
Ant v4 Components
Humanoid-vv Components
8,000
6,000
8,000
Reward
6,000
Reward
4,000
Reward
6,000
4,000
4,000
2,000
2,000
2,000
Env. Steps (x10")
Env. Steps (x10*)
Env. Steps (x10%)
Figure 4: Left. Ablation study on the impact of replay buffer size on policy performance. For smaller replay buffers learning becomes unstable or does not converge, while larger sizes tend to lead to similar final performances. Right. Investigation in the importance of various design choices of Vlearn on Ant-v4 and Humanoid-v4.
From MyoSuite (Vittorio et al., 2022), we evaluated all tasks involving the 39-dimensional myoHand (see Figure 3). Vlearn shows consistently strong performance across wide range of tasks, achieving the highest aggregated performance. Although it does not always outperform all baselines in specific cases, Vlearn tends to learn more stable behavior compared to SAC and V-trace. This stability is particularly evident in the easy versions of the tasks.
In direct comparisons between Vlear and V-trace, Vleam consistently demonstrates superior perfor- mance across all tasks. In particular, V-trace demonstrates minimal learning within a fixed number of environment interactions. For the dog tasks, the V-trace estimator fails to learn and is among the worst baselines, even falling behind on-policy methods. Our investigations revealed that while V-trace eventually able learn, experiences significant drop in performance after an extended training period. This discrepancy can be attributed to the sole difference between the two experiments, namely the V-trace objective. The key distinction between our objective and V-trace lies in how they handle situations where the importance ratio approaches zero and the position of the expectations. In our objective, samples get assigned a weight close to zero, minimizing their influence on the gradient. Conversely, V-trace attempts bring these samples closer to the target network, potentially leading to performance degradation. Additionally, estimating the joint statc-action expectation is typically more stable. This observation aligns with findings in Mahmood et al. (2014); Dann al. (2014); Graves & Ghiassian (2022), which shows that importance sampling exclusively for the Bellman targets can lead to inferior performance. While they evaluated this only for the linear case, we found similar results for general nonlinear function approximation as shown in Figure 2.
4.2 IMPORTANCE OF DESIGN CHOICES
We investigate the effect of replay buffer size as well as the individual components of our method on the learning process. We trained multiple agents the Gymnasium Humanoid-vä v4 with varying replay buffer sizes from 2e3 up to 1e6, for 10 seeds each. Note that while the original V-trace (Espeholt et al.. 2018) relies on a relatively large replay buffer, the massively parallel computation used there implies that policy distribution differences are not as pronounced as in standard off-policy methods like Vlearn or SAC. As shown in Figure 4 (left), significant improvement occurs when transitioning from updating the policy and critical setting near an on-policy setting, using the smallest replay buffer size, to using medium buffer. In particular, moving from a small buffer medium buffer yields significant benefits, while there is little difference between buffer sizes at the upper end of the range. In our experiments, we found that a replay buffer size of 5e5 consistently produced optimal performance across all tasks and was used for the results in Figure 2, Figure 3, and Figure 5.
In order to analyze our design choices, we trained several variations of Vlearn for Ant-v4 and Humanoid-v4 to investigate the effects of each component (Figure 4 right). As naive baseline, we removed importance sampling (No IS) and assumed that all samples in the replay buffer were from the current policy. Without the importance sampling correction, the agent environments. Replacing the TRPL policy loss with the clipped PPO loss (PPO loss), our results suggest that the heuristic trust region provided by PPO is insufficient for the off-policy case, where
unable learn in both

---

Published as a conference paper at ICLR 2025
Vlear
SAC
MPO
PPO
TRPL
V-rrace
Hopper-v4
Walker2D.vv
HallChestah-ve v4
15,000
3,000
Reward
Reward
4,000
Reward
10,000
2, 2,000
1,000
2,000
5,000
0
Env. Steps (x10
Env. Steps (x10%)
Env. Steps (x10%)
Figure 5: Mean over 10 seeds and 95% bootstrapped confidence intervals for the low-dimensional Gymnasium tasks. While SAC achieves the best performance on the HalfCheetab-v4, Vlearn still performs similar MPO and achieves and equivalent performance for Hopper-v4 and Walker2D-v4. V-trace once again cannot make any meaningful progress in the off- off-policy setting.
stabilizing learning is more important. While the PPO loss achieve reasonable performance on Humanoid-v4, its asymptotic performance lags behind that of the agent using TRPL. and is not able to learn for Ant-v4. For importance weight truncation, we followed previous work (Espeholt et al., 2018; Munos et al., 2016) by choosing é, = 1 to reduce variance and potentially avoid exploding gradients. Similar to the PPO loss, saw that learning becomes unstable for larger truncation level (cp 20). Finally, we trained an agent without the twin critic networks (No Twin), which performs worse in both environments. We assume that the twin network can be seen as small ensemble that provides a form of regularization that becomes more beneficial for higher dimensional problems.
4.3 Low- DIMENSIONAL CONTROL TASKS
While the core benefit of our method is in ligh-dimmnsional action spaces, we ensure Vlearn is able to achieve competitive performance on lower-dimensee lierrdimensiionl tasks. Generally, Vleam performs well on the lower dimensional Gymnasium tasks shown in Figure 5. Its asymptotic performance comparable to that of other baselines, except for SAC's performance on HalfCheetah-vá While Vlearn outperforms on-policy methods on the HalfCheetah- it exhibits slower convergence rate compared to SAC and reaches lower local optimum, similar to the on-policy methods. Generally, HalfCheetah-v4 seems to an outlier that challenges the learning capabilities of all trust-region methods. Even MPO, employing related KL regularization concept, does not match the performance of SAC. Moreover, the environment itself has shown extreme behavior in the past (Zhang et al., 2021). However, for the other lower dimensional tasks, Vlearn and MPO perform comparably to the remaining baselines.
5 CONCLUSION AND LIMITATIONS
In this work, have demonstrated efficient approach for learning V-functions from off-policy data using an upper bound objective to subsequently update the policy network. Our approach not only ensures computational efficiency, but also demonstrates improved stability and performance compared to existing methods. Integrating this idea with TRPL yields an effective off-policy method that is particularly well-suited for scenarios involving high-dimensional action spaces.
Although our method excels in handling high-dimensional action spaces, it may still require a substantial amount of data to achieve optimal performance. Hence, sample efficiency remains challenge that needs to be addressed further. In future work, we aim to further enhance the stability of off- off-policy state-value dunction-based methods and explore their potential benefits tackling even higher-dimensional problems, such those found in overactuated systems. Additionally, we interested in which ideas for Q-function-basee methods also translate to the V-function only setting, such ensembles (Chen et al., 2021). distributional (Bellemare et al., 2017) or categorical critics (Farebrother et al., 2024) as well as simplified ways to stabilize off policy learning (Bhatt et al., 2024; Gallici al., 2024). Furthermore, which offers the opportunity to leverage pre-collected data efficiently. opening doors to real-world applications and minimizing the need for extensive data collection.
are looking to extend our method to the realm of offline RL,
10

---

Published as a conference paper at ICLR 2025
6 REPRODUCIBILITY STATEMENT
In our experiments, we utilized standard RL benchmarks and followed the evaluation protocols from Agarwal et al. (2021). The pseudo-code of our method, implementation details, and hyperparameters used in our experiments can be found in the Appendix. Readers can replicate our results by following the procedures outlined in these sections.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum posteriori policy optimisation. In International conference on Learning Representations, 2018.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J Zico Kolter. Differentiable convex optimization layers. Advances in neural information processing systems, 32, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXhv:1607.06450 2016.
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. International conference on machine learning, pp. 458. PMLR, 2017.
Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pp. 679-684, 1957.
Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In The Twelfth International Conference on Learning Representations, 2024.
Steven J Bradtke and Andrew Barto. Linear least-squares algorithms for temporal difference learning. Machine learning, 22(1):33-57, 1996.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double g-learning: Learning fast without a model. In International Conference on Learning Representations (ICLR), 2021.
Christoph Dann, Gerhard Neumann, Jan Peters, et al. Policy evaluation with temporal differences: survey and comparison. Journal of Machine Learning Research, 15:809-883. 2014.
Thomas Degris, Martha White, and Richard S. Sutton. Off-policy actor-critic. In Proceedings of the 29th International Conference on International Conference on Machine Learning, ICML'12, pp. 179-186, Madison, WI, USA, 2012. Omnipress. ISBN 9781 450312851.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. In International Conference on Learning Representations, 2020. URL http: //wrxiv..rg/aaaaaaiee..aaaa attlaa///////2/ 12729.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. International conference on machine learning. pp. 1407-1416. PMLR, 2018
Jesse Farebrother, Jordi Orbay. Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arX1v:2403.03950. 2024.
11

---

Published as a conference paper at ICLR 2025
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Joseph Futoma, Michael C Hughes, and Finale Doshi-Velez. Popcor: Partially observed prediction constrained reinforcement learning, arXiv preprint urXiv:2001.04033 2020.
Matteo Gallici, Mattie Fellows, Benjamin Hllis,, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.048/1 2024.
Eric Graves and Sina Ghiassian. Importance sampling placement in off off-policy temporal-difference methods. arXiv preprint arXiv:2203.10172. 2022.
Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard Turner, and Sergey Levine. Q-prop: Sample-efficient policy gradient with an off-policy critic. arXiv preprint urXiv:1611.022477 2016
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learming, pp. 1861-1870. PMLR, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint vrXiv:1509.0297/ 2015.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8:293-321, 1992.
Michael Luo, Jiahao Yao, Richard Liaw, Eric Liang, and Ion Stoica. Impact: Importance weighted asynchronous architectures with clipped target networks. In International Conference on Learning Representutions (ICLR), 2020.
A Rupam Mahmood, Hado P Van Hasselt, and Richard S Sutton. Weighted importance sampling for off-policy learning with linear function approximation. Advances in neural information processing systems, 27, 2014.
Wenjia Meng, Qian Zheng, Yue Shi, and Gang Pan. An off-policy trust region policy op- timization method with monotonic improvement guarantee for deep reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 33(5):2223-23 22335, 2022. doi: N0.1109/NNNSS..2000.000
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13 2013.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Trust-pel: An off-policy trust
region method for continuous control. In International Conference on Learning Representations (ICLR), 2018. URL https: openreview. net /ttpt ipdffiiiiyrreeeccs
Gerhard Neumann and Jan Peters. Fitted q-iteration by advantage weighted regression. Advances in neural information processing systems, 21, 2008
Fabian Otto, Philipp Becker, Ngo Anh Vien, Hanna Carolin Ziesche, and Gerhard Neumann. Dif- ferentiable trust region layers for deep reinforcement learning. In International Conference on Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2025
Fabian Otto, Onur Celik, Hongyi Zhou, Hanna Ziesche, Ngo Anh Vien, and Gerhard Neumann. Deep black-box reinforcement learning with movement primitives. arXiv preprint arXiv:2210.09622. 2022.
Fabian Otto, Hongyi Zhou, Onur Celik, Ge Li, Rudolf Lioutikov, and Gerhard Neumann. Mp3: Movement primitive-based (re-) planning policy. arXiv preprint arXiv:2306.12729 2023.
Art Owen. Monte carlo theory, methods and examples, 2013.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. trrantage-weightee regresioooorriioo regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.001170 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region
Policy Optimization. In Proceedings of Machine Learning Research, pp. 1889-1897, 2015. URL
http:// httt:::/procceeiine ngs ml r. press vv37 schuccllmnn schulmant 5, html.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. In arXiv preprint, 2017. URL
06347.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Joannis Antonoglou, Veda Panncershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-48) 2016.
Richard S Sutton. Learning to predict by the methods temporal differences. Machine learning, 9-44, 1988.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. advances in neural information processing systems, 28, 2015.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012. doi: 10. 109/1ROS. 2012 2.666609.
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL https::////........ zeeooo..o..... org/ wtt.o//cooooooooooo 8127025.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm dm-control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi: https://doi.
org/10. 1016/...016/.simmpp.. 2020. 100022. URL https::///// http://wwwieeeeeeeeeeeeeeeeeeeee wttpeeettcttteeeeeeeeeeeeeeeeeee scienceee/////// httpccceeepiii/265eee22
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
Caggiano Vittorio, Wang Huawei, Durandau Guillaume, Sartori Massimo, and Kumar Vikash. Myosuite contact-rich simulation suite for musculoskeletal motor control. https://githu: com/ htt.fa/ebboorrrarrrc/aaaaaaaaaae 2022. URL https: : /////a/xvv.//////////////////a// http://wwwwg///a////2////2 13600.
13

---

Published as a conference paper at ICLR 2025
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2003. PMLR, 2016.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. In International Conference on Learning Representations (ICLR), 2017.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.
Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp. Kurtland Chua, Frank Hutter, and Roberto Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp- 4015 4023. PMLR, 2021.
Shangtong Zhang, Wendelin Bochmer, and Shimon Whiteson. Generalized off-policy actor-critic. Advances in neural information processing systems, 32, 2019.
Yuhua Zhu and Lexing Ying. Borrowing from the future: An attempt to address double sampling. In Mathematical and scientific machine learning, pp. 246-268. PMLR, 2020.
14

---

Published as a conference paper at ICLR 2025
A DERIVATION
We now show the proof of our new objective function. We start with a standard loss function for learning a V-value function for policy as defined in Equation 2. Then we derive its upper bound shown Equation For simplicity, we denote V r(s, + + (a()s)
A.1 UPPER BOUND ORJECTIVE
(x) (s)
(V) - P))
i(als) V (a(ass)
21
= (s) (s)
(x](s) (awors(-ls)
27
(Taas)
T (aass v To (a|s)
-Esner (x)
Bawno(-ls)
(a(s)) Vi (s) Bawn
(V7 does not depend on a)
(aas)) (a(as)
F)] 1) (V) -s)]-p)
=Bsnd (x)
Eawmo((l))
/ To(aas)
(als) (als)
=B (x)
(v) - da
weight terms
s ( Laddy
=Eerdt(e) (e)
<Esnd (x)
tf (x) da
(aass) (V) - F)3]
=Esnd (x)
(-
Tala)s))
(Jensen's inequality)
(als)
(V/ (s) v)2
-E (m)()))
Ty (a(ss)
Here, we denote the convex function f(x) = x² the weight terms are in [0, and normalized, tda f no (a(s)) s(als) da f (als da 1 and ds the stationary distribution induced by policy π. Therefore, one can estimate the loss using Monte Carlo samples from the joint state-action distribution
(o]s)
T ((t.)1ss)) ΣΣ 3 {s())
(V - (ris + (Vo(s41))))
A.2 CONSISTENCY OF UPPER BOUND OBJECTIVE
We follow a similar derivation as in Neumann Peters (2008) (for the case of state-action value function Q(,, a)) prove the consistency between I," (0) and I.(0), i.e., the solution for minimizing
15

---

Published as a conference paper at ICLR 2025
L(Ø) is the same for the original objective (0).
[VV (s) (1))
L (0) (s)
(a)
Radd (+)
V8 (s)2 2V 2V) (s) [V] + E (
(a)))(1))) (p (VV - 1
(( Exnd (x)
Esdd (1) (s)] - 2V5 (s) + V²]]
Ea-a (V - 2V (s) [pV] + (-)o)())))) [pD2]]
(Vo" does not depend on a)
(o 2 (a) [V] -Bant(-aa) [V]]]
where p (alss) (Ta(ss)) The above results show that L' (0) and L(0) are identical except for constant term added, which remains independent of Vū.
A.3 ANALYZING THE VARIANCE OF THE DIFFERENT IMPORTANCE SAMPLING ESTIMATORS
As shown in Section 3.2, for the bandit case the V-trace and weighted importance sampling estimators
are given by
Vivarce Eaway Par(a) Eewny pe
and Vwis Eowny Par(a)
Earns Pa
with the crucial difference that the importance weights are squared for V-trace. We can now consider the effective sample sizes for both, the standard nelf-norma importance sampling estimator, pitz/ E, Pj and the squared version p/ 23 03. They are given by
off = 1E Pt)
(EP-P3)) Di
and n
Σn Pf
respectively. Given that P with Σ Pj being strictly positive and it holds that naa 2 ne (ef. Owen (2013) Chapter 9.3). Consequently, the variance in the squared case is always larger or equal to that of the standard estimator.
B PSEUDO CODE
Algorithm Pseudo Code for Vlearn
1: Initialize policy Φ; critics 01, 02; target critics 01, 82
2: Initialize wold Ra; trust region bounds Cu. <
3: Initialize replay buffer D; truncation level Ep
4: while not converged do
5: Collect sample (s, 0, r, 8 (', and add to D
6: Sample batch
{(s, a, r. To (c)s)))t=.... from D 7: Get current policy (als)) for all in B
8: Compute projected policy #. = TRPL((TT, Tolds Eus ex) for
9:
Update critic network E (1,2) with gradient
Vo, K EB min (a)=)) mn(als) <p) (V (r (V (s)))
10: if update policy then
11:
Compute advantage estimates Å = r min Vo min;=1,2 (s)
Update policy with trust region loss V end if
K Σ min
(a(aas)
(a)aa) cp) A (ad(Ts, #)
13:
14:
0, < Tθ, + (1 -1-18 T)0, for {1,2]
15:
if update old policy for trust region then
16: Vold f- T
end if
18:
end while
16

---

Published as a conference paper at ICLR 2025
C
ADDITIONAL RESULTS
Viearn
V-trace
PPO
No IS
Twin
6
20
Hopper-v4
Walker2D-v4
HaliCheetahh-v
8,000
3,000
Reward
Reward
Reward
2,000
4,000
1,000
2,000
2,000
2 Env. Steps (x (x10%)
2 Env. Steps (x 109)
Env. Steps 1010))
Figure 6: Performance of Vlearn without importance sampling (no IS) and when replacing TRPL with the PPO loss low-dimensional Gymnasium tasks. Shown are the over 10 seeds and 95% bootstrapped confidence intervals. Similar to the high-dimensional setting, using the PPO loss ignoring importance sampling altogether is not sufficient ensure stable training and the performance suffers for most tasks.
D HYPERPARAMETERS
We conducted a random grid search to tune all hyperparameters for both Gymnasium' ! (Towers et al., 2023) and DMC (Tunyasuvunakool al., 2020), which are based Mujoco (Todorov et al., 2012). We included learning rates within the range [10-
10 1004,00400000 10 5 10 10 10-3) mean 10-1,1 x
and covariance bounds within [10-5 10-4 5
10 10-20-22000 2 10
10-"). batch sizes of [16, 32, 64, 128, 256, 512], policy update intervals of [1, 2]. and network sizes consisting of two hidden layers with [64, 128, 256, 512) neurons each. All models were trained on an internal cluster on one Nvidia V100 for approximately 1-3 days, depending on the task. The best hyperparameter sets, as shown in Tables and 2, were then trained for multiple seeds.
We found it is important of achieving harmonious balance between the selected learning rate and the mean/covariance bound of the trust region. We observed that an aggressive learning rate does not pair well with tight bound, and vice versa. Interestingly, when compared to other off- off-policy methods, we found that Vlearn and off-policy V-trace performed better with smaller batch sizes, and we did not observe any benefits from using warm start samples. In general, we noticed that the hyperparameters were a blend of configurations established from existing off- off-policy and on- on-policy methods. We attribute this to the structural similarities between our approach and both on-policy methods, such as PPO (Schulman et al., 2017) and TRPL (Otto et al., 2021), as well as off-policy methods, such as SAC (Haarooj et al., 2018) and MPO* (Abdolmaleki et al., 2018).
"Published under MIT license at httt::///thhhu..oo//////e/eoeaan
PPullished under Apache-2.0 license at https: http://:///gtttuu..oo/////////e/ c
control
*Published under Apache-2.0 license at wwwp:tttoo..ccooooooooooooooooot deepmieeeiiee nd/ //mjjooo
*Published under Apache-2.0 license /ttps:://ittub..cmm
deepmi nd/ acme/
17

---

Published as a conference paper at ICLR 2025
Table Hyperparameters for the Gymnasium (Towers al., 2023) experiments in Figure and Fig- ure The larger sample size for the the two on-policy methods is for the Humanoid-v4 experiments. Vlearn/V-trace
PPO
TRPL
SAC
MPO
number samples
2048/16384
2048/16384
1
1
GAE X discount factor
0.95
0.95
0.99
0.99
0.99
0.99
0.99
n.a.
0.05
n.a.
le-3
ex:
n.a.
0.0005
0.0005
n.a.
2c-6
optimizer updates per epoch
adam
adam
adam
adam
adam
10
20
1000
1000
1000
learning rate
3e-4
5e-5
5e-4
3c-4
3e-4
epochs critic learming rate critic (and alpha)
10
10
n.a.
n.a.
n.a.
3c-4
3c-4
5e-4
3c-4
3e-4
learning rate dual number minibatches
n.a.
n.a.
n.a.
n.a.
le-2
32
64
n.a.
n.a.
n.a.
batch size
n.a.
64
256
256
buffer size
n.a.
n.a.
5c5
lc6
le6
learning starts policy update interval
0
10000
10000
n.a.
n.a.
2
polyak, weight
5e-3
5e-3
5e-3
trust region loss weight
T.a.
10
n.a.
n.a.
num action samples
n.a.
n.a.
n.a.
20
normalized observations normalized rewards
True True
True False n.a.
Truc False n.a.
False False n.a.
Falsc
False n.a.
observation clip
reward clip
10.0
n.a.
n.a.
n.a.
n.a.
critic clip importance ratio clip
0.2
0.2
n.a.
n.a.
D.a.
n.a.
hidden layers
[64, 64] [64, 64] tanh 1.0
[64, 64] [64, 64]
[256. 256]
[256,256]
[256.256]
hidden layers critic
[256, 256]
[256,256] relu
[256,256]
hidden activation
tanh 1.0
relu 1.0
relu
initial std
1.0
1.0
18

---

Published as a conference paper at ICLR 2025
Table 2: Hyperparameters for the dog tasks from DMC (Tunyasuvunakool al., 2020) and the
MyoHand tasks from Myosuite (Vittorio et al., 2022) in Figure 2 and Figure 3, respectively.
PPO
TRPL
Vlearn/V-trace
SAC
MPO
number samples
16384
16384
1
GAE À discount factor
0.95
0.95
n.a.
n.a.
0.99
0.99
0.99
0.99
0.99
C
0.05
0.1
1e-3
n.a.
0,0005
0.0005
n.a.
le-6
optimizer updates per epoch learning rate epochs critic learning rate critic (and alpha)
adam
adam
adam
adam
adam
10
20
1000
1000
1000
3e-4
5e-5
le-4
3e-4
3e-4
10
10
3e-4
3e-4
le-4
3e-4
3e-4
number minibatches
32
64
n.a.
n.a.
le-2
batch size
n.a.
n.a.
64
256
256
buffer size
n.a.
n.a.
5e5
le6
le6
learning starts
0
0
10000
10000
policy update interval
n.a.
n.a.
1
polyak weight
n.a.
n.a.
5e-3
5e-3
5e-3
trust region loss weight
n.a.
10
n.a.
n.a.
normalized observations normalized rewards
True True
True
True False
False False n.a.
False
False
False
observation clip
10.0
n.a.
reward clip critic clip importance ratio clip
10.0
n.a.
n.a.
n.a.
n.a.
0.2
n.a.
n.a.
n.a.
n.a.
0.2
n.a.
n.a.
n.a.
n.a.
hidden layers
[64, 64]
[64, 64]
[512, 512]
[512,512]
[512,512] [512,512]
hidden layers critic
[64, 64]
[64, 64)
[512, 512]
[512,512] relu 1.0
hidden activation
tanh 1.0
tanh 1.0
relu 1.0
relu
initial std
1.0
19


Reviews:
[{"review_id": "qaRC6AU2Ix", "reviewer": "Reviewer_ej3J", "summary": "This work proposes a fully off-policy reinforcement learning algorithm based on policy gradient and importance sampling. Crucially, the algorithm does not attempt to estimate a state-action value function, but only a value function. As a consequence, the algorithm is well-suited for large action spaces. The authors build upon existing analysis in the linear case suggesting that the entire Bellman error should be reweighted, instead of the Bellman target alone. A simple analysis in bandit settings is provided, confirming that the chosen objective has less variance. The authors follow this idea in the deep reinforcement learning settings. The algorithm is evaluated in high-dimensional continuous control environments, showing that it is essential to combine the objective with a stable algorithm such as TRPL. The algorithm outperforms all baselines in large action spaces, and remains competitive in standard environments with smaller action spaces.", "questions": "- Can the authors comment on the first weakness?\n- A lot of work seems to be necessary to ensure stable training of Vlearn. How is it expected to perform in noisy settings, or under non-stationary rewards? Would its performance be impact similarly to that of baselines, or more?\n- In Appendix C, \\citet commands should be replaced with \\citep.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper is clearly written, and the contribution is well outlined.\n- Figure 1 is very helpful in describing the different behavior of Vlearn and Vtrace.\n- The experimental evaluation is rigorous, involving multiple seeds and controlling for implementation differences and hyperparameter selection.\n- Authors are transparent regarding limitations of the method on standard benchmarks (see Figure 4).", "weaknesses": "- This work opens an important question: why is a PPO-style loss insufficient for Vlearn, which thus requires TRPL? Figure 3 shows that changing the objective impacts performance in a drastic way. The answer provided (\"the heuristic trust region provided by PPO is insufficient for the off-policy case\", as well as lines 290-295) is not entirely satisfactory. A more detailed empirical or formal analysis would be very interesting.\n- The contribution of this paper is largely in bringing ideas from the linear setting to deep RL. While this implies that the objective and the guarantees are not exactly novel, verifying that formal ideas can work in practice remains very valuable. In particular, the application of WIS in deep RL is, to the best of my knowledge, novel.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bU7KTsP3hX", "reviewer": "Reviewer_HaA8", "summary": "This paper introduces Vlearn, a novel off-policy policy gradient method for reinforcement learning. Extensive experimental results demonstrate that Vlearn achieves promising performance compared to existing approaches.", "questions": "I am curious about the distinction between “high-dimensional action space” and “low-dimensional action space.” In Section 4.3, experiments indicate that Vlearn performs less effectively than SAC. Could you elaborate on why Vlearn may not perform as well in low-dimensional action spaces?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Vlearn demonstrated promising performance compared to existing reinforcement learning algorithms in high-dimensional action spaces.\n- Extensive experiments and ablation studies confirmed the effectiveness of Vlearn.\n- Theoretical analysis provided insight into the motivation and design choices behind Vlearn.", "weaknesses": "Please refer to the questions part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tOuiXncnGk", "reviewer": "Reviewer_mtMd", "summary": "The paper proposes to use state-only value functions to model the critic during off-policy learning instead of using a state-action critic. It introduces a method to use importance sampling to correct for the distribution mismatch between data in the replay buffer and the policy being learned. The method has a theoretical justification and is applied to various high-dimensional domains.", "questions": "I am having trouble wrapping my head around the fact that since each transition sample in the replay buffer has different associated log probabilities, learning will not be unstable. Typically, in importance sampling, the target and reference distributions are fixed. In this case, the target distribution is changing (since the policy is learning) and the reference is also changing since different transition samples in the buffer have different log probabilities. Do the authors have a sense of why this is not affecting learning? I would have expected instability. It would be interesting if there was an experiment to test this out.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The technique is simple and relatively straightforward to adopt based on the presented pseudo-code.\n- It is theoretically justified. Their proposed loss function upper-bounds the loss we actually want to minimize, and it shares the same solution as the loss we care about.\n- Experiments are run over multiple seeds and results present confidence intervals. \n- The presented ablations show the importance of different components of the paper’s algorithm.\n- Figure 1 nicely illustrates some intuition about the method.", "weaknesses": "- The exact motivation of the paper is unclear to me. There are different thoughts scattered throughout, but I think these need to be consolidated to better understand what is actually being solved here. For example there are several seemingly related and unrelated comments that I think need to be consolidated: a) why are high-dimensional actions the actual issue? Our methods already handle high-dimensional states. If high-dimensional actions are an issue, it should be made clearer why. b) “Q-functions dependency on actions prevents it from updating actions that were not used while acting” (line 40); I agree this is an issue, but its only mentioned in passing. c) or is the main goal to scale off-policy algorithms?; if so there, are many other ideas and algorithms that need to be considered. d) or is the idea to just make an algorithmic improvement to V-trace? (line 173).\n- Related to the above point. Since the motivation is unclear, I am not sure if the proposed baselines in the experiments are fair. The comparison to V-trace makes sense, but all the other algorithms are base/original implementations without any other techniques, so I would expect them to perform poorly anyway. For example, if the goal is to stabilize off-policy learning then there are plenty of algorithms that can be evaluated such as using layernorm [1] or batch norm [2], or removing the target network [1, 2] etc.\n- The abstract makes a claim that the method improves upon exploration and exploitation. I dont think this is necessarily accurate. The experiments do not explicitly test for say exploration. It would be safer to say that it yields high-return agents.\n- Lines 119 to 121 need a citation.\n- The paper needs to fix its use of “target”. The RL literature typically uses it to refer to the target network and the target that the current estimate is trying to regress to. However, the paper uses target to refer to the current estimate as well (line 177), which makes parsing the paper challenging.\n- Line 163 onwards discusses essentially the double sampling problem. A citation is needed here (see https://koppelresearch.wordpress.com/2018/09/13/double-sampling-a-quirk-of-reinforcement-learning/) and the term double sampling should be used. Also the way double sampling is presented here makes it seem that its an issue with equation (2) only and not equation (1), but its an issue for (1) as well.\n- In my opinion, if the goal is for improved off-policy learning, the proposed algorithm introduces many more complicated pieces such as twin value function networks that are more complicated than methods that improve off-policy learning with simpler techniques (such as removing the target network and simply using layer norm [1]).\n\n[1] SIMPLIFYING DEEP TEMPORAL DIFFERENCE LEARNING. Gallici et al. 2025.\n\n[2] CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity. Bhatt et al. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uhw9ccyKFx", "reviewer": "Reviewer_urYk", "summary": "The authors introduce Vlearn, an algorithm based on weighted importance sampling with state-value functions V, rather than state-action value functions Q. The authors argue this approach eliminates the complexity of learning action-values, particularly in domains with high dimensional action spaces and demonstrate performance benefits over SAC in MuJoCo environments.", "questions": "- I am surprised by the effectiveness of IS in a setting with high dimensional actions, where the ratios may have particularly high variance. Clipping can help avoid overly large values, but what about overly small values? Do the ratios remain close to one especially as the replay buffer grows and the behavior and current policies become more distinct?\n\n- DMC also has a humanoid environment with a high dimensional action space. Does Vlearn also achieve a high performance in that task?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Vlearn provides a modern version of an old idea (weighted IS), including techniques from recent deep RL algorithms and showing competitive performance with deep RL algorithms. \n\n- This is a (mostly) new flavor of algorithm in the context of deep RL and may open up the design space for further algorithms.", "weaknesses": "- Conceptual novelty is not high since the algorithm is based on well-known ideas in the literature. I think the “modern” version of these ideas has value, however it’s unclear how much value that is exactly. \n\n- While the results are strong against SAC, SAC is definitely not state of the art in any of the tested domains. For example, TQC or TD7 for gym and BRO or iQRL for DMC are all model-free RL algorithms with much stronger performance than SAC. Consequently, it’s hard to get excited about these results since they underperform more modern algorithms, and I worry that the results may have overly benefited from environment specific tuning (dog) and may not generalize to more complex algorithms. For me to be convinced, I would like to see stronger results or covering more tasks/domains, to convince me that this algorithm is generally useful and not over-tuned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)