Value Improved Actor Critic Algorithms
Authors: Yaniv Oren
Moritz Akiya Zanger
Pascal R. Van der Vaart
Mustafa Mert Çelikok
Wendelin Boehmer
Matthijs T. J. Spaan
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
actor critic
policy improvement operators
Primary area: reinforcement_learning

Abstract: To learn approximately optimal acting policies for decision problems, modern Actor Critic algorithms rely on deep Neural Networks (DNNs) to parameterize the acting policy and greedification operators to iteratively improve it. 
The reliance on DNNs suggests an improvement that is gradient based, which is per step much less greedy than the improvement possible by greedier operators such as the greedy update used by Q-learning algorithms.
On the other hand, slow and steady changes to the policy can also be beneficial for the stability of the learning process, resulting in a tradeoff between greedification and stability.
To address this tradeoff, we propose to extend the standard framework of actor critic algorithms with value-improvement: a second greedification operator applied only when updating the policy's value estimate.
In this framework the agent can evaluate non-parameterized policies and perform much greedier updates while maintaining the steady gradient-based improvement to the parameterized acting policy.
We prove that this approach converges in the popular analysis scheme of generalized Policy Iteration in the finite-horizon domain.
Empirically, incorporating value-improvement into the popular off-policy actor-critic algorithms TD3 and SAC significantly improves or matches performance over their respective baselines, across different environments from the DeepMind continuous control domain, with negligible compute and implementation cost.

Value Improved Actor Critic Algorithms
Yaniv Oren
Moritz A. Zanger
Department Intelligent Systems Delft University of Technology 2628 Delft, The Netherlands y. oren@tudelft. ml
Department of Intelligent Systems Delft University of Technology 2628 CD Delft, The Netherlands zanger@tudelft. ml
der Vaart Department of Intelligent Systems Delft University of Technology 2628 CD Delft, The Netherlands p.r. w.r.vandeervarrt-11tt.. 18tudelf
Pascal R.
Mustafa Mert Çelikok Dept. of Mathematics & Computer Science University of Southern Denmark
Odense, Denmark
celikok@imada. swteeiimaaaddd....eeommm sdu. .kk
Wendelin Böhmer
Matthijs T. J. Spaan Department of Intelligent Systems Delft University of Technology 2628 CD Delft, The Netherlands m.t. spaan@tudelft....
Department of Intelligent Systems Delft University of Technology 2628 CD Delft, The Netherlands
j w. bowmer@tudelle
Abstract
To learn approximately optimal acting policies for decision problems, modern Actor Critic algorithms rely deep Neural Networks (DNNs) parameterize the acting policy and greedification operators to iteratively improve it. The reliance on DNNs suggests an improvement that is gradient based, which per step much less greedy than the improvement possible by greedier operators such as the greedy update used by Q-learning algorithms. On the other hand, slow changes the policy can also be beneficial for the stability of the learning process, resulting in a tradeoff between greedification and stability. To better address this tradeoff, we propose to decouple the acting policy from the policy evaluated by the critic. This allows the agent to separately improve the critic's policy (e.g. value improvement) with greedier updates while maintaining the slow gradient-based improvement to the parameterized acting policy. We investigate the convergence of this approach in the finite-horizon domain using popular analysis scheme which generalizes Policy Iteration with arbitrary improvement operators and approximate evaluation. Empirically, incorporating malue-improvement into the popular off-policy actor- critic algorithms TD3 and SAC significantly improves or matches performance over the baselines respectively, across different environments from the DeepMind continuous control domain, with negligible compute and implementation cos(")
Introduction
The objective of Reinforcement Learning (RI.) is to learn acting policies π, probability distribution actions, that, when executed, maximize the expected return (i.e., value) in given task. Modern RL. methods of the Actor-Critic (AC) family (e.g.. [Schulman et al.) 2017] Fujimoto et al.) 2018
Haarnoja et al. 2018b Abdolmaleki al. 2018 use deep neural networks to parameterize the acting
'Code is available at http:///gthhb.omm///a///1/1///1
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

policy, which is iteratively improved using variations of policy improvement operators based in stochastic gradient-descent (SGD), e.g., the policy gradient (Sutton et al.) 1999). These methods rely on specific type of policy improvement operators called greedification operators, which produce a new policy x' that increases the current evaluation Q" (see Definition
2 for more detail).
In gradient-based optimization the magniude of the update to the policy- the amount of greedification is governed by the learning rate, which cannot be tuned independently to induce the maximum greedification possible at every step, the greedy update π (T()) = arg max, Q= ()) (which we define generally as any policy и that has support only on maximizing actions). Similarly, executing N repeating gradient steps with respect to the same batch will encourage the parameters to over-fit to the batch (as well as being computationally intensive) and is thus does not address the problem of limited greedification of gradient based operators. For these reasons, the greedification of DNN-based policies is typically slow compared to, for instance, the arg max greedification used in Policy Iteration (Sutton& & Barto 2018) and Q-learning (Mnih et al. 2015).
While limited greedification can slow down learning, previous work has shown that too much greedification can cause instability in the learning process through overestimation bias (see van Hasselt et al. 2016, Böhmer et al. 2016,] Fujimoto et al., 2018), which can be addressed through softer, dess-greedy updates (Fox et al. (016)) This leads to a direct tradeoff between greedification and learning stability.
Previous work partially addresses this tradeoff by decoupling the policy improvement into two steps. First, an improved policy with controllable greediness is explicitly produced by a greedification operator as target. Second, the acting policy is regressed against this target using supervised learning loss, such as cross-entropy. The target policy is usually not a DNN, and can be for instance a Monte Carlo Tree Search-based based policy, variational parametric distribution, or nonparametric model (see Haarnoja et al. 2018b) Abdolmaleki et al. 2018:; Grill et al., 2020. Hessel et al. 2021; Danihelka et al. 2022). Unfortunately, this approach does not address the tradeoff fully: The parameterized acting policy is still improved with gradient-basee optimization which imposes similar limitations on the rate of change the acting policy.
To better address this tradeoff, we propose to explicitly decouple the acting policy from the evaluated policy (the policy evaluated by the critic), and apply greedification independently for (i) the evaluation of policies that need not be parameterized and can he arbitrarily greedy, while (ii) maintaining the slower policy improvement to the acting policy that is suitable for DNNs and facilitates learning stability. We refer to an update step which evaluates an independently-impron policy as a value improvement step and to this approach as Value-Improves Actor Critic (VIAC). Since this framework diverges from the assumption made by the majority of RL methods (evaluated policy = acting policy) it unclear whether this approach converges and for which improvement operators. Our first result is that policy improvement is not sufficient condition for convergence to the optimal policy of even exact Policy Iteration algorithms because it allows for infinitesimal improvement. To classify improvement operators that guarantee convergence, we identify necessary and sufficient conditions for operators to guarantee convergence to an optimal policy for a family of generalized Approximate Policy Iteration algorithms, a popular setup for analysis of AC algorithms (Tsitsiklis) 2002; Smirnova & Dohmatob) 2019).
both. This allows
We prove convergence for this class of operators in both generalized Approximate Policy Iteration and Value- Improved generalized Approximate Policy Iteration algorithms in finite-horizon MDPs. Prior work has shown that the generalized Approximate Policy Iteration setup converges for specific operators, as well as for all operators that induce deterministic policies (see Williams & Baird III 1993 Titsiklis 2002. Bertsekas 2011 Smmirnvva & Dohmatob 2019). Our results complement prior work by extending convergence to stochastic policies and a large class of practical operators, such as the operator developed for the Gumbel MuZero algorithm (Danihelka et al., 2022), as well as the Value- Improved extension to the algorithm. We demonstrate that incorporating value- improvement into practical algorithms can be beneficial with experiments in Deep Mind's control suite (Tunyasuvunakool et al. 2020) with the popular off-policy AC algorithms TD3 (Fujimoto et al. 2018) and SAC (Haamoja al. (0T88), where in all environments tested VI-TID3/SAC significantly outperform or match their respective baselines.
2 Background
The reinforcement learning problem is formulated as an agent interacting with a Markov Decision Process A, P. R, p. H), where S state space, A an action space, P: S A - (S)
2

---

is a conditional probability measure over the state space that defines the transition probability P(s, a). The immediate reward R(s, a) is a state-action dependent bounded random variable. Initial states are sampled from the start-state distribution In finite horizon MDPs, H specifies the length of trajectory in the environment. Many RI. setups and algorithms consider the infinite horizion case, where H + oo, but for simplicity's s sake our theoretical analysis in Section [ remains restricted to finite horizons, discrete state spaces and finite (and thus discrete) action spaces |A] < co. Note that in finite horizon MDP the policy is not stationary, as the same state can have different optimal actions at different timesteps of an episode. We model this without loss of generality still as a stationary policy problem by augmenting the state with the decision time t, which casts underlying state that is visited twice in an episode as two different states, which preserves stationarity of the policy in the augmented state space. The resulting state and transitions of the MDP then become directed acyelic graph (DAG). Throughout the paper, we will assume all states in the MDP contain the timestep as part of the representation.
The objective of the agent is to find a policy π S + (A), a distribution over actions at each state, that maximizes the objective J. the expected return from the starting state distribution p. We denote the set of all possible policies with IT. This quantity can also be written as the expected state value v with respect starting states so:
1 7T
((x) = E V (5p) m = E
Σ
S0 Se+1 Stt. P(se,ae)) acron(s)), Te R(
The discount factor < y S is traditionally set to 1 in finite horizon MDPs. The state value V can also be used to define a state-action Q-value and vice versa, i.., Vs E Va E A:
= 1 (s')s
8' d' ] (*(s)= = E00 (s, a>)(s)]
We refer to the optimal policy n- = arg max, V and its value as V and Q" respectively.
Policy Improvement To find n", many RL and Dynamic Programming (DP) approaches based in
approximate or exact Policy Iteration (Sutton & Barto (018)) can be cast as iterative processes that aim to produce a sequence policies Tr that improve over iterations such that the exact values V
or approximate values TTTH z VT= satisfy VTn+1 >> V*a (or in the approximate case Ra+1 2 v*))
using policy improvement operators:
Definition (Policy Improvement Operator). an operator I - satisfies:
VT E II, E S: (I(m)(s 2 (*),, Vx E S V=(m)( > V
(I)
(i.e., policy improvement), as long as π is not yet an optimal policy v * V*, we call I policy improvement operator.
Greedification The policy improvement theorem (Sutton & Barto 2018 is a fundamental result in RL and DP theory, which connects the policy improvement optimization process to specific maximization problem referred to in literature as greedification (see Chan et al. (022). Greedification the process of finding policy n' which increases another policy, s, evaluation Q" (Equation 2
Theorem (Policy Improvement). Let and nt' be two policies such that Vs S:
Σ (=)(s.))((aas)) > ((aaaa)aaaass)=== == aEA
(2)
REA
Then:
(s) (s).
(3)
In addition, there is strict inequality of Equation at any state, then there must be strict inequality of Equation 3 at least one state.
We refer to Sutton& & Barto 2018) for proof. Theorem D proves that when the evaluation QT is exact, greedification with respect to т., Q" produces an improved policy x'. If the inequality in Equation 2 is strict > we call n' greedier than π and any policy n' such that EasA (s. m (* a) greedy policy with respect to Q*.
Greedification Operators The policy improvement theorem and greedification give rise to the most popular class policy improvement operators, greedification operators, which produce policy improvement (Equation3 3 specifically by greedification:
3

---

Definition 2 (Greedification Operator). If an operator I: II x Q + II satisfies:
Σ ()(a)s) N
Σ
T(als) VT E I1, Aq e Q V,
(4)
aEAA
acA
as well as Is E S such that:
Σ
(Tm a)
Σ x(als)a(s.a) VT ET, Vg E 2
(5)
aEA
unless is already greedy with respect to q: T( I a greedification operator.
max, g(s, a), Vs E S, we call
The set Q denotes all bounded functions q: S A - R. Since practical operators are not generally designed to distinguish between exact QT and approximated q Q", we formulate the definition more generally in terms of q E Q. Greedification operators are policy improvement operators for q = = (i.e., Theorem D. Although most of the analysis in this paper will focus on greedification operators, the problem we point to in our first theoretical result is not unique to greedification and applies to policy improvement operators in general, which motivates us to explicitly distinguish between the two. We provide an example of a policy improvement operator that is not greedification operator in Appendix [A.2] to demonstrate that greedification operators are a strict subset of policy improvement operators (when =Q)..
Perhaps the most famous greedification operator is the greedy operator Targ max (T, (s) arg max, { a), which drives foundational algorithms such as Value Iteration, Policy Iteration and Q-learning (Sutton & Barto 2018). Many modern RL methods on the other hand are based in the actor critic (AC) framework, which we generally refer to as the iteration of (approximate) policy improvement (improving the actor), (approximate) policy evaluation (evaluating the actor, i.e., updating the critic). ACS traditionally rely on variations of the policy gradient operator (Sutton et al. 1999), which is well suited for the greedification of parameterized policies.
Other popular greedification operators are deterministic greedification operators Lnst (Williams) Baird III 1993) which produce policies that are greedier (Fquation 2 and deterministic. The regularized-a policy improvement operator (see Grill et al. 2020) used by Gumbel MuZero (Dani) helka et al., 2022) Iyma(Tr (s) = softmax( (q((5 )) + logn(s) T(s) (for a a monotonically in- creasing transformation). Best-of-N (BoN). popular operator in large language model align- ment with RL (Gui et al. 2024 Huang et al., (025).. At a state 8 BoN samples N actions AN fai,............... /(1,....,a,, aN } from (s) and evaluates them using q(s,,,). BON returns the maximiz- ing action: IBON (TT, ())s = arg max ((8, a). As the sample size N increases, BoN better approximates the arg aax 9(s.a)
Implicit greedification operators Recently, Kostrikov et al. (2022) proposed that it is also possible to produce implicil greedification, by training a critic approximate the value of greedier policy directly, without that policy being explicitly defined. The authors demonstrate that by training critic the with the asymmetric expectile loss L3 on data set D drawn with policy n.
(6) for 1/2 the critic uy (s) directly estimates the value of policy than is greedier than n, with 1 corresponding to the value of an arg max policy. This operator is then used to drive their Implicit Q-learning (IQL) algorithm for offline-RL, where the L5 enables the critic to approximate the value of an optimal policy without the bootstrapping of actions that are out of the training distribution.
((0)= B|C (s), Q (s((ssss) (s)a))ss, DD] (I = = 1y-2col (V-
Generalized Policy Iteration A popular approach for analyzing the underlying convergence behavior of large families of RL algorithms is to analyze the convergence of DP algorithm which abstracts the underlying learning dynamics of the RL algorithms (for example, see (mminnova & Dohmatob) 2019). A common setup used for this analysis is that of an approximate Policy Iteration algorithm (sometimes called specifically Optimistic or Modified Policy Iteration, see (Bertsekas) 201I) which iterates policy improvement, approximate policy evaluation. The policy improvement step most often uses the greedy operator. The policy evaluation step generalizes across Value Iteration and Policy Iteration, by doing a finite number ls of Bellman updates with the same policy (s, a) = TT (s, (s, a) [E[(s, + "EEgr
(a') Is's s')q;((' (,,') a')). 1..... k. Since our objective in this work is to analyze the underlying convergence of RL algorithms with respect to as many possible policy improvement operators, we formulate a variation of this algorithm which is generalized to all policy improvement operators I (Algorithm D. which we refer to as Generalized Policy Iteration (GPI). T' denotes the Bellman optimality operator, T (x a) = B[R(s, a)] + yE ~ waa.maaaee...a E. g(ssss/ q(s),a')).
1+1(s,
4

---

Algorithm 1 Generalized Policy Iteration
1: For starting functions E 2 т E II greedification operator I, k > 1 and e > 0
2: while Eaea (m(als)g(s;a)) max, q(s, > 0, Vs S and ((s.a)) T (s a) > do
3: q(s, a) t (T (T*)* V(s, a) ESx
4: (s) t I(T. E
3 Value Improved Generalized Policy Iteration Algorithms
Since the framework of Value-Improved AC/GPI generalizes AC/GPI beyond algorithms that evaluate their own policy, our first objective to analyze the the underlying convergence of this family of algorithms under very general setup. We begin by extending the DP framework of GPI, which underlies ACs, to DP framework which underlies VIACs. The framework extended by decoupling the improvement of the acting policy from that of the evaluated policy (line 3 Algorithm D. We refer to this new framework as Value-Improved GPI (Algorithm{2 Z} Modifications to the original algorithm are marked in blue. Since the acting and evaluated policies are improved with different operators I and Zz, it is not apparent whether π of Algorithm 2 whether decoupling the policies is sound. Therefore, our aim is to establish general pairs of operators for which this process converges.
converges to the optimal policy, i.e.
Algorithm 2 Value- Improved Generalized Policy Iteration
1: For starting vectors q E 2, π E II, policy improvement operators I1, 1
2: while LaEa (T(als)q(s, a)) - max, ((s, b)] > 0, Vs E S and (((ss)) T ("((s, a)l > do
3: ((saa) c (T (ss,a) ESx A
4: (s) t (8), Vs ES
fundamental result in RL is that policy iteration algorithms converge to the optimal policy for any policy improvement operator (Definition and by extension, all greedification operators) that produces deterministic policies. This holds because a finite MDP has only finite number of deterministic policies through which the policy iteration process iterates (Sutton & Barto 2018). This result however does not generalize to operators that produce stochastic policies, which are used by many practical RL algorithms such as PPO (Schulman al. 2017). MPO (Abdolmaleki et al. 2018). SAC (Haamooj et al. (2018b), and Gumbel MuZero (Danthelka et al.) (022). Our first theoretical result is that for stochastic policies, the policy improvement property (whether satisfied through greedification or not) is not sufficient to guarantee convergence, even in the limiting case of Policy Iteration with exact evaluation.
Theorem (Improvement is not enough). Policy improvement not a sufficient condition for the convergence of Policy lteration algorithms (Algorithm 7 with exact evaluation) to the optimal policy for all starting policies Tg E II all finite-state MDPs.
Proof sketch. With stochastic policies, an infinitesimal policy improvement is possible, which can satisfy the policy improvement condition at every step and yet converge in the limit to policies that are not arg max policies. Since every optimal policy is an arg max policy (note that we define arg max policies as policies with support only on maximizing actions, not necessarily as deterministic policies), Policy Iteration with such operators cannot be guaranteed to converge to the optimal policy. For complete proof see Appendix A.
Why is this problem? Many algorithms (for example, Gu et al. 2022 are motivated by policy improvement through demonstrating greedification. Theorem 2 demonstrates that this is not sufficient to establish that the resulting policy improvement will lead to an optimal policy. For that reason, convergence for these algorithms must generally proven individually for each new operator (e.g., see MPO and GreedyAC Chan et al. 2202)), which is often an arduous and nontrivial process.
Furthermore, Theorem 2 and its underlying intuition highlight a critical gap: we currently lack guiding principles for designing novel greedification operators in the form of necessary and sufficient conditions for convergence to the optimal policy. To illustrate that this can lead to problems in practice, we show in Appendices A.5 and that choices of the used by the greedification operator
5

---

Iym: can render this operator either sufficient or insufficient. To address this problem, we identify necessary condition and two independent sufficient conditions for greedification operators, such that they induce convergence of Algorithm [
Definition (Necessary Greedification). In the limit of n applications of a greedification operator I on value estimate q E Q and starting policy no E II, the policy nu, converges greedy policy with respect to 9: Vss E S:
limn Σ (((sa))nn(ul
max ((8, a), where Tn+1(s) I(TTT:
Intuition. Since every optimal policy is arg max policy (has support only actions that maximize Q*), if greedification operator cannot converge to an arg max policy even in the limit and for fixed y then this operator cannot converge an optimal policy in general. See Appendix for a concrete example where such a condition is necessary for convergence of a Policy Iteration algorithm. It is possible to formulate the same condition more specifically for the set of all Q functions {Q" VT E I1J. However, since we are interested in algorithms that may not have access to exact values Q", seems more prudent define it more generally Vyq E Q. Since practical operators are not generally designed to distinguish between exact Q and approximated ~ Q", we formulate the definition more generally in terms of q E Q.
Unfortunately, the necessary greedification condition is not sufficient, even in the case of exact evaluation. This is due to the fact that assuming convergence to a greedy policy in the limit for a fixed function does not necessarily imply the same when the function changes between iterations. There exist settings where the ordering of actions a, a', q(ssssss a) < q(s. a') can oscillate between iterations, preventing the convergence to greedy policies (See Appendix [A.3 for concrete example). Below, we identify two additional conditions which are each sufficient for convergence in finite horizon and finite action spaces. The first condition resolves the issue by lower-bounding the rate of improvement, which guarantees that the oscillation does not continue infinitely. The second simply augments the necessary greedification condition to require convergence for any sequence of Q functions.
Definition 4 (Lower Bounded Greedification). We call an operator I a lower-bounded greedification and for every q E Q, Be > 0, such that operator if I is a greedification operator (Definition 2 vs E S and Vr E II:
Σ I( [(T.))aa)))((s.)) - Σ ((als))(ssa) >
6,
aEA
unless EGEA I(TT))((ss)((ss.) max, q(s, Vs S.
Intuition. Since the lower bound is constant with respect stationary g, it eliminates the possibility of infinitesimal improvements and guarantees convergence to an arg max policy in finite iterations with respect to the stationary q (See Lemma 8 and Appendix [A.12] for proof). We note that this definition does not guarantee convergence to an optimal policy nor an arg max policy with respect non-stationary qn-
Definition 5 (Linit-Sufficient Greedification) Let qo Q1 E Q be sequence of functions such that limn- In = q for some q Q. Let no: Ti.... be a sequence of policies where Tn+1 I( 9n+1) for some operator I. We call operator I a Sufficient greedification operator If is a greedification operator (Definition 2 and in the limit n + 0o the improved policy an+1 converges to greedy policy with respect the limiting value q, Vs E S:
lim 120000
Ta(als)un(s,d) a) = max ((8,
AEA
Intuition. Even in the presence of infinitesimal improvement and non-stationary estimates qn- limit sufficient greedification operator is guaranteed to converge to a greedy policy in the limit n + 00, as long as there exists a limiting value on the sequence of value estimates lim On g.
Practical operators that are sufficient operators Lower bounded greedification is used establish convergence for MPO (see Appendix A.2, Proposition 3 of (Abdolmaleki et al. 2018)). Similarly. deterministic operators Lett are also lower bounded greedification operators (see Appendix A.4 for proof). Lower bounded greedification operators however cannot contain operators that induce convergence to the greedy policy only in the limit, because the convergence they induce in finite steps. Iymm on the other hand induces convergence only in the limit, and in fact is more generally

---

limit-sufficient greedification operator (see Appendix A.5 for proof), The deterministic greedification operator on the other hand does not converge with respect arbitrary non-stationary sequences lim On (see Appendix A.7). which leads us to conclude that both sets are useful in that they both contain practical operators and neither set contain the other. The greedy operator on the other hand member of both sets, demonstrating that the sets are also not disjoint (see Appendix A.8 proof).
Equipped with Definitions 4 and 5 we establish our main theoretical result, convergence for both Algorithms and for all sufficient greedification operators:
Theorem 3 (Convergence Algorithms and 2.. Generalized Policy Iteration algorithms and their Value Improved extension (Algorithums and respectively) converge for sufficient greedification operators, in finite iterations (for operators defined in Definition 4 or in the limit (for operators defined in Definition S in finite-horizon MDPs.
Using induction from terminal states, the proof builds on the immediate convergence of values of terminal states $H, convergence of policies at states 8и-1 and finally showing that given that g. converge for all states 8t+1, they also converge for all states 8 (in finite iterations or in the limit, respectively). The evaluation of greedier policy (value improvement, line in Algorithm 2) is accepted by the induction that underlies the convergence of Algorithm which allows us to the same method establish convergence for Algorithm 2 The full proof is provided in the Appendix. In A.9 for Algorithm | with limit sufficient operators and = 1, extended to k 2 in A0.0 to Value-Improved algorithms AA.1I and to bowerebounded operators in
Proof sketch;
A corollary of Zym= being a limit-sufficient greedification operator along with Theorem is the convergence of process underlying the Gumbel MuZero algorithm.
Corollary 1. The Generalized Policy Iteration process underlying the Gumbel MuZero algorithm family converges to the optimal policy for finite horizon MDPs, for all my E II such that log T (( defined vs E S.aEA
Interestingly, Iz does not need to be a sufficient or even a necessary greedification operator for convergence to the optimal policy, as is established by the following corollary:
Corollary 2. Algorithm converges to the optimal policy for any non-detriment operator In (e.g.. operators that satisfy the non-strict inequality of Equation D) as long as In is itself sufficient.
As another consequence, CorollaryDalks establishes that the algorithm converges when implicit greedification operators such the expectile loss used for value improvement. Motivated that the Generalized Policy Iteration process underlying VIAC algorithms converges, we proceed to empirically evaluate practical RI. VIAC algorithms.
For proof see Appendix(A.I II
4 Value Improved Actor Critic Algorithms
Talue-improvement can be incorporated into existing AC algorithms in one of two ways: (i) Incor- porating an additional explicit greedification operator to produce greedier evaluation policy, and use the greedier policy to bootstrap actions from which to generate value targets. (ii) Incorporating an implicit greedification operator, for example by replacing the value loss with an asymmetric loss (Algorithms 3 and 4 respectively in Appendix Cannd implementation details in Appendix [D). We begin by testing the hypothesis that additional greedification of the evaluation policy (e.g.. value improvement) can directly lead to accelerated learning by extending TD3 (Fujimoto et 2018 with value improvement (VI-TD3, Algorithm 3)
7

---

happer-stadd
25
irerevementt 5ua
0.05
20
Value extimation
15-
400
0.02
10-
Average
200
0.01
0.00
166
hopper-hop
Value improvemntt 038
3-
200
Value eztimation biss
vitdB 1pg-13
2-
(oq
150-
vi td3 (pg=10)
(d3 200
200
Environment steps
Environment
Environment steps
Figure 1: Mean and one standard error shaded area across seeds for VI-TD3 with Zz the deterministic policy gradient and increasing number of gradient steps (pg=n), with baseline (pg=0) TD3 for reference.
We choose the same improvement operator already used by TD3 I2 I1, the deterministic policy gradient, as a value improvement operator, in order to decouple possible effects stemming from the combination of different operators. In order compare different degrees greedification, a different number py of repeating gradient steps with respect to the same batch are applied to the evaluated policy, which then discarded after each use. We evaluate the performance the agents classic control environments from the DeepMind continuous control benchmark ('unyasuvunakool et al. 2020). Results are presented in Figure ! where performance increases with greedification (left).
As expected, greedier targets are larger targets (center), that is the difference between the value bootstrap that uses the greedier policy n' and the baseline bootstrap increases with greedification. An interaction with over-estimation bias (which we compute in the same manner as Chen et al. (202I) exists in some environments like hopper-stand, but cannot explain the improved performance exclusively, shown in the hopper-hop environment (Figure right).
hogper-stand
hopper-hop
Average evalustion retum
WLMM3 [T-
vi.d3 [e-
vi_t3 tt= 0.75)
200
vi,dd3 [r-
600
Average
200
vl. td3 It= 0.95)
0.25
1.25
1.50
1.75
2.00
0.50
1.25
1.50
Environment
1e6
Environment steps
1wh
Figure 2: Mean and one standard error across 10 seeds for VI-TD3 with expectile loss with different values of the expectile parameter т. Performance increases up 0.8 and then decays.
Repeating gradient steps are very computationally expensive however. Implicit policy improvement on the other hand provides value improvement for negligible compute and implementation cost. In addition, the greedification amount can be chosen with the greedification parameter 7 directly. In Figure we evaluate VI-TD3 with implicit value improvement (Algorithm 4 based in the expectile loss operator and increasing values of the greedification-parane monotonically improves performance up to point, from which performance monotonically degrades, suggesting that the greedification of the evaluated policy can be tuned as a hyperparameter. This sup- ports the conclusions of previous literature that there a tradcoff between stability and greedification, and suggests that this tradeoff can be at least partially addressed with value improvement.
r. The increased greedification
In Figure 3 we compare baseline TD3 and SAC (Haarnoja et al., 20185) to VI-TD3 and VI-SAC with implicit value improvement (7 0.75, blue) across a larger number of control environments.

---

hopper stand
hopperhap
stand
humanoid
1000
400
750
500
400-
400
Average
100
200
200
0-
1e6
1e6
led
1ed
humanoid-run
fist-wwi
scrobot-swingup
quadruped-walk
200
800
300
10000
Average evstuatio
800
200-
400
200
800
50-
100
400
200
146
146
105
leb
quadruped-rum -uu
walkar
walker-run
chectah-run
800-
1000
800
800
800
690
600
sac td3
400
400
400
Vi. vas (expectill)
vi. 1d.td3 (expectile)
Average
200
200
200
vi. (BoN)
Environment steps 1e6
Environment steps
1e6
Environment steps
Environment steps
Figure 3: Baseline TD3 (dashed) and SAC (solid) in red vs. VI-TD3/SAC (blue) with expectile loss and BoN (dot-dashed). Mean and two standard errors (=s 95% Gaussian CI) in the shaded area of evaluation curves across 10 seeds for BoN and 20 for the other agents.
Across environments, the VI (blue) variations significantly outperform or matches their respective baseline (red) while introducing negligible additional compute and implementation cost.
The significant performance increase provided by the expectile operator Figure 3 raises the possibility that the performance gains are not due to the greedification of the evaluated policy, however. Rather, the implicit expectile operator itself may be responsible for the performance gain. To investigate whether this is the case, include an additional VI-TD3 variation with an explicit improvement operator operator: Best-of-N (BoN, dot-dashed, see Section 2 BoN-based VI-TD3 (N = 64) provides similar performance gains to expectile-based VI-TD3 most environments, demonstrating that significant performance gains in these domains are not unique to the expectile operator.
Along similar lines, although Figure demonstrates that it is possible to see performance gains that decoupled from overestimation increase, that may not be the case for the expectile operator. It is possible that the (overall more significant) performance gains of the expectile operator are instead resulting from -increeedeeee over-estimation bias. In Figure 4 in Appendix B we investigate in more detail the interaction between greedification, performance and over estimation bias per environment for the expectile operator. We find that in the majority of the environments significant performance gains are observed without any over estimation increase. We summarize these results in Table Din in Appendix B
We include additional results in Appendix B (i) In Figure include results for the recent algorithm TD7 (Fujimoto al., 2023), which shows similar performance gains with value improvement. (ii) In Figure 6 we investigate increasing the greediness of the update to the parameterized acting policy, Repeating gradient steps on the same batch do not effectively increase sample efficiency, as discussed in Section [
(iii) In Figure 7 we investigate the tradeoff between spending the compute of additional gradient steps for value improvement vs. for increased replay ratio. In line with similar findings in literature (Chen et al. 2021), replay ratio provides a strong performance gain for small ratios. However, as the ratio increases, performance degrades, result which the literature generally attributes to instability. The VI agent on the other hand does not degrade with the same increase to compute. (iv) In Figure 8 we investigate more generally how the greedification of the acting policy compares to the greedification of the evaluated policy is one more important than the other? Can value improvement replace policy improvement? Our results suggest that as long as the Value-Impin Salue-Improvement operator is rooted in policy maintained during the learning process, the rate at which this policy improved is the more impactful factor. On the other hand, slow policy improvement does not prevent inleoiimprvvementt from significantly accelerating the learning process.

---

5
Related Work
VIAC generalizes the underlying learning scheme of several recent methods. XQL Garg et al. 2023). SQL,, and EQL (Xu et al. 2023) build the implicit policy improvement setup of IQL to design algorithms that iterate a value improvement + policy extraction loop, rather than the policy improvement- + policy evaluation loop of standard AC methods. When the policy extraction step (learning a policy given a value function) can be interpreted as a policy improvement operator, these algorithms become members of the VIAC family with specific instantiations of operators. BEE (Ji] et al. 2024) is another recent method that explicitly considers value target that includes its own maximization operator, albeit from the perspective of mixing targets that take into account exploration / exxllittatinn tradeoffs in the environment. Similarly. OBAC Luo et al. (2024) learns pair of value functions, one for the acting policy and one using implicit policy improvement over data from the replay buffer and uses both pairs of value functions to train the acting policy.
In model-based RL, it is popular to employ improvement operators for action selection and policy improvement, and sometimes even to generate value targets (e.g., value improvement, for example see Moerland et al. 2023). The more common setup employs the same operator for action selection and policy improvement (for example, AlphaZero (Silver et al. (2088). MuZero Reanalyze (Schrittwieser et al. 202T) an example of an algorithm that considers using the same operator (tree search in this case to produce value targets well, and thus can be thought of as belonging to the setup of VIAC. These algorithms however are traditionally motivated from the perspective that the acting policy, target policy and evaluated policy all coincide as they are all produced by the same operator.
TD3 can be thought of as an example of an agent which acts, improves, and evaluates three different policies: The acting policy is improved using the deterministic policy gradient, during action selection the acting policy is modificd with noise in order to induce exploration, and finally the evaluated policy is regularized with Although only one policy improvement operator is used, TD3 can be thought of as an algorithm which decouples the acting policy from the evaluated policy. GreedyAC (Neumann et al. 2023) inspired a family of algorithms (see Lingwei Zhu| (025)) which share similarities with the VIAC framework in that they explicitly maintain two different policies, although both policies are used for policy improvement, as opposed to value improvement.
iiteerenlly-arrmettrriiie
noise in order to improve learning stability,
6 Conclusions
In this work we have proposed to decouple the evaluated policy from the acting policy and apply policy improvement additionally to the evaluated policy (value improvement) in order to better control the tradeoff between greedification and stability in Actor Critic (AC) algorithms. We refer this
approach as Value-Improved AC (VIAC). VIAC provides a unified perspective on recent online and offline RL algorithms which combine different improvement operators (Garg et al. 2023 Xu et al. 2023; Ji et al.) 2024, Xu et al., 2025). We' ve shown that policy improvement is not sufficient condition for convergence of Dynamic Programming / RL algorithms with stochastic policies. We've identified necessary and sufficient conditions for convergence of such algorithms and demonstrated that Generalized Policy Iteration (Algorithm []]. which underlies the learning dynamics of ACs, converges to the optimal policy for such sufficient greedification operators in the finite horizon domain. With this groundwork laid, we've 'e demonstrated that Falue-Immroved Generalized Policy Iteration (A (Algorithm{). which respectively underlies VIACs, converges to the optimal policy in the finite horizon domain.
Empirically, VI-TD3 and VI-SAC significantly improve upon or match the performance of their respective baselines in all DeepMind control environments tested with negligible increase compute and implementation costs. We hope that our work will act as motivation to design future AC and general Approximate Policy Iteration algorithms with multiple improvement operators in mind, as well as extend existing algorithms with value-improvemeet.
Acknowledgments
We thank Joery de Vries and Caroline Horsch for timely assistance with compute. Mustafa Mert Çelikok was partially funded by the Hybrid Intelligence Center, 10-year programme funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research, http://ttbridnnellinnc-ccttttttt grant number 024.004.022. The project has
10

---

received funding from the EU Horizon 2020 programme under grant number 964505 (Epistemic AI). The computational resources for the experiments were provided by the Delft High Performance Computing Centre (DHPC) and the Delft Artificial Intelligence Cluster (DAIC).
References
Abbas Abdolmalcki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, and Martin A. Riedmiller. Maximum a posteriori policy optimisation. In The 6th International Conference Learning Representations, 2018.
Dimitri P Bertsekas. Approximate policy iteration: A survey and some new methods. Journal of Control Theory and Applications, 9(3):310 10333,033,1100 2011.
D.P. Bertsekas. Approximate dynamic programming. In Dynamic Programming and Optimal Control,
number v. 2 in Athena Scientific optimization and computation series, chapter 6. Athena Scientific, 2007. ISBN 9781 9781886529304.
David Blackwell. Discounted dynamic programming. The Annals of Mathematical Statistics, 36(1): 226 235, 1965.
Wendelin Böhmer, Rong Guo, and Klaus Obermayer. Non-determinnsttcc policy improvement stabi- lizes approximated reinforcement learning. arXiv preprint arXiv:1612.07548, 2016.
Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, A. Rupam Mahmood, and Martha White. Greedification operators for policy optimization: Investigating forward and reverse KL divergences. J. Mach. Learn. Res., 23:253:1 -253:79,, 2022.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double Q-learning: Learning fast without a model. In The 9th International Conference on Learning Representations, 2021.
Mark Collier, Basil Mustafa, Efi Kokiopoulou. Rodolphe Jenatton, and Jesse Berent. simple probabilistic method for deep classification under input-dependent label noise. arXiv preprint arXiv:2003.06677. 2020.
Ivo Danihelka, Arthur Guez, Julian Schrittwieser, and David Silver. Policy improvement by planning with Gumbel. In The Tenth International Conference Learning Representations, 2022.
Delft AI Cluster (DAIC). The Delft AI Cluster (DAIC), RRID:SCR RR15:,,,_025091,, 2024.
Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. In The Thirty-Second Conference Uncertainty in Artificial Intelligence. AUAI Press, 2016.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In The 35th International Conference on Machine Learning, volume 80, pp- 1582-1591. PMLR, 2018.
Scott Fujimoto, Wei-Di Chang, Edward J. Smith, Shixiang Gu, Doina Precup, and David Meger. For SALE: state-action representation learning for deep reinforcement learning. In Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems, 2023.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme Q-learning: Maxent RI. without entropy. In The Eleventh International Conference on Learning Representations, 2023.
Jean-Bastien Grill, Florent Altché, Yunhao Tang, Thomas Hubert, Michal Valko, loannis Antonoglou, and Rémi Munos. Monte-carlo tree search regularized policy optimization. In The 37th International Conference on Machine Learning, volume 119, pp. 3769-3778. PMLR, 2020.
Lin Gui, Cristina Garbacea, and Victor Veitch. BoNBoN alignment for large language models and the sweetness of best-of-n sampling. Advances in Neural Information Processing Systems 38: Annual Conference Neural Information Processing Systems, 2024.

---

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor- citorccritic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In The 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1856-1865. PMLR, 2018a.
Tuomas Haarnoja. Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 10t:::822... 1825055 2018b.
Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theophane Weber, David Silver, and Hado van Hasselt. Muesli: Combining improvements in policy optimiza- tion. In The 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4214 4226. PMLR, 2021.
Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Akshay Krishnamurthy, and Dylan Foster. Is Best-of-N the best of them? Coverage, scaling, and optimality in inference-time alignment. In The Forty-second International Conference on Machine Learning. 2025.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G. M. Araújo. CleanRL: High-quality single-file implementations of deep reinforcement learning algorithms. J. Mach. Learn. Res., 23:274:1-274: 2022.
Tianying Ji, Yu Luo, Fuchun Sun, Xianyuan Zhan, Jianwei Zhang, and Huazhe Xu. Seizing serendipity: Exploiting the value of past success in off-policy actor-critic. In The Forty-first International Conference on Machine Learning, 2024.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In The Tenth International Conference on Learning Representations, 2022.
Yukie Nagai Lingwei Zhu, Han Wang. Fat-to-thin policy optimization: Offline reinforcement learning with sparse policies. In The Thirteenth International Conference Learning Representations, 2025.
Yu Luo, Tianying Ji, Fuchun Sun, Jianwei Zhang, Huazhe Xu, and Xianyuan Zhan. Offline-booster actor-critic: Adaptively blending optimal historical behaviors in deep off-policy RL. In The Forty-first International Conference on Machine Learning, 2024.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(75409:529-333, 2015.
Thomas M. Moerland, Joost Broekens, Aske Plaat, and Catholijn M. Jonker. Model-based reinforce- ment learning: A survey. Found. Trends Mach. Learn., 16(1):111))))))) 1-18 2023.
Samuel Neumann, Sungsu Lim, Ajin George Joseph, Yangchen Pan, Adam White, and Martha White. Greedy actor-critic: A new conditional cross-entropy method for policy improvement. In The Eleventh International Conference Learning Representations, 2023.
Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with learned model. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems, pp. 27580-27599 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Arthur Guez, Mare Lanetot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters Chess, Shogi, and Go through self-play. Science,
2018.
12

---

Elena Smirnova and Elvis Dohmatob. On the convergence of approximate and regularized policy iteration schemes. arXiv preprint arXiv:1909.09621, 2019.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S. Sutton, David A. McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Informa- fion Processing Systems 12: anmual Conference on Neural Information Processing Systems, pp. 1057-1063. The MIT Press, 1999.
John N. Tsitsiklis. On the convergence of optimistic policy iteration. Mach. Learn. Res., 3:59-72. 2002.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Frez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_ control::: Software and tasks for continuous control. Software Impacts, 6: 6010022 2020. ISSN 2665-9638.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp. 2000, AAAI Press, 2016.
Ronald J Williams and Leemon C Baird III. Analysis of some incremental variants of policy iteration: First steps toward understanding actor-critic learning systems. Technical report, 1993.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Wai Kin Victor Chan, and Xianyuan Zhan. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, 2023.
Haoran Xu, Liyuan Mao, Hui Jin, Weinan Zhang, Xianyuan Zhan, and Amy Zhang. Uni-rl: Unifying online and offline rl via implicit value regularization. Advances Neural Information Processing Systems 39; Annual Conference on Neural Information P Processing Systems, 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In Section 4 Figure we demonstrate that TD3 and SAC with value improve- ment significantly improve or match performance over baseline in mujoco. In Section Figure and 2 we demonstrate that it is possible to directly increase sample efficiency through greedification of the evaluation policy (value improvement) up to point from which performance degrades, suggesting that greedification can be tuned as a hyperparameter. In Section 3 Theorem 2 and its proof in Appendix [A.] that improvement is not enough for convergence to the optimal policy for policy itcration algorithms. The necessary and sufficient conditions identified (Definitions]) 3 and 5 and the operators belonging to them are supported proof' in Appendix A Specifically, the necessary condition is supported by the example in Appendix A That is not sufficient is proven in Appendix A.3 That the sufficient conditions are sufficient as well as convergence of Algorithms [ and 2 Theorem B is proven in Appendices: A.9 for Algorithm I with limit sufficient operators and k 1, extended to & 2 in A.10 to malue-Improve algorithms in A. and to lower-bounded
operators in A.12
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our theoretical analysis is limited to finite horizon MDPs with finite action spaces and discrete state spaces, which is described in the background and in the theorems.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
14

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All lemmas and theorems are supported by complete proofs that are to the best of our knowledge correct. The main assumptions are finite horizon, finite action spaces and discrete state spaces MDPs, and they are described in the background. To the best of our knowledge all assumptions are specified.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide pseudocode to the algorithms used our experiments in Algo- rithms
The full set of hyperparameters and implementation details are specified in Appendix
3 and 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
15

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide supplementary materials with code and instructions faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7/nips. //nips.cc/publl.. wwtpc//ggs////eeeeocooococ ppbllic/guudess/dddeuubiii. wwweesuubmisssooPe..
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All experimental details are specified in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
16

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We use the standard statistical evaluation approach for performance comparison of standard errors of the mean. We use two standard errors for the main results (Figure 3) and one standard error for the ablations (all other figures).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify compute resources in Appendix D.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification; the research conducted the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
17

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our contributions are to foundational research in RL, and for that reason, do not have direct societal impacts of which we are aware.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe that our work does not pose high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
18

---

Justification: We reference the resources we' ve used (namely, other papers and the CleanRL (Huang al. 2022) repository), and do not use any assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappesswitttteee thcode cantsatsssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
19

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not non non- stanstandard components.
involve LLMs as any important, original,
Please refer to our LLM policy for what should or should not be described.
20

---

A Proofs
A.1 Theorem 2:
policy improvement is not enough
Theorem 2 states: Policy improvement not sufficient condition for the convergence of Policy Iteration algorithms (Algorithm)) with exact evaluation) to the optimal policy for all starting policies To E 11 in all finite-state MDPs.
Proof sketch: We will construct simple MDP where the Q" values remain the same for all policies π, and show that even in this simple example, it is possible for a Policy Iteration algorithm to converge to non-optimal policies, with policy improvement operators that allow for stochastic policies. In addition, while adjacent to the narrative of this paper, we demonstrate that the same problem persists with deterministic policies in continuous action spaces in Appendix A.I.1
Proof. Consider very simple deterministic MDP with starting state so and two actions a1. 02 that lead respectively to two terminal states $1, $2. The reward function R(5,, a1) = and R(so; a2) = 2 and transition function P(s (o: 01) = 1, P(s2 (oo 02) = 1 and zero otherwise. We have Q (8o; (1) = and Q" (sp, az) = 2 for all policies π E IT. The optimal policy is therefor x" (s()) 0g that is x² (a1 (so) = 0 and x (a2)))))) 1.
Consider the very simple policy improvement operator Iinadequate. When EMEA ([aass)Q((ssa))
EeeA softmax((" (als) ("alsss([[(]] (s,a), the operator is defined as follows: Zinadequate (T) aT(s) + (1 a )softmax Q²' (s, (s). Otherwise, when the policy is already greedier than the softmax policy, the operator is defined as follows: Linadequate (T) = arg max, Q" a).
In natural language: when the policy less greedy than the softmax policy, the operator produces a mix between the current policy and the softmax policy. This is always greedier than the current policy, and thus will act as a greedification operator for policies such policies. When the policy is as greedy or greedier than the softmax, the operator produces directly a greedy policy.
The policy improvement theorem proves that this operator is Policy Improvement operator, because for every policy ı it greedifies the policy with respect to Q" (that is, the policy's 's evaluation is strictly larger unless the policy is already greedy).
We now apply this operator in the Policy Iteration scheme to the simple example MDP speci- fied above, with a starting uniform policy [aa (sa) (a22 (s)))))) Since this operator produces a mixture of the current and softmax policy, in the limit it will converge to the softmax policy lim, nn softmax(Q(s,) policy improvement operator. This proves that policy improvement operators cannot in general guarantee convergence to the optimal policy even in the limiting setting of Policy Iteration, and thus also not in more general setting such as Approximate Policy Iteration.
F arg max, Q(s, a), i.e. to a sub-optimal policy. despite being
In this example, since Qa does not change across different iterations n of Policy Iteration algo- rithm, we can identify the following: For convergence to the optimal policy, it is necessary that a greedification operator un+1 I(T q) will converge a greedy policy, with respect to the same stationary q Q":
lim
Σ (al()))((ss))) = max ((,, a), Vs E. BEA
7 +00
a
More generally, this example shows that we can construct general operators I that produce arbitrary se- quences of policies un+1 = I( 2 0 with values 4*+++ such that V* (s) 2 VTT (s), Vs E S, with a strict > in at least one state. Since this is the only condition required by policy improve- ment, operators I are policy improvement operators. However, since the improvement need not be bounded it is possible to construct sequences that converge to arbitrary values in the interval (s)) V* (s)], Vs E S. I.e., because the improvement property allows for infinitesimal improve- ments, it does not guarantee convergence to the optimal policy.
A.1.1 Policy improvement in continuous action spaces
A similar problem applies to continuous action spaces. Imagine similar MDP as above with continuous action spaces A = (0, 1). reward function R(s0, a) = a, Va E A and zero otherwise and every transition is terminal. Now imagine an operator that produces deterministic action I( (s). such that J (s, (m, Q*) (a)s) ds > [ (s, a) (a s) s)ds unless the policy is already optimal. I is an improvement operator and satisfies the greedification property.
21

---

Let us again choose xo the uniform policy across actions. At the first step. I can produce just-above the middle action ai 0.5. At each step, I can produce new action I(m (*)) = T2 02 > 01 However, since the space (0, 1) is the continuum and non-countable, there are more actions to select that any iterative process will ever have to go through. Therefor, even the limit n -> inf, the operator will never have to choose m( Q*) =1
A.2 Policy Improvement operators that are not Greedification operators
Lemma 1. There exist operators I 11 Q II that are Policy Improvement operators, and but are not Greedification operators according to Definition
therefore fulfill Equation 3
Proof sketch: We will prove that a random-seane operator that mutates the policy я randomly into a new policy n'. evaluates and keeps if V > V= is not greedification operator, even though is policy improvement operator by definition. We do this by constructing an MDP and choosing a specific initial policy To. Greedification with respect to the initial policy, at state so, results in Ti (8o) d1- However, the optimal policy in this state actually chooses action dg, because the optimal policy can take better actions in the future than policy T1. Such an example proves that is possible construct policy improvement while violating greedification, demonstrating that the condition only goes one way: every greedification operator is policy improvement operator, not vice versa.
Proof. Consider the following finite-horizon MDP: State space S = (......, $10}. Action space A a1, 02 0z. States (Ss...., .10} are terminal states. Transition function: f(s,, a1) $2. f(s1, a2) = 83, f(1), 0s) = $4. f(32, 01) = S5, f(22, 02) S6. f(8s, a1) = 87, f(ss, a2) = 88.
f(s a1) 89, [s1. 02) $10.
Rewards: R(52. 95) = 2, R(s2, 86) = -1, R(33,87) 87) = 1, R[s3, ss) 0, R(sa, Sn) = 3. R(sa, S10) 2.
Actions that are not specified lead directly to terminal state with zero reward.
Let us begin by identifying the optimal policy in this MDP, in states 81 and 84: क (s1) 0g and x" (*())) = 01, with a value of 3 without a discount factor.
Let us construct starting policy mp:
(o()1) = 01 (x())) = 42, *0(8s) = 02 (s(s)) = d2 The other states are terminal and there are no actions to take, and therefor no policy.
Consider the following Policy Improvement operator Ig II x Q + II, which this example will demonstrate is not a greedification operator. IE takes policy π, and mutates it with random process to n². Ig proceeds to conduct exact evaluation of n², to find V= If (s) 2 V on all > V (s) in least one state, Ig outputs n'. Otherwise, the process repeats. This process guarentees policy improvement. However, this process may directly produce the optimal policy in this MDP, which in states $1, 83 x (s1) = a3 and π rT*(s)) a d1.
states, and V= V=(s) (s)
Note however, that the optimal policy is not greedier policy with respect to the value of To. For To- we have: Q* (1) 1, QT (s), 02) 0, QT (s1,
2. A greedier policy with respect to these values cannot deterministically choose action a3, which is the action chosen by the optimal policy in this state.
Therefor, this example demonstrates that it is possible for a policy to be improved (higher value in least one state, and greater or equal on all states), without being greedier with respect to some original policy's ' value. In turn, this demonstrates that there exist Policy Improvement operators that are not Greedification operators.
A.3 Necessary greedification operators may not be sufficient
Lemma 2. Greedification operators (Definition) 2) which have the necessary greedification property (Definition 3 may not be sufficient for Policy Iteration algorithms to converge to the optimal policy.
Proof sketch: First, we demonstrate the problem: certain operators with the necessary property, such as the deterministic greedification operators, may not converge to greedy policy with respect to non-stationary Q*= Second, we will show that in Policy Iteration it is possible to experience non stationary evaluations Q*a that will prevent a
algorithm from converging to a greedy policy. We will show this by constructing an operator that performs deterministic
22

---

greedification in some states, and greedification that converges only in the limit in other states (both n greedification operators), and show that the problem can persist practical MDPs. This operator is a necessary-greedifne will converge to greedy policies. However, since in Policy Iteration algorithm the evaluation q is not necessarily stationary, it is possible that Policy Iteration algorithm based in this operator will not converge to the optimal policy.
operator in all states. That is, with respect a stationary q. it
Proof. Let A= fai, a2, 03} and a sequence qa (a(ar) - ((a2), and gn (n(4a) = g(aa), with a limiting value 1, We omit the dependency of g state as it is unnecessary in this example. In this case, the optimal policy with respect to any qn is ag.
9n (a2) ) = (-1)*+1/2"
Take the least-greedifynig deterministic greedification operator Imin (g, a) mil (a)) This operator produces the worst action, with respect to q. that is better than the current action selected by the policy, and as such, is a greedification operator by definition, with respect to deterministic policies. Since there are finitely many deterministic policies on a finite action space |.|| < oc, this operator will converge to lim Fn = arg max, g(a) with respect stationary 9-
Take To = ag- Using the operator Imin_det det we have Tn = -Imin_det det (9n, 11 When n is odd, Tn 01. and when is even, nn d2, without ever converging to the optimal policy ag.
Next we will construct an example MDP and improvement operators which this situation can happen in practice. Consider a finite-state, finite horizon MDP with states 81,...,. Sn. We are interested in the behavior at state Sy specifically, which similarly has actions 01, a2 az with re- wards R(s1, 0z) = 3, R(s1, a1) R(51, 02) = 0. The transition f(1 as) so is terminal and f(81, 01) = 82, ((11,
83.
Consider the following improvement operator: On state $1, this operator is Iminget- However, on all other states, this is a necessary greedification operator, which converges only in the limit, and in non-constant rate. It is possible to construct the rest of the MDP and starting policies To such that the sequence alternates >> QT ($1, 01) > QT (81, a2) when n is odd, and 1 >> QTT (#1 a2) > (81,
(51, a1) when is even, while both are smaller than QT (81,0g)
3. This is
possible because the policies T. (#2), (8)) can be soft, and it is possible to construct an MDP which produces arbitrary values bounded between 0, 1 by setting R(s2, a1) 1, R(s2,
0, R(s2, 03) = 0 and R(33, a1) = 1, R(s3, 02) = O, R[s3,, ag) = 0. In such MDP, lima Tn will never converge 0g, the optimal policy in this state.
A.4 Deterministic greedification operators are lower-bounded greedification operators
Lemma 3. Deterministic greedification operators Idet, i.e. greedification operators (Definition) 2 that produce deterministic policies are lower-bounded greedification operators 4 in MDPs with finite action spaces.
that is, the minimum difference across two actions that do not have the same value (i.e. the minimum greater than zero difference). If there is no greater than zero difference, then all actions are optimal and every policy is already optimal. Otherwise, the greedification imposed by choosing at least one better action in at least one state has be greater than the minimum difference between two actions.
Proof. Take c S., (... lg(s, -
A.5 The operator Iyms is a I Simit-Sufficient Greedification operator
The operator proposed by Danihelka et al. (2022) is defined as follows:
Igme (T,, (als)) = softmax((r a)) + log [(als)) = E. a expllog (oo
exp(log exp(lugstall))oo((ssaaa))
Lemma 4 (Igms with a stationary a is a Limit- Sufficient Greedification Operator). For any starting policy To E II such that log To (als) is defined and sequences qi..... In such that lim In q E Q iterative applications Tu+1 Iama (Tn, qn) converge to greedy policy with respect to the limiting value q. That is.
lim 700
Σ
(n(alss)n max ((s,),, Vs S.
23

---

Proof sketch: We will prove that repeated applications of the Igm: operator converge to softmax policy of the form
Tn (als) exp(log (a(a))) + no (gn(( (8.a))) a)));
which itself converges to an arg max policy as lim,-+20- For simplicity, we will first prove for stationary q, and then repeat the same steps for non-stationary In, lim for some limiting value q.
Proof. We will show that the Gumbel MuZero operator Z9mz with monotonically increasing transformation, is a Limit-Sufficient Grecdification operator.
Danihelka et al. (2022) have shown that this operator is Greedification operator (Section 4 and Appendix of (Danihelka et al., 2022)). remains for us to demonstrate that the sequence Fn converges for Igmz. such that
lim
Tn (a]s))(((s, max (s,
for any To and Vs E S. Step Convergence with stationary q For a stationary q, at any iteration n, the policy nn can be formulated as:
Tn (a) exp(a a)) + log ( 2n Σ
explo(g(s, log nn 1 ss)
Zn
(10)
Where zn is the normalizer of the softmax operator. We can expand n, backwards as follows:
mn((a= = exp(o((((s a)) + log nn (als()))))
(11)
Zn
1
exp (o(u(s, ) + log alg(s, a)) + An (
(12)
Zn
zn
exp (a(1(3, a)) + ( + log n 2((((s) - log zn-1)
(13)
Zn
exp (20 a)) + log mng(n)(())s))
(14)
2n
(15)
exp (qa( a)) + log no(als))
(16)
As nn is softmax policy, i.e. EaEd nn (als) 1, the product IT z
must act as normalizer:
exp (no a)) log (a((s))
(17)
aEA
We can now directly take the limit lim n..
lim Nu (a) lim (IT 000
exp (norg((s, a)) + log mo(als))
(18)
22000
2,
is well established that as the temperature 1/n goes to zero, the softmax converges an arg max (Collier et al., 2020). With non-stationary 4n we get a slightly more involved sequence, and the formulated proof that the softmax converges to an arg max will serve us demonstrate convergence with 9n- We include the proof that the softmax converges to an arg max below in step 1.5.
Step 1.5: Convergence of the softmax to an arg max Define omaz = max, o(q(s,a)). Let us expl -nd We have:
now multiply by
expl max
(als) (II exp (no(g(s, a) + log no(als))) exp( exp(-n0maa max)
(19)
exp( -no0
To (a(ss))
1 - exp (n(o(((ss))) a)) Tmax)
(20)
exp( -110
24

---

We now note that (g a)) ) max 0 0f ())((s))) & maxa a)) = o maz and otherwise (1(8, a)) max = 0 if (1(8, a)) = max, (1(8, a)) = maz- In that case, exp (n(a(g(8,a))
mman)) exp (0) 1. We substitute that into the limit:
-os exp( nono))
(alss)
1 exp (n(o(a(s,a)) amaz)) =0, if if a)) mar
lim Tn (a|s)= n=+00
limn- exp( exp(-nomuux mux) (1
Nn(als)
1 1
if o a)) maz
(21)
Note that:
1. The numerator where (q((s, a)) 0 maa converges to e
2. The numerator where (g(s, a)) # mea converges to lim e = 0,, 0,,00 0.
3. The denominator always normalizes the policy such that EaeA Tn(als) = 1, E S, due to the definition of the softmax.
As a result, we have:
ifo(q(s,a) # o mea
lim Ty (a(s) =
(22)
22 700
if o a)) = 6 max
For some normalization constant z. lee. the policy lim Tn is an arg max policy with respect to y. that is, the policy has probability mass only over actions that maximize (q)-
Step 2: Convergence with non-stationary qn We will now extend the proof to non-stationary On that assumed to have a limiting value, limn-soo On 4 line with definition Sufficient Greedification.
First, we have:
Ta (a) = exp(o(1n(8 a)) + log mn-1((
(23)
Zn
1
exp (o(g, a)) log (9 a)) n- (alss)
(24)
Zn
2n-1
exp (s(gn(s; + o (s, + log Tn 2((ss))
(25)
2n -1
1
-1==1-) exp
(q.(s. log to(ss))
(26)
Based on the same expansion of the sequence as above. Multiplying by maaa and formulating the limit in a similar manner to above, we then have:
lim Tn (als) lim *u(a|s) n=000
exp (II (a(g((s,a) - &maa))
(27)
100000
mao
Let us look at the term ( Omax).
First, where o(g(s, + omans we have
lim Σ (((g(() - -Omar)
lim Σ (l((((s,))) -(((ss))) - (amax - [{g(s,))) (28)
20000
n
1=1
lim -n(omaz - olg(s, a))) + Σ - (29)
n 200
As the term (s),(s))) - ( goes to zero due to the definition of qns the term (o a)) (g( a)) goes to constant, and the term -n(o -0(g(s a))) goes to
-oo due to the definition of Omaz. Therefor, the limit:
lim (a(a(s,aa) Fmaw) = 00
lim exp (o(g((s))) a)) - -mmaw) = (30)
25

---

Let us look at the second case, where ((((s, a))
Omax. the sequence converges.
lim
E(o a)) 0maz) cr(88a) a)
(31)
A=
For some constant ar(s, u), as lim, o (fn(s, (s,,)) = Omax. Thus, we have again:
if of a)) # omar
lim Tn(a|s) ((s.a)
(32)
if o a)) maar
Demonstrating that nn converges to arg max policy with respect (q)). Since is monotonically increasing, g(s, a) and (g((s, a)) are maximized for the same action thus nn is also an arg max policy with respect to q. Therefor. Igms is Limit-Sufficiee Greedification operator.
A.6 Iyms can be formulated as an insufficient caafffcieet-greeeiffccttto operator
Lemma 5. The Igmz greedification operator with non-stationary a transformation can be formu- lated as an insufficient greedification operator.
Proof sketch: We construct variation of the Iyme operator with an increasing transformation on- which is different at each iteration. Because the transformation is not constant, it converges to some softmax policy rather than an arg max policy.
Proof. The function a used by Igme is only required to be an increasing transformation (sce Danihelka al. (2022) Section 3.3). That is if q(s, a) q(s.a') then we must have that (q(s, a)) > a (g(s a') a')). In practice, the function proposed by Danihelka et al. (2022) is of the form (q(s, a)) (NN) a), where / is function of the planning budget N of the MCTS algorithm.
practitioner might be interested in running the algorithm with a decreasing planning budget over iterations (perhaps the value estimates become increasingly more accurate, and therefor there is less reason to dedicate much compute into planning with MCTS). In that case, we can formulate on (n(9n( (s, a)) 3a 4u (s, a). This transformation is always increasing in q(s, a), adhering to the requirements from a. Nonetheless, the sequence nr will not converge an argmax policy for this choice of o:
lim nn lim M1
exp
(8, + log (a(as)
(33)
23
isn
lim
II
exp
[qn a)] + log (a(as))
(34)
21
Which will converge to some softmax policy as the following limit converges to a constant: lim 3ht Eisn In (s, a)] = c(s, a), and thus the policy remains a softmax policy n, (als) softmax(c(s,c) a) + log (a(ss)))
A.7 Lower Bounded Greedification operators y Lifit-Sufficient Greedification operators
Lemma 6. The set of all lower-bounded greedification operators (Definition 4 not subset of the set of all iimit-sufficient greedification operators (Defiiition B). That is, there exists a lower-bounded greedification operator which not a limit-sufficient greedification operator.
Proof sketch: Convergence with respect to arbitrary sequences limm- On is a strong property. and it is possible come up with sequences for which specific lower-bounded greedification operator do not result in convergence. By constructing such a sequence and choosing such an operator, we will show that there are lower-bounded greedification operators which are Limit- Mumit-Sufficient Greedification operators, demonstrating that lower-bounded greedification operators & limit-sufficient greedification operators.
Proof. Let A = {a1, a2, 033 and a sequence g. (a,) ( 9n (a2) (-1)n+1/2")
+
((22). and qn (a3) = g(az), with a limiting value q = 11,
Let mo = a1. The minimal deterministic Greedification operator Ieee (s) arg min, q(8. a) > En naa' (s) a'), that is, the deterministic Greedification operator which chooses the least- greedifying action at each step will not converge to the optimal policy on this sequence. At each
26

---

iteration, Idee Tn) = 0122 (as in, or 0z), because In alternates (n(a1) ) > 9n (az) for even n. and 9n (a, < On (ag) for odd n. Since this operator lower-bounded greedification operator (see Appendix A.4) this demonstrates that lower-bounded greedification operators & limit-sufficient greedification operators.
A.8 The greedy operator both a limit-sufficient as well as a lower-bounded greedification operator
Lemma 7. The greedy operator Larg max both lower-bounded greedification operator (Definition well as limit-sufficien greedification operator (Definition 5y.
Proof. The greedy operator is greedification operator by definition. We will show that can have both the lower-bounded greedification property as well as the limit sufficient greedification property. Step I): We will show that the greedy operator is a lower-bounded greedification operator (Definition
The greedy operator produces the maximum greedification in any state by definition. Therefor:
Σ
Iargmaz (T, (a)s)) > Iact (T. ()(a)))
aEA
where Ldet is the deterministic greedification operator, As E S,a E A. Since the de- terministic greedification operator itself bounded by an e (see Appendix A.4). we have D.eE Targmaz (T, 4) ()(ss)((s a) - A π (Tas))(( a)l >
Step 2): We will show that the greedy operator is limit-sufficient greedification operator (Definition 5).
We will prove that the sequence (#n; (n) defined for Targ max above converges, such that limn-too Luc Fn (a))))n(ss,)) max; ((, b)l = 0, for any no- That is, the policy converges to arg max policy with respect the limiting value q-
For any In in the sequence, we have by definition max, gn)(s,a). We substitute that into the
the operator
EAEA Targ max (yn; (a)l)(a))) limit:
lim Σ (alss)n max (s, b))
(35)
aEA
lim
max q(s, b)
(36)
n=000
<
lim max l (s, g(s,a) a)l
(37)
n R-+00
max lim (q, a) q(s,, a)l
(38)
R-+00
max lim (n, a) g(s, a)]
(39)
n-000
The first step holds by substitutions. The inequality is well known property used to prove that the greedy operator a contraction, see (Blackwell] 1965). In Equation 38 the limit and max operators can be exchanged because the action space is finite, and finally the limit and absolute value can be exchanged because the absolute value is a continuous function.
A.9 Proof for Theorem 3 for As 1 and In the identity operator
We will prove Theorem 3 first for lo = 1 for improved readability, and in the following Appendix A.10 we will extend the proof for k 2 1. In Appendix A.II we will further extend the proof for value-improvement.
A.9.1 Notation
We use R denote the mean-reward vector R E RIS|AAI, where R,,a E a). We use pa E R|S|||[((S||A) to denote the matrix of transition probabilities multiplied by a policy, indexed as follows: Pat aa = P(s'ls, a) (a') (') We denote the state-action value q and the policy æ as vectors in the state-action space s.t. n. h E RIS[[AA The set C RISIAI contains all admissible policies that define probability distribution over the action space for every state. For convenience,
27

---

we denote (s,, a) as specific entry in the vector indexed by a and g()), T(s) as the appropriate |A| dimensional vectors for index s. In this notation, we can write expectations as the dot product ((8) (s)) = (s)((1) a)] u(s). With slight abuse of notation, we use q IT = v, v RISI
to denote the vector with entries v(s). We use max, E S] to denote the vector with entries max, g(5) max, ((,,a) a).
We let &, denote a state (.,t) E S, that is, a state in the environment arrived at after t transitions. The states 5и are terminal states, and the indexing begins from so- We let q"t ,",,"" xh denote the vectors at iteration m of Algorithm[] [ We let of" oF" denote the sub-vectors of all entries in q nh associated with states 8t. In this notation - is the q vector for all terminal transitions ($)-1;-) after the one iteration of the algorithm.
Proof sketch Our proof follows induction from terminal states, For all terminal states s., the value V (*(HH)
for all policies π. Similarly, ( a) = (= r(s, a) for all policies и. That is, the Q values converge after one update, and from then on remain stationary. Given that the {( remains stationary for all states - limit sufficient greedification guarantees that policy π 11 at state -1 converges to an arg max policy, which guarantees that the value estimates v($H-1)= = Ene x
converge. Finally, as the state-value estimates converge, this process repeats backwards from states - all the way to states 8o, at which point the value y and policy æ converge to the value of the optimal policy and an optimal policy respectively, for all states in the MDP.
A.9.2 Complete proof
Proof. Convergence for Generalized Policy Iteration with k =
We will prove by backwards induction from the terminal states that the sequence limm- ( induced by Algorithm| I converges for any qº, T7, sufficient greedification operator I and k > 1.
That is, for every > 0 there exists a M, such that l|g" - q- l| S E and || q" max, q¹ll || < for all m 2 M,. 9º E I |A| and T." E 11.
Induction Hypothesis: For every
0 there exist M6+1 such that for all M we have |17771 - g S e, and || 4 - max, 4++1 S e.
Base Case / = H 1: Let > 0. Since states 8и are terminal, and have therefore value 0, we have 4 = RH = - and therefore | - - S 6 trivially holds for all m 2 1.
By the Sufficiency condition of the sufficient greedification operator which induces convergence of Tt to an arg max policy with respect to q there exists M such that:
|
max 9n - |-
-4r max 9 S
for all 2 - Thus the Induction Hypothesis holds at the base case.
Case < - 1: We will show that the Induction Hypothesis holds for all states t + 1, it also holds
for states t.
Step 1: Let E > 0. Assume the Induction Hypothesis holds for states t + 1. Then there exists Mi+1 such that | 9/41ll S and || 1 max. L4z ll S for all m 2 Mi+1
Let us define the transition matrix P [S)A with Pa...' = P(s'|s,0)
First, for all m 2 Mi+1 we have:
1 m+
9. 1l = || + ( (.+1) R 7P 1mmaaxp 9411l
(40)
= =>IIP(TTT+ ((t1) - P max الtد.l
(41)
< || m+ 9 max
(42)
< | 4h max g4
(43)
<6
(44)
(40 is by substitution based on step 4 in Algorithm I for = (42) is by the definition of the operator norm |PI (43 is by the fact that the operator norm in sup-norm of all transition matrices is slightly more involved, and follows from the Induction Hypothesis and the limit-sufficient greedification.
1 (Bertsekas) 2007).
28

---

Let us show that 44 i.e. |
9:41 - maxa Z++1 || I holds. Under the infinity norm holds
point-wise for each state S:
m-11 of (0) max (v,
(45)
a
< |म ( - max Pi+1 (8,
(46)
S max qith 1 (s, max 1
(47)
max (9 (s,1(s,)))
(48)
se.
(49)
the induction hypothesis || q max, F+tll S e, which holds under the infinity norm point wise, (46) uses the sufficient greedification operatorproperty [m#t1 (f72l(s) S [++1 (471((s).
44
44 7 the inequality [Tt41| f [ss) I max,' q.t1 a'), (48) the inequality - max, (,,
T:+1 d'), Va' E A, and (49) the induction hypothesis 1i+1 1 S.
Step 2: Pick M2 2 M,+1 such that for all 2 Mi have - max 9f'l S Such M; must exist because of the following: In Step 1, we proved that the first part of the inductive step holds. That is, that df has e-converged to the value of the optimal policy. Such of" satisfies the conditions of the sequence of the limit-sufficient greedification operator. For that reason, the policy n."" must converge (that is || q.th max, qi c), and M; must exist.
Thus, the Induction Hypothesis holds for all states if it holds for states 1 1.
O..... : H there exists M; such that for all m 2 M we have ||9," 9-| S c, and || | qf" max, q7 S c. Therefore, we can pick N, max, o..... 11 M such that 11 - 9. S c, and || q." max, q/ll I for all m 2 N, and / = ...... H 1, proving that Algorithm [ converges to an optimal policy and optimal q-values
Finally, let > 0. By backwards induction, for each
for any T E 11, 9d E AI, k = and sufficient greedification operatorZ.
We proceed extend the proof for k > 1 below.
A.10 Extension of the Proof for Theorem B to k and In the identity operator
In this section we will extend the proof of Theorem need not be modified. In order to extend the proof to k 2 1, we only need to show the following: For all k: 2 and every >> 0 such that the Induction Hypothesis holds, there exists an M such that ||g." m+1
3 from Appendix A.9 to k 2 1. Much of the proof
4f' <.
Proof. We will first extend the notation: let q: m. denote the vector q at states after m algorithm iterations and i 2 1 Bellman updates, such that m.d (T gm de m. d. g=+1 -q m.k
o% and finally
Second, we will extend the Induction Hypothesis:
Extended Induction Hypothesis: For every 0 there exist Mf+1 such that for all m M5+1 and
i 2 we have |4 - 4++1l S e, and || m.. - max, G S The Base Case does not change, that there exists an Mf such that m. oe l| < for all 2 0 and m 2 Mf.
will proceed to Step I in the Inductive Step. We need to show
Let E > 0 and m 2 M 2 M
First, for any 2 1:
11 mn
qi'l +>|R+>((TTHt m.-1)) 9:
< ||||||
+
m+1
4:11
m.i-1 max maxqi+1ll
<6
The first equality is the application of the Bellman Operator in line 4 in Algorithm the ith time. The rest follows from Proof A.9 and the extended Induction Hypothesis.
Second, we need to show that this holds for i as well:
|| m. qil || = 1| m-- 1, of < ||
m-1.8-1 max Git1ll S
Ge+1
29

---

The first equality is by definition, and the the first and second inequalities are by the same argumenta- tion as above. The rest of the proof need not be modified.
A.11 Extension for I2 a general improvement operator
We extend the proof from the above section for all non-detriment operators (that is, non-strict greedification operators) L2 used for value improvement.
Proof. Similarly to the proof of Theorem 3 from Appendix A.9
(and A.10 we will prove by
backwards induction from the terminal states 8h that the sequence lim (TTh )mh) induced by Algorithm 2 converges for any gº =,, sufficient greedification operator I, greedification operator In and 2 1. That is, for every c > there exists a M, such that -1 q"ll I c and ||>T
4Th max, q"Il < for all m 2 Mc. d² E RISSIAI and N E II. The proof follows directly from the proof in Appendix A.9. The base case is modified the qs converge immediately and the policy convergence is not influenced by the introduction of Iz. The Induction Hypothesis need not be modified. In the inductive step. Step follows directly from the Induction Hypothesis, and Step
need not be modified for the same reason the base case need not be modified.
A.12 Convergence of Algorithm 2 with lower bounded greedification operators
We extend the proof from Appendices A.9 and A. to bounded greedification operators.
Proof sketch The proof is almost identical to that of limit- greedification, with one major difference. Lower-bounded greedification allows for convergence to a greedy policy in finite iterations (see Lemma 8 below) with respect to any stationary For that reason, the values at states - become exact in finite iterations (unlike limit-sufficient, where they converge only in the limit). As they become exact, they also become fully stationary, and as they become stationary, greedification guarantees that the values (and policy) at states $H-2 become exact and stationary in finite iterations, and the process repeats by induction all the way back to states 8g-
Below we first prove Lemma and then use Lemma to complete the induction proof.
A.12.1 Lower bounded greedification converges to an arg max policy finite steps We will begin by proving that operators with the Bounded Greedification property:
Σ I(T.g)(als))(3..)
Σ M(als)q(s,a) >
unless EAEA I( (alss)a(s) policy with respect to any q E Q, in a finite number of steps.
max, q(s, a) are guaranteed to convergence to an arg max
Lemma 8. Let I be a bounded greedification operator and let a sequence Ta+1 I(g, na). For any starting To E I1, q E Q, there exists an M for which:
Tu (a)()))(( a) max g(8, 0), Vn > M.
a
That is, the policy nn converges greedy policy with respect to q in finite number of steps n M.
Proof. Let I be a Bounded Greedification operator. At each iteration, the sequence E.eA Tx (a)s))((ss)) a) must increase, i.e. EneA Th (als))((s,)) > EneA 1n- (als)g(s, a), > O, for least one state E S. The same sequence is monotonically non-decreasing. by definition of greedification, for all other states. Therefore, the sequence E.es EaEd In (n(a)s)q(s, a) is monotoni- cally increasing (for each state Eas T.. (a (a)s))(s, a) is at least as large as in the past step, and in at least one state it is distinctly higher), unless EaEA Tn (n(a)s)(ss, a) = max, q(s, a).
Due to the Bounded Greedification property, the minimum increase is bounded by e, that is:
min Mim
(als))((s)
-E (ali(ass)(ssa)) n > 0.
A
SES aEA
30

---

The sequence EeEs Eos Nn (n(a)s)g(s, a) is bounded by Eees max, g(s, a) from above, and by Exes min, 7(8, a) from below. The increases between any two iterations is bounded from below by e by definition unless the policy is already greedy, as / a lower-bounded greedification operator.
Since the sequence is bounded from below and above and the increase bounded a constant amount 0, it must converge in finite n. < oo the maximum of the sequence DnEA Wn (a))))(())))))))))))))))()))))) a) max, g(s, a). That is, n converges to greedy policy with respect q in finite number of iterations n.
A.12.2 Modified Induction for Bounded Greedification
We modify the induction of the proof of Theorem Bl with finite- sufficient greedification operators. that converge to an arg max policy in finite number of iterations.
Proof. Modified Induction Hypothesis: There exist M441 such that for all m 2 Mc+1 we have f+ T+-1 and MHz u++1 maxa L:+1'
Modified Base Case: Because the convergence to the arg max is finite time (Lemma g) there exists MH-1 such that:
7 H
4H max TH-1
for all m 2 MH-1- Thus the Modified Induction Hypothesis holds at the base case.
Modified Case < H Step (1): Similarly, for all 2 Mr+1 we have:
11a=+1 9i'l || = || 7P (4+1) - R max I411l
(50)
[111P( m-1 (441) max gi+1ll
(51)
- |Pl| [ m+1 g max 4+1ll
(52)
(53)
Since also | 1g 2.'l 2 0, we have ||q.t 4. 0 and a. a== of
Step (2): Pick M, 2 M1+1 such that for all m 2 M2 we have || qfth max. q |l = 0 which must exist due to convergence to the argmax in finite time of this operator class. Thus, the Modified Induction Hypothesis holds for all states / if it holds for states + 1.
B
Additional Results
B.1
Value improvement and over estimation
Explicit improvempto results in greedier evaluation policies. As such, it should directly increase the value targets compared to no value improvement (demonstrated empirically in Figure Figute]center), The same can be expected to happen when the value improvement relies implicit improvement operators such as the expectile operator. Respectively, any increase to the value targets can be expected to interact with (and more specifically, increase) over-estimation bias.
is well known that overestimation bias can induce pseudo optimistic exploration. This is because increased overestimation bias is more likely to increase overestimation of unvisited state-actions, thus driving exploration into these unvisited state-actions. For that reason, while can be detrimental in certain environments (as demonstrated by Fujimoto et al.] 2018). it can be beneficial in others.
In Figure 4 we investigate the interaction between implicit improvement and over estimation with the expectile operator, with VI-TD3. Each row presents results for a pair of environments. We plot (i) final averaged evaluation return after million environment interactions vs. T (the first and third columns from the left). And (ii) we plot final over estimation bias after 3 million environment interactions vs. 7 (the second and last columns from the left).
Generally increases over estimation bias increases. On the other hand, in many environments the majority of the performance gain is observed for values of S 0.6 (for example hopper-hop, for which none to negligible over estimation bias is observed. This is summarized in Table
We conclude that while in this domain overestimation can be beneficial (e.g. fish-swim), benefits of VIAC are not limited to the benefits of overestimation (r S 0.6 in hopper-hop, humanoid- stand/walk/run, for example).
31

---

Fiaal Evalustion hopper-stnnd
Dverectimation hopperstand
Final Evaluation hopper-hop
Overectimation hopper-hop
Fetal value GAtirna 2-1 5-38
1.3
Ertal ENANAtio nE..m
-20
22
30
50
83
40
0.5
28
033
-20
0.6
577
38
0.3
0.6
5.7
6.3
08
5.7
63
Evaluation humanoid--on
Final humanaid-wn
Nmaal VALUM SSmmI bua
Rraal avaltaation sct.m
sssmationn
Final Evaluation humanoid-run
120
Overestimatton humanotd-mg run
Final Evaluation quadrupes--rn
120
Overestimation undrupedd
1.3
Rral Leaduatonn stum
90
0.6
3.
0.9
-20
05
0.6
0.9
0.66
0.7
6.9
05
0.6
0.2
6.9
quadruped
Overres guddhppal--ull
swinn
Ovarre
rocur
037
Final Evolustion walker-walk
Ocemesimmdion malcee-malt
Final Evaluation walker-nu run
Overestimaion waleneea
023
342
0.6
2.8
0.6
0.4
0.6
0.7
6.1
0.7
3.5
6.1
chastsh-nun .un
Drecestimation cheelehotun
bcrobot-swingup
Ovemesimation acrebot-wwingu
897
19-M MU GOSTNatio 2014
Final sooliaal on recaan
733
593
90
Figure 4: Mean and standard across 10 seeds. Left: Final evaluation greedification parameter т for VI-TD3 with implicit improvement after 3m environment interactions. 7 0.5 is baseline TD3. Right: Final overestimation bias vs. after 3m environment interactions. The majority of the performance increases are independent from an increase in over estimation bias.
32

---

Table 1: Performance and overestimation statisties across environments.
Performance gain without overest. increase
Max
Overest. as of max overest.
Overest.
Environment
7
performance gain
T
% of
performance
hopper-stand
353.80 + 60.77
0.75
5%
2%
hopper-hop
88.01 + 24.31
0.59
217.25 + 26.38
0.8
12%
2%
humanoid-stand
206.79 + 64.93
0.55
503.93 + 14.72
0.75
31%
humanoid-walk
320.64 + 43.42
0.6
415.28 + 29.29
0.7
14%
3%
humanoid-run
107.50 + 5.38
0.6
125.25 3
0.75
22%
5%
quadruped-run
166.1844..66
0.6
3%
1%
quadruped-ve waak
1: 2
0.59
1%
fish-swim
0.9
87%
4%
walker-walk
27.04 9.63
0.7
28%
1%
walker-rum
17.65 + 29.07
0.53
98.23 25.79
0.75
27%
1%
cheetah-run
10.37 11.18
0.53
10.37 = 11.18
0.53
4%
1%
acrobot-swingup
22.74 3.94
0.6
257.24
0.8
11%
6%
B.2
Value Improved TD7
hoppor-stand
hopper-hop
2.00
750
750
600
5.00
400
250
250
200
Average
0-
106
1es
1u6
humanold-mu
fish
acrobot- swingup
acrobot-swingup
qua quadruped-walk
300
800
200
800
100
200
60
166
166
quadruped-run
walker-wallk
walker-run
cheetah-run
200
800
400
400
td7
200
200
400
vi. td7 (expectill)
Environiment
106
Environment
166
Environment
Xc6
Environment
185
Figure 5: Mean and two standard errors across 10 seeds of VI-TD7 with expectile loss vs. TD7 on the same tasks as Figure Similar performance gains are observed for VI-TD7 in this domain.
B.3
Increased greedification of the acting policy
hopper-stand
hopper-hop
Avarage evaluation retum
150
100
td3 (baseline)
200
vi td3 20)
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Environment steps
1ec
Environment steps
Figure 6: Mean and one standard enror across 10 seeds in evaluation of VI-TD3 with Policy Gradient as the value improvement operator, vs. TD3 with 20 repeating policy gradient steps in each update, vs. baseline TD33. Increasing the number of sating-policg updates on the same batch does not contribute to performance.
33

---

B.4 Increased value improvement vs. increased replay ratio
If one is able to spend additional compute on gradient updates, an increased replay ratio is an attractive alternative value improvement. Figure 7 we compare VI-TD3 with increasing number of gradient steps to TD3 with increasing replay ratio. In line with similar findings in literature (Chen et al. 2021). replay ratio provides a very strong performance gain for small ratios. As the ratio increases, performance degrades, result which the literature generally attributes to instability. 'he VI agent on the other hand does not degrade with increased compute. This suggests reduced interaction between greedification of the evaluated policy and instability compared to that of the acting policy.
hopperstand
hopper-hop
Average exaluation return
(baseline)
125
td3 (mr=4)
tdB (m-15)
400
75-
200
25
0.25
0.50
1:00
1:25
1.50
1.75
2.00
0.00
0.50
1.25
1:50
1.73
2.00
steps
1e6
Enviranment steps hopper-hop
leh
hopper-stand
Average evaluation ritur
800-
200
vl. 13 (pg=5)
(pg-10)
600
150
v1. 13 (pg-20)
400
200
50-
0.25
0.50
1.00
1.25
1.50
1.75
2.00
0.00
0.50
1.25
1.50
2.00
Invironment steps
1c6
steps
Figure Mean and one standard errors across seeds evaluation of VI-TD3 with policy-gradien based value improvement vs. td3 with increased replay ratio. Number of gradient steps are equated across rr / VI agent pairs as a pseudo metric for compute. The performance of TD3 generally degrades with increased replay ratio, in line with the results of Chen et al. (202I). In contrast, the performance of VI-TD3 increases with compute, without access to additional mechanisms to address instability.
B.5 Value Improvement vs. Policy Improvement
A few questions that VIAC naturally raises are how does the greedification of the acting policy compare to the greedification of the evaluated policy? one more important than the other? Does one render the other unnecessary?
To answer these questions stripped off many additional influences as possible, we construct simple experiment in toy grid environment with Value Improved Generalized Policy Iteration (VIGPI). The VIGPI algorithm uses the operator IGMæ for both policy improvement (PI) as well as value improvement (VI) with increasing S. The value table is initialized with Os. Evaluation uses the Bellman update until (s) - (s) < 0.0005 for all states. We plot the value at iteration of the starting state 8o function of i.
In the left plot we compare agents with increasing for the acting policy improvement (PI), In the right plot we compare agents with increasing 3 for the evaluated policy improvement (VI). For both agents we include a final variation which uses the greedy operator for either PI or VI, as well as versions that use 3 = 0 (i.e. no greedification).
Since any value improvement that less greedy than the greedy update depends on the acting policy, slow greedification of the acting policy slows down the convergence of the VI variants. On the other hand, VI + Pl is able to significantly increase the rate of convergence compared
same without 0. with PI there is no learning at all (because the
VI. Similarly, when the rate of greedification policy never changes) and with there is no issue, because remains non-detriment operator, and thus does not prevent the improvement of the acting policy (Corollary 2
34

---

Increasing Policy Improvement
Increasing Value Improvement
04
V1-33
3-10
-94 20
100
100
ltcraiion
Iteration
Figure 8: Starting value iteration.
C Explicit and implicit Value Improved Actor Critic algorithms
In Algorithms 3 and modifications to baseline off- off-policy Actor Critic marked in blue.
Algorithm 3 Explicit Off-policy Value-Improne Actor Critic
1: Initialize policy network Tp, Q network ęş. Greedification Operators I₁ and Iz, replay buffer B
2: for each episode do
3: for each environment interaction do
4: Act ag ~ (s:)
5: Observe 8t-1,
6:
Add the transition (st, a, rt, $1+1) to the buffer B Sample batch b from B of transitions of the form (St. Oz, Pt, S1+1)
7:
8:
Update the policy Te(s;) + I1 (Ta. (st), As, E b
9:
Further improve the policy x (8++1) [2( 2o (sL+ Vsr+1 E
10:
Sample an action from the improved policy a ~ T' (8++1) V8441 b
11:
Compute the value targets y(st, a,) rt (s a), ((s,, rt, 8411) E b
12:
Update Is with gradient descent and MSE loss using targets
Algorithm 4 Implicit Off-policy Value-Improved Actor Critic
1: Initialize policy network wo. Q network 4s- Greedification Operator I,, implicit greedification parameter and replay buffer B for each episode do
2:
3: for each environment interaction / do
4: Act dz (=()))
5:
Observe 8;+1,
6:
Add the transition (st, d2, Fr, 8(+1) the buffer B
7:
Sample a batch b from of transitions of the form (st, (s, T2, S1+1)
8:
Update the policy TT()t) + I, (Ta (s)),, Vs, Sample an action from the policy a ~ T($+11, V b
9:
Compute the value targets y(st, a.) +- rt 790 (8++1, a), V(st, at rt, 8111 E b
11:
Update to with gradient descent and L loss using targets y, see Supplement D.2
D Experimental Details
D.1 Gradient-Basee V1-TD3
Gradient-based VI-TD3 copies existing policy used compute value targets (the target policy, in TD3) т&r into a new policy T'P- The algorithm executes N repeating gradient steps on Thp with respect to states 8++1 E with the same operator TD3 uses to improve the policy (the deterministic policy gradient) and with respect to the same batch b. The value-improved target yst, a,) is computed in the
35

---

same manner to the original target of TD3 but with the fresh greedified target network Té.. In TD3, that summarizes to sampling an action from a clipped Gaussian distribution with mean T (st-1): 1). variance parameter and clipped between (-;
d' ~ ( (s++1), c..pp))pp.p3pp
(54)
And using the action a' to compute the value target in the Sarsa manner:
S ax) = re + min qd, (s++1, a'), V(s oe, rt, St41) E b
(55)
(E(1.2)
The policy used to compute the value targets no is then discarded.
D.2 Implicit Policy Improvement with Expectile Loss
The expectile-loss C3 proposed by Kostrikov et al. (2022) as an implicit policy improvement operator for continuous-domain Q-leaming can be formulated as follows: when y(st, az) > g(s,, at) (the target is greater than the prediction), the loss equals ( a,) - q(s), ((8t, (t) (the target is smaller than the prediction) the loss equals (1 - (v((s, at) - (((),,))) = 0.5, this loss is equivalent to the baseline Le loss. Intuitively, when т > 0.5 the agent favors errors where the prediction should increase, over predictions where it should reduce. L.e. the agent favors targets where n' (st++1) (the implicit policy evaluated on the next state) chooses "better" actions than the current policy, directly approximating the value of an improved policy.
When y(s, a+) S
By imposing this loss on the value network, in stochastic environments the network may learn to be risk-seeking, by implicitly favoring interactions 8t. 0z. Tt; 8141 where the observed rt. was large the state 8:+1 was favorable. This is addressed by Kostrikov al. (2022) by learning an additional vy network that is trained with the expectile loss, while the q network is trained with SARSA targets T + you (81+1) and the regular L2 loss, while the Vy network is trained with targets y(s,, a,) = (s:, ax) and the expectile loss. In deterministic environments this is not necessary however, and in our experiments we have directly replaced the C2 loss the value q6 with the expectile loss.
The value target y(s,, a,) remained the unmodified target used by TD3 / SAC respectively.
Another aspect where our use of the expectile loss diverges from that of Kostrikov et al. (2022), is that we use it with respect the online acting policy ne. rather than with respect to the policy captured
by the replay buffer. That is, the original targets of IQL are:
y( a.) rt + (s41)) (st, Te,
Pe, S441) E D.
(56)
On the other hand, our usage is with respect to me:
y(8,, az) = + (0 (82+1, m)(s+1)), (8t, (t, 8141) E D.
(57)
This results in targets that are much more on policy (more "fresh", if you will), compared to the information in the replay buffer, unless the replay buffer contains trajectories exclusively from ma. This also enables compute more accurate targets by averaging across any number of fresh samples {a:)W ~ T0(8++1) with: y(sr,a) a4) rt + Nx Σi (0(s1+1,0;) az), although this was not used in our experiments.
D.3 Evaluation Method
We plot the mean and standard error for evaluation curves across multiple seeds. Evaluation curves are computed as follows: after every n = 5000 interactions with the environment, m = 3 evaluation episodes ran with the latest network of the agent (actor and critic). The score of the agent is the return averaged across the m episodes. The actions in evaluation are chosen deterministically for TD3, SAC and TD7 with the mean of the policy (the agents use Gaussian policies). The evaluation episodes are not included in the agent's 's replay buffer or used for training, nor do they count towards the number of interactions.
D.4 Compute
The experiments were run on the internal compute Delft of the following GPU architectures: NVIDIA Quadro K2200, Tesla PTO0, GeForce GTX 1080 Ti, GeForee RTX 2080 Ti, Tesla V100S and Nvidia A-40. Each seed was ran on one GPU, and was
Delft AI Cluster (DAIC) (0024) using any
36

---

given access to 6GB of RAM and 2 CPU cores. Total training wall-clock time averages were in the range of 0.5 to hours per 10 environment steps, depending on GPU architecture, the baseline algorithm and VI variations. For example, baseline TD3 wall-clock time averages were roughly 1. 1.25 hours per 10 environment steps on average. The total wall clock time over all experiments presented in this paper (main results, baselines and ablations) is estimated at z 12000 wall-clock hours of the compute resources detailed above: s 7320 for the results in the paper and R 4300 for the ablations in the appendix. Additional experiments that are not included in the paper were run in the process of implementation and testing.
D.5 Implementation Hyperparemeter Tuning
Our implementation of TD3 and SAC relies on the popular code base CleanRL. (Huang et al. 2022).
CleanRL. consists of implementations of many popular RL. algorithms which are carefully tuned to match or improve upon the performance reported in the original paper. The implementations TD3 and SAC use the same hyperparameters as used by the authors (Fujimoto et al. (2018) and Haarnoja al. (2088)) respectively), with the exception of the different diffennnteearning rates for the actor and the critic i SAC, which were tuned by CleanRL.
For the TD7 agent, we use the original implementation by the authors (Fujimoto the action space to the DeepMind control's 's in the same manner as CleanRL's implementation of TD3, Additionally, non-prioritized replay buffer has been used for TD7 which was used by the TD3 and SAC agents as well. The hyperparameters are the same used by the author.
2023), adapting
The VI-variations of all algorithms use the same hyperparameters as the baseline algorithms without any additional tuning, with the exception of grid search for the greedification parameters presented in Figure
D.6 Network Architectures
The experiments presented in this paper rely standard architectures for every baseline. TD3 and SAC used the same architecture, with the exception that SAC's policy network predicts a mean of a Gaussian distribution well as standard deviation, while TD3 predicts only the mean. TD7 used the same architecture proposed and used by Fujimoto et al. (2023).
TD3 and SAC:
Actor: layer MLP of width 256 per layer, with ReLU activations on the hidden layers. The final action prediction is passed through tanh function.
Critic: 3 layer MLP of width 256 per layer, with ReLU activations on the hidden layers and no activation on the output layer.
TD7: Has a more complex architecture, which is specified in (Fujimoto al. 2023
D.7
Hyperparemeters
TD3
SAC
TD7
exploration noise
0.1
exploration noise
0.1
Target policy noise
0.2
Target policy noise
0.2
Target smoothing
0.005
Target smoothing
0.005
noise clip
0.5
auto tuning of entropy
True
noise clip
0.5
Critic learning rate
le-3
Critic learning rate
3e-4
Learning rate
3e-4
Policy learning rate
3e-4
Policy learning rate
3e-4
Policy update frequency
2
Policy update frequency
2
Policy update frequency
2
0.99
?
0.99
?
0.99
Buffer size
10
Buffer size
10"
Buffer size
10a
Batch size
256
Batch size
256
Batch size
256
learning start
10
lcarning start
10*
learning start
10*
evaluation frequency
5000
evaluation frequency
5000
evaluation frequency
5000
Num. eval, episodes
3
Num. eval. episodes
3
Num, eval. episodes
37


Reviews:
[{"review_id": "h0ei9aLoFp", "reviewer": "Reviewer_TiJB", "summary": "This paper introduces a new framework called Value-Improved Actor-Critic (VIAC), which separates the policy that interacts with the environment from the one being evaluated by the critic. The main idea is to let the critic evaluate a more \"greedy\" policy to get better value estimates, while the actor still updates slowly and stably using gradients. The authors back this up with a solid theoretical analysis—showing under what conditions this separation still leads to convergence—and test the idea both explicitly (with multiple critic-side updates) and implicitly (using expectile loss). They apply it to TD3 and SAC and find that the implicit version, in particular, gives nice performance gains with almost no extra compute.", "questions": "1. Could the authors respond to the concerns raised in the weaknesses section, particularly the first point regarding whether the implicit method genuinely performs value improvement?\n\n2. Could the authors elaborate on how the mismatch between the actor and critic policies affects learning? For example, if the two policies diverge significantly, does this impact training stability or sample efficiency?\n\n3. The theoretical results are grounded in finite MDPs, but modern deep RL involves continuous state and action spaces, along with approximation error. How well do the guarantees hold up under function approximation and in the presence of non-stationary Q estimates?\n\n4. The balance between greedy updates and stable improvement somewhat reminds me of TRPO, which deliberately avoids overly greedy policy updates. Do the authors see any conceptual connection here? It would be interesting to hear their thoughts.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper presents a novel and elegant way to address a long-standing tradeoff in actor-critic methods: the tension between greediness and stability.\n2. The theoretical analysis is thorough and introduces meaningful distinctions between types of policy improvement operators, offering a clear view into when convergence is (or isn’t) guaranteed. I also like the intuition for the theories.\n3. The empirical results are compelling. The authors show that integrating value improvement into TD3 and SAC leads to consistent performance gains across a range of continuous control tasks.\n4. The implicit implementation using expectile loss is particularly practical—it’s simple to implement, introduces negligible overhead, and still captures the benefits of value improvement.\n\n**Weaknesses:**\n\n1. I’m not entirely convinced that the implicit version using expectile loss fits neatly within the VIAC framework. There is an important distinction between genuinely greedy value estimation and simple overestimation—larger value estimates may stem from estimation bias rather than deliberate greedification. It remains unclear whether the observed improvements are due to a shift toward a greedier policy or simply the result of over- or underestimation. This is my primary concern, and I would appreciate a more detailed explanation from the authors on this point.\n2. While the explicit variant is mainly included to motivate the framework conceptually, its high computational cost makes it impractical for most real-world use. That wouldn’t be a problem on its own—after all, the implicit version is the one intended for practical use. However, given my concern above about whether the implicit method truly reflects value improvement in the VIAC sense, the lack of a practical, clearly greedy implementation slightly weakens the overall strength of the framework. If expectile loss isn’t truly implementing a greedy operator, then the framework may not be as broadly applicable in practice as it first appears.\n3. While some of the evaluated environments (e.g., cartpole-swingup, hopper-stand) exhibit partially sparse rewards, the benchmark suite is still relatively standard. It would be interesting to test the method in more extreme sparse-reward settings, such as long-horizon navigation or delayed-reward tasks, to further validate its robustness.\n\n\n\nOverall, I found the paper clear and well-motivated. If I misunderstood anything, I'm happy to be corrected, and I’m open to revisiting my evaluation based on the authors’ responses.", "was_revised": true, "final_justification": "The rebuttal resolved my main concerns: IQL’s fit within VIAC is now clear, bias analysis shows gains beyond estimation changes, and BON results confirm practical value. Only minor uncertainty remains on theory under deep RL. Given these, I am raising my score.", "rebuttal": "Dear reviewer, \n\nThank you for your time, thoughtful review and detailed comments and questions.\n\n**1a. Does IQL fit within VIAC?**\n\n*TLDR: We conclude that yes, the expectile operator fits within the VIAC framework.*\n\n[1] prove that the expectile loss explicitly evaluates greedier policies with respect to some sampling policy $\\pi_b$. \nIn their setting of offline RL, $\\pi_b$ is captured as the distribution over the actions per state in the replay buffer.\nHowever, since we are not in the offline RL setting, we use the expectile loss with respect to fresh actions from $\\pi_\\theta$ directly (see Appendix D.3), which results in an operator that explicitly greedifies with respect to $\\pi_\\theta$.\nFurther, [2] show that the policy $I_{IQL}(\\pi, Q)$ can be extracted explicitly.\nWe conclude that when $ \\tau > 0.5: \\sum_{a \\in A} I_{IQL}(\\pi, Q)(a|s) Q(s,a) > \\sum_{a \\in A} \\pi(a|s) Q(s,a) $, i.e., the expectile loss results in explicit greedification, and by using it with respect to fresh actions from $\\pi_\\theta$, it explicitly greedifies with respect to $\\pi_\\theta$.\n\nThe expectile operator however will not converge to the greedy policy unless $\\tau \\to 1$, and for that reason, it is not a *sufficient* improvement operator unless $\\tau \\to 1$.\nThis raises again the question whether IQL fits within the VIAC framework or if it conflict with the theory. Corollary 2 addresses this by proving that even when insufficient (\"non-detriment\") operators are used for value improvement, the algorithm still converges.\n\n**1b. It remains unclear whether the observed improvements are due to a shift toward a greedier policy or simply the result of over- or underestimation.**\n\n*TLDR: We conclude that the results presented in Figures 1 & 4 & additional results included here, suggest that performance gains of VIAC go beyond simply the result of over or underestimation.*\n\nIn Figure 1 (second row, right) we see performance improvement without change in estimation bias.\nThis shows that performance gains with VIAC can go beyond simply the result of over or underestimation.\n\nOverall, we would expect to see 3 behaviors, differing across environments:\n(i) Where some overestimation is beneficial, VIAC may benefit from increased greedification up to the point of over-estimation increase. As long as over estimation is beneficial, it can benefit from the over estimation induced by increased greedification.\n(ii) Where overestimation is mostly detrimental, VIAC may benefit from increased greedification up to the point of over-estimation increase. Additional greedification will tradeoff against any negative effects of overestimation.\n(iii) As long as the effects of overestimation are minor, VIAC may benefit from increased greedification irrespective of overestimation.\n\nIn Figure 4, we see that humanoid-run belongs essentially to (iii): the entire performance improvement is experienced when greedifying $\\tau \\to 0.6$, up to which point the estimation bias is not strongly influenced. \nFrom $\\tau > 0.6$, the estimation bias monotonically increases while performance remains roughly the same, until a point where the overestimation is too high and performance decays.\n\nWe include a table which extends and summarizes Figure 4 to all environments.\nWe report:\n\n(i) The maximum final performance increase (final performance - baseline final performance) observed over $\\tau$s, without (<1%) increase in overstimation. The $\\tau$ associated with this final performance.\n\n(ii) The maximum final performance increase observed over $\\tau$s. The associated $\\tau$. The % overestimation increase out of the maximum overestimation observed in this experiment. The % overestimation / performance for this $\\tau$.\n\nAveraged across 10 seeds per $\\tau$, and along with SEM. Stat. signif. differences from baseline are marked in bold.\n\nIn 5 environments (hopper-hop, humanoid-stand/walk/run, acrobot-swingup) there is significant performance gain without an increase in over estimation.\nIn 6 environments (hopper-stand, quadruped-walk/run, fish-swim, walker-walk/run) any performance gain is correlated with some increase in over estimation.\nIn 1 environment (cheetah-run) there is no statistically significant difference between the baseline and VI-TD3.\n\nWe conclude that the benefits of VIAC go beyond increasing over estimation bias. \nOn the other hand, VIAC can be used to increase estimation bias, and when it is beneficial, VIAC can benefit from it (and when it is not beneficial, it's one of the reasons it is important that the amount of greedification be tuned).\n\n|Environment|Gain w.o. overestimation increase|$\\tau$|Max. perf. gain|$\\tau$|Overestimation % of max. overestimation|Overestimation as % of perf.|\n|-|-|-|-|-|-|-|\n|hopper-stand|-|-|353.80 ± 60.77|0.75|5%|2%|\n|hopper-hop|88.01 ± 24.31|0.59|217.25 ± 26.38|0.8|12%|2%|\n|humanoid-stand|206.79 ± 64.93|0.55|503.93 ± 14.72|0.75|31%|5%|\n|humanoid-walk|320.64 ± 43.42|0.6|415.28 ± 29.29|0.7|14%|3%|\n|humanoid-run|107.50 ± 5.38|0.6|125.25 ± 3.78|0.75|22%|5%|\n|quadruped-run|-|-|166.18 ± 47.69|0.6|3%|1%|\n|quadruped-walk|-|-|276.55 ± 59.85|0.59|1%|1%|\n|fish-swim|-|-|308.07 ± 19.02|0.9|87%|4%|\n|walker-walk|-|-|27.04 ± 9.63|0.7|28%|1%|\n|walker-run|17.65 ± 29.07|0.53|98.23 ± 25.79|0.75|27%|1%|\n|cheetah-run|10.37 ± 11.18|0.53|10.37 ± 11.18|0.53|4%|1%|\n|acrobot-swingup|22.74 ± 3.94|0.6|257.24 ± 6.13|0.8|11%|6%|\n\n**2. Practical value improvement with other, low cost, explicit greedification operators:**\n\n*TLDR: In additional results we include here other - low cost - explicit greedification operators demonstrate significant  and comparable increases to performance.*\n\nWe include an example of an explicit improvement operator that is inexpensive and shows strong performance benefits in these domains: Best-of-n (BON). BON samples N actions, takes the value of the maximizing action as the bootstrap for the value target.\nThis is a popular operator in RL for language models [3].\nWe include a table that summarizes a figure like Figure 3 for baseline TD3, and the IQL and BON variations of VI-TD3.\n$N=64$ for BON and for IQL we use the same $\\tau=0.75$.\n\nBON shows similar performance gains to IQL across domains.\n\nBelow we report final-performance results normalized per environment between minimum and maximum final return over all seeds (20 for TD3 and VI-TD3 (IQL), and 12 for VI-TD3 (BON)), averaged per agent over environments, with +- one SEM.\nThe IQL and BON variations are within each others 2 SEM confidence bound.\n\n|Agent|Normalized Final Perf.|\n|-|-|\n|Baseline TD3|0.41 ± 0.05|\n|VI-TD3 (IQL)|0.75 ± 0.04|\n|VI-TD3 (BON)|0.68 ± 0.04|\n\nFor results per env., due to limited space, see the reply to reviewer HLEH.\n\n**3. Sparse rewards:**\n\nDue to the possibility of explicitly tuning over-estimation bias through the tuning of the greedification with VIAC, we would expect VIAC to be able to induce pseudo-optimism-in-the-face-of-uncertainty exploration which should perform even better than baselines that do not have it, in sparse reward domains.\n\n**Q2. Could the authors elaborate on how the mismatch between the actor and critic policies affects learning?**\n\nIn our experiments, we have only tried operators that sample from the acting policy $\\pi_\\theta$ and thus do not majorly diverge from it (like BON or IQL). We would conjecture that the important factor is whether the critic predictions are reliable for the evaluted policy. if OOD actions are evaluated, we would expect off-policy-evaluation concerns.\n\n**Q3. The theoretical results are grounded in finite MDPs, but modern deep RL involves continuous state and action spaces, along with approximation error. How well do the guarantees hold up under function approximation and in the presence of non-stationary Q estimates?**\n\n*Cont. actions*: While this is not usually analyzed in the setup of DP (to our knowledge), our theory is not fundementally limited to discrete actions. The induction does not depend on the action space, only on whether the improvement operators can greedify the policy all the way to an argmax.\n\n*Approximation error*: When taking specific operators (say policy gradient for PI and the argmax for VI), under specific function approximation assumptions (presence of local optima, bounds on the possible error, etc), we don't see a reason to assume VIAC will not converge: VIAC simply maintains a Q estimate that is larger at any step than that of AC, but is still anchored in real policies.\n\nNote that our analysis is already in the presence of non-stationary and approximate Q estimates. Similarly, we conjecture that the results can generally extend to other standard methods to approximate Q, such as Monte-Carlo / TD-$\\lambda$ targets.\n\n**Q4. The balance between greedy updates and stable improvement somewhat reminds me of TRPO, which deliberately avoids overly greedy policy updates. Do the authors see any conceptual connection here?**\n\nYes absolutely.\nWe touch on a similar connection in the introduction, in lines 43-49.\nIf we think of the improved policy of TRPO $\\pi'$ as the \"optimally greedy\" policy, the evaluated policy is still limited to $\\pi_\\theta$, which is only moved towards $\\pi'$ with gradient steps.\nVIAC in contrast would suggest directly evaluating $\\pi'$, which seems very natural for TRPO. Does that make sense?\n\n*All results presented in the rebuttal will be included in the appendix.*\n\n*Please let us know if there remain any unaddressed concerns, comments or questions.*\n\n[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline reinforcement learning with implicit q-learning.\" arXiv preprint arXiv:2110.06169 (2021).\n\n[2] Hansen-Estruch, Philippe, et al. \"Idql: Implicit q-learning as an actor-critic method with diffusion policies.\" arXiv preprint arXiv:2304.10573 (2023).\n\n[3] Huang, Audrey, et al. \"Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment.\" arXiv preprint arXiv:2503.21878 (2025)."}, {"review_id": "QezqbIBzka", "reviewer": "Reviewer_HLEH", "summary": "The paper studies a class of actor critic algorithms that separates between an \"evaluated policy\" and the \"acting policy\". The evaluated policy is induced by a greedification operator that is separate from the greedification operator of the acting policy.\nAs a result, the policy evaluation phase effectively generates additional \"value improvement\" by means of evaluating an improved policy.\nThese algorithms are called Value Improved Actor Critic Algorithms (VIAC).\n\nThe core idea is that value improvement can be controlled separately from the the greedification of the acting policy, which may be constrained to support greedification operators that are more limited than those of the value improvement.\n\nThe paper studies this class of algorithms within the generalized policy iteration framework, and presents elementary convergence results. Further, experiments are carried that indicate VIAC offer clear improvement over non value improved baselines TD3, TD7, and SAC on continuous control tasks.", "questions": "- Line 247: Typo \"the sets are not also not disjoint\"\n- Is $I_{gmz}$ just a slight generalization (by use of $\\sigma$) of the standard soft policy iteration greedification operator?\n- The notion of \"non-detriment\" operator is first introduced in the statement of Corollary 2, and it's meaning is unclear (the \"e.g.\" just points back to the definition of a greedification operator).\n- Lines 282-283: It is unclear what is the difference between questions 1 and 2.\n- It is unclear how you map the greedification operators in Alg2 to those of Alg3. When you increase the $n$ of $pg=n$, do you increase it for both the evaluated policy and acting policy?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n* The paper makes an appealing observation, that value improvement and policy improvement may be subject to different considerations, and thus it is worthwhile exploring approaches that decouple them.\n* Experiments indicate the approach has merit, improving over existing baselines.\n\n### Weaknesses\n* It is ultimately unclear to me where the performance gain really comes from. Is there really an interaction of the two improvement operators, or is this algorithm more of an online version of IQL? See comments 1+2 below.\n* Much of the paper is dedicated to introduction of the theory, that ultimately does not reveal much other than sort of a sanity check with very limited scope. (While I consider this beneficial to some extent, it is currently occupying quite a bit of real estate considering its added value.)\n\nComments:\n1. I was left uncertain as to whether the generalized policy iteration is the right framework to study the VIAC algorithm in its implicit value improvement flavor. Is the greedification of the acting policy important in any way, or is the implicit value improvement really doing all the work, in which case the algorithm is question is an online (offpolicy) version of IQL? In particular, perhaps the experiments involving implicit value improvement (i.e., Figure 3), would benefit from another IQL-like baseline, where the improvement of the acting policy is ablated in some way.\n2. The presentation of Figure 1 with x axis being the environment steps is questionable, given that the compute would be the right reference to use ofor the comparison.", "was_revised": true, "final_justification": "Following the authors detailed rebuttal and discussions that followed, most of my concerns have been addressed. In particular, the authors have convinced me the framework through which they study VIAC makes sense, and that the performance gains reported in the paper originate from the combination of the two (value / acting policy) improvement mechanisms.\nAs a result, I choose to raise my rating 4 --> 5, favoring clear acceptance.", "rebuttal": "Dear reviewer, \n\nThank you for your valuable time, detailed review and many thoughtful comments and questions.\n\n**1. It is ultimately unclear to me where the performance gain really comes from. Is the greedification of the acting policy important in any way, or is the implicit value improvement really doing all the work, in which case the algorithm is question is an online (offpolicy) version of IQL?**\n\n*TLDR: experiments with other value-improvement operators show that signif. performance gains from VI are not unique to IQL. We conclude that VIAC is not simply online IQL.*\n\nBest-of-n (BON) is an explicit value improvement operator which samples N actions and takes the value of the maximizing action as the bootstrap for the value target.\nBON is a popular operator in RL for language models [1].\nWe include a table that summarizes the final performance for baseline TD3, and the IQL and BON variations of VI-TD3, where BON shows similar performance gains to IQL across most domains.\nFor BON, we use $N=64$, and for IQL we use the same $\\tau=0.75$.\nNormalized per environment between minimum and maximum final return over all seeds (20 for TD3 and VI-TD3 (IQL), and 12 for VI-TD3 (BON)), averaged per agent over environments, with +- one SEM.\nThe IQL and BON variations are within each others 2 SEM confidence bound.\n\n|Agent|Normalized Final Perf.|\n|-|-|\n|Baseline TD3|0.41 ± 0.05|\n|VI-TD3 (IQL)|0.75 ± 0.04|\n|VI-TD3 (BON)|0.68 ± 0.04|\n\nBelow are the unnormalized final-performance results per environment along with 1 SEM which show the similarities in performance gains across the majority of environments in more detail.\n\n|Environment|Final perf. TD3|Final perf. VI-TD3 (expectile)|Final perf. VI-TD3 (BON)|\n|-|-|-|-|\n|hopper-stand|670.9 ± 87.5|**936.4** ± 5.1|**900.4** ± 24.4|\n|hopper-hop|135.8 ± 29.6|**377.6** ± 34.0|250.2 ± 38.9|\n|humanoid-stand|26.4 ± 17.8|666.6 ± 15.2|**723.1** ± 24.9|\n|humanoid-walk|38.8 ± 34.5|**540.5** ± 12.5|**560.5** ± 52.4|\n|humanoid-run|0.9 ± 0.1|**172.4** ± 6.3|**180.7** ± 7.3|\n|fish-swim|489.9 ± 74.6|**753.2** ± 21.6|492.9 ± 60.1|\n|acrobot-swingup|9.4 ± 3.3|**317.2** ± 36.6|5.4 ± 4.3|\n|quadruped-walk|706.9 ± 86.1|**923**.6 ± 9.7|**943.1** ± 5.3|\n|quadruped-run|**764.3** ± 51.6|**838.6** ± 19.5|**863.1** ± 30.3|\n|walker-walk|**971.0** ± 2.0|**966.3** ± 3.6|**974.9** ± 2.7|\n|walker-run|792.4 ± 16.0|**852.4** ± 10.5|**833.8** ± 5.0|\n|cheetah-run|**897.3** ± 4.4|**831.3** ± 24.5|**848.9** ± 27.4|\n\nWe will include these results in the appendix.\n\n**2. Much of the paper is dedicated to introduction of the theory, that ultimately does not reveal much other than sort of a sanity check with very limited scope.**\n\nCombining value improvement with policy improvement is already done in practice (see lines 326 to 336), albeit generally without this perspective in mind. \nFor that reason, we think that the sanity check is important - if this approach is not generally sound we should not or should be careful when we use it. \nThe choice of very abstract, very general scope of GPI allows us to establish Theorem 2, policy improvement is not enough, and the nec. / suff. conditions of Definitions 3-5, under the most abstract conditions, that are not limited to function approximation or even to RL.\nAnalyzing VIAC in this scope allows us to further establish that this novel algorithms approach to RL/DP is sound under the most general conditions (excepting for finite horizon).\nWe believe that it is an important first step towards establishing the theoretical understanding that underpins this general perspective which diverges from the standard RL iteration of policy improvement / policy evaluation.\n\n**C1. I was left uncertain as to whether the generalized policy iteration is the right framework to study the VIAC algorithm in its implicit value improvement flavor.**\n\n*TLDR: We conclude that: 1. implicit improvement is within VIAC (including specifically the way we use IQL), and 2. that the value and policy improvement compliment each other, and unless one of them is explicitly the argmax, they do not render each other obselete.*\n\nIf we understand correctly (please correct us otherwise), the concern is that the improvement in the value is doing all the work, and the improvement in the policy becomes unnecessary / unimportant.\n\nWe first note that if the greedy operator is used the VI-GPI collapses to Value Iteration / Policy Iteration (which are the same algorithm, from the perspective of GPI). In which case, the value improvement and the policy improvemnt coincide.\nHowever, as long as both operators take $\\pi$ into account (for example, operators like $I_{GMZ}$ or policy gradient), every improvement step depends strongly on $\\pi$. If $I(\\pi, Q)$ does not diverge far from $\\pi$ the value-improved value function does evaluate policies that are very different from $\\pi$.\n\nIn fact, in a very simple set of experiments with dynamic programming and VI-GPI, discrete actions, and the same operators for both value and policy improvement ($I_{GMZ}$), we found that the improvement to the policy is the dominating effect.\nIf this experiment sounds interesting despite its simplicity, please let us know and we will include it.\n\nSpecifically in relation to the implicit improvement of the expectile - note that we use it with respect to freshly sampled actions from the acting policy $\\pi_\\theta$, not with respect to actions stored in the replay buffer. [2] prove that this evaluates an explicitly greedier policy with respect to $\\pi_\\theta$, and [3] show that this policy is real and can be extracted.\n\nPlease let us know if the comment was not understood correctly, and this does not address / answer the concern the reviewer has in mind.\n\n**C2. The presentation of Figure 1 with x axis being the environment steps is questionable, given that the compute would be the right reference to use for the comparison.**\n\nThanks for raising this point.\nThe objective of Figure 1 is to show that when stripping other effects (such as IQL as the additional operator), there remains an advantage in sample efficiency from the greedification of the evaluated policy. \nTo answer this question, we cannot put compute cost on the x axis.\nWe agree indeed that the agents in Figure 1 result in additional compute cost (which we also mention following the figure, line 299).\nWe will make sure the objective of the experiment is made more clear.\n\n**Q1. Typo:** Thanks, good catch! Corrected to \"the sets are also not disjoint\".\n\n**Q2. Is $I_{GMZ}$ a generalization of soft policy iteration?**\n\nInterestingly, we believe the answer is no: In soft (max. entropy) policy iteration (/SAC), we aim to get $\\pi_n = softmax (q_n)$.\nThat is in fact not an improvement operator outside of soft-RL (an easy proof is that $softmax (q) \\neq argmax (q)$, and we know that any optimal policy is an argmax policy).\nIn contrast, when we use $I_{GMZ}$ we aim to get $\\pi_{n+1} = softmax (q_n + \\log \\pi_n)$.\nInterestingly, this *is* a policy improvement operator [4]. \nIn fact, this is also a limit-sufficient policy improvement operator (because $\\pi_\\theta$ will converge to an argmax policy that is uniform with respect to all maximizing actions, see Lemma 5 in Appendix A.6 in our paper).\n\n**Q3. non-detriment is not clear:** \n\nThank you for pointing it out. We mean any operator that satisfies at least (in)equality 4. \nNote that this is not the same as greedification, which requires satisfying also inequality 5 (Which is the difference between the two). \nWe will make sure this is made clear in the text.\n\n**Q4. Lines 282-283 are not clear**: \n\nThank you for pointing that out.\nThe first question aims to establish that the effect is there:\nthat the greedification of the evaluated policy can by itself increase sample efficiency.\n\nThe second question aims to establish that the effect justifies the cost:\nthat it is also possible to see significant performance gains without significant increases in implementation complexity / compute.\n\nWe will make sure to make it clear in the text.\n\n**Q5. It is unclear how you map the greedification operators in Alg2 to those of Alg3. When you increase the $n$ of $pg=n$, do you increase it for both the evaluated policy and acting policy?**\n\nThanks for pointing that out. \nThe increases are only to the evaluated policy (line 291). \nWe will make sure that this is made more clear in the text. \nWe investigate the effects of increases to the acting policy in the Appendix in Figure 6 (which is detrimental for PI while beneficial for VI).\n\n*Please let us know if there remain any unaddressed concerns, comments or questions.*\n\n[1] Huang, Audrey, et al. \"Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment.\" arXiv preprint arXiv:2503.21878 (2025).\n\n[2] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline reinforcement learning with implicit q-learning.\" arXiv preprint arXiv:2110.06169 (2021).\n\n[3] Hansen-Estruch, Philippe, et al. \"Idql: Implicit q-learning as an actor-critic method with diffusion policies.\" arXiv preprint arXiv:2304.10573 (2023).\n\n[4] Danihelka, Ivo, et al. \"Policy improvement by planning with Gumbel.\" International Conference on Learning Representations. 2022."}, {"review_id": "ImEu9VaxUT", "reviewer": "Reviewer_SmgB", "summary": "This paper introduces Value Improved Actor-Critic, a general technique for AC algorithms to balance the greediness and stability during training. The key idea behind is to follow the value improvement -> policy improvement paradigm. Moreover, the authors choose to apply slower policy improvement for the actual policy for stability reason. The method is experimented on both TD-3 and SAC on dm-control benchmark, verifying that VIAC can consistently improves the performance and has the capability to adjust the tradeoff via simple hyper-parameter tuning.", "questions": "- Is there any other greedy operators that also works well and where is the experiment for Gumbel MuZero operator? Have the author explored this direction?", "limitations": "I didn't find any place the author discussed the limitation.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n- Well-motivated to solve the trade-off between greediness and stability.\n- Has strong theoretical foundation.\n- Proposes explicit and implicit methods, and well addresses the computational overhead for repetitive gradient update. \n- Generality is well-supported by both deterministic and stochastic policies. \n\nWeakness:\n- The presentation has a lot of room to be improved. Too much space is spent for the justification of generalized policy iteration. The authors should clearly explain the motivation and connection to AC algorithms at first, otherwise it is hard to follow.\n- The authors claim that the slower policy improvement brings stability, which I believe is also a key difference between the method and other off-policy algorithms that improve UTD ratios. However, there is no ablations to show the effectiveness of this design choice. \n- There is no proofs of convergence or theoretical guarantees of improvement over standard actor-critic for both deterministic and stochastic policies. Compared to the dense proofs for Section 3, this looks weird. \n- For experiments in dm-control suite, the highest rewards the method achieved are still far from sota performance. For example, for humanoid stand and run, sota algorithms can reach 1000~ and 600~ [1,2], which are almost doubled compared to the proposed method. I note this is a minor issue. \n\n[1] Nauman, Michal, et al. \"Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control.\" arXiv preprint arXiv:2405.16158 (2024).\n\n[2] Celik, Onur, et al. \"Dime: Diffusion-based maximum entropy reinforcement learning.\" arXiv preprint arXiv:2502.02316 (2025).", "was_revised": true, "final_justification": "The authors have clarified the policy stability claim and provided a detailed explanation on potential operators with a comprehensive evaluation.", "rebuttal": "Dear reviewer, \n\nThank you for your valuable time and clear comments and questions.\n\n**1. The presentation has a lot of room to be improved. Too much space is spent for the justification of generalized policy iteration. The authors should clearly explain the motivation and connection to AC algorithms at first, otherwise it is hard to follow.**\n\nThanks for pointing that out, we will make sure that motivation and connection to ACs is made earlier and clearer.\n\n**2. The authors claim that the slower policy improvement brings stability, which we believe is also a key difference between the method and other off-policy algorithms that improve UTD ratios. However, there is no ablations to show the effectiveness of this design choice.**\n\nThe claim that the slower policy improvement can bring stability is motivated by prior work: \n[6] explicitly delay the policy updates for learning stability. [7] and [8] explicitly limit the maximum change allowed to the policy before each new data-gathering step.\n[9] explicitly refers to and addresses issues that rise from too-fast changes to the policy.\n\nIn regards specifically to VIAC, Figure 7 demonstrates that increased UTD destabilizes the learning process much earlier than the increased greedification of the evaluated policy, equated over the same budget of gradient steps.\n\n**3. There is no proofs of convergence or theoretical guarantees of improvement over standard actor-critic for both deterministic and stochastic policies. Compared to the dense proofs for Section 3, this looks weird.**\n\n*In regards to convergence:* \nUsing DP to analyze convergence properties of AC algorithms is rather standard [1, 2].\nThe main advantage of this analysis, from our perspective, is in its generality: the results are not limited to specific improvement operators, gradient based updates, presence of local optima, etc. and thus can be used to underly the convergence of algorithms that go beyong AC, such as MuZero [10].\n\n*In regards to improvement over standard actor-critics (or even policy iteration):*\nUnder strict-greedification ($>$) assumptions, we would expect that it is possible to show that value-improved GPI can outperform GPI for specific combinations of operators. \nSince a small number of recent algorithms already fall within the framework of VIAC in practice, albeit without the perspective of the framework (see lines 323 to 336 in the related work), we viewed proving soundness of this framework as more important than proving the possibility of benefit (which from that perspective, has already been observed).\nHowever, we agree that this result is important as well and we hope it will be addressed in future work.\n\n**4. For experiments in dm-control suite, the highest rewards the method achieved are still far from sota performance. For example, for humanoid stand and run, sota algorithms can reach 1000~ and 600~ [1,2], which are almost doubled compared to the proposed method. I note this is a minor issue.**\n\nBRO [3] specifically focuses on scaling up the DNN architectures and pairing the agent with dedicated optimism-based exploration, while Dime [4] explicitly changes the policy class and DNN architectures. In contrast, the baseline TD3 and SAC over which we investigate the effect of VI do not use any of these advancements. However, the results in Figure 5 in the appendix (VI-TD7) show that the improvements of VIAC are not limited to the older architectures of TD3 and SAC, and indeed VI-TD7 results in better performance across several environments compared to VI-TD3 / VI-SAC on both humanoid-stand and humanoid-run, suggesting that the improvement from VIAC is orthogonal to improvement from DNN architectures.\n\n**Q1. Is there any other greedy operators that also works well and where is the experiment for Gumbel MuZero operator? Have the author explored this direction?**\n\n*TLDR: yes, and they have also showed significant performance gains in this domain.*\n\nWe have tried several explicit greedification operators, in addition to PG and the implicit IQL. Overall, all resulted in performance gains in this domain. \nWe include best of N (BON) as an example: \nBON samples $N$ actions and takes the value of the maximizing action as the bootstrap for the value target, and is popular in RL for language models [5].\nThe main advantage of IQL in comparison to the explicit operators is not only in its minimal computational and implementation cost but also that the greedification can be tuned with $\\tau$ without increasing cost. In contrast, in operators such as BON, increasing greedification translates to increasing $N$, where the compute cost of the operator also increases with $N$.\n\nIn the experiments reported below, BON shows similar performance gains to IQL across most domains.\n\nWe first report final-performance results normalized per environment between minimum and maximum final return over all seeds (20 for TD3 and VI-TD3 (IQL), and 12 for VI-TD3 (BON)), averaged per agent over environments, with +- one SEM.\nFor BON, we use $N=64$, and for IQL we use the same $\\tau=0.75$.\nThe IQL and BON variations are within each others 2 SEM confidence bound.\n\n|Agent|Normalized Final Perf.|\n|-|-|\n|Baseline TD3|0.41 ± 0.05|\n|VI-TD3 (IQL)|0.75 ± 0.04|\n|VI-TD3 (BON)|0.68 ± 0.04|\n\nBelow are the unnormalized final-performance results per environment along with 1 SEM which show the similarities in performance gains across the majority of environments in more detail.\n\n|Environment|Final perf. TD3|Final perf. VI-TD3 (expectile)|Final perf. VI-TD3 (BON)|\n|-|-|-|-|\n|hopper-stand|670.9 ± 87.5|**936.4** ± 5.1|**900.4** ± 24.4|\n|hopper-hop|135.8 ± 29.6|**377.6** ± 34.0|250.2 ± 38.9|\n|humanoid-stand|26.4 ± 17.8|666.6 ± 15.2|**723.1** ± 24.9|\n|humanoid-walk|38.8 ± 34.5|**540.5** ± 12.5|**560.5** ± 52.4|\n|humanoid-run|0.9 ± 0.1|**172.4** ± 6.3|**180.7** ± 7.3|\n|fish-swim|489.9 ± 74.6|**753.2** ± 21.6|492.9 ± 60.1|\n|acrobot-swingup|9.4 ± 3.3|**317.2** ± 36.6|5.4 ± 4.3|\n|quadruped-walk|706.9 ± 86.1|**923**.6 ± 9.7|**943.1** ± 5.3|\n|quadruped-run|**764.3** ± 51.6|**838.6** ± 19.5|**863.1** ± 30.3|\n|walker-walk|**971.0** ± 2.0|**966.3** ± 3.6|**974.9** ± 2.7|\n|walker-run|792.4 ± 16.0|**852.4** ± 10.5|**833.8** ± 5.0|\n|cheetah-run|**897.3** ± 4.4|**831.3** ± 24.5|**848.9** ± 27.4|\n\nWe will include these results in the appendix.\n\n**I didn't find any place the author discussed the limitation.**\n\nThe main limitation we observe for our contribution is the limitations of the theoretical results (finite horizon MDPs, DP, etc) which we mention per theorem. We will make sure this is made more clear and explicit in the text.\n\n*Please let us know if there remain any unaddressed concerns, comments or questions.*\n\n[1] Tsitsiklis, John N. \"On the convergence of optimistic policy iteration.\" Journal of Machine Learning Research 3.Jul (2002): 59-72.\n\n[2] Smirnova, Elena, and Elvis Dohmatob. \"On the convergence of approximate and regularized policy iteration schemes.\" arXiv preprint arXiv:1909.09621 (2019).\n\n[3] Nauman, Michal, et al. \"Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control.\" arXiv preprint arXiv:2405.16158 (2024).\n\n[4] Celik, Onur, et al. \"Dime: Diffusion-based maximum entropy reinforcement learning.\" arXiv preprint arXiv:2502.02316 (2025).\n\n[5] Huang, Audrey, et al. \"Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment.\" arXiv preprint arXiv:2503.21878 (2025).\n\n[6] Fujimoto, Scott, Herke Hoof, and David Meger. \"Addressing function approximation error in actor-critic methods.\" International conference on machine learning. PMLR, 2018.\n\n[7] Schulman, John, et al. \"Trust region policy optimization.\" International conference on machine learning. PMLR, 2015.\n\n[8] Schulman, John, et al. \"Proximal policy optimization algorithms.\" arXiv preprint arXiv:1707.06347 (2017).\n\n[9] Capone, Cristiano, and Paolo Muratore. \"Learning fast while changing slow in spiking neural networks.\" Neuromorphic Computing and Engineering 4.3 (2024): 034002.\n\n[10] Danihelka, Ivo, et al. \"Policy improvement by planning with Gumbel.\" International Conference on Learning Representations. 2022."}, {"review_id": "6jhrRoHfnF", "reviewer": "Reviewer_y6Pz", "summary": "This paper studies Value Improved Actor Critic (VAIC) to address the tradeoff between fast improvement (greedification) and stability. The authors propose a novel framework that decouples the acting policy from the evaluated policy in actor-critic. This allows for greedier value improvements to the critic, while keeping stable updates for the actor. Empirically, the proposed VAIC variants of TD3 and SAC outperform or match the original methods across several DeepMind control suite tasks.", "questions": "In explicit VI-TD3, does the extra greediness (e.g., more PG steps) lead to an increase in total compute/wall-clock time, even if environment step efficiency is improved?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n+\tTheoretical contribution about convergence in actor critic is new and insightful, AFAIK.  Has necessary and sufficient conditions to guarantee that policy iteration converges to optimal policy. Shows that Policy improvement is not a sufficient condition for the convergence of Policy Iteration. \n\n+\tPractical algorithmic suggestion about decoupled acting and evaluated policy.\n\n+\tConcrete implementation with explicit and implicit greedification strategies, with associated positive empirical results on VAIC for TD3 and SAC. \n\n+\tGenerally clearly written paper and insightful ablation and analysis experiments. (Analysis around Fig4-7 are all appreciated).\n\nWeaknesses:\n\n-\tAnalysis seems to be limited to finite-horizon MDPs. \n\n-\tLimited empirical study on stability-greediness tradeoff. This is a keystone of the paper’s narrative, but there is only limited empirical analysis of this.  \n\n-\tEmpirical validation limited to  low-dimensional continuous control, not discrete action or high-dimensional inputs. \n\n-\tIntroduces a new greedification hyperparameter to tune in RL.", "was_revised": true, "final_justification": "I do not have deep expertise in this topic, so please defer to any more expert reviewer than me. \nHowever, from reading all the reviews and author discussion, I don't see any fatal flaws, and there seem to be several good theoretical insights and practical algorithmic contributions coming out.\nSo I recommend accept.", "rebuttal": "Dear reviewer, \n\nThank you for your valuable time, thoughtful review and accurate comments.\n\n**2. Limited empirical study on stability-greediness tradeoff. This is a keystone of the paper’s narrative, but there is only limited empirical analysis of this.**\n\nWe agree with the reviewer that the tradeoff can be investigated further. \nWe note that aspects of it have been investigated in prior work (for example [1], beyond the works already cited in the paper), and that different faces of it are investigated in Figures 1, 2, 4, 6 and 7.\n\n**4. Introduces a new greedification hyperparameter to tune in RL.**\n\nOne could say that the tradeoff was there before VIAC, and with it being there, there was a possibility to tune it.\nHowever, we are of course in agreement with the reviewer that the introduction of any additional hyperparameter is unfortunate.\nWe will include in the appendix a figure like Figure 4, extended for all environments, that sheds more light on the effects of different $\\tau$.\n\n**Q1: In explicit VI-TD3, does the extra greediness (e.g., more PG steps) lead to an increase in total compute/wall-clock time, even if environment step efficiency is improved?**\n\n*TLDR: Yes, absolutely. However, it is possible to get signficant (stronger, in fact) performance improvement with much smaller compute cost than PG with other explicit greedification operators. Results for which are included below.*\n\nYes indeed, increasing PG steps comes with an increase in compute cost, which is important to mention (included in line 299). However, it is also possible to see significant performance improvement with explicit greedification of the evaluated policy without insignificant compute increase:\n\nA simple example of a value improvement operator that is inexpensive and shows strong performance benefits in these domains is best-of-n (BON): sample N actions, take the value of the maximizing action as the bootstrap for the value target.\nThis is a popular operator in RL for language models [2].\nWe include a table that summarizes final performance for VI-TD3 with this operator across the same domains as in Figure 3.\nWe include baseline TD3, VI-TD3 with the expectile operator ($\\tau=0.75$) and VI-TD3 with BON ($N=64$).\n\nBON shows similar performance gains to IQL across most domains.\n\nBelow we report final-performance results normalized per environment between minimum and maximum final return over all seeds (20 for TD3 and VI-TD3 (IQL), and 12 for VI-TD3 (BON)), averaged per agent over environments, with +- one SEM.\nThe IQL and BON variations are within each others 2 SEM confidence bound.\n\n|Agent|Normalized Final Perf.|\n|-|-|\n|Baseline TD3|0.41 ± 0.05|\n|VI-TD3 (IQL)|0.75 ± 0.04|\n|VI-TD3 (BON)|0.68 ± 0.04|\n\nBelow are the unnormalized final-performance results per environment along with 1 SEM which show the similarities in performance gains across the majority of environments in more detail.\n\n|Environment|Final perf. TD3|Final perf. VI-TD3 (expectile)|Final perf. VI-TD3 (BON)|\n|-|-|-|-|\n|hopper-stand|670.9 ± 87.5|**936.4** ± 5.1|**900.4** ± 24.4|\n|hopper-hop|135.8 ± 29.6|**377.6** ± 34.0|250.2 ± 38.9|\n|humanoid-stand|26.4 ± 17.8|666.6 ± 15.2|**723.1** ± 24.9|\n|humanoid-walk|38.8 ± 34.5|**540.5** ± 12.5|**560.5** ± 52.4|\n|humanoid-run|0.9 ± 0.1|**172.4** ± 6.3|**180.7** ± 7.3|\n|fish-swim|489.9 ± 74.6|**753.2** ± 21.6|492.9 ± 60.1|\n|acrobot-swingup|9.4 ± 3.3|**317.2** ± 36.6|5.4 ± 4.3|\n|quadruped-walk|706.9 ± 86.1|**923**.6 ± 9.7|**943.1** ± 5.3|\n|quadruped-run|**764.3** ± 51.6|**838.6** ± 19.5|**863.1** ± 30.3|\n|walker-walk|**971.0** ± 2.0|**966.3** ± 3.6|**974.9** ± 2.7|\n|walker-run|792.4 ± 16.0|**852.4** ± 10.5|**833.8** ± 5.0|\n|cheetah-run|**897.3** ± 4.4|**831.3** ± 24.5|**848.9** ± 27.4|\n\n[1] Thrun, Sebastian, and Anton Schwartz. \"Issues in using function approximation for reinforcement learning.\" Proceedings of the 1993 connectionist models summer school. Psychology Press, 2014.\n\n[2] Huang, Audrey, et al. \"Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment.\" arXiv preprint arXiv:2503.21878 (2025)."}]

Author rebuttal:

Meta-review: This paper introduces Value-Improved Actor-Critic (VIAC), a general framework that decouples the acting policy from the evaluated policy in actor-critic algorithms. The central idea is to apply a greedier operator when evaluating the policy (value improvement), while maintaining the slower, gradient-based updates for the acting policy to ensure stability. This separation directly addresses the long-standing tradeoff between greedification and stability in actor-critic methods. The framework is grounded in generalized policy iteration, with convergence guarantees in finite-horizon settings. Empirical results show that incorporating VIAC into TD3 and SAC leads to consistent improvements across multiple continuous control benchmarks, with negligible additional computational cost.

The reviewers raised thoughtful questions about whether the implicit version (based on expectile loss) truly implements value improvement, or whether its gains stem from estimation bias. The authors’ rebuttal addressed this by demonstrating that improvements are not solely due to over/underestimation (e.g., performance gains occur even without bias change) and that value and policy improvement complement each other rather than rendering one another redundant. While some concerns remain about the simple discrete state-action space setting, limited scope of experiments (e.g., sparse reward, high-dimensional inputs) and the practicality of the explicit variant, all reviewers agree that the technical contribution is novel, well-motivated, and supported by both theoretical analysis and empirical evidence.
Decision: Accept (poster)