Title: Value Improved Actor Critic Algorithms
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

actor critic

policy improvement operators
Abstract: To learn approximately optimal acting policies for decision problems, modern Actor Critic algorithms rely on deep Neural Networks (DNNs) to parameterize the acting policy and greedification operators to iteratively improve it. 
The reliance on DNNs suggests an improvement that is gradient based, which is per step much less greedy than the improvement possible by greedier operators such as the greedy update used by Q-learning algorithms.
On the other hand, slow and steady changes to the policy can also be beneficial for the stability of the learning process, resulting in a tradeoff between greedification and stability.
To address this tradeoff, we propose to extend the standard framework of actor critic algorithms with value-improvement: a second greedification operator applied only when updating the policy's value estimate.
In this framework the agent can evaluate non-parameterized policies and perform much greedier updates while maintaining the steady gradient-based improvement to the parameterized acting policy.
We prove that this approach converges in the popular analysis scheme of generalized Policy Iteration in the finite-horizon domain.
Empirically, incorporating value-improvement into the popular off-policy actor-critic algorithms TD3 and SAC significantly improves or matches performance over their respective baselines, across different environments from the DeepMind continuous control domain, with negligible compute and implementation cost.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies Value Improved Actor Critic (VAIC) to address the tradeoff between fast improvement (greedification) and stability. The authors propose a novel framework that decouples the acting policy from the evaluated policy in actor-critic. This allows for greedier value improvements to the critic, while keeping stable updates for the actor. Empirically, the proposed VAIC variants of TD3 and SAC outperform or match the original methods across several DeepMind control suite tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n+\tTheoretical contribution about convergence in actor critic is new and insightful, AFAIK.  Has necessary and sufficient conditions to guarantee that policy iteration converges to optimal policy. Shows that Policy improvement is not a sufficient condition for the convergence of Policy Iteration. \n\n+\tPractical algorithmic suggestion about decoupled acting and evaluated policy.\n\n+\tConcrete implementation with explicit and implicit greedification strategies, with associated positive empirical results on VAIC for TD3 and SAC. \n\n+\tGenerally clearly written paper and insightful ablation and analysis experiments. (Analysis around Fig4-7 are all appreciated).\n\nWeaknesses:\n\n-\tAnalysis seems to be limited to finite-horizon MDPs. \n\n-\tLimited empirical study on stability-greediness tradeoff. This is a keystone of the paper’s narrative, but there is only limited empirical analysis of this.  \n\n-\tEmpirical validation limited to  low-dimensional continuous control, not discrete action or high-dimensional inputs. \n\n-\tIntroduces a new greedification hyperparameter to tune in RL.\n\nQUESTIONS:\nIn explicit VI-TD3, does the extra greediness (e.g., more PG steps) lead to an increase in total compute/wall-clock time, even if environment step efficiency is improved?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Value Improved Actor-Critic, a general technique for AC algorithms to balance the greediness and stability during training. The key idea behind is to follow the value improvement -> policy improvement paradigm. Moreover, the authors choose to apply slower policy improvement for the actual policy for stability reason. The method is experimented on both TD-3 and SAC on dm-control benchmark, verifying that VIAC can consistently improves the performance and has the capability to adjust the tradeoff via simple hyper-parameter tuning.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- Well-motivated to solve the trade-off between greediness and stability.\n- Has strong theoretical foundation.\n- Proposes explicit and implicit methods, and well addresses the computational overhead for repetitive gradient update. \n- Generality is well-supported by both deterministic and stochastic policies. \n\nWeakness:\n- The presentation has a lot of room to be improved. Too much space is spent for the justification of generalized policy iteration. The authors should clearly explain the motivation and connection to AC algorithms at first, otherwise it is hard to follow.\n- The authors claim that the slower policy improvement brings stability, which I believe is also a key difference between the method and other off-policy algorithms that improve UTD ratios. However, there is no ablations to show the effectiveness of this design choice. \n- There is no proofs of convergence or theoretical guarantees of improvement over standard actor-critic for both deterministic and stochastic policies. Compared to the dense proofs for Section 3, this looks weird. \n- For experiments in dm-control suite, the highest rewards the method achieved are still far from sota performance. For example, for humanoid stand and run, sota algorithms can reach 1000~ and 600~ [1,2], which are almost doubled compared to the proposed method. I note this is a minor issue. \n\n[1] Nauman, Michal, et al. \"Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control.\" arXiv preprint arXiv:2405.16158 (2024).\n\n[2] Celik, Onur, et al. \"Dime: Diffusion-based maximum entropy reinforcement learning.\" arXiv preprint arXiv:2502.02316 (2025).\n\nQUESTIONS:\n- Is there any other greedy operators that also works well and where is the experiment for Gumbel MuZero operator? Have the author explored this direction?\n\nLIMITATIONS:\nI didn't find any place the author discussed the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a class of actor critic algorithms that separates between an \"evaluated policy\" and the \"acting policy\". The evaluated policy is induced by a greedification operator that is separate from the greedification operator of the acting policy.\nAs a result, the policy evaluation phase effectively generates additional \"value improvement\" by means of evaluating an improved policy.\nThese algorithms are called Value Improved Actor Critic Algorithms (VIAC).\n\nThe core idea is that value improvement can be controlled separately from the the greedification of the acting policy, which may be constrained to support greedification operators that are more limited than those of the value improvement.\n\nThe paper studies this class of algorithms within the generalized policy iteration framework, and presents elementary convergence results. Further, experiments are carried that indicate VIAC offer clear improvement over non value improved baselines TD3, TD7, and SAC on continuous control tasks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n* The paper makes an appealing observation, that value improvement and policy improvement may be subject to different considerations, and thus it is worthwhile exploring approaches that decouple them.\n* Experiments indicate the approach has merit, improving over existing baselines.\n\n### Weaknesses\n* It is ultimately unclear to me where the performance gain really comes from. Is there really an interaction of the two improvement operators, or is this algorithm more of an online version of IQL? See comments 1+2 below.\n* Much of the paper is dedicated to introduction of the theory, that ultimately does not reveal much other than sort of a sanity check with very limited scope. (While I consider this beneficial to some extent, it is currently occupying quite a bit of real estate considering its added value.)\n\nComments:\n1. I was left uncertain as to whether the generalized policy iteration is the right framework to study the VIAC algorithm in its implicit value improvement flavor. Is the greedification of the acting policy important in any way, or is the implicit value improvement really doing all the work, in which case the algorithm is question is an online (offpolicy) version of IQL? In particular, perhaps the experiments involving implicit value improvement (i.e., Figure 3), would benefit from another IQL-like baseline, where the improvement of the acting policy is ablated in some way.\n2. The presentation of Figure 1 with x axis being the environment steps is questionable, given that the compute would be the right reference to use ofor the comparison.\n\nQUESTIONS:\n- Line 247: Typo \"the sets are not also not disjoint\"\n- Is $I_{gmz}$ just a slight generalization (by use of $\\sigma$) of the standard soft policy iteration greedification operator?\n- The notion of \"non-detriment\" operator is first introduced in the statement of Corollary 2, and it's meaning is unclear (the \"e.g.\" just points back to the definition of a greedification operator).\n- Lines 282-283: It is unclear what is the difference between questions 1 and 2.\n- It is unclear how you map the greedification operators in Alg2 to those of Alg3. When you increase the $n$ of $pg=n$, do you increase it for both the evaluated policy and acting policy?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new framework called Value-Improved Actor-Critic (VIAC), which separates the policy that interacts with the environment from the one being evaluated by the critic. The main idea is to let the critic evaluate a more \"greedy\" policy to get better value estimates, while the actor still updates slowly and stably using gradients. The authors back this up with a solid theoretical analysis—showing under what conditions this separation still leads to convergence—and test the idea both explicitly (with multiple critic-side updates) and implicitly (using expectile loss). They apply it to TD3 and SAC and find that the implicit version, in particular, gives nice performance gains with almost no extra compute.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper presents a novel and elegant way to address a long-standing tradeoff in actor-critic methods: the tension between greediness and stability.\n2. The theoretical analysis is thorough and introduces meaningful distinctions between types of policy improvement operators, offering a clear view into when convergence is (or isn’t) guaranteed. I also like the intuition for the theories.\n3. The empirical results are compelling. The authors show that integrating value improvement into TD3 and SAC leads to consistent performance gains across a range of continuous control tasks.\n4. The implicit implementation using expectile loss is particularly practical—it’s simple to implement, introduces negligible overhead, and still captures the benefits of value improvement.\n\n**Weaknesses:**\n\n1. I’m not entirely convinced that the implicit version using expectile loss fits neatly within the VIAC framework. There is an important distinction between genuinely greedy value estimation and simple overestimation—larger value estimates may stem from estimation bias rather than deliberate greedification. It remains unclear whether the observed improvements are due to a shift toward a greedier policy or simply the result of over- or underestimation. This is my primary concern, and I would appreciate a more detailed explanation from the authors on this point.\n2. While the explicit variant is mainly included to motivate the framework conceptually, its high computational cost makes it impractical for most real-world use. That wouldn’t be a problem on its own—after all, the implicit version is the one intended for practical use. However, given my concern above about whether the implicit method truly reflects value improvement in the VIAC sense, the lack of a practical, clearly greedy implementation slightly weakens the overall strength of the framework. If expectile loss isn’t truly implementing a greedy operator, then the framework may not be as broadly applicable in practice as it first appears.\n3. While some of the evaluated environments (e.g., cartpole-swingup, hopper-stand) exhibit partially sparse rewards, the benchmark suite is still relatively standard. It would be interesting to test the method in more extreme sparse-reward settings, such as long-horizon navigation or delayed-reward tasks, to further validate its robustness.\n\n\n\nOverall, I found the paper clear and well-motivated. If I misunderstood anything, I'm happy to be corrected, and I’m open to revisiting my evaluation based on the authors’ responses.\n\nQUESTIONS:\n1. Could the authors respond to the concerns raised in the weaknesses section, particularly the first point regarding whether the implicit method genuinely performs value improvement?\n\n2. Could the authors elaborate on how the mismatch between the actor and critic policies affects learning? For example, if the two policies diverge significantly, does this impact training stability or sample efficiency?\n\n3. The theoretical results are grounded in finite MDPs, but modern deep RL involves continuous state and action spaces, along with approximation error. How well do the guarantees hold up under function approximation and in the presence of non-stationary Q estimates?\n\n4. The balance between greedy updates and stable improvement somewhat reminds me of TRPO, which deliberately avoids overly greedy policy updates. Do the authors see any conceptual connection here? It would be interesting to hear their thoughts.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}