Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
Authors: Motoki Omura
Kazuki Ota
Takayuki Osa
Yusuke Mukuta
Tatsuya Harada
Venue: ICML 2025 poster (2025)
Keywords: online reinforcement learning
q-learning
bellman operator
Primary area: reinforcement_learning->deep_rl

Abstract: For continuous action spaces, actor-critic methods are widely used in online reinforcement learning (RL). However, unlike RL algorithms for discrete actions, which generally model the optimal value function using the Bellman optimality operator, RL algorithms for continuous actions typically model Q-values for the current policy using the Bellman operator. These algorithms for continuous actions rely exclusively on policy updates for improvement, which often results in low sample efficiency. This study examines the effectiveness of incorporating the Bellman optimality operator into actor-critic frameworks. Experiments in a simple environment show that modeling optimal values accelerates learning but leads to overestimation bias. To address this, we propose an annealing approach that gradually transitions from the Bellman optimality operator to the Bellman operator, thereby accelerating learning while mitigating bias. Our method, combined with TD3 and SAC, significantly outperforms existing approaches across various locomotion and manipulation tasks, demonstrating improved performance and robustness to hyperparameters related to optimality. The code for this study is available at https://github.com/motokiomura/annealed-q-learning.

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
Motoki Omura' Kazuki Ota
Takayuki Osa
Yusuke Mukuta 12 Tatsuya Harada
1
Abstract
used for discrete action tasks, estimate the optimal Q-value via the Bellman optimality operator. However, in tasks with continuous action spaces, computing max," Q(s', a') for an infinite number actions is challenging. Actor-critic- based algorithms address this by estimating the Q-value for the current policy using the Bellman operator. these cases, policy improvement is achieved solely through policy updates, leading to slower performance improvement and reduced sample efficiency (Ji et al., 2024). In tasks with continuous action spaces, such as robotic control, sample collection is costly, which makes low sample efficiency a critical challenge. Although some prior studies have ex- plored modeling (soft) optimal values instead of the values of the current policy, they faced challenges such increased computational cost and instability (Haarnoja et al., 2017; Garg et al., 2023; Kalashnikov et al., 2018). 18).
For continuous action spaces, actor-critic methods are widely used in online reinforcement learning (RL). However, unlike RI. algorithms for discrete actions, which generally model the optimal value function using the Bellman optimality operator, RL algorithms for continuous actions typically model Q-values for the current policy using the Bellman operator. These algorithms for continu- ous actions rely exclusively on policy updates for improvement, which often results in low sample efficiency. This study examines the effectiveness of incorporating the Bellman optimality operator into actor-critic frameworks. Experiments in a simple environment show that modeling optimal values accelerates learning but leads to overesti- mation bias. To address this, we propose an an- nealing approach that gradually transitions from the Bellman optimality operator the Bellman operator, thereby accelerating learning while mit- igating bias. Our method, combined with TD3 and SAC, significantly outperforms existing ap- proaches across various locomotion and manipula- tion tasks, demonstrating improved performance and robustness to hyperparameters related to op- timality. The code for this study is available at https:: //gttubb www/moto///oooooooooooooooo htt////tooooaaaaaaaaamaam annealed-g-learning
This study first examines the effectiveness of modeling opti- mal values in actor-critic methods for online RL. Prelimi- nary experiments were conducted with a low-dimensional environment using tabular actor-critic method. These ex- periments compared SARSA-based updates, which used the Bellman operator, to Q-learning- based updates, which used the Bellman optimality operator, for critic learning. The results showed that in Q-lcarning- Q-value directly approached the optimal value, whereas in SARSA-based updates, the Q-value improved only after the policy improved. This delay in policy learning caused the Q-value to converge its optimal value more slowly in the SARSA-based approach.
methods, the
This finding motivates the application of the Bellman op- timality operator in actor-critic methods. However, using the Bellman optimality operator with function approxima- tors to model Q-values introduces overestimation bias due to the stochasticity of Q-values and the max operator in target values (Thrun & Schwartz, 1993; Lan et al., 2020; Chen et al., 2021). Simulating this stochasticity by injecting noise into Q-values in our preliminary experiments showed that Q-learming-baie based methods converged to values larger than the optimal Q-value due to overestimation bias. Con- versely, SARSA-based methods, which do not involve a max operator, exhibited lower sensitivity noise.
1. Introduction
In recent years, online reinforcement learning (RL) has been widely applied to robotics (Schulman et al., 2015; Haarnoja et al., 2018) and gaming (Mnih et al., 2013; 2015; Badia et al., 2020; Tessler et al., 2017), demonstrating remark- able performance, particularly in tasks with discrete action spaces like games. Q-learning-based algorithms, commonly
'The University of Tokyo PRIKEN. Correspondence to: Motoki Omura <omura@ wi...uttokoo.ac.ppp
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In summary, while the Bellman optimality operator acceler-

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
ates learning, it also increases overestimation bias, leading to convergence inflated Q-values. Conversely, the Bell- man operator suppresses overestimation bias but results in slower learning.
faster learning with reduced bias. This method can be easily implemented in existing actor-critic methods using the expectile loss.
We demonstrated that Annealed Q-learning combined with TD3 and SAC achieves significantly better per- formance than widely used algorithms in continuous action tasks, while annealing improves performance and robustness to optimality-relatee hyperparameters.
To address this, we propose a gradual transition from the Bellman optimality operator to the Bellman operator. This approach accelerates learning while eventually estimating values with reduced bias. In the preliminary experiments, linearly interpolating target values calculated with each op- erator, combined with annealing the degree of optimality, resulted in faster learning and less biased estimation.
2. Preliminaries
This method has additional benefits. While overestimation bias may occur in the early stages of learning depending on the annealing schedule, it can promote exploration, which advantageous at this stage (Lan et al., 2020). In the later stages of learning, the current policy is expected converge, making it undesirable to introduce bias for further policy improvement. Therefore, applying updates based on the Bellman operator, which inherently reduces bias, becomes a reasonable approach.
2.1. Reinforcement Learning
In RL, the problem is defined within Markov deci- sion process (MDP) framework presented by the tuple (S, A. P. r. 7, d). Here. S denotes the set of all pos- sible states, A denotes the set of all possible actions, P(s at) is the transition probability from one state to another given specific action, r(s, a) represents the reward function assigning values to each state-action pair, 7 the discount factor that diminishes the value of future rewards, and d(())) is the probability distribution of initial states. The policy Taa s) is the probability of taking specific action in a given state. The objective of RL is to discover a policy that maximizes the expected sum of discounted rewards, denoted as E R[ x). where Rg is the return calculated as R₁ = - ,² (r(s, ax) and T is a task horizon.
In tasks with continuous action spaces, which we focus on in this study, the Bellman optimality operator is computa- tionally intractable due to the max operation. To address this, we use the expectile loss (Kostrikov et al., 2022), which enables operations analogous to the max operator in contin- uous action spaces and a smooth interpolation between the Bellman optimality operator and the Bellman operator. By gradually reducing a parameter 7 representing the estimated to 0.5, our operator tran-
2.2. Bellman Operator and Bellman Optimality Operator
expectile from a value close sitions smoothly from the Bellman optimality operator to the Bellman operator. This approach can be implemented by replacing the L2 loss used for critic learning in existing actor-critic methods with the expectile loss and annealing the parameter T related to optimality.
In RL, the Bellman (expectation) operator and Bellman optimality operator (Sutton, 1988) play fundamental roles in defining the iterative updates for value functions in MDPs. Here, we describe the update of the action-value function Q(s, a) instead of the state-value function V(s).
Experiments on various locomotion and manipulation tasks in continuous action spaces tested the proposed Annealed Q-learning (AQ-L) method combined with TD3 and SAC. The proposed methods significantly outperformed widely used algorithms such as TD3 and SAC. Additionally, an- nealing improved robustness to hyperparameters related to optimality and enhanced performance compared to static optimality estimates.
Bellman Operator For given policy т, the Bellman operator 7 is defined on the Q-function Q= a) as:
T ((,,)) = E (s),a) + BB ,d)]
Applying T repeatedly leads to the functioo satisfying the Bellman equation: Q" (s, a) = T (s)
The contributions of this study are as follows:
The value function in SARSA (Rummery & Niranjan, 1994) and the critic in actor-critic methods are generally updated based on the Bellman operator T These algorithms follow the policy iteration framework, where policy evaluation is performed using updates based on Tx, followed by policy improvement using the updated value function to obtain a better policy. In other words, to derive an improved pol- icy, the value function must accurately evaluate the current policy (Sutton & Barto, 2018).
Preliminary experiments provided the insight that in actor-critic methods, using the Bellman optimality op- erator accelerates learning but introduces overestima- tion bias, while the Bellman operator reduces bias at the cost of slower learning.
We proposed a method for annealing from the Bellman optimality operator to the Bellman operator, enabling
2

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
Bellman Optimality Operator The Bellman optimality
operator T is defined obtain the optimal Q-function Q", assuming the agent selects the action that maximizes expected future rewards:
S1
ag
5
0.1
Ag
("((s,) = ENND (r(s.)) a) + max (
So
8g
This operator is contractive, ensuring that iterative appli- cations converge to the optimal action-value function Q", and lead to the Bellman optimality equation: Q" (8, T*Q" (s."(s,))
S2
S4
a
1/- 0.5
4--0.11
a1
In methods such as Q-learning (Watkins, 1989) and DQN (Mnih al., 2013; 2015), the action-value function is up- dated directly toward the optimal action-value function ing the Bellman optimality operator T
Figure 1. A simple MDP used in preliminary experiments. A re- ward is obtained upon reaching state s; and the episode termi- nates when the agent reaches either state 83 or 84. The discount rate 7 is set to 0.9.
2.3. Expectile Loss
The expectile loss is asymmetric loss function with a pa- rameter and minimizing this loss enables the estimation of the expectile T of the given data. When = 0.5, the loss function is equivalent to the L.2 loss, allowing the estimation of the mean. As T approaches 1, estimates values closer to the maximum. This loss function is used in offline RI. to compute the Bellman optimality operator T*, and the cor- responding method is known Implicit Q-learning (IQL) (Kostrikov et al., 2022).
3. Gradual Transition from Bellman Optimality Operator to Bellman Operator
In this section, we first discuss the effectiveness of the Bell- optimality operator T and the Bellman operator T in actor-critic methods by employing Q-learning- -baedd and SARSA- updates, respectively. Our preliminary exper- iments demonstrate that Q- -aassd updates, which further enhance improvement in the critic, accelerate Iearn- ing but may lead to overestimation bias. Based on these findings, we propose an annealing approach that gradually transitions from Q-learming-based i.e., from the Bellman optimality operator T to the Bellman operator T by reducing the influence of the max operator, thereby mitigating the residual bias in the later stages of training.
IQL is an algorithm widely used in offline RL to learn the value function and the policy using only actions from the offline dataset, without relying on actions from the policy being learned. IQL can learn near-optimal value functions in an in-distribution manner. This is achieved by considering the target value of the value function random variable depending on the action and estimating the upper expectile, e.g.. т 0.9, with the expectile loss. Thus, the loss for learning the Q-function as follows:
SARSA-based updates,
3.1. Bellman Optimality Operator Accelerates Learning
Many commonly used algorithms for continuous action tasks in online RL (Fujimoto et al., 2018; Haarnoja et al., 2018; Schulman et al., 2017) update the critic with SARSA- based updates under the Bellman operator T The value function estimates the value under the current policy, relying solely on policy updates for policy improvement. Replac- ing the critic's ' update mechanism with blearning-based approach under T may accelerate policy improvement and improve learning efficiency. To investigate this, we con- ducted preliminary experiments in a simple environment, as shown Figure In this experiment, we evaluated how many steps it took for the learned Q-values to approach the optimal values in the environment. We used conventional SARSA-based actor-critic method and Q-learning-based actor-critic method for learning algorithms. In the SARSA- based method, Q-values and policy logits 0..a were stored in a table, and the Q-values were updated in a SARSA-like manner (Q(s, a) +- Q(s, a) + a(r + 7 (Q a')]
((0) -
(L)
Qs(s. a))],
where (z( |- 1uu < 0) Qe is an approximated
Q-function parameterized by 0, and Q3 is the target network with parameters 0.
In stochastic environment, not only the randomness of the action but also the randomness of the state transition affects the target value, so the following loss function involving the V-function is used:
I((b) = ( Vs().
L(Ø)= = E ((los,))=((((s)))++))))
Qo(s,
Thus, IQL computed the Bellman optimality operator's max- imum over actions using only actions from the dataset.
3

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
Optie Ast
Crleariing O1ss.Ao Ao)
SARESA (e) SARSA o1))
Arrwaded Olte, del Arrwated Olie, anl
tion is maximized. They expressed the randomness of the Q- values by adding noise 6, The bias is then defined as: Z
Optan Qlss.
Q-leaming C6ks, R2)
r r+ 7 maxa (Q(s', a') + Ear - (r + y max,' (ss,aa)))
Even when the mean of car,ar is O, Jensen's 's inequality shows:
54
O
E. [Z| = 1m0 Egt eg 2 y max =0, (3)
a²
2000 Environment Steps
indicating overestimation occurs.
Environment Steps
To simulate the overestimation bias due to the random- ness, we conducted an experiment in which noise was added to Q-values when calculating the target value as + Y (Q(x, a') + Estaa' for Q-learning and r + yo [Q(s', a') + Est,a'] for SARSA, where E was sampled from normal distribution with mean O. The re- sults are shown in the right panel of Figure 2. In the Q- learning- based method, convergence was faster, but due to the noise, the estimated Q-values converged to values larger than the optimal ones, confirming overestimation. In the SARSA-based case, the expected value is used instead of the maximum value, and since the mean of es',a' a! is 0, its impact is minimal.
The estimated values of ($a, 0a) and Q(su, a1) when using tabular actor-critic with Q-leariin--aase critic up- date under 7 compared to a SARSA- based critic update based on 7 in the environment of Figure 1. Right: The estimated Q-values when Gaussian noise from (0, 0.3) is added to the Q-value be- fore calculating the target value to simulate the randomness in Q-values when approximated with functions. In Annealed, both Q-Ieamiin--aas based and AARSA-based target values are computed, and the learning process transitions lincarly from Q-Icaming-based at the beginning to SARSA-based at the end.
Figure 2. Left:
a)s, while the policy was updated using a policy gradient method (0,,a + 0s.a + V log (ss a)), where represents the step size. In the Q-learning- based method, Q-values were updated in a Q-learning-like manner (Q(s,a) a) + ((s, a) + a( +y max,' Q(s', a²) - (s, a))). and the policy was updated in the same manner using the policy gradient method.
3.3. Gradual transition from Q-learning to SARSA
The results presented in the previous section illustrate the advantages and disadvantages of SARSA-based and Q- bearning-based updates in actor-critic methods. Q-learning- based updates accelerate learning but lead to convergence to higher values, whereas SARSA-based updates exhibit slower learning but reduced bias. To leverage the strengths of both approaches, we propose method that initially em- ploys Q-learning-
The Q-values learned in this experiment are shown in the left panel of Figure 2. The Q-learning-base approach was able to estimate the optimal Q-values faster than the SARSA- based approach. In the SARSA-based method, the policy must first improve before Q-values can move toward the optimal value function, causing delay compared to the Q-learning- blearning-based method.
updates and gradually transitions to A-baedd updates as learning progresses. Specifi- cally, in our preliminary experiments, compute both the target values of Q-learning and SARSA, given by QQL = r+7 max, Q(s', a') and QSARSA = T-yBBa' [Q(s', .a'), and use a weighted average as the target value:
3.2. Bellman Operator is Less Biased
As demonstrated, Q-learning-based updates in actor-critic methods can accelerate learning compared to SARSA- updates. However, calculating the target value using the max operation in Q-learning-basee based methods can introduce overestimation bias. When representing the Q-values with a function approximator, the approximation errors and the stochasticity of gradient methods result in noisy estimated Q-values. Computing the maximum of these noisy Q-values leads to overestimation. This can be explained as follows: the ideal target value is r + y max, E[Q(s', ad)], but the computable target value is "E[max Q(s', a')]. Ac- cording to Jensen's 's inequality, + [mmma. Q(s', a')] > r + y m [sQ(s', d')] Thrun & Schwartz (1993); Lan et al. (2020); Chen et al. (2021) provided more specific defi- nitions of bias. They assumed that the v-values are equal for all actions, corresponding to the scenario where overestima-
QAOI. = TQQi. + (1 - QSARAAA
(4)
where w is a weight that linearly decays from I to O. The estimated Q-value is updated as follows:
as + Q(s, a) + a(QAQL Q(s,
(5)
The results, as represented by the orange curve in Figure 2, demonstrate fast and less biased Q-value estimations. The settings of these preliminary experiments and other details are provided in Appendix A.
3.4. Annealed Q-learning: Gradual Transition of Operator in Continuous Action Spaces
In the previous section, we showed that estimating the op- timal Q-value through the max operation can accelerate
4

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
800
Table 1. The average score 3M steps across the DM Control tasks. The range in parentheses represents the confidence interval.
700
Average Return
600
Method
Mean
IQM
500
AQ-TD3
740.3 (731.0 749.0)
820.0 (811.5 826.8)
AQ-SAC
746.1 (736.3 775.00)
832.4 (820.5 841.6)
400
AQ-TD3
TD3
492.7 (459.5
516.0 (461.8- 571.9)
SAC
657.9 (623.6 688.9)
765.0 (712.8 800.9)
300
AQ-SAC
XQL
564.4 (522.2 - 604.7)
628.8 (560.6 688.3)
200
TD3
SAC
100
XQL
the later stages of training reduce bias, even though these up- dates no longer lead to further improvement. By this stage, the policy is expected to have converged; thus, prioritizing bias reduction over further improvement is reasonable.
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Environment Steps
x10"
Figure 3. The average scores across the 10 locomotion tasks in DM Control.
In the proposed method, T is simply linearly decayed over time. While experimenting with several annealing patterns, linear schedule performed sufficiently well. Specifically, when the initial value of is Tiits the maximum timestep T, and the current timestep is then timestep t can be expressed as:
learning and that annealing this optimality allows for ef- ficient estimation of less biased values in the final stage. However, in continuous action tasks, it is generally not fea- sible to directly compute the maximum vavalue. To address this, we use the expectile loss, introduced in Kostrikov et al. (2022), which allows for the implicit computation of Q-learning-based target values without employing the max operation. This approach also enables smooth interpolation between Q-learning- and SARSA-based target values.
T (t) Tiit (Tini 0.5) T
We named this method Annealed Q-learning (AQ-L) and conducted experiments using methods combined with TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018), referred as AQ-TD3 and AQ-SAC, respectively. These methods are implemented through a simple modification, where the squared error in the critic update is replaced with the expectile loss, and T is annealed as shown in Equa- tion (7).
In online RL, the expectile loss used to estimate the opti- mal value based on actions sampled from the current policy. The loss function for the value function is expressed as follows:
T(00- =
Q - Qu((s a))].
(6)
4. Experiments
In the expectile loss, setting 7 = allows for estimating Q-learning-basee target, while setting = 0.5 estimates a SARSA-based target. In this study, we propose method that begins with a F value close at the start of training and gradually anneals it to 0.5 by the end.
To verify the effectiveness of the proposed method, conducted several experiments. Specifically, we examined how the proposed AQ-TD3 and AQ-SAC perform compared to widely used online RL methods such as SAC and TD3. Additionally, we investigated the impact of the annealing of t on task scores and the robustness of hyperparameters.
As discussed in the previous section, estimating the Q- learning-bas based optimal Q-value at the early stages of train- ing and transitioning to estimating the SARSA-based Q- value for the current policy in the later stages enables ef- ficient learning. While overestimation might occur in the early stages depending on the annealing schedule, this can facilitate exploration (Lan et al., 2020). The relationship between this bias and exploration is also discussed in Ap- pendix I. Overestimation increases the chance of the agent selecting overestimated actions. correcting them in the pro- cess, and thus broadening the range of actions tried. This bias is particularly beneficial in the early stages of learning when exploration is critical. The AASAA--aaee updates in
4.1. Experimental Setup
we conducted online RL training using 10 challenging loco- motion tasks from DM Control (Tassa et al., 2018; Tunya- suvunakool al., 2020) and 10 difficult manipulation tasks from Meta-Worke (Yu al., 2019). Although Meta-World is often used for multi-task learning, this study focuses ex- clusively on single-task learning. As for the 10 Meta- World tasks, we selected difficult tasks reported to have low suc- cess rates in their paper. We employed TD3 and SAC as baseline methods, widely used in continuous action tasks for online RI, along with XQL. (Garg et al., 2023). XQL uses
5

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
AQ-TD3
AQ-SAC
TD3
SAC
XQL
cheetah-run
fish-swim
hopper-hop
hopper-stand
humanoid-run
Average Return
Aver age Recum
Average Return
Aver Average Retur
Average
Environment Steps
Environment Steps
Environment Steps
Environment Steps
Environment Stops
humanoid stand
humanoid-walk walk
quadruped-rui
quadruped walk
walker-run
Average Return
Average Return
Avorage Return
Average Return
Average Ret Return
Environment Steps
Environment
Steps
Environment Steps
Environment Steps
Enviranment Steps
Figure 4. The average return for each task in DM Control.
Gumbel regression, assuming Gumbel distribution for the error distribution, and estimates the soft-optimal value in untropy-maximizin RL, offering comparison as another method for estimating the optimal value. While Kostrikov et al. (2022) incorporated both Q-function and a V-function to mitigate performance degradation observed in stochastic environments, AQ-L demonstrated superior performance without employing a V-function, even in extremely stochas- tic environments. Therefore, we utilize only the Q-function in approach. Further details on this comparison are provided in Appendix G. The hyperparameters related to annealing are shown in Appendix B. The results for using different annealing durations are presented in Appendix E. For AQ-TD3 and AQ-SAC, all other hyperparameters are kept identical those of TD3 and SAC, respectively. When is fixed at 0.5, AQ-TD3 and AQ-SAC are precisely the same as TD3 and SAC.
maximum return achievable for these tasks is 1000. In hopper-hop, humanoid-run, and humanoid-walk, where ex- isting methods achieved low scores after 3 million training steps, the scores of AQ-TD3 and AQ-SAC increased sig- nificantly. In tasks such as hopper-stand, quadruped and quadruped- walk, where existing methods attained con- sistently high scores, AQ-TD3 and AQ-SAC achieved high scores with fewer steps and exhibited reduced variance, re- sulting in higher final scores. This suggests improved sam- ple efficiency from optimal value estimation and enhanced stability due to increased exploration early in the training. AQ-TD3 and AQ-SAC also outperformed XQL, another method for estimating the (soft-)optimal value, demonstrat- ing that expectile-barte based squared loss in online RL is more stable and superior to XQL's exponential apponential-based loss. A straightforward approach computing the max operation max-backe max-backup. which involves sampling Q- values for multiple actions and using the maximum value. AQ-L outperformed max-backup in terms of both computational efficiency and score. The results and details of this comparison are shown in Appendix D.
4.2. Comparison with Prior Studies
The average scores across 10 tasks from DM Control for AQ- TD3, AQ-SAC and the baseline methods from prior studies are shown in Figure 3. The average scores, interquartile mean (IQM), and their confidence intervals, measured using Agarwal et al. (2021), are listed in Tables 1 and 4. Both AQ- TD3 and AQ-SAC significantly outperformed prior studies. Furthermore, they demonstrated substantial improvements over their respective base algorithms, TD3 and SAC. These results indicate that the simple modifications of employing the expectile loss and annealing T in online RL can lead to remarkable performance enhancements. As shown in Table 4, AQ-TD3 and AQ-SAC achieve high scores even with smaller steps, demonstrating a significant improvement in sample efficiency.
Figure 5 shows the average success rate across the manipu- lation tasks in Meta- Meta-World, and the success rates for each task are presented in Figure 6. TD3 exhibited significantly lower success rate than SAC, suggesting the necessity of exploration enhancement, such as entropy maximization in SAC. Nevertheless, AQ-TD3 achieved a slight perfor- mance improvement over TD3, which struggled to learn ef- fectively. AQ-SAC outperformed SAC in most tasks, achiev- ing a higher success rate in fewer steps and significantly improving asymptotic performance and sample efficiency. The proposed method also demonstrated effectiveness in manipulation tasks.
The scores for each task are shown in Figure 4. The
6

---

0.9. Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
1.0
AQ-TD3
Table 2. The average final score across 10 DM Control tasks when is annealed compared to when is fixed in AQ-SAC. Annealing 7 improves scores and enhances robustness to 7 settings.
AQ-SAC
TD3
0.8
SAC
Success Rate
Method
Mean
IQM
0.6
Annealed (0.7)
720.2 (694.8 741.0)
821.7 (803,4 836.0)
Annealed (0.8)
742.1 -729.11754.7)
824.0 (805.4 840.2)
0.4
Annealed (0.9)
746.1 (732.0-558.5
832.4 (815.3 844.8)
Annealed (0.95)
736.1 (725.7- 745.9)
815.6 (798.8 827.1)
Fixed (0.6)
713.6 -190.5-7344
822.7 (800.1 834.9)
0.2
Fixed (0.7)
730.7 711.3-745.77
825.0 (806.1 840.0)
Fixed (0.8)
683.4
775.9 (756.5 - 790.4)
0.0
Fixed (0.9)
588.2 -13.6)
632.7 (594,9- 665.9)
Fixed (0.95)
364.9 (338.8 390.6)
303.5 (265.4 341.5)
0.4
0.6
0.8
1.0
Environment Steps
x107
We measured the bias of the estimated Q-value with respect to the Monte Carlo return, and the results are presented in Figure 12. The bias increases as T becomes larger, and in AQ-SAC, it eventually reaches a level comparable that of SAC. This outcome is consistent with the preliminary experiments shown in Figure 2.
Figure 5. The average success rate across the 10 manipulation tasks in Meta-World.
4.3. Effects of r Annealing
We evaluated the effectiveness of annealing T using AQ- SAC. Table 2 presents the final average scores across all tasks for AQ-SAC when 7 is annealed compared to when т fixed. AQ-SAC with T annealing, starting from 0.9, out- performed the versions with fixed 0.6, 0.7, 0.8, 0.9, and 0.95, highlighting the effectiveness of annealing. Figure il- lustrates the results for two challenging tasks: humanoid-run and hopper-hop. In hopper-hop, when is fixed, increasing from 0.5 (SAC) to 0.6 and 0.7 improves the score, but the learning process becomes unstable, leading to a decline in scores at r = 0.8, with 7 = 0.7 producing the best result. In contrast, AQ-SAC, which anneals r from 0.9, maintained stable learning and achieved higher final scores than the fixed-7 versions, even with the optimal fixed 7 value of 0.7 showed better
In AQ-L, a linear annealing schedule was used. We also conducted experiments with several non-linear annealing patterns. The results demonstrated that linear annealing achieves sufficiently good performance. The details of the experiments on non-linear annealing patterns are provided in Appendix H.
5. Related Work
Off-Policy Online RL In continuous action space online RL tasks, methods such as SAC (Haarnoja et al., 2018) and TD3 (Fujimoto al., 2018 are widely used. SAC is an actor-critic method based on Haarnoja et al. (2017) that in- corporates entropy maximization. The value function adds the policy's s entropy to the target value and models the soft value function for the current policy. TD3 is an extension of DDPG (Lillicrap, 2015), incorporating techniques like clipped double Q-learning, action noise, and delayed policy updates. Like SAC, TD3 updates the value function based SARSA, learning the value function under the current policy. Both methods employ SARSA-based updates, fo- cusing on learning the value function for the current policy, Our study demonstrates that by replacing the value function update with the expectile loss, shifting to a Q-learning- maximization, we can achieve improved performance.
0.7. In humanoid-run, the fixed early-stage learning than AQ-SAC, but the learning became unstable over time, resulting in significant score drop by the end of the training. For larger T values, the scores were considerably lower, suggesting high sensitivity to r when it is not annealed. AQ-SAC, on the other hand, maintained relatively stable score improvement. The results for other tasks are provided in Appendix C.
Even when annealing T, the initial value of - remains a hyperparameter. However, compared to fixed r, annealing makes the method less sensitive to this hyperparameter. As shown in Table 2, in the fixed 7 case, = 0.7 performed best, but increasing it to 0.8 led significant drop in scores, with even lower scores for T 0.9 and 0.95. In the annealed r case, the best initial value was However, scores did not change significantly even when the initial value varied between 0.7 and 0.95, demonstrating that AQ-SAC is more robust to hyperparameter variations than the fixed 7 case.
Overestimation bias is a recurring issue in RL. Thrun & Schwartz (1993) formalized how noise in function approx- imators leads to overestimation bias, showing how noise variance and the number of actions contribute to the increase in bias. Lan et al. (2020) and Chen et al. (2021) suggest that overestimation can sometimes be beneficial for exploration while using ensembles to control the mean and variance of
7

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
AQ-TD3
AQ-SAC
TD3
SAC
pick-place
stick-pull
push
bin-picking
basketball
Success Rate
Succes Rate
Saccess Rate
Success Rate
Succes
Environment Steps
Environment Steps
Environment Steps
Environment Steps
Environment Steps
assembly
box-close
push-wall
shelf- shelf-place
Success lttee
Success Rate
Succc5s Pate
Succes Hate
Success Rate
622
0.4 Environment Steps
02
6.6
3.1
Enviranment Steps
Environment Stops
Environment Steps
Environment Steps
Figure 6, The average success rate for each task in Meta-World.
54C
(3:9)
Fbud (9.7)
Fias (3:3]
Fied (9:9)
Tout (3.85
online fine-tuning following offline RL, and it is not an algorithm designed for learning without an offline dataset. XQL (Garg et al., 2023) models the soft optimal value in maximum entropy RL using Gumbel loss, derived from maximum likelihood estimation under the assumption of Gumbel error distribution. MXQL (Omura et al., 2024) stabilizes the Gumbel loss by employing Maclaurin ex- pansion. By increasing the order of the expansion from 2 to infinity, MXQL. transitions from SARSA A-bsed learn- ing to soft Q-learning- may benefit from annealing the expansion order to improve performance, tuning hyperparameters can be challenging due to the infinite range of the optimality parameter. The experiment combining MXQL and AQ-L is described in Appendix J.
hoppar-hap
humanaid-run
Aparge
120
08
5.8 Environment Steps
2.0
25
30
2.8
2,0
Ensirunment Stapss
learning. Although MXQL
Figure 7. The average return of AQ-SAC on hopper-hop and humanoid-run when 7 is annealed compared to when is fixed. Annealing r improves asymptotic performance and stability.
bias. Incorporating these ensemble techniques into our pro- posed method could potentially enhance sample efficiency, which could be promising direction for future work.
Kalashnikov et al. (2018); Kumar et al. (2019; 2020) im- prove sample efficiency by modeling the optimal Q-value through sampling multiple actions from the policy distribu- tion and selecting the action that maximizes the Q-value, These methods can shift from SARSA-baset to Q-learning- based by increasing the number of samples from 1 to infinity. However, tuning poses challenges, and computational costs rise significantly as the number of samples increases.
Value Function Maximization in Continuous Action Space In continuous action tasks, calculating the max- imum Q-value is generally intractable. Various methods have tackled this issue by using asymmetric loss functions (Kostrikov et al., 2022; Garg et al., 2023; Xu et al., 2023; Omura et al., 2024; Sikchi et al., 2024), evaluating multiple actions (Kalashnikov et al., 2018; Kumar et al., 2019; 2020), or discretizing actions (Tavakoli et al., 2018; Seyde et al., 2023; 2024).
Some methods, such as Tavakoli al. (2018); Seyde et al. (2023; 2024); Ireland & Montana (2024), achieve optimal Q-value estimation through action discretization. However, if the number of intervals is small, accurate action selec- tion becomes difficult while increasing the intervals reduces sample efficiency.
IQL (Kostrikov et al., 2022) approaches Q-value as a ran- dom variable with inherent randomness related to the ac- tions, modeling the maximum value using the expectile loss with an expectile parameter close to 1. Adjusting the hyperparameter т from 0.5 to 1 shifts the value function estimate from SARSA-based to e-learning-based How- ever, it should be noted that IQL. focuses on offline RI. and
The most closely related prior work is Ji et al. (2024). Moti- vated by the underestimation of the value function in actor- critic methods, they compute a weighted sum of target val- ues from the Bellman optimality operator, where the max
8

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
actions is computed within the replay buffer via in- sample maximization, and from the Bellman operator. In contrast, our study aims to accelerate early-stage learning by introducing a beneficial bias, and to reduce this bias in the later stages. Rather than combining the outputs of the Bellman optimality operator and the Bellman operator, we propose a gradual transition from the Bellman optimal- ity operator, where the max computed under the current policy, to the Bellman operator over the course of training- While Ji et al. (2024) estimate two separate Q-values, our ap- proach allows us to maintain a single Q-function throughout training. Therefore, our method can be viewed as simplifi- cation of et al. (2024), incorporating a novel motivational rationale and a scheduling mechanism to gradually reduce optimality, while relying on a single Q-function.
References
Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
Badia, A. P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, Z. D., and Blundell, C. Agent57: Outperforming the Atari human benchmark. In III. H. D. and Singh, A. (eds.), Proceedings of the 37th Interna- tional Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 507-517. PMLR, 13-18 Jul 2020.
Chen, X., Wang, C.,, Zhou, Z., and Ross, K. W. Randomized ensembled double q-learning: Learning fast without model. In International Conference on Learning Repre- sentations, 2021.
6. Conclusion
This study investigated the challenges and potential of mod- eling optimal value in online RL, particularly within actor- critic methods for continuous action tasks. Through experi- mental validation, we demonstrated that using the Bellman optimality operator accelerates learning but introduces over- estimation bias, while the Bellman operator ensures stable value estimation at the cost of slower learning. To address this trade-off, we proposed Annealed Q-learning, which smoothly transitions from the Bellman optimality operator to the Bellman operator using the expectile loss. This ap- proach initially enhances learning speed while mitigating bias in the later stages. Experiments on diverse locomotion and manipulation tasks validated the effectiveness of the pro- posed method, achieving superior performance compared to widely used algorithms like SAC and TD3, and demon- strating robustness to hyperparameter settings through the annealing process. While linear annealing contributed to performance improvement, further optimization of the an- nealing schedule is a promising direction for future research.
D' D'Oro, P., Schwarzer, M., Nikishin, E.,, Bacon, P.-L.., Belle- mare, M. G., and Courville, A. Sample-efficient rein- forcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations, 2023.
Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Dy, and Krause, A. (eds.). Proceedings of the 35th In- ternational Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1587- 1596. PMLR, 10-15 Jul 2018.
Garg, D., Hejna, J., Geist, M., and Ermon, S. Extreme q-learning: Maxent RL without entropy. In International Conference on Learning Representations, 2023.
Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Rein- forcement learning with deep based policies. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, vol- ume 70 of Proceedings of Machine Learning Research, pp. 1352-1361. PMLR, 06-11 Aug 2017.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, Soft actor- critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Ma- chine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018
Acknowledgments
This research is partially supported by JST Moonshot R&D Grant Number JPMJPS2011, CREST Grant Number JP- MICR2015 and Basic Research Grant (Super AI) of Insti- tute for AI and Beyond of the University of Tokyo. T.O was supported by JSPS KAKENHI Grant Number JP25K03176. K.O. was supported by JST SPRING, Grant Number JP- MJSP2108.
Ireland, D. and Montana, G. Revalued: Regularised ensem- ble value- deluueddcoopositionn for factorisable markov decision processes. In The Twelfth International Conference on Learning Representations, 2024.

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
Ji, T., Luo, Y., Sun, F., Zhan, X., Zhang, J., and Xu, H. Seizing serendipity: exploiting the value of past success in off-policy actor-critic. In Proceedings of the 41st In- ternational Conference on Machine Learning, 1CML'24. Jorg. 2024.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region poliey optimization. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Confer- ence on Machine Learning. volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897, Lille, France, 07-09 Jul 2015. PMLR.
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Van- houcke, V., et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, pp. 651-673. PMLR, 2018.
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms, 2017.
Seyde, T., Werner, P., Schwarting, W., Gilitschenski, I,, Riedmiller, M., Rus, D., and Wulfmeier, M. Solving continuous control via q-learning. In The Kleventh Inter- national Conference Learning Representations, 2023.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q- q-learning. In International Conference on Learning Representations, 2022.
Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Sta- bilizing off- off-policy q-learning via bootstrapping error duction. In Wallach, H., Larochelle, H.,, Beygelzimer, A., d'Alché-Buee F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Seyde, T., Werner, P., Schwarting, W., Wulfmeier, M., and Rus, D. Growing Q-networks: Solving continuous control tasks with adaptive control resolution. In Abate, A., Can- non, M., Margellos, K., and Papachristodoulou, A. (eds.), Proceedings of the 6th Annual Learning for Dynamics Control Conference, volume 242 of Proceedings of Ma- chine Learning Research, pp. 1646-1661. PMLR, 15 Jul 2024.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conser- vative q-learning for offline reinforcement learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Process- ing Systems, volume 33, pp. 1179-1191. Curran Asso- ciates, Inc., 2020.
Sikchi, H., Zheng, Q., Zhang, A., and Nickum, Dual RL: Unification and methods for reinforcement and imi- tation learning. In The Twelfth International Conference on Learning Representations, 2024.
Lan, Q., Pan, Y., Fyshe, A., and White, M. Maxmin q- learning: Controlling the estimation bias of q-learning. In International Conference on Learning Representations, 2020.
Sutton, R. Learning to predict by the method of temporal differences. Machine Learning, 3:9.44, 08 1988. doi: 10. 10.1007/BF001155 15009.
Lillicrap, T. Continuous control with deep reinforcement learning. arXiv preprint 150v.02509 2015.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonogiou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning, 2013.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., de Las Casas, D., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq. A., Lillicrap, T., and Riedmiller, M. Deepmind control suite, 2018
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540): 529-533, 2015.
Tavakoli, A., Pardo, F., and Kormushev, P. Action branch- ing architectures for deep reinforcement learning. In AAAI Conference on Artificial Intelligence, pp. 4131- 4138.
Morerio, P., Cavazza, J., Volpi, R., Vidal, R., and Murino, V. Curriculum dropout. In ICCV, 2017.
Tessler, C., Givony, S., Zahavy, T., Mankowitz, D., and Mannor, S. A deep hierarchical approach to lifelong learning in minecraft. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 31(1), Feb. 2017. doi: 10. 10.16099aaii.3311....7.. 10744.10744
Omura, M., Osa, T., Mukuta, Y., and Harada, T. Stabi- lizing extreme q-learning by maclaurin expansion. In Reinforcement Learning Conference, 2024.
Rummery, G. and Niranjan, M. On-line q-learning us- ing connectionist systems. Technical Report CUED/F- INFENG/TR 166, 11 1994.
Thrun, S. and Schwartz, A. Issues in using function ap- proximation for reinforcement learning. In In Fourth Connectionist Models Summer School, 10 1993.
10

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bohez, S., Merel, J., Erez, T., Lillicrap, T., Heess, N., and Tassa, Y. dm-control: Software and tasks for continuous control. Software Impacts, 6: 100022, 2020. ISSN 2665-9638 doi: http:////....111/...maaaaaaaaa20 100022.
Watkins, C. J. C. H. Learning from Delayed Rewards. PhD thesis, King's College, Cambridge, UK, 05 1989.
Xu, H., Jiang. L.,, Li, J., Yang, Z.,, Wang, Z., Chan, V. W. K., and Zhan, X. Offline RL with no OOD actions: In-sample learning via implicit value regularization. In The Eleventh International Conference Learning Representations, 2023.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evalua- tion for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRl.), 2019.

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
A. Details of Preliminary Experiments
In the preliminary experiment, we compared Q-learning- updates and SARSA -based updates in actor-critic models using the environment shown in Figure 1. The details of this experiment are described below. The critic manages the estimated Q-values for each state-action pair in table and updates them based either Q-learing or SARSA. The actor manages logits for each action pair in table, calculates the distribution using the softmax function, and samples actions from this distribution. Thus, the policy is expressed as follows:
cxp( (0s,a)
Ta |s)
Σ exp (0,b))
(8)
The update of these logits is performed using the policy gradient method, with the update equation given follows:
0 0 + V log ae(as | st) Q(s at), V ee log (a s) = 60.a' - = ss,
(9)
where da.a' is the Kronecker delta. The step size used for updates in both the critic and the actor was set to le-3, While increasing the step size accelerates learning, the observation that Q-learning--aaae -based updates are faster than SARSA-bass updates remained consistent. step size that yielded smooth learning curves was chosen. When using 8p as the only initial state, the lower-value state 82 was visited less frequently than 81, making it challenging to accurately estimate Q-value for #2- address this, the initial state was randomly selected from 8g, 81, and 82. Additionally, although actions were sampled from the policy, the estimation of Q-values for low-value actions progressed slowly. To mitigate this, 10% probability of taking random actions, akin to c-greedy policy, introduced.
In Figure 2, the estimated Q-values for so are shown, while the Q-values for 81 and are presented in Figure 8. These results are from experiments where noise was added to the Q-values. However, since the next state for and 82 is a terminal state and Q-values are learned solely from the reward, they are unaffected by the noise. Furthermore, the updates are identical for both Q-learning-basee and SARSA-based methods under these conditions, leading the same Q-value estimation in both cases.
6,000
0.055
0.075
Q151, 30/a1)
Optimai 0o)
(2 @0/a1)
Optimal O(5z,
Optimal O(s,, 9z)
Optimal Qls;, ax)
0.005
SARSA ag)
0.025
SARSA 0 ap)
SARSA Q18, a1)
0.000
Q-leariing O(s), as]
0.000
Q-leaming 01sz, ao)
QJearning Q a
Q-leaming Ols,, ax)
Arnealed (65,, an)
Annealed O15;
Annealed ((5,, ax)
Annesled Q15, a;)
0775
-2.075
10000
20000
20000
#2000
70000
10000
20200
30000
40000
50000
60000
70000
Environment Steps
Environment Steps
Figure 8. The estimated Q-values for and in the experiment where noise was added to reproduce the randomness of Q-value estimation the environment of Figure
A.1. Results under Different Settings
Variance of Noise As the standard deviation of Gaussian noise NNO, a)) increases, the randomness of the estimated Q-value is expected to increase, Icading to larger overestimation bias. Therefore, experiments were conducted using various values of a. The MDP the same as in Figure The results, shown Figure 9, indicate that the bias increases as a becomes larger. Even with Annealed, a larger a causes bias the early stages of learning; however, this promotes exploration, and ultimately, the values converge to those obtained with SARSA.
12

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
Optimal (So. aq)
Qllearning QSso, ao) Q-learning Q5s., a1)
SARSA (5o. ao)
Annealed ((5o. ao)
Optimal (5,, 21)
SARSA Q(sp, a1)
Annealed Q(5o, a1)
o 0.1
-= 0.2
a= 0.3
0 = 0.4
51 an
Q15.. 3r
Q15
eode Environment Steps
Envirooment Sleps
Environment Steps
Environment Steps
Figure 9. The estimated Q-values obtained using various standard deviations of Gaussian noise in the environment of Figure 1.
ra and 54 If the Q-values Q(5', a') vary significantly depending on the action, then even if the Q-values are noisy, the maximum action Q-value max, (s',, a') can be reliably selected each time, making overestimation less likely. Conversely, if (s',, aa) does not change much across different actions a', overestimation becomes more significant. To verify this, we varied r3g and гΛ and estimated the Q-values at so- The standard deviation of the noise is 0.3. The results are shown in Figure 10. It was observed that when the difference between r3 and ra is large, the bias decreases.
Optimal (So, ag)
Q-learning QSo,, ao)
SARSA QSo,, ag)
Annealed Q(5o, an)
Optimal (So, a1)
Q-learning QSSo, a1)
SARSA (So. a1)
Annealed Q(5,, a1)
04
0001
0.1,/4 = 0.1
ry 0.2,02 - 0.2
= 0.5. f4 = - 0.5
(1... 2a/a1)
@ 2n/ar)
(5.,, 2g/a1)
Environment Sleps
Environment Steps
Environment Steps
Environment Steps
Figure 10. The estimated Q-values obtained using various values of r3 and 14 with the standard deviations of Gaussian noise is 0.3 in the environment of Figure! 1.
r1 and 12 In the preliminary experiment conducted in the environment shown in Figure 1, the Q-values at 8o were both positive. We conduct experiments under conditions where the Q-value is negative. In Figure 1, the reward values were set as r1 = and r2 = 0.5. While keeping the interval unchanged, we modify them to rı 0.25 and r2 = -0.25. The results obtained by varying the variance of the noise are shown in Figure 11. The results were generally similar those in the case of positive rewards. However, due the initial Q-value being set to zero, actions corresponding negative Q-values were less likely to be selected, resulting in slower learning. Therefore, we extended the training steps. Despite this adjustment, SARSA, which inherently learns more slowly, failed to converge.
B. Details of Experiments on DM Control and Meta-Workle
The implementations of TD3, SAC, AQ-TD3, and AQ-SAC are based on D'Oro et al. (2023). To ensure fair comparison, all methods employed batch size of 256, and both the actor and critic networks used two hidden layers consisting of 256 units each. XQL uses the official implementation, and used XSAC, which integrates XQL with SAC. As described in the XQL paper, the temperature parameter 33 was evaluated for values [1, 2, 5J. and the best value 3 5 was chosen. Other hyperparameters for all methods follow the values reported in their respective papers. For tasks in the DM Control, the results are averaged over 10 random seeds, while, due to computational considerations, 5 random seeds are used to report the results for Meta- World tasks. The annealing period T corresponds to the total number of training steps. Specifically, is set to million steps for DM Control tasks and million steps for Meta- World tasks. The initial expetile value for
13

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
Optimal (So. ag)
Q-learning (So, ao)
SARSA (50. an)
Annealed ((5o. ao)
Optimal (5,, 21)
Q-learning Q(s a1)
SARSA Q a1)
Annealed Q(5o, a1)
0, 0.25, f2
0.25
0.1, n. 0.25, - 0.25 ar
o 0.2, 0.25, 0.25
0 0.3, = 0.25, r> = 0.25
Q15c, an(a)
Q150. ax/a
Q 2y/0))
O OL 2g/a
2000
1984 soDe Envirorment Steps
uspo T0808 12000 1400
2030
1000 Environment Steps
upp0 sip00 19000 1400
2000
+000 seaa Environment Steps
US00 10800
50000
M400
2000
4000 Environment Steps
vapo saa0e 10000 M400
Figure 1. The estimated Q-values obtained using 11 0.25 and 12 = -.25 with the various standard deviations a of Gaussian noise the environment of Figure 1.
annealing, Tinit 0.8 for AQ-TD3 and 0.9 for AQ-SAC the DM Control tasks, and 0.7 for both AQ-TD3 and AQ-SAC in the Meta- Woold tasks. The hyperparameters of AQ-TD3 and AQ-SAC are summarized in Table 3.
We measured the bias of the estimated Q-value using AQ-SAC. Following the methodology of Chen et al. (2021), evaluated the bias with respect to the Monte Carlo return. The results presented in Figure 12. The bias increases becomes larger, and in AQ-SAC, it eventually reaches level comparable to that of SAC.
Table 3. Hyperparameters for AQ-TD3 and AQ-SAC.
Parameter
AQ-TD3 AQ-SAC
Discount factor
0.99
Minibatch size Optimizer
256
Adam
Learning rate
0.0003
Activation function
ReLU
Number of hidden layers
2
Hidden units per layer
256
Replay buffer size
106
(EMA coefficient for target networks)
0.995
Tini (DM Control)
0.8
0.9
Tiit "Meta-World
0.7
Table 4. The average IM steps across the DM Control tasks.
Method
Mean
IQM
AQ-TD3
620.8 (609.9 631.5)
655.4 (637.0 673.9)
AQ-SAC
624.8 (611.4-63) 637.5)
651.5 (635.0 669.6)
TD3
376.8 (348.1 407.3)
327.0 (281.5 377.1)
SAC
493.7 1354.6551133
516.1 1.51.7-578..
XQL
433.6 (394.8 472.1)
433.5 (372.2 492.9)
C. Experimental Results Using Fixed 7 and Annealed 7
To evaluate the effect of annealing, we compared the results of annealing 7 and fixing T using AQ-SAC. Figure 13 shows the average return for each task when annealing from various Tinit values. Figure 14 shows the average return for each task when using various fixed 7 values. When annealing, the performance is less sensitive to the 7 value, demonstrating increased robustness to hyperparameters through annealing.
14

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
AQ-SAC
SAC
Fixed (0.6)
Fixed (0.7)
Fixed (0.8)
Fixed (0.9)
Fixed (0.95)
0.0
0.5
1.0
1.5
2.0
2.5
3.0 x110P
Environment Steps
Figure 12. The bias of the estimated Q-value with respect to the Monte Carlo return. The larger т, the greater the bias, and AQ-SAC, the bias eventually became comparable that of SAC.
D. Experimental Results of Max-backup
In continuous action tasks, one straightforward way to compute maximum Q-values is by sampling multiple actions from the current policy, calculating their Q-values, and selecting the maximum. This method, known as max-backup was employed by Kumar et al. (2020). In this study, we compared max-backup with AQ-SAC using different sampling counts. Both methods were implemented based on SAC. Figure 15 presents the average return for each task when using max-backup with various action sampling numbers. Max-backup is implemented based on Kumar al. (2020). While max-backup has the drawback of increased computational cost, demonstrated scores comparable AQ-SAC in some tasks. However, in more challenging tasks such hopper-hop and humanoid-run, AQ-SAC outperformed max-backup. This suggests that expectile-basele methods for maximum value estimation are more effective than the sampling-base approach used in max-backup. Since the computation cost of max- max-backup increases with the number of samples, AQ-SAC is also superior in terms of both performance and computational efficiency.
E. Analysis of Annealing Duration
In AQ-L, annealing is performed over the entire training steps, but the annealing duration can also be hyperparameter. Therefore, experiments were conducted using AQ-SAC with varying annealing durations. The results shown in Figure 16. After the annealing phase, learning proceeds with r = 0.5, consistent with the standard SAC setting. Notably, even when T is as small as 1 million steps, there is a significant performance improvement compared to SAC. The final return does not vary significantly across different values T. indicating that AQ-SAC robust to within the range tested.
F. Results from Extended Training on humanoid-run and humanoid-walk
In Figure 4, both humanoid-run and humanoid walk had not yet converged even after 3 million steps, and their performance continued improve. Therefore, the results of extending training up to 10 million steps for both AQ-SAC and SAC are shown in Figure 17. Even in terms of the asymptotic performance after 10 million steps of training, the proposed method AQ-SAC outperformed SAC.
G. The Necessity of the V-function
In IQL, both the Q-function and V-function were trained to handle environmental stochasticity. However, shown in Figure 18, this study suggests that training with only the Q-function achieves better performance even in stochastic environments. Therefore, AQ-L utilizes only the Q-function.
Figure 18 presents results comparing AQ-SAC using only the Q-function versus AQ-SAC using both the Q-function and V-function in the stochastic hopper-hop environment. Stochasticity in the environment is introduced by adding Gaussian noise with mean of the actions input into the DM Control tasks. The left figure shows the results when 7 is annealed,
15

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
cheetaln-un
fish swim
hopper-hop hop
hopper-stand
Aurrane Retuur
Apsrage Raturn
400
16
20
70
75
Envtronment Steps
Enviromment Stops
Envirommnnt Stops
Enviroment: Steps
humancid-run
humanoid-stare stand
humanoid-walk
quadruped-run
Avenag Raturn
600
Average
Anszago
30
60
8.5
2.0
3.5
2.8
2.5
62
24
25
000
115
20 25
85
100
15
23
25
34
Ensimoment Steps
Fastrommet Steps
Emuinomment Strps
Ensinorment Steps
quadruped-walk
walker-run
Aatraga Retum
Averaee Return
000
(0.7)
Anneaeed (0.8)
Annealed (0.9)
Annealed (0:95)
Steps
Figure 13. The average return for each task when annealing with various Taa values in AQ-SAC.
while the right figure corresponds to fixed r. The values used were Tinit 0.9 for annealing and T 0.7 for the fixed case, these settings yielded the best performance.
In both cases, whether annealing or fixed, using only the Q-function consistently outperformed using both the Q-function and V-function across all levels of noise standard deviation. This performance difference is attributed to the additional network required when using the V-function, which increase approximation error and degrade performance.
For the annealing case in the left figure, the degradation in average return as noise standard deviation increases similar whether or not the V-function is used. This is likely because annealing results in learning behavior similar SAC toward the end of training, mitigating the impact of environmental randomness the IQL loss.
In the fixed 7 case shown in the right figure, performance degradation due to increased noise standard deviation is more pronounced when the V-function not used. Nevertheless, using only the Q-function still outperforms the setup with both Q-function and V-function.
H. Non-linear Annealing Patterns
We also tested non-linear annealing patterns with AQ-SAC. Figure 19 illustrates different annealing patterns, including exponential annealing (Expl) as proposed by Morerio et al. (2017) in the context of dropout scheduling, its inverse (Exp2), and sigmoid-basee annealing (Sigmoid). The results, as shown in Table 5, indicate that Sigmoid and Expl performed similarly to linear annealing, but Exp2 resulted in worse performance. This suggests that prolonging the high 7 period leads to excessive initial bias, which negatively impacts the later stages of learning. These observations indicate that linear annealing proves to be effective enough, and future research could explore dynamic adjustments based on bias.
16

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
cheetaln-run
fish swim
hopper-hop hop
hopper-stand
Auatrngy Return
Return
300
Envtronment Steps
Enviromment Stops
Envirnment Stops
Envirommen: Steps
humancid-run
humancid--e stand
humanoid-walk
quadruped-run
Nowrags Ratur
Acozago Rutur
Aterage
130
60
3.5
2.8
2.5
Lin Frvironment
100
20 Emuinomment Strpp
Ensimoment Steps
Steps
Frsinonmmnt Steps
quadruped-walk
walker-run
Aatraga Retum
Averug Return
(0.6)
Fixed (0.7)
Fixed (0.8)
Fixed (0.9)
Fixed (0.95)
Steps
Figure 14. The average return for each task when using various fixed values in AQ-SAC.
Table The average scores across the 10 DM Control tasks for AQ-SAC using various anncaling patterns. A simple lincar annealing showed the best performance.
Method
Mean
IQM
Linear
746.1 (732.0 - -58277)77777
832.4 -88151184444 (815.1-844.88
Expl
728.1 (716.7 739.4)
809.1 (792.9 823.7)
Exp2
694.9 (680.5 706.8)
772.4 (759.3-8444
Sigmoid
742.2 (731.3 -752.9)
812.9 (795.7 829.4)
I. The Relationship Between Bias and Exploration
We measured policy entropy during early training. The table below shows average entropy over DMC tasks for SAC (Fixed (0.5), (0.7), and (0.9)), corresponding SAC with expectile loss with 0.5, 0.7, 0.9. As shown in Figure 12, these methods exhibit overestimation bias in the order: Fixed (0.9) > Fixed (0.7) > SAC. The entropy follows the same trend, suggesting higher bias leads to broader exploration.
This supports the intuitive hypothesis that overestimation can lead to suboptimal actions due to inflated Q-values, promoting exploration. Prior work (e.g., Section of Lan et al. (2020)) also demonstrates that overestimation can be beneficial in tasks where exploration important, while it degrades performance tasks where exploration undesirable. These results support the claim that overestimation bias promotes exploration.
J. Combination of Annealed Q-learning and MXQL
In our study, we enable the transition from the Bellman optimality operator to the Bellman operator by using expectile regression from IQL (Kostrikov et al., 2022). Gumbel regression, proposed in XQL (Garg et al., 2023), allows the computation of the soft Bellman optimality operator. In MXQL. (Omura et al., 2024), which applies the Maclaurin expansion
17

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
cheetal--un
fish swim
hopper-hop hop
hopper-stand
000
Auarange Retuur
Anerage Raturn
400
166.
20
20
75
Environment Steps
Enviromment Stops
Envirnmenn Stops
Enviroment: Steps
humancid-run
humanoid-stane stand
humanoid-walk
quadruped-run
Avenagy faturn
Average
Anszago
8.5
2.9
3.5
2.8
2.5
68
1.3
24
000
105
20
25
10
85
130
1.5
23
25
34
Ensimoment Steps
Fastronment
Steps
Emuinomment Strps
Ensinorment Steps
quadruped-walk
walker-run
Aatraga Retum
Average Retur
AQ-SAC
max( (2)
maxQ (5)
maxQ (10)
maxQ (20)
+00
Steps
Figure 15. The average return for each task when using max- max-backup with various action sampling numbers.
Table The average policy entropy in 10 DM Control tasks. 1.5k steps corresponds to shortly after training begins at 10k steps. These results suggest that a large in the early stages of training increases policy entropy and promotes exploration.
Method
15k steps
100k steps
200k steps
Fixed (0.5) (SAC)
6.78 + 0.30
6.23 1 0.39
5.97 1 0.38
Fixed (0.7)
7.41 + 0.36
6.40 + 0.40
6.11 + 0.38
Fixed (0.9) 8.28 + 0.44
6.94 + 0.39
6.52 + 0.38
to XQL, it is possible to transition from the soft Bellman optimality operator to the Bellman operator by varying the order of the expansion. Therefore, in this section, we conduct experiments by combining MXQL and AQ-L. The expansion order of MXQL was linearly decayed from Miii in steps of 2. Experiments were conducted in combination with TD3 and SAC. We tested Ainitt 4,8, 12 and adopted 4, which yielded the best average score. The hyperparameter 3 of MXQL was tested over 0.1, 0.5, 1, 2, 5, and we adopted 1, which resulted in the best average score. The overall average score in DM Control is shown in Figure 20, and the scores for individual tasks are shown in Figure 21. Even when Nimit = 4, the computation includes fourth-order terms, the loss function changes discretely, and the training is sensitive to B, which makes the learning unstable. Nevertheless, the performance is significantly improved when combined with TD3.
18

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
hopper-hop
humanoid-run
350
300
Average Return
Average Return
250
SAC
100
AQL (IM Anneal)
100
50
AQL (3M Anneal)
AQL (5M Anneal)
Environment Steps
Environment Steps
Figure 16. The average return of AQ-SAC when the annealing duration is varied. The dashed line represents the step count at which anncaling ends, after which learning proceeds with r 0.5, the same as the SAC.
humanoid-run
humanoid-walk
400
800
300
Average Return
Average Return
600
200
400
100
200
SAC AQ-SAC
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps
x10
Environment Steps
x10
Figure 17. The averate return of SAC and AQ-SAC. In AQ-SAC, training continues with 0.5 after 3M steps of annealing.
hopper-hop
hopper-hop
000
Q (std 0.05)
Average Return
Average Return
Q (std 0.1)
200
200
Q (std 0.5)
Q+V (std 0.05)
100
V (std 0.1)
Q+V (std 0.5)
Environment Steps
Environment Steps
(a) Annealed from
(b) Fixed at
Figure 18. The average scores for AQ-SAC trained using only the Q-function compared to AQ-SAC utilizing both the Q-function and V-function in stochastic hopper-hop hop environment. The stochastic environment is created by adding zero-mean Gaussian noise to the actions fed into the DM Control environment. The different colors represent the varying standard deviations of the Gaussian noise applied. The solid lines represent results obtained using only the Q-fuctioon, while the dotted lines indicate those obtained using both the Q-function and V-function. The left figure shows the results when annealed from initial value of 0.9, and the right figure shows the results when is fixed at 0.7.
19

---

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning
0.90
0.50
Average Return
0.75
Linear
Expl
Exp2
Sigmoid
AQ-TDS (MXQL)
0.60
200
AQ-SAC (MXQL)
TD3
0.55
SAC
Environment Steps
x10*
Environment Steps
x10
Figure 19. The annealing patterns of r used in the experiments.
Figure 20. The average scores across the 10 locomotion
tasks in DM Control.
AQ-TD3 (MXQL)
AQ-SAC (MXQL)
TD3
SAC
choetah-run
fish-swim
hopper-hop
hopper-stand
humanoid-run
Average Reeum
Average Return
Average Return
Average Return
Average
Environment Steps
Environment Steps
Environment Steps
Environment Steps
Environment Stops
humanoid-stand
humanoid-walk
quadruped-run
quadruped-walk
walker-run
Average Return
Avenage Return
Averago Return
Average Return
Average
10
Environment Steps
Environment Steps
Environment Steps
Environment Steps
Figure 21. The average for each task in DM Control.
20


Reviews:
[{"review_id": "OK65UNEmXs", "reviewer": "Reviewer_VRtg", "summary": "This paper proposed Annealed Q-learning to gradually transition from the Bellman optimality operator to the Bellman operator, leveraging early optimistic exploration while reducing overestimation bias during convergence. The approach is introduced via an illustrative example and performance is verified on a selection of continuous control tasks, where it performs favorably compared to established baseline algorithms. Overall, the approach is relatively simple and performs well on the tasks considered.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful and helpful comments. Below, we address each of the points raised:\n\n**Additional experiments (Longer runs on Humanoid-Walk/Run, DMC Dog tasks, action penalty env, D4PG baseline):**\n\n We appreciate the suggestion to conduct additional experiments. Due to limited computational resources, it is difficult to perform all the experiments; however, we have started running several of them and will share the results once they are complete.\n\n**Related work [1, 2]:**\n\n Thank you for pointing out the name collision with Amortized Q-learning. We will rename our method to AQ-L in the camera-ready version to avoid confusion. We also appreciate the pointer to [2], and we will add a discussion on this work as part of the related research on action discretization.\n\n**Potential extensions towards distributional RL:**\n\n As in QR-DQN [3], estimating Q-values for each discretized expectile and progressively reducing the target expectile may improve the consistency of the loss function during training. We believe this is a promising direction for extending our approach.\n\n**Line 409 – Counter-argument to \"accurate action selection becomes difficult\":**\n\nWhile discretization can be sufficient in certain tasks, it may lead to reduced sample efficiency in high-dimensional settings. For example, in the humanoid-run task, our method achieves a return of 250 within 3M steps, whereas [2] requires over 5M steps to reach the same performance. We will incorporate this discussion, including the effectiveness of discretization in lower-dimensional tasks, into the camera-ready version.\n\n**Figure 5 caption — \"locomotion tasks\" → \"manipulation tasks\":**\n\n Thank you for catching this error. We will correct it in the final version.\n\n\nOnce again, we sincerely thank you for the constructive and valuable feedback. Please feel \nfree to reach out if there are any further questions or points that require clarification.\n\n[1] T. Van de Wielde, et al. \"Q-learning in enormous action spaces via amortized approximate maximization.\" arXiv, 2020.\n\n[2] D. Ireland, and Giovanni Montana. \"Revalued: Regularised ensemble value-decomposition for factorisable markov decision processes.\" ICLR, 2024.\n\n[3] Will Dabney, Mark Rowland, Marc G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.", "claims_and_evidence": "The approach is fairly straight-forward, introduced via an illustrative example and validated on continuous control benchmarks, with insightful ablations in the appendix. The provided evidence supports the claims, while it could be nice to run individual tasks to convergence to further strengthen the paper.", "methods_and_evaluation_criteria": "The evaluation criteria are in line with the literature. Potential areas for improvement are mentioned below.", "theoretical_claims": "The paper focuses more on quantitative validation without specific theoretical claims.", "experimental_designs_or_analyses": "The experimental validation is in line with prior works and shows the method comparing favorably. The following would further strengthen the paper:\n- Longer runs on Humanoid-Walk/Run would be helpful to actually judge convergence.\n- Have you tried to run experiments on DMC Dog tasks? Would be interesting to see regarding scalability?\n- It would be interesting to see how the AQ-XXX agents compare to their XXX counterpart in action-penalized environments. Would they converge to local cost-minimization optima more quickly and get stuck there?\n- Figure 4: SAC has commonly struggled on DMC tasks and AQ-SAC’s improvements are great to see. Please also add a D4PG baseline, as this would provide a strong reference (original DMC reference agent).", "relation_to_broader_scientific_literature": "The general literature review is good, please see below for additional papers to consider.", "essential_references_not_discussed": "- Line 243: please revise the naming to avoid collision with Amortized Q-learning (AQL) in [1]. AQ-XXX should be fine, so maybe go for AQ-L to avoid an exact match with AQL?\n- Please also check [2] for an alternative approach to reducing overestimation bias in Q-learning for (discretized) continuous control\n\n**Additional references:**\n\n[1] T. Van de Wielde, et al. \"Q-learning in enormous action spaces via amortized approximate maximization.\" arXiv, 2020.\n\n[2] D. Ireland, and Giovanni Montana. \"Revalued: Regularised ensemble value-decomposition for factorisable markov decision processes.\" ICLR, 2024.", "other_strengths_and_weaknesses": "- Having an adaptive method for selecting the annealing constant would significantly strengthen the contribution", "other_comments_or_suggestions": "- Line 409: a potential counter-argument to “accurate action selection becomes difficult” is that highly discretized control appeared to be sufficient for the tasks evaluated in this paper\n- Figure 5, caption: locomotion tasks —> manipulation tasks", "questions_for_authors": "- Have you thought about potential extensions towards “distributional RL” methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bQ39zXQr2I", "reviewer": "Reviewer_dvAE", "summary": "This paper proposes to modify the Q-learning update in SAC and TD3 with an expectile loss like IQL, where the crucial proposition is to anneal the value of expectile $\\tau$ from values close to 1 (representing max Q update) to 0.5 (representing SARSA update). The paper claims that \"overestimation\" in early stages of learning is beneficial for exploration and is shown to accelerate learning. However, at the end of training, reducing the overestimation bias is claimed to be desirable.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the highly valuable feedback. The concerns are comprehensively covered in the \"Claims and Evidence\" section. Below, we provide responses to each of the specific points mentioned there.\n\n**1. Why use annealing between max-Q and SARSA? If max-Q causes overestimation, why not reduce it directly? And if SARSA is suboptimal or slow, why use it at all?**\n\nMax-Q introduces an overestimation bias, but as discussed later, this can encourage exploration. Therefore, reducing this bias directly may not always be desirable.\n\nThe reviewer asks, “If SARSA leads to suboptimal or slow learning, why use it at the end of training?” However, we emphasize that SARSA’s value estimates are not suboptimal. As an on-policy method, SARSA avoids the distribution mismatch between the actions used for target estimation and those actually taken. Furthermore, since SARSA does not involve Max, it introduces less bias. Although it may learn more slowly, it provides more accurate value estimates, as shown in Section 3.2.\n\n**2. How early max Q update leads to better exploration. There is no experiment that validates this claim.**\n\nWe agree and appreciate this point. To address it, we measured policy entropy during early training. The table below shows average entropy over DMC tasks for SAC (Fixed (0.5), (0.7), and (0.9)), corresponding to SAC with expectile loss with $\\tau = 0.5, 0.7, 0.9$. As shown in Figure 12, these methods exhibit overestimation bias in the order: Fixed (0.9) > Fixed (0.7) > SAC. The entropy follows the same trend, suggesting higher bias leads to broader exploration.\n\nThis supports the intuitive hypothesis that overestimation can lead to suboptimal actions due to inflated Q-values, promoting  exploration. Prior work (e.g., Section 3 of [1]) also demonstrates that overestimation can be beneficial in tasks where exploration is important, while it degrades performance in tasks where exploration is undesirable. These results support the claim that overestimation bias promotes exploration.\n\nWe will include these points in the camera-ready version to clarify the link between exploration and overestimation.\n\n\n[1] Q. Lan, Y. Pan, A. Fyshe, and M. White. Maxmin q-learning: Controlling the estimation bias of q-learning. In ICLR, 2020.\n\n**Table**: The average policy entropy in DM Control tasks. 15k steps corresponds to shortly after training begins at 10k steps.These results suggest that a large $\\tau$ in the early stages of training increases policy entropy and promotes exploration.\n| Method           | 15k steps        | 100k steps       | 200k steps       |\n|------------------|------------------|------------------|------------------|\n| SAC (Fixed (0.5))  | 6.78 ± 0.30      | 6.23 ± 0.39      | 5.97 ± 0.38      |\n| Fixed (0.7)      | 7.41 ± 0.36      | 6.40 ± 0.40      | 6.11 ± 0.38      |\n| Fixed (0.9)      | 8.28 ± 0.44      | 6.94 ± 0.39      | 6.52 ± 0.38      |\n\n\n**3. Why not just use max-Q throughout, since overestimation can be mitigated by exploration? Why is it necessary to balance max-Q and SARSA-style updates?**\n\nWhile exploration can reduce overestimation, it doesn't eliminate it entirely. As shown in Figures 2 (right) and 12, higher $\\tau$ values lead to greater final bias, exceeding that of SAC, a SARSA-style method. AQ-SAC starts with $\\tau = 0.9$ to encourage exploration but gradually reduces bias to SAC's level, highlighting the benefit of balancing max-Q and SARSA updates. Fixed $\\tau$ settings (e.g., 0.9, 0.95) perform poorly (Table 2), indicating that persistent overestimation harms optimality. While overestimation tends to diminish with more data, prior studies [1,2,3] have shown that actively suppressing it can significantly enhance sample efficiency. Balancing both perspectives, our method offers a natural compromise.\n\n[2] Deep reinforcement learning with double Q-Learning. Hado van Hasselt, Arthur Guez, David Silver. AAAI 2016.\n\n[3] Addressing Function Approximation Error in Actor-Critic Methods. Scott Fujimoto, Herke van Hoof, David Meger. ICML 2018.\n\n\n\n**4. If a well-chosen fixed τ performs similarly to annealing, is the added complexity of annealing really necessary?**\n\n We acknowledge that a carefully selected fixed $\\tau$ can yield good performance. However, as shown in Table 2, performance becomes highly sensitive to $\\tau$ when it increases beyond a certain threshold, and performance can deteriorate significantly due to the increased bias.\n\nIn RL tasks where sample efficiency is critical and many hyperparameters are inherently delicate, it is often impractical to finely tune all hyperparameters. In this context, our annealing strategy, which provides robustness against hyperparameter sensitivity, is a meaningful advantage.\n\nOnce again, we sincerely appreciate the thoughtful and constructive feedback. It has helped us clarify and strengthen the motivation and evidence behind our proposed method. Please feel free to let us know if you have any further questions or concerns.", "claims_and_evidence": "There is no theoretical or empirical evidence of the two key hypotheses:\n1. why annealing should be done at all, as opposed to either of the two objectives. For instance, if max Q leads to overestimation, then one could use methods to reduce overestimation. Why do this specific kind of annealing to reduce overestimation? Similarly, if SARSA update leads to suboptimal or slow learning, why is it desirable to do it at all at the end of training?\n2. how early max Q update leads to better exploration. There is no experiment that validates this claim.\n\nTherefore, while the IQL-inspired method performs well in the experiments, the motivation behind the two key ideas of IQL-inspiration and annealing seem to be made up. Ideally, these ideas should be empirically validated to clearly delineate where exactly the gain is coming from.\n\n----\n\nFurthermore, in Section 3.2, the overestimation due to Q-learning is explored, but it is not clear whether this overestimation would change the optimal policy to the extent that one would prefer the Bellman operator instead of the Bellman optimality operator. In fact, we care about speed of convergence and optimality, and the overestimation should be self-resolved due to exploration in online RL (as mentioned in the paper) — so, it is not clear why one would not just choose to do max-Q like RL altogether. There needs to be a clear justification of the idea of balancing between max-Q and SARSA-style update.\n\n----\n\nThe effectiveness of annealing in Section 4.3 is incorrectly claimed. In Table 2, the difference between Fixed (0.7) and Annealed (0.9) is so small that one might just prefer the fixed IQL method instead of the added complexity of annealing. Concretely, if we were to run two experiments: one with annealing from various values and one with fixed with various values, we would arrive at almost the same optimal result. Then, there is no meaningful benefit of the added complexity of annealing.", "methods_and_evaluation_criteria": "No clear justification for why a fixed $\\tau$ is not enough and why one needs to do annealing to balance the max-Q and SARSA objectives, instead of a weighted balancing.", "theoretical_claims": "1. The paper lacks a strong theoretical justification for the problem that annealing solves, and why annealing is a sound method to solve this problem.\n2. There is no convergence or improvement proof that shows that the proposed change in the algorithm is a valid change that preserves stability and leads to better stability or convergence speed.", "experimental_designs_or_analyses": "1. The experiments do not justify the key problem that is present that is solved by the annealing. While there is a clear benefit offered due to the proposed method in the experiments, the source of this improvement is not well motivated or empirically justified.\n2. The empirical result of Fixed (0.7) is quite strong, and it would be worthwhile to explore why that works as well. Annealing adds another layer of complexity that achieves the same effect as Fixed (0.7), while complicating the analysis to find the core reason for improvement.", "relation_to_broader_scientific_literature": "The idea extends IQL algorithm from offline RL to online RL, which already results in good results: Fixed (0.7). The addition of annealing does not bring any further statistically significant improvements.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths\n- The observed gains in the experiments are statistically significant.\n\n## Weaknesses\n- The reasons for the gains are not attributed and motivated convincingly.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- L268: \"Overestimation increases the chance of the agent selecting overestimated actions, correcting them in the process, and thus broadening the range of actions tried.\"\nHow does it broaden the range of actions tried?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "denKEztGvp", "reviewer": "Reviewer_Q7Mh", "summary": "The manuscript proposes a gradual transition from a Bellman optimality operator to a Bellman operator by using a linearly annealed parameter to blend two Q-target estimates. I have noticed that a previously published paper introduced a BEE operator that similarly combines an exploitation-based update from historical best actions with an exploration-based update from the current policy using a λ parameter. Although the two works target different bias issues—overestimation in this manuscript versus underestimation in the earlier work—their underlying methods and even choices of algorithm backbone show striking similarities. Moreover, the manuscript does not discuss or reference this earlier work, making it challenging to distinguish its unique contributions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for pointing out that interesting and important prior work.\n\nBEE and AQL share a similarity in that both utilize the Bellman optimality operator based on in-sample maximization as well as the Bellman (expectation) operator. As the reviewer correctly noted, the key difference lies in the motivation. While BEE focuses on addressing underestimation, our approach emphasizes the potential benefits of overestimation in promoting learning, as supported by our preliminary experiments. Moreover, our method introduces a scheduling strategy that gradually decays optimality over time—a scheduling not explored in BEE.\n\nAdditionally, unlike BEE, which combines the outputs of two separate Q-functions using different operators, our approach transitions gradually from one operator to the other. This allows us to maintain a single Q-function throughout training, simplifying the overall design.\n\nTherefore, our method can be seen as a simplification of BEE that introduces a novel motivation and scheduling mechanism to decay optimality, while relying on a single Q-function.\n\nWe agree that the relationship with BEE is highly relevant, and we will revise our camera-ready version to explicitly discuss this connection and clarify our contributions accordingly.\n\nThe hyperparameters used in our experiments are listed in Appendix B. To further improve clarity, we will also include an overview table summarizing them in the camera-ready version.\n\nOnce again, we sincerely appreciate the valuable feedback. We hope that the inclusion of this discussion in the paper addresses your concerns. If not, we would greatly appreciate it if you could let us know what further explanations or revisions would be needed to improve the score.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "See the next part.", "essential_references_not_discussed": "The manuscript proposes a gradual transition from the Bellman optimality operator to the Bellman operator by linearly annealing a parameter over time, thereby blending two Q-target estimates to accelerate learning while mitigating overestimation bias. The BEE paper[1], on the other hand, introduces an operator that linearly combines two different Bellman updates: one that exploits historically best-performing actions and another that uses the current policy for exploration, with the balance controlled by a trade-off parameter (λ). Although one work primarily focuses on reducing overestimation and the other on addressing underestimation in later training stages, both share a very similar underlying motivation—improving the accuracy of Q-value estimates within off-policy actor-critic frameworks by fusing two complementary update strategies. Moreover, the algorithmic implementation in both works is remarkably similar: each computes two separate Q-targets and then combines them using a parameter that is scheduled over time (BEE discussed some different scheduling mechanisms in its appendix), and they even make unexpectedly similar choices in terms of the algorithm backbone and experimental settings. This striking resemblance in both method and experimental design raises serious concerns regarding the originality of the manuscript, especially as it does not reference or discuss the BEE paper, leaving readers without a clear understanding of how the contributions of the manuscript differ or extend the existing work.\n\nIn light of these concerns, I recommend that the authors provide a detailed comparative discussion to clarify the unique contributions of their work. Specifically, they should explicitly highlight any theoretical or practical refinements that distinguish their approach from the one presented in the BEE paper. Without such a discussion, the high degree of overlap in core methodology, and even backbone selection significantly undermines the manuscript’s originality and its contribution to the field.\n\n[1] Ji, T., Luo, Y., Sun, F., Zhan, X., Zhang, J., & Xu, H. (2024, July). Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic. In International Conference on Machine Learning (pp. 21672-21718). PMLR.", "other_strengths_and_weaknesses": "The experimental results are well-executed and provide strong empirical support.", "other_comments_or_suggestions": "The main contribution of this manuscript could be considered as building upon the BEE paper by designing a more adaptive and intelligent trade-off mechanism for the $\\lambda$ (or the $w$ parameter in this manuscript), allowing it to evolve naturally throughout the training process. If the manuscript seriously acknowledges this connection and revises its contribution accordingly, it would be a solid and acceptable work. Given that the experimental results are well-executed and provide strong empirical support, this refinement could make the contribution even more valuable.", "questions_for_authors": "I find it lacks an overview table of the hyperparameters.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9OpVxKjZwT", "reviewer": "Reviewer_KfGB", "summary": "This paper is mainly concerned about improving the online policy learning. The paper is especially focusing on the respective advantages and disadvantages of Bellman (SARSA-like) and Bellman optimality (Q-learning-based) operators. Through training in a simple discrete action environment, they empirically showcase that Bellman optimality operator accelerates learning, while Bellman operator is less biased. Motivated by such balance, they use a linear combination of both target values from the two groups of methods to be their actual target in training. And to approximate taking maximum for Q-learning part, an expectile loss with linearly decayed threshold value is applied. The performance of such proposal is tested in several environments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedback. Please find our responses below.\n\n**What is the justification for the linear decay in w?**\n\n As shown in Table 4 and Figure 18, we experimented with several annealing patterns and found that linear decay achieved sufficiently good performance. While other patterns or adaptive strategies could also be viable design choices, we prioritized simplicity in this work.\n\n**Why in the cheetah-run and humanoid-stand environments, are the proposed methods less efficient than some oracle methods? Especially, it seems that SAC is still the best in the latter one?**\n\nIn the cheetah-run environment, even the base algorithms such as SAC and TD3 converge relatively quickly to high performance. This suggests that sufficient exploration and improvement in policy optimality are already achieved without the need for further bias-induced acceleration. In such cases, the additional bias introduced by AQL may outweigh its benefits, leading to lower overall efficiency. This interpretation is supported by Figure 13, where reducing the initial value of $\\tau$ helps mitigate the bias and results in convergence speeds comparable to SAC.\n\nA similar explanation applies to the humanoid-stand environment. Here, a high initial $\\tau$ value of 0.9 likely introduces excessive bias, again offsetting the intended benefits of AQL. When $\\tau$ is reduced to 0.7 or 0.8, the convergence becomes comparable to SAC, which further supports this hypothesis.\n\nThese observations indicate that the inefficiency of our method in these environments can be mitigated through appropriate tuning of $\\tau$. Importantly, regardless of the $\\tau$ setting, the final returns achieved by our method remain competitive highlighting the robustness of the proposed approach.\n\n**For the results of biases in Figure 12, how about annealed $\\tau$?**\n\n The AQ-SAC results in Figure 12 correspond to the case where $\\tau$ is annealed from 0.9 . In AQ-SAC, the initial bias increases, which can contribute to exploration, and then gradually decreases to a level comparable to the SARSA-based approach (SAC) toward the end. This behavior aligns with our intention. Similar trends were observed when annealing from different initial values of $\\tau$, and we will include these additional results in the camera-ready version.\n\n**Line 212–213: what is the expectation over (where is the randomness)?**\n\n The expectation is over the randomness in the Q-function, specifically the variable $\\epsilon$ mentioned in the following sentence. We will add a clearer explanation to improve readability.\n\nWe sincerely appreciate your helpful feedback once again. We hope that our responses have sufficiently addressed your concerns. If there are any remaining issues, we would greatly appreciate your guidance on what further explanations or revisions would be necessary to improve your score.", "claims_and_evidence": "Although mainly using empirical evidence, the efficiency of the proposal is quite convincing.", "methods_and_evaluation_criteria": "- What is the justification for the linear decay in $w$?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- The motivating examples in Section 3 provides an intuition of the balance between two operators.\n- Why in cheetah-run and humanoid-stand environments, the proposed methods are less efficient than some oracle methods? Especially it seems that SAC is still the best in the latter one?\n- For the results of biases in Figure 12, how about annealed $\\\\tau$?\n- Different annealing patterns are tested for the final choice of linear decay.", "relation_to_broader_scientific_literature": "Such method provides a promising approach to balance between training efficiency and bias control.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "> **Clarity**\n- Line 212-213: what is the expectation over (where is the randomness)?", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H4L2cZO1Ru", "reviewer": "Reviewer_v2xt", "summary": "The paper studies the contrast and connections between Bellman operator and Bellman optimality operator in online Reinforcement Learning (RL). Bellman operator is widely used for on-policy actor-critic methods but often hurts to sample efficiency. Bellman optimality operator, on the other hand, is widely used for discrete RL but suffers from overestimation bias in target values. The paper bridges the gap between the two operators by leveraging IQL's expectile value loss. Specifically, authors propose to gradually transition from the optimality operator to the traditional SARSA operator by annealing the expectile coefficient $\\tau$. Contrary to IQL, updates only utilize the $Q$ value function in the target instead of the state-value function. Experiments on a small toy MDP, DM Control tasks and MetaWorld benchmark validate that annealing addresses overestimation and provides sample efficiency.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the highly constructive feedback. We provide several responses below.\n\n**Evaluation using optimality gap**\n\n Thank you for the suggestion. As we understand it, the optimality gap refers to how much the agent's performance falls short of a target score, such as a human-level or oracle performance. In our tasks, however, such target scores are not always clearly defined. If the maximum achievable score in the task (e.g., 1000 in DM Control) is considered as the target, then we assume the optimality gap corresponds to the difference between 1000 and the agent’s average score. If this interpretation is incorrect, we would greatly appreciate clarification.\n\n**Evaluating overestimation via Q-value gap**\n\n In Figure 12, we report the gap between the estimated Q-values and the actual Q-values computed via Monte Carlo return, across our proposed method and several ablations. In AQ-SAC, our proposed method, we observe that the bias is initially larger—potentially encouraging exploration—but gradually decays to a level comparable with the SAC (SARSA-based method) toward the end. This behavior aligns with our intended design.\n\n\n**Theoretical basis and impact of the annealing scheme**\n\nIf we focus solely on optimality, when $\\tau$ = 0.5, the max operator in Equation (3) of our paper is replaced by an expectation, turning the inequality into an equality. As a result, no bias is introduced, allowing for a more optimal value estimation. On the other hand, $\\tau \\simeq 1$ corresponds to the max operator, which introduces bias. Therefore, we anneal $\\tau$ toward 0.5 during training to reduce bias and eventually enable the estimation of an optimal policy.\nIt is an open question, as the reviewer correctly pointed out, how the max operator and bias affect the speed of improvement, and it is theoretically challenging to characterize this relationship. The table included in our response to Q2 of reviewer dvAE provides additional empirical evidence suggesting that bias can promote exploration, thereby highlighting the effectiveness of our method.\n\n\nThank you again for the valuable feedback. If you have any further concerns, we would be happy to address them.", "claims_and_evidence": "Please refer to strengths and weaknesses.", "methods_and_evaluation_criteria": "Please refer to strengths and weaknesses.", "theoretical_claims": "Please refer to strengths and weaknesses.", "experimental_designs_or_analyses": "Please refer to strengths and weaknesses.", "relation_to_broader_scientific_literature": "Please refer to strengths and weaknesses.", "essential_references_not_discussed": "Please refer to strengths and weaknesses.", "other_strengths_and_weaknesses": "### Strengths\n\n* This paper is exceptionally well written and easy to understand.\n* The work investigates the annealing scheme thoroughly using comprehensive experiments.\n* The paper tackles important problems of sample efficiency and overestimation bias by proposing a simple idea.\n### Weaknesses\n\n*  **Evaluation:** While the paper studies and analyzes the annealing scheme, its evaluation and overall comparative presentation could be improved. Authors could utilize the rliable library [1] or metrics to compare between different baselines. For instance, while authors compare the IQM, the optimality gap should also be compared and presented. Similarly, authors should also consider evaluating the gap between actual and estimated target $Q$ values to evaluate overestimation bias for atleast one of the DM Control tasks.\n* **Contribution and Novelty**: While the paper makes a solid empirical contribution, its theoretical understanding remains an open question. Could the authors elaborate on how the annealing scheme affects the ability of IQL in extracting the optimal policy? Additionally, authors could discuss how the annealing scheme is different from prior efforts that aim to address overestimation bias and sample efficiency. In its current form the work only ablates and tunes $\\tau$, a trick partially explored by IQL [2].\n\n[1]. Agarwal et al, Deep Reinforcement Learning at the Edge of the Statistical Precipice, NeurIPS 2021.\n[2]. Kostrikov et al, Offline Reinforcement Learning with Implicit Q-Learning, ICLR 2022.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please refer to strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}]

Author rebuttal:

Meta-review: This paper studies the effective integration of the Bellman optimality operator into actor-critic algorithms for online reinforcement learning in continuous action spaces, aiming to balance the fast but biased Q-learning with the stable but slower SARSA-based updates. To address this trade-off, the study introduces an annealing strategy that gradually shifts from the Bellman optimality operator to the standard Bellman operator, using expectile loss to enable smooth interpolation. The proposed Annealed Q-learning (AQL) is further implemented with TD3 and SAC to experimentally improve sample efficiency while mitigating overestimation bias.

In general, the proposed method appears to be simple yet effective. While some reviewers raised concerns about the lack of theoretical justification and comparisons to closely related work (notably the BEE paper), the authors addressed these concerns in their rebuttal by clarifying distinctions, extending discussions on prior work, and promising additional empirical comparisons in the camera-ready version. Furthermore, as acknowledged by reviewers, the empirical evaluation appears to be solid and thorough, demonstrating AQL’s effectiveness across a range of standard benchmarks. Overall, this paper introduces a timely and practically useful contribution to the deep RL research community.
Decision: Accept (poster)