AsymQ: Asymmetric Q-loss to mitigate overestimation bias in off-policy reinforcement learning
Authors: Qinsheng Zhang
Arjun Krishna
Sehoon Ha
Yongxin Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
estimation bias

Abstract: It is well-known that off-policy deep reinforcement learning algorithms suffer from overestimation bias in value function approximation. Existing methods to reduce overestimation bias often utilize multiple value function estimators. Consequently, these methods have a larger time and memory consumption. In this work, we propose a new class of policy evaluation algorithms dubbed, \textbf{AsymQ}, that use asymmetric loss functions to train the Q-value network. Departing from the symmetric loss functions such as mean squared error~(MSE) and Huber loss on the Temporal difference~(TD) error, we adopt asymmetric loss functions of the TD-error to impose a higher penalty on overestimation error. We present one such AsymQ loss called \textbf{Softmax MSE~(SMSE)} that can be implemented with minimal modifications to the standard policy evaluation. Empirically, we show that using SMSE loss helps reduce estimation bias, and subsequently improves policy performance when combined with standard reinforcement learning algorithms. With SMSE, even the Deep Deterministic Policy Gradients~(DDPG) algorithm can achieve performance comparable to that of state-of-the-art methods such as the Twin-Delayed DDPG (TD3) and Soft Actor Critic~(SAC) on challenging environments in the OpenAI Gym MuJoCo benchmark. We additionally demonstrate that the proposed SMSE loss can also boost the performance of Deep Q learning (DQN) in Atari games with discrete action spaces.

Under review as a conference paper at ICLR 2023
ASymQ: ASYMMETRIC Q-LOSS TO MITIGATE OVER- ESTIMATION BIAS IN OFF-POLICY REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
It is well-known that off-policy deep reinforcement learning algorithms suffer from overestimation bias in value function approximation. Existing methods to reduce overestimation bias often utilize multiple value function estimators. Con- sequently, these methods have a larger time and memory consumption. In this work, propose
class of policy evaluation algorithms dubbed, AsymQ. that use asymmetric loss functions to train the Q-value network. Departing from the symmetric loss functions such
squared error (MSE) and Huber loss on the Temporal difference (TD) error, we adopt asymmetric loss functions of the TD-error impose higher penalty on overestimation error. We present one such AsymQ loss called Softmmx MSE (SMSE) that can be implemented with minimal modifications to the standard policy evaluation. Empirically, we show that using SMSE loss helps reduce estimation bias, and subsequently improves policy performance when combined with standard reinforcement learning algo- rithms. With SMSE, even the Deep Deterministic Policy Gradients (DDPG) al- gorithm can achieve performance comparable to that state-of-the-ar methods such the Twin-Delayed DDPG (TD3) and Soft Actor Critic (SAC) chal- lenging environments in the OpenAl Gym MuJoCo benchmark. We additionally demonstrate that the proposed SMSE loss also boost the performance of Deep Q learning (DQN) in Atari games with discrete action spaces.
INTRODUCTION
Learning an accurate value function in Deep Reinforcement Learning (DRL) is crucial; the value function of policy is not only important for policy improvement (Sutton & Barto 2018) but also useful for many downstream applications such as risk-aware planning (Kochenderfer) 2015) and goal-based reachability planning (Nasiriany et al. 2019). However, most off-policy DRL algorithms accompanied by estimation bias in policy evaluation and how to remove this estimation bias has been a long-standing challenge. In this work, we revisit the estimation bias problem in off- policy DRL. from new perspective and propose lightweight modification of the standard policy evaluation algorithm to mitigate estimation bias.
Value estimation bias in DRL Thrun & Schwartz (1993) firstly shows that maximization of noisy value estimation consistently induces overestimation bias in Q-learning, where the learned value function overestimates the learned policy, i.e., the prediction from the learned value function is higher than the ground truth value of the policy. Several methods have been proposed to reduce estimation bias policy evaluation and policy improvement. Hassell (2010); Van Hasselt et al. (016)) propose double Q-learning, which trains two independent estimators to suppress overesti- mation. In the continuous state-action space setting, FFujimoto et al. (2018) shows the existence of overestimation in popular Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al.) 2015) and proposes Twin Delayed Deep Deterministic Policy Gradient (TD3) to alleviate the overestimation issue, which utilizes the minimum value estimation from two critic networks to fit Q value function. However, these approaches that involve multiple value function estimators together with a minimum
operator, may succumb to underestimation bias (Ciosek et al. 2019: Pan et al.) 2020). where the value prediction of the learned function is lower than the real policy performance.
Lyu et al. (2022):
Wu et al. 20200; Wei al. (2022) Kuznetsov et al. (2020): Chen et al. (2021) Liang et al. 2022

---

Under review as a conference paper at ICLR 2023
Lee et al. ) propose new critic update schemes to reduce the estimation bias. However, all these methods require multiple actors or an ensemble of critics and usually involve other convoluted tricks that often incur additional computation and memory overheads to improve policy performance. In this work, we explore an efficient approach that reduces value estimation bias without incurring extra computational costs.
Loss function in policy evaluation The policy evaluation in DRL is typically based on the Bellman update equation (Bellman (966) where the discrepancy between the predicted and target values are minimized. The target value is the combination of the immediate reward and subsequent value function prediction on future states. There are two main components in policy evaluation based on Bellman temporal difference (TD) learning, the fitting target that acts as supervised signals to train neural networks and the loss function that serves as an objective metric. Most of the existing work in reducing estimation bias focuses on the first component, and typically try to construct more robust target value, such as a lagged target network (Mnih et al. 2015) and ensemble multiple value networks (Fujimoto et al. 2018). The choice of the loss function used for fitting somehow receives much less attention. In practice, most RL algorithms choose symmetric mean square error (MSE) Huber loss (Patterson et al. (2022) as metric in fitting value functions. Once zero MSE loss is reached for each state, the prediction of the value network can match policy performance exactly.
In this work, we propose a novel approach based on modifying the loss function to control estimation bias in DRL. We discover that the optimization landscape used to train value networks, governed by the loss functions, plays a crucial role in policy evaluation. A proper choice of the value-fitting loss function can effectively control estimation bias. In particular, we show asymmetric functions can be used for policy evaluation to reduce estimation bias and propose a class of algorithms called AsymQ. We specifically evaluate one family of AsymQ loss functions called SoftMax MSE (SMSE) to learn the Q value function in both continuous and discrete action environments, but we validate the benefit of using other AsymQ loss functions as well. We find asymmetric loss functions can inject inductive bias to the learning process and thereby control estimation bias present in DRL. policy evaluation updates. We highlight the intuition behind our approach in Fig where asymmetry can assign different weights for both overestimated and underestimated states and help alleviate bias in policy evaluation. Notably, compared with existing methods (Hasselt 2010: Fujimoto et al. 2018), our approach needs only one critic and actor and negligible overhead computational and memory cost.
We summarize our contributions as follows: (1) We show asymmetric loss functions can be used for learning the value function, and introduce simple asymmetric loss for policy evaluation, namely, Softmax MSE (SMSE) parameterized by temperature parameter, that can be easily combined with existing RI. algorithms. (2) We further propose an auto-tuning algorithm for the temperature to reduce the burden of tuning parameters for our proposed approach. (3) We show that SMSE can significantly reduce estimation bias and improve the performance of popular baseline algorithms such as DDPG on MuJuCo environments and DQN on Atari games. To the best of our knowledge, this is the first algorithm that achieves such competitive performance without the need for other tricks such as multiple critic networks and crafted exploration methods that often incur extra computational and memory costs.
PRELIMINARIES: VALUE-BASED DEEP REINFORCEMENT LEARNING
The interaction of the RL agent with the environment can be formalized as a Markov Decision Process (MDP). The MDP is represented as a tuple (S, A. p. r, r.)). where S,A represents the set of states and actions respectively, represents the transition probabilities, r(s, a) the reward function, and y represents the discount factor. The goal of an RI. agent such that the expected return J(=s) E 0o
learn a behavior policy ne(a, (st) 2 7 TT is maximized.
In RL the value function usually learned based on temporal difference (TD) learning, an update scheme based on the Bellman equation /Sutton & Barto (2018); Bellman| (1966). The Bellman equa- tion formulates the value of a state-action pair (8, a) in terms of the value subsequent state-action pairs (s', a'):
(" = r(s, a) FE aa (=" a')), ~ (T-1s)),
2

---

Under review as a conference paper at ICLR 2023
Overestimation Gradient
Symmetric Loss
Qcurrent
Q targot
Max Bias
Asymmetric Loss
Underestimation Gradient
Figure 1: Effect of asymmetric loss on estimation bias in DQN and DDPG. Aside from the gradients from overestimated and underestimated state-action pairs, there exists an additional bias induced by the maximum operator and random noise in policy improvement (Thrun & Schwartz, 1993). Symmetric loss functions assign the same weights for the gradients from overestimated and under- estimated state-action pairs thereby leading to overestimation in Q learning and DDPG. Whereas asymmetric loss functions can assign different weights for these even when states-action pairs have the same absolute TD error. The figure the right illustrates the intuition for why AsymQ loss functions can help counteract the overestimation bias.
where Q" a E y'r(st, at) (s, a), denotes the expected return when performing action in state
0 and following policy æ for all the subsequent states. Q" is also known as the value function or the critic.
policy can be represented and learned explicitly as done actor-critic methods or inferred implic- itly from the value function in methods such as Q-learning. In continuous control, the parameterized policy Ta can be updated through the deterministic policy gradient algorithm Silver et al. (2014):
VoJ(Fs) V (s, a)la=n((s) (s)].
(2)
In DRL, the value function is estimated by a neural network, which we denote as Qo a) parame- terized by network parameters 0. Based temporal difference learning, existing works update by minimizing the mean squared TD-error, 0 + 0 a VvE... [5(s, a)² where the TD-error 8(s, a) defined by
8(8,a) a) -rsrs a) - QQr (s',
(s',a') a') - Qo(s, a), ~ TT(('))
(3)
Here Qø is usually chosen frozen target network. The weights O' are usually updated in an exponential moving average approach, i.e., 0' + 70 + (0--) +)0' for some < 1.
We start our investigation by revisiting estimation bias and necessary conditions for fitting the Q value function in [Section 3 and show there exist many asymmetric loss functions that also satisfy these conditions. We demonstrate the advantage of using asymmetric loss functions by examining specific family of AsymQ loss functions that we term as SoftmaxMSE. (SMSE) in Section 4 and provide more empirical results and ablation studies in [Section 5
ESTIMATION BIAS AND ASYMMETRIC LOSS FUNCTION
Estimation bias and supervised target The overestimation bias in DRL believed to due to its greedy update and the noise present in training. The overestimation issue was initially described in Thrun & Schwartz (1993), where the value estimation updated with a greedy target y = r + Y maxa Q(s', a'). When there exists noise approximation error in the Q value function, the maximum over it along with the enror will generally be greater than the true maximum, namely, E. Imax, (Q(s', aa',, a') + <)] max, Q(s', a'). Similar results are also found in the continuous setting. We briefly summarize these results the following lemma (Fujmoooo et al. 2018) (more details provided in Appendix A).
3

---

Under review as a conference paper at ICLR 2023
Lemma 1. For pair of actor-critic {Te, Qo). assume step update based on policy gradient results in a new policy Tz. Under mild assumptions, the value estimate will be overestimated: B[Qu(s,Tg(s)) (x()) 2 (s,
The overestimation bias turns out to be a serious problem in DRL. The bias may accumulate over iterations and result in an inaccurate value function. Thus, an inaccurate value function will ex- acerbate sub-optimal actions and lead to poor policies. To alleviate overestimation bias, a popular approach is to construct robust supervised signals from multiple crities to fit the Q function. For instance, TD3 takes minimum of two crities to update the target in TD-error as
6(,, = - Qo(s,))
= r(s, a) + min(Qo, (s', (T((s))))))) Qo (s', (s())))
(4)
which can effectively suppress overestimation. However, this pessimistic approach often induces underestination bias (Ciosek et al. 2019 Pan al. 2020) as demonstrate in Fig A key challenge in tackling this issue is that we do not know during training if a particular estimate is over- estimated as computing the ground truth Q-value is expensive, therefore, we have to rely on proxy TD error signals during training. If the supervised target value is close to ground truth values then overestimated state-action pairs have negative TD errors, while underestimated pairs associated with positive TD errors.
Estimation bias and loss function Another important component of policy evaluation is the loss landscape. MSE of 'TD errors is the default option for most existing works in learning value func- tions. Based on Bellman update formula (Bellman 1966), zero TD-error, i.e., 8(s, a) 0 for every state-action pair, indicates that the value network can predict value function perfectly. For MSE Q loss, besides its simple formula, the global minima will coincide with a well-fitted value functionn that has zero TD error. The magnitude of MSE fitting loss quantifies the discrepancy between the learned Q value function and the value prediction target, such as Eq (4) However, the constructed supervised target usually depends on bootstrapping as in Eq (4) and the loss is not the distance from the ground truth value function of the policy. fact, due to nonconvexity and the limited capacity of neural networks, zero TD error is almost impossible to reach. In practice, finite even one update step of stochastic gradient descent is employed to optimize value networks after each policy update in DRL.
However, symmetric losses like MSE or Huber are not the only options for optimizing the value network that zero TD error is reached at its global minimum. In this work, we investigate loss functions that are asymmetric. Compared with MSE loss, in which overestimated states with neg- ative TD error are indistinguishable from underestimated states with positive TD error, asymmetric loss functions can create different loss landscapes for statc-action pairs with positive and negative TD errors.
Due to its asymmetry with respect to state-action pairs with positive and negative TD-error 6, we coin our new losses as asymmetric Q (AsymQ) loss functions. Among many choices for AsymQ loss functions, we start our investigation with a simple function that is obtained by making a small modification to the standard MSE function. From MSE CMSE (MSE(().)) Di=l 406. we introduce asymmetric loss whose global minimum coincides with zero TD-error
B LSMSE ((0:),(( Σ
8: s.g. (Sot(maxxx
T 11111 =
Σ s.g.
B
Σ exp( 6 10?
exp(- (-))
(5)
where the operator s.g.[=] denotes the stop gradient operator during backpropagation. The design of this asymmetric loss has several merits. First, it parameterizes a family of loss functions including MSE as special case when T - oc. Second, compared with standard MSE loss, it introduces minimum overhead computation and can be interpreted as weighted MSE loss. Instead of assigning the same weights for all samples, [Eq (5) determines weights based on the TD-error distribution of
samples in one batch. Since the weights in [Eq (5) are obtained from boftmax-opprator on TD-errors, we coin the asymmetric loss as Softmax MSE (SMSE).
Although Eq (5) depends on the samples in a batch, i.e., a particular sample's weight with the same TD-error can vary across batches, we can still compare samples in batches and their underlying base functions. MSE loss function is based quadratic function EMSE(();) EE 83, while the loss
4

---

Under review as a conference paper at ICLR 2023
function for SMSE is (the detailed derivation is included in Appendix A);
6 + exp (-r)). T.T exp
8
SSMSE (().),
(6)
This loss landscape is depicted in Fig 2 and versatility as we can control the loss asymmetry and loss landscape by the sign and magnitude of temperature. While in this paper we primarily explore the SMSE loss function, we investigate some additional choices of AsymQ loss functions in [Section
Note the choice of SMSE is mainly due to its simplicity
Leveraging the asymmetric loss landscape of SMSE, we can effectively encode inductive bias for policy evaluation. For the inherent overestimation bias present in algorithms such as DQN and DDPG, SMSE with positive temperature will penalize states with negative TD error. We illustrate this point with a toy example Fig 2, where we simulate value function updates with varying SMSE landscapes with all simulations starting with batch where the TD-errors of all samples set to a constant negative value by fixing appropriate target values. We study how the TD errors of sampled-bated evolve over updates and find that the loss landscape changes the value function fitting dynamics significantly. Samples in the batch shift from negative TD-error to positive TD-error as lower the temperature. With the temperature in the right range, estimation bias can be lowered quickly in our synthetic simulation of value function update. We demonstrate that this indeed scales mitigating estimation bias during RL training on challenging continuous control benchmarks in the next section.
T +
T-20
T
T-2
20
TO Enner
30
10
15
Loss
110
step
10 TiRz
-10
step
10
0.15
jis.ogromm
5
0.10
0.05
0
-10
-5
0
10
15
20
TD-Error (4)
0.00
15
-10
10
35
(d)
Though it shares the same global minimum point with MSE, SMSE has different and asymmetric loss landscape as in Fig 2a. reduces to MSE when 7 = oo. Fig 2b,2c simulate SGD trajectories of TD-error on SMSE loss landscape for 10 steps where sampled state-action pairs have negative TD-error (note that negative TD-error corresponds to overestimation). We further visualize the final histogram of TD-error for different temperatures in Fig 2d. The shaded regions depict histograms for the final TD-error results of each temperature with solid- lines indicating the fitted Gaussian distributions. The intuition found in the toy example can be generalized more general RL envs, where asymmetric Q loss function can control bias preference of policy evaluation and alleviate inherited overestimation in DDPG and DQN.
Figure 2: SMSE loss parameterized by temperature Eq (5) and (6)
ASYMMETRIC Q LOSS: CONTROL ESTIMATION BIAS
In this section, we show that the small change value-fitting loss has a large impact on reinforce- ment learning and that different loss landscapes will lead to different estimation biases. Our simple asymmetric loss can drastically boost policy learning in existing RL methods without involving other computationally expensive tricks. We experiment with algorithm the suite of MuJoCo continuous control tasks (Todorov et al. 2012), interfaced through OpenAl Gym (Brockman et al. 2016).
5

---

Under review as a conference paper at ICLR 2023
4.1 CONTROL ESTIMATION BIAS
We visualize the estimation bias of DDPG, TD3, and DDPG+SMSE (ours) for the OpenAl gym environments Walker2d-v3 and Ant-v3 in angFig The estimation bias measured by the average dif- ference between Q network prediction and ground truth value for 10,000 randomly sampled action- state pairs. More details regarding how the estimation bias computed can found in Appendixx B) Clearly, DDPG suffers significant overestimation, which is expected due to inherited maximum bias (see lemma 1). In contrast, TD3 has a serious underestimation issue, a phenomenon that was also reported in Pan et al. 2020 Lyu et al. (2022).
As different temperatures parameterize different loss landscapes, we conduct an ablation study on estimation bias with different temperatures. Lower temperatures put penalty on action-state pairs whose predictions are larger than their supervised labels and more relaxed for pairs with pos- itive TD-error. As it validates in Fig 3c and 3d] positive temperatures issues in DDPG and extremely small positive temperatures even lead to slight underestimation. The results suggest we can some extent control estimation bias through the parameterized loss func- tion, therefore these asymmetric loss functions can be used to introduce inductive bias for learning value functions. When the temperature is within proper range, we can significantly reduce estima- tion bias and learn accurate value functions.
relieve overestimation
302
2003
66%0
Average (Op- Q
Average 1Qu- Q7)
200
Average (Qp
Average (Qu
1:00
50
50
300
-150
-10
-100
x10"
x10"
x10
Step x10*
Walker2d
(b) Ant-v3
(c) Walker2d
(d) Ant-v3
Figure 3: Effect of parameterized SMSE loss estimation bias. Fig 3a 3bddepccss the estimation
bias of DDPG, TD3, and DDPG+ Walker2d-v3 and Ant-v3 respectively. The baseline (T = oo) DDPG with default MSE loss shows tremendous overestimation effect whereas TD3 exhibits underestimation when compared with their value prediction with ground truth state-action Q value. For SMSE, Fig 3c) and 3d illustrates the effect of temperature on estimation bias and highlight the significance of choosing a proper temperature to reduce estimate bias. The shaded region here indicates 0.25 of the standard deviation of the estimation bias.
4.2 BOOST POLICY PERFORMANCE
We expect RL algorithms to benefit from more accurate value functions. We investigate the policy performance of DDPG with our SMSE loss. Fig 4ajand 4b the effects of different temperature. As T decreases from positive infinity, the more accurate value function helps boost policy perfor- mance. At the same time, we also find learning at extremely small temperatures may deteriorate policy performance. The issue may be explained by the following factors. A lower temperature makes learning more pessimistic learning may ignore samples with positive TD error and penal- izes both true positive and false positive overestimation. As an extreme case, as T - 0, the weight from softmax En (5)of of SMSE will concentrate on samples with the smallest TD-error, which suc- cumb to outliers and noise in the training. To illustrate this effect, we introduce the metric effective batch ratio (EBR),
EBR
B w
where {w:} quantifies the contribution weights for each sample in one batch. Note that for MSE (equivalently SMSE with T = 2)) w; = B and EBR = so each sample contributes equally to the value function update. On the other extreme, when T - 0, the weights (w:} = Softmax((- 4 for SMSE concentrate on one sample and FBR B- In general, EBR measure the effective

---

Under review as a conference paper at ICLR 2023
percentage of samples that contribute to the value function update. EBR is inspired by effective sample size in statistics literature (Murphy) 2012). The statistics also indicate the variance {wx} in one batch.
We visualize EBR for different temperatures in Fig 4g during training. As expected, EBR decreases decrease temperature. We additionally adopt clipped weights in practical implementation Algorithm Tb to prevent very small EBR and improve training stability.
92100
100
Average return
Awerage return
3320
03
2002
9320
2220
1020
1+50
0.5
Steo
Step
Step x10
Step x10
(a) Walker2d
(b) Ant-v3
(c) EBR Ant-v3
(d) Auto tunning T Ant-v3
Figure 4: Impact of SMSE on policy learning with DDPG. Fig 4a and 4b shows policy performance of DDPG and DDPG+SMSE in Walker2d-v3 and Ant-v3 environments. It depicts that by reducing estimation bias through the parameterized SMSE loss, we are able boost the learning efficiency and policy performance of DDPG. Fig 4c depicts the effective batch ratio (EBR) for different tem- peratures of SDPG+SMSE in Ant-v3. Fig 4d illustrates the automatically adjusted temperature over the training steps for DDPG+SMSE* in Ant-v3. The learning curves for DDPG+SMSE* included Fig 5 to contrast the performance against fixed temperature SMSE.
4.3 AUTOMATIC TEMPERATURE ADJUSTMENT FOR SMSE
The experiments on estimation bias demonstrate the importance of choosing the proper temperature for various environments, and it can be expensive to do exhaustive grid search for every envi- ronment, so auto-tur auto-tuning of this parameter should greatly reduce the burden of adopting our method. Though there exist many metrics can employ perform automatic temperature adjustments (Harnoja et al. 2018), we present one auto-tuning algorithm tailored for SMSE DDPG with the EBR metric.
In our experiments with DDPG, we find the optimal fixed temperatures for different environments can vary from 3 to 50 for the proposed SMSE loss. An ideal temperature should balance the trade- off between the estimation bias and variance, where a lower temperature puts more penalty on overestimation than underestimation the weights uniform which is preferred to resist noise and reduce variance during learning. Here we introduce one simple approach to adjust the temperature based on EBR. Empirically, we find experiments with competitive performance usually have EBR in fixed interval, which is around [0.95, 0.98). Therefore, a simple tuning scheme can be proposed based on EBR. We increase the temperature when EBR is too small and decrease the hyperparameter when EBR is too large over the expected interval. We summarize the final algorithm in Algorithm I (More discussions and details of hyperparameters as *MSE* and illustrate the learning curve and the auto-tuned temperature for DDPG-SMSE in Ant-v3 in Fig and Fig 4d respectively.
counteract inherent bias while higher temperatures make
included in Section 5) and Appendix B). We term the auto-tuned loss
7

---

Under review as a conference paper at ICLR 2023
Algorithm AsymQ: Policy evaluation with SMSE-loss with automatic temperature adjustment
Input: Sup: Btown: interval for desired EBR, Amultiplier: temperature multiplier
Sample mini-batch of B transitions (s,, at, ri st) from replay-buffer B TD-error 6; = ri + 7Q (8/, Ta (8()) Qo a.)
Weights {ws} = Softmax(clip((
[6] T 2, Iu 2))
B Loss Cg Σ (w;))
I/ s.g.: stop gradient
Update -
Every d iterations Calculate EBR based on Eq (7)
T + x Smultiptier EBR
Sup
T x- T/ mmulipheer if EBR Bdown
5 EXPERIMENTS
We present more experiments to answer the following questions: (1) How is the performance of DDPG+SMSE compared with the existing algorithms with multiple critics? (2) Can our auto-tuning algorithm help find the proper temperature to reduce estimation bias? (3) Can other asymmetric loss functions apart from SMSE improve policy learning? (4) Can our approach work for environments with discrete action space?
MuJoCo ENVIRONMENTS
DDPG
SAC-D
SAC-S
DDPG+SMSE
SAC-S+SMSE
DOPG+SMSE
Ant-v3
Walker22-v3
Hooper-v3
Swimmerw3
Aucrage return
Average return
3020
Average retuin
2530
100
Average
2002
2010
2002
2mm
1020
62
1000
1930
520
20
0.5
100
044
0.
0.0
0.4
0.
00
0/
98
10
Step
x10*
Step
x10
Step x10*
Step
x100
Humanoidiv3
HalrCheetah-vv3
10000
Avarage recurr
Average retum
0002
2002
6000
2000
0.25
6.9
175 1.59
22
Step
x10
Step
x10
Figure 5: Learning curves for the OpenAl gym continuous control tasks. The shaded region repre- sents half standard deviation over trials.
Benchmark We demonstrate the effectiveness of our approach in the six MuloCo environments. We compare our approach with popular baselines, DDPG, TD3, and Soft Actor-Critic with automat- ing entropy adjustment (Haarnoja et al. 2018) (SAC). For SAC, we include two variants, SAC-D which uses two critics, and the Clipped Double Q-Learning trick in Eq (4) and SAC-S which only has one critic. For reproducibility, based on CleanRL (Huang et al. (20TT). We use default hyperparameters unless stated otherwise for all experiments. We apply the SMSE trick in DDPG, denoted DDPG+SMSE, Our results
implementation of selected baselines and algorithm is
8

---

Under review as a conference paper at ICLR 2023
presented in Tab I with learning curves in Fig 5 For each environment, we select the temperature by sweeping temperature over values {3, 5, 10, 20, 50}. We include more details of our implementation and hyperparameters for both our algorithms and compared basclines in Anppeddi B
(RL with a single critic) The experiment results show our algorithm can significantly improve DDPG by simply replacing MSE loss with the asymmetric SMSE loss for fitting Q functions. Our algorithm achieves performance comparable with the SOTA algorithms with multiple critics. It indicates our algorithm can save half of the computational resources (see Appendix for update time comparisons) for policy evaluation while retaining policy performance, in some cases even outperforming existing algorithms in challenging environments like Ant-v3. (Automating temperature adjustment) We also include learning curves obtained by our proposed automating temperature adjustment, denoted as DMPG+ Our algorithm can find the proper temperature for most environments. (Composi- tionality with other techniques) We also apply the SMSE trick in SAC-S, denoted as SAC-S+SMSE. The asymmetric loss can also benefit the off-policy algorithm with a stochastic actor. (Other asym- metric loss function) Our asymmetric algorithm is not limited to a specific loss. We experiment with two new asymmetric losses:
(16:),.T) -Sottmax (1- & (1211)03, (2(())),))
E sign(6))))) 87 (T> 1).
(8)
Though these new asymmetric losses have different landscapes, they also penalize negative TD- error more and positive TD-error less. [Tb I shows DDPG with these new asymmetric losses can also boost policy learning of DDPG in challenging environments. Interestingly, we find the fixed temperature T = 1.5 for Lg works for all environments.
Elvironment
DDPG
SAC-D
SAC-S
TD3
DMPG-SMSE
SAC-SASMSE
DDPGGSMSF
DDPG-
DDPG+
Swimmee-x3 10
1.47::0.06
0.76:::0:24
1,30--0322
0.6
+0.27
1.43+0008
1.33+-030
1.16:0.21
1221:009
1,46 + 0.08
[aaliCheetah-33 (x 10")
1.12:1:003
1.010:001
1,081:0.00 0.33
102:1005
1.0110.10
1.0:100.06
101110111
1:07::0.05
Walker2d (x10P)
1.67+-0.77
4.105-488
3055
3784-033
4.1540466
4.65--018
4.16:00.0
0.674:2255
4:26+-0:38
Hoppes-xt (x 10
1.87::0.33
1.83-::224
3402041
1.9::..7
3.21
3.2420.18
3.2740233
5.54200311
(x
4:29:::49
38311.11
1,121041 5372048
5.6610.25
5.1521112
5.4220032
53440.44
4:8.0:38
Hamrnodd-v (x)0P)
1.74 5.00:0337 1.37 5.34:0.11
5.16202.0
4.85-50.28
5.16:00.00
4.2040.43
4.05
Table I: Average Return over 5 seeds with corresponding to one standard deviation. SMSE* uses automatic lemperature adjustment while for SMSE we report best performance from a fixed grid- searched temperature. We also include results for DDPG with asymmetric losses in Eq (8) which validates the benefit of using asymmetric loss functions in general
5.2 DQN EXPERIMENTS
We demonstrate that our proposed SMSE loss can also be beneficial for discrete action scenarios by demonstrating it's effectiveness when combined with the DQN algorithn| Mnih et al. (2015). We evaluate our proposed method on the MinAtar benchmark Young & Tian (2019) and some Atari games Bellemare et al. (2013) comparing against both baseline DQN and DoubleDQN up- dateVan Hasselt et al. ( with all the variants operating with common set of hyperparameters. In Fig 6) we report the best cest-performing learning curves for some of the games in benchmarks ob- tained by grid search in (0.5, 1, 5, 10, 20, 100) for the temperature hyperparameter in DQN+SMSE keeping the clip parameter at Results for more games in the benchmarks are available in Ap- pendix B.7
CONCLUSIONS AND LIMITATIONS
We introduce a lightweight approach to control estimation bias for policy evaluation in DRL. Instead of constructing robust supervised targets at the cost of multiple value networks, we find the fitting loss function in policy evaluation has a surprising effect on controlling estimation bias. We are able to reduce estimation bias and boost policy performance by simply changing MSE loss to an asymmetric loss in DDPG without increasing the computation cost by much. We point out several limitations of approach here. Though we propose SMSE which includes MSE as special case and an automatic temperature adjustment algorithm for SMSE, how to design a proper or even optimal loss function is out of the scope of our work. In fact, We actually find SMSE is not so

---

Under review as a conference paper at ICLR 2023
DON
DoubleDQN
DQN-SMSE
Minzhaar/Speeeirvdderss
MinAtaarAAterixx-00
BreakkottooFranccsip.v
ReadRunne www.eeoooooroo---....
250
Average S
Average return
200
23000
Average
23200
15000
10
22000:
15
10
5500
10
Step X10
Step x10
Step x10
Step x10
Figure 6: Learning curves for some games in MinAtar and Atari benchmarks. The shaded region represents one standard deviation the average evaluations over trials.
robust in DQN setting, several fail cases and possible explanations are included in Appendix B) Our results provide one approach inject inductive bias in DRL via loss functions for policy evaluation. It remains an open problem to adjust and customize the loss landscape automatically for different environments.
REPRODUCIBILITY STATEMENT
The detailed discussion assumptions and proofs are included in Appendix For experiments, hyperparameters and other implementation details Algorithm and compared algorithms are included in the main paper and Appendix B This should provide sufficient information for the reader of interest to reproduce our results.
REFERENCES
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Richard Bellman. Dynamic programming. Science, 153(3731):34 37, 1966.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01144 2016.
Xinyue Chen, Che Wang, Zijian Zhou, Keith Ross. Randomized ensembled double q-learning: Learning fast without model. arXiv preprint arXiv:2101.05983 2021.
Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
in actor-
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. Technical report, 2018.
Hado Hasselt. Double q-learning. Advances in neural information processing systems, 23, 2010.
Shengyi Huang. Rousslan Fernand Julien Dossa, Chang Ye, and Jeff Braga. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. 2021.
Mykel Kochenderfer. Decision making under uncertainty: theory and application. MIT press, 2015.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overesti- mation bias with truncated mixture continuous distributional quantile critics. International Conference Machine Learning, pp. 5566. PMLR, 2020.
10

---

Under review as a conference paper at ICLR 2023
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning Controlling the estimation bias of q-learning. arXiv preprint arXiv:2002.06487, 2020.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified frame- work for ensemble learning in deep reinforcement learning. International Conference on Machine Learning, pp. 6131-6141, PMLR, 2021.
Litian Liang, Yaosheng Xu, Stephen McAleer, Dailin Hu, Alexander Ihler, Pieter Abbeel, and Roy Fox. Reducing variance in temporal-difference value estimation via ensemble of deep networks. In International Conference on Machine Learning, pp. 13285-13301. PMLR. 2022.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1XXiv:150900299 2015.
Jiafei Lyu, Xiaoteng Ma, Jiangpeng Yan, and Xiu Li. Efficient continuous control with double actors and regularized crities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7655-7663, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51867540):529-533, 2015.
Kevin P Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies, Advances in Neural Information Processing Systems, 32, 2019.
Ling Pan, Qingpeng Cai, and Longbo Huang. Softmax deep double deterministic policy gradients. Advances in Neural Information Processing Systems, 33:1 1767-11777, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances neural information processing systems, 32, 2019.
Andrew Patterson, Victor Liao, and Martha White. Robust losses for learning value functions. arXiv preprint wxXi:22205.08464 2022.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pp. 387-395. PMLR, 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum, volume 6, pp. 1 1-9, 1993.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Wei Wei, Yujia Zhang, Jiye Liang, Lin Li, and Yyuze Li. Controlling underestimation bias in reinforcement learning via quasi-median operation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 8621-8628, 2022.
Jiyy Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. arXiv preprint ar arXiv:2107.14171
2021. 2021.
11

---

Under review as a conference paper at ICLR 2023
Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, and Shuicheng Yan. EnvPool: A highly parallel reinforcement learning environment execution engine. arXiv preprint arXiv:2206.1053 2022.
Dongming Wu, Xingping Dong, Jianbing Shen, and Steven CH Hoi. Reducing estimation bias via triplet-average deep deterministic policy gradient. IEEE transactions on neural networks and learning systems, 91(11):493344953 2020.
Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.00 1903.03176, 2019.
12

---

Under review as a conference paper at ICLR 2023
A
PROOF
A.1
LEMMA
The proof is heavily based on conclusions from Section 4.1 and Appendix B from Fujimoto et al. 2018). We summarize the finding below. Given a pair of actor (To, Q8), we can define two new actors from the policy gradient algorithm
Φ Evs V
oT (s) ())=
ø + E.s V VoTT
a
a) a))a=
(10)
Z2
We note ④ is based on estimated value function Qo while @ is for ground truth value function Q" Q" is only for analysis purposes and not available during training. We assume Z1 and Zz are chosen normalize the gradient. Due gradient ascend, there exists 6 such that if a S G the estimated value of 7 will be bounded below by the estimated value of TT.
2 E[Qa( 88, n [+)
(11)
Similarly, there exists 62 such that if S 62 then the value of म will be bounded above by the true value of T.
E (s, ग. (T(s) EsQ
(12)
If in expectation the Qo is at least as large as the Q for policy Ts E[Qo(8, 0z(())
>
E[Q (s, (s)), then Eq (11) and (12) indicate that policy T will also be overestimated if step size a & 62)(
(2 (sQ"
(13)
The above conclusion is possible when we have the normalized gradient during gradient ascent. For the unnormalized gradient, stronger assumption on Q" and Qs to make Eq (13) hold (Fuji- moto et al. 2018). Although [Eq (I3)is made step policy gradient, the overestimation may accumulate as we train policy for more steps.
A.2
EQ (6)
To prove Eq (6) equivalent to Eqq (5) up normalization constants, show their gradients with respect to are equivalent.
We denote loss in Eq (5)
L and Eq (6) L2- Therefore, the gradients can be obtained
dL
2 2exp(- - %)
5, )6:- 6, exp(
(14)
dō,
Σ exp(-
where Σn exp( 4) is a normalization constants shared for every sample 8, in one batch. On the other hand,
dl2 - T exp (-
8 - 5, exp (- Texp
8/ T exp
6 - = 6, exp
8
(15)
dō,
Based on Eqq (14) and (15) we can conclude that gradient during one step gradient descent for Li and L2 are equivalent up normalization constants.
B EXPERIMENT DETAILS AND DISCUSSIONS
B. IMPLEMENTATION OF SMSE FOR POLICY EVALUATION
For Softmax MSE loss, we include the key modification based on PyTorch (Paszke et al. 2019) in Fig 7 We clip the weight logits for numerical stability. For all experiments, we find clipping range - log 2, log 2] works best for various experiments. log 2 is chosen to guarantee the ratio between the largest weight and the smallest weight is less than 4. Without clipping, Softmax MSE may assign unreasonable large and small weights for samples in one batch and result in small EBR which leads to very noisy updates and could destabilize the training.
13

---

Under review as a conference paper at ICLR 2023
tdeerror target__ estimated__
torch.pae ltd_errr,,,,, 2.0)
$ e_lose acrors.meate 7 MSB Impiementation
clip weights value for numerical stability
weight_lagitss (rrch.clip--sd_err / T, math. (1-0hheell math. log (thres))
0/threst, weights sorchmsofrmax (veight__ogitaa)) betch_size
q loas (weights errors) .mean
(a
Figure 7: PyTorch code sample for modifying MSE loss to SMSE loss with fixed temperature
B.2 COMPARED BASELINES IN MUJOCO ENVIRONMENTS
For DDPG, SAC-D, and TD3, implementation is based CleanRL (Huang et al. 2021 which provides a single-file implementation for each algorithm. We add an extra evaluating step that tests each RL. agent in the same environments with 10 different random seeds. All learning curves and reward return numbers are based on experiments in testing environments despite we find that episode return curves during training are similar testing ones. For DDPG, use the imple- mentation from Fujimoto et al. (2018) instead of the original DDPG (Lillicrap et al. 2015), which includes several tricks that improve DDPG performance. Our SAC-D implementation matches the performance of the official codebase For fair comparison, we use the same learning rate 3 x 10" and start- tirettiiesseps 25000 for all experiments. SAC-S based on SAC-D but only uses one critic instead of taking the same minimum value of two independent crities. Unless stated otherwise, the hyperparameters and implementation details follow the default choice in CleanRL.
B.3 HYPERPARAMETERS FOR SMSE
For SMSE algorithm with fixed temperatures for policy evaluation, we sweep the temperature op- tions across environments, For each environment, we report the performance with the best tempera- ture and present the best performing temperature parameter in Tab
For weights' clip value, we try (- In 2, 2), (- In In 3), (- In In 5) and (- In 10, In 10), and find (- In 2) and (- 3, 3) can learn good policies most of the time, therefore we use (-In 2, In 2) by default across environments. In Fig 8 we compare policy performance and EBR for different clipping ranges for fixed temperature T 3 in Walker2d-V3 V3. The experiment shows that clipping helps stabilize training and learn better policies. We also note that clipping plays less important role when we use higher temperatures in SMSE.
For auto-tuned temperature SMSE* we use interval (0.95, 0.98) for desired EBR range in Algo- rithm I The range is selected based on EBR curves for different environments. We have not further swept EBR options and better performance may be possible with fine-tuned hyperparameters.
We believe that the above hyperparameters might not be optimal for SMSE and better hyperpa- rameters be obtained through more extensive search. However, results still demonstrate a surprising performance boost obtained by simply replacing MSE with SMSE, warranting further investigation of asymmetric loss landscapes for value function learning.
Sensitivity of Fmmuitiplier For Ant-v3 we try three different Smaltiplier parameters and report the per- formance curves for DDPG+SMSE* in Fig We see that the performance is robust to the choice of this hyperparameter when contrasted with the difference in performance observed while tuning the temperature parameter directly.
B.4 OTHER AS ASYMMETRIC LOSS
We also conduct some initial experiments with more asymmetric loss functions Eq (8) It is worth noting the main difference between ć and SMSE is that SMSE has stop gradient operator in calculating SoftMax. Although they appear be very similar, Ê, still presents different loss landscape and has different gradient descent dynamics. L2 is another even simpler asymmetric
14

---

Under review as a conference paper at ICLR 2023
Walker2d-v3
Walker2d-v3
(-In2.In2)
0.9
1000
(nIn3,In3)
(-In5.In5)
Average return
(n1110,
0.7
3000
EBR
0.6
2000
0.5
0.4
1000
(-In2.In2)
(-In3,In3)
0.2
(-In5,In5)
(-In10, I0)
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Step x x106
Step x x106
Figure 8: Effect of clipping on policy learning and EBR with T Clipping can help stabilize learning when using SMSE with a lower temperature by avoiding very small EBR. Without clipping, SMSE may assign large weights for overestimated samples and ignore underestimated samples.
1.01
1.001
1.05
Ant-v3
5000
Average return
3000
2000
1000
0.0
0.5
1.0
1.5
2.0
Step x06
Figure 9: Performance obtained with different Amaliiptier on Ant-v3 (average over seeds are re- ported) with DDPG+SMSE* (auto-tune algorithm). We observe reasonable performance for differ- multiplier hyperparameters demonstrating that the performance sensitivity lower
Environment
DDPG+SMSE
SAC-S+SMSE
Ant-v3
20.0
5.0
Walker2d-v3
3,0
5.0
Hopper-v3
3.0
5.0
Swimmer-v3
50.0
10.0
Humanoid-v3
20.0
5.0
HalfCheetah-v3
50.0
50.0
Table Temperature for different environments
15

---

For each environment, we pick 20 checkpoints uniformly from the train steps to test RL agents' performance. For each checkpoint, we evaluate the learned Q values on 10,000 state-action pairs that are uniformly sampled from the replay buffer. The predicted Q-value is obtained by simply feed-forwarding the state-action pair into the Q-network from the checkpoint. The ground truth Q" is estimated in Monte-Card Carlo fashion, where we execute the policy checkpoint and collect rewards along trajectories to estimate average discounted return (T), the trajectories are clipped 500 steps and the average over 3 such trajectories are treated as the ground truth value, use only Under review as a conference paper at ICLR 2023
loss function, which can be interpreted as combination of two quadratic functions. We initially parameterize it as
((16)),))
Σ T (igg(())) o7
(16)
We perform a grid-search for T over {1.2, 1,5, 2.0}. and find that = 1.5 works well for all environments. We emphasize that both functions put penalization state-action pairs whose TD-Error negative.
B.5 ESTIMATION BIAS FOR FIG
trajectories because the variance observed across trajectories were low as the policy is deterministic. We plot the mean and standard derivation of estimation bias over 10,000 state-action pairs in Fig The pode for this procedure is provided Algorithm
Algorithm 2 checking estimation bias in the value function for a given checkpoint
Let π,, Q" be the policy and value function to evaluate.
Explore environment with ı + +0(0, and store N transitions replay-buffer Sample M state-action pairs {(s, {()) from B
Let Q PRED = [']xMM and Q TRUE = [
[ 1xM
for = to M do
Q_PRED[i] = (s;; Q4)
Q_TRUE[i] = Monte- Carl---stiiatteedeeeeeeeee----ee aaabo-sstiiaee(dsssomeed..ttt a, (T,TT)))
end for return mean and standard deviation of (Q_PRED Q Q_TRUE)
B.6 UPDATE TIME ANALYSIS
We benchmark the average update time for the algorithms DDPG, TD3, SAC-D, SAC-S, DDPG SMSE, and SAC-S SMSE for training the Ant-v3 environment. All algorithms run with the same batch-size hyperparameter on an Intel i9-9900 16 core + RTX 2080 Ti desktop. From Fig 10, observe that SMSE loss function adds very minimal cost (0.2 ms/step) to the update time on-top of DDPG but still achieves performance comparable to the state-of-the-art algorithms.
B.7 ENVIRONMENTS WITH DISCRETE ACTIONS
The DQN implementation for the Atari benchmark is based on the Tianshou Weng et al. 202T package that leverages envpool (Weng et al. 2022) implementations of the Atari environments to speed up the run-time. The architecture and pre-processing of frames are similar to the ones found the original DQN paper(Mnih et al., 2015). The hyperparameters used to come from the default parameters present in the Tianshou code-base, we just experimented with different temperature pa- rameters of the additionally introduced SMSE loss, and temperatures from the set (0.5, 1, 10, 20, 100) were tried. In Tab 3 we outline the best performing run't 's temperature and final performances for Atari games and contrast it against the baseline DQN (T oo) and DoubleDQN. The results reported are the mean and standard deviation of returns over random seeds.
16

---

Under review as a conference paper at ICLR 2023
Average update time (ms)
Qa update
update
target (0, update
8
6
5
4
2
1
SAC-D
SAC-S
TD3
DDPG
DDPG
SAC-S
+ SMSE + SMSE
Figure 10: Average update time for each training step for different algorithms. DDPG + SMSE adds very little computational cost over DDPG but achieves performance comparable that of TD3 or
SAC
Environment
Temperature
DQN SMSE
DQN
DoubleDQN
Breakout
205.31=21.52
101.56=11..70
118.64
SpaceInvaders
596.75:14..89
533.14:196.00
502. 17:7599
Seaquest
2924.72::413.79
2939.121:89.32
1904.1943544
RoadRunner
13935.1
11866.93=1786.06
30798.37+:308..0
BeamRider
20
3379.35=611.30
3368.274571.66
2026.12::445..1
Riverraid
100
7463.80 1002.75
7651.94-L681.36
827..78+778.29
Asterix
10
2983.21 685.62
2714.63+440..77
2079.31+344.00
Table 3: Temperature for best-performing DQN SMSE Loss Atari games benchmark
(Gym:NoFrameskip-. We report the average Return over 5 seeds with + corresponding to one standard deviation.
For the MinAtar (Young & Tian 2019) games benchmark, we use the CleanRI. (Huang et al. 2021)
implementation of DQN and leverage the hyperparameters suggested the MinAtar paper along with additionally limiting the environment steps for the MinAtar/Seaquestt-ee environment 10,000
steps as done in (Lan et al. 2020). Similar the Atari games benchmark we try different temper-
ature parameters from the set (0.5, 5, 10, 20, 100) and report the best performing in Tab 4
where the result represents the mean and standard deviation over 5 random seeds.
Environment
Temperature
DQN + SMSE
DQN
DoubleDQN
MinAttarAsttriix-a.
26.79-11.85 4.774
11.26::1.88.
11.77#2.19
MinnAtarrSpaceenvaders
44.86:14.25
39.9243.84
MinAtarrBreakout-v00
20
17.53-1-2.28
17.1
14.98-10.84
MinAtaar/Seqquestvv0 MinAtar/Freeway-vo
10
4.76
4.53+1.41
3.24+10.3
20
48.6040.41
48.7410.52
48.07+1.44
Table 4: Temperature for best-performing DQN SMSE Loss MinAtar benchmark. We report the average Return over 5 seeds with + corresponding to one standard deviation.
While the performance improvement observed on some of the games in both the Atari and Mi- nAtar benchmarks, it still fails to show major improvements for good fraction of the environments, unlike the continuous control MuloCo benchmark. This could be due to several reasons, one major factor is that estimation bias in DQN is different from that of DDPG, and the loss developed and
17

---

Under review as a conference paper at ICLR 2023
Environment
Algorithm
Performance
Steps
Wall-clock Time (hours)
Estimated Total
FLOPS (x10"5)
Hopper-v3
REDQ
3528.57+1144.1
125K
20.46
3.303
Ilopper-v3
SMSSE
3213,84.1205.3 56
1000K
11.93
0.161
Walker2d-v3
REDQ
5426.60++626.55
300K
50.32
8.291
Walker2d-v3
SAC-S+SMSE
1653.17=187.51
1000K
11.21
0.168
Ant-v3
REDQ
6086.05::122.02
300K
53.31
11.144
Ant-v3
SAC-
5172.35:112.12
1500K
16.16
0.338
2021). Our SAC-S+SMSE can achieve competitive results with much fewer computational resources measured in wall-clock time and estimated FLOPs count.
Table 5: Comparison against REDQ (Chen et al.
tested extensively in DDPG doesn't t simply transfer and work for all DQN environments. is to be noted that even DoubleDQN updates sometimes result in lower performance compared DQN so it can be expected that just reducing estimation bias does not always necessarily translate to higher performance. Additionally, we leverage the typer-parameters suggested in popular open-source packages and do not perform any tuning apart from small grid search on the SMSE-temperraure parameter. An extensive hyper-parameter search along with the exploration of new asymmetric loss functions for environments with discrete action spaces is an exciting direction for future work.
B.8 SMSE WITH ENSEMBLE Q-NETWORKS
Arguably one of the most effective approaches to reducing the bias and having stable value- function targets is the usage of an ensemble of critics (Kuznetsov et al. [0020] Chen et al.) [021] Liang et al.) 2022 Lee et al. 2021 Lyu et al. 2022. Wu et al. 2020 Wei et al.) 2022). this section, we com- pare SMSE with sfate-of-the-arr algorithms based on ensemble of critics. We find our approach can achieve competitive results in a much more computational efficiency approach. Besides, we surprisingly find our asymmetric loss is orthogonal to the ensemble method to some extends.
SMSE vs REDQ (Chen et al., 2021) We first compare our work with an ensemble Q algorithm called Randomized Ensembled Double Q-Learning (REDQ) (Chen et al. (021). REDQ leverages an ensemble of Q-networks with additional tricks such as high update-to-data ratio and target- minimization from subset of these Q-networks demonstrate strong performance in sample- efficient manner. In Tab 5] we compare the best performance (mean across 3 seeds) of REDQ and our proposed SAC-S+SMSE on some Mujaco environments and report the number of environment steps and wall-clock time duration to achieve the same. For this comparison, we use the same un- derlying hyperparameters such as architecture, learning rate, etc., with REDQ specifically using 10 networks, random subset size of 2 for target construction, and an update-to-data (UTD) ratio of 20. The performance SAC-S+SMSE is reported for the best temperature parameter selected by the procedure outlined in Appendix B.2 as done in the main paper.
REDQ indeed achieves a very high performance, which can be attributed to stable targets obtained from ensemble networks and the high UTD ratio. While sample-efficient performance is achieved, the run-time and memory consumed by such ensemble techniques are high. If we N networks, compare M networks to provide target estimate, and perform G updates per step (UTD ratio) we can approximate the FLOPs used in REDQ for the network size used in our experiments as G x ((2 FwdPass x M) + N x (FwdPass + (2 BckwdPass))).
The FLOPs count of SAC-S+SMSE can estimated with G = 1, M 1, and N = 1. We obtain the FLOPS for FwdPass from fycore library and approximate the cost of BckwdPass 3 FwdPass.
SMSE improves Ensemble To experiment we can incorporate an ensemble of networks into the SMSE framework, we improve on the ensemble averaging target (AVG) baseline also used in (Chnn et al. 202I), by plugging in our proposed SMSE loss to update the Q-networks from targets predisposed to overestimation bias. In Fig 11 compare the impact of using SMSE loss instead of the baseline MSE loss. We see that the proposed loss function significantly improves the AVG baseline. To further see if method helps reduce estimation bias we compare REDQ, AVG,
18

---

Under review as a conference paper at ICLR 2023
Hopper-v3
Walker 2d-33
MMS RD SUSE
ASTING SMCE
.000
5350
Averoge retur
2560
Average return
8000
2000
2303
1200
2323
2003
60
06
2.0
00
0.5
3.0
20
20
environment Interactions
environment Interactions
Figure 11: Impact of SMSE loss on the performance of ensemble average targets (AVG).
Hopper-v3
Hopper-v3
R/3 16
3015
AVS 10 SPICE
Average normalized blas
200
0.4
Average blas
63
100
0.
ub
-0.00
044 environment interactions
environment interactions
Hopper-v3
3500
3000
Average retum
2300
300
environment Interactions
385
Figure 12: The plot of the top-left indicates the average value function bias (i.e. prediction esti- mated true value). The plot the top-right indicates the average normalized bias (i.e average bias divided by estimated true value). The plot at the bottom compares the performance of REDQ, AVG, and AVG SMSE. While observe slight underestimation REDQ and overestimation in AVG, the average bias in AVG + SMSE is close to O, showing that the proposed loss function effectively reduces bias using averaged ensemble target estimates.
and AVG+SMSE Hopper-v3 and present the results in Fig 12 Average bias is the mean of the differences between the predicted Q-values and the estimated true Q-values (estimated by Monte Carlo simulation) over state-action pair samples. The normalized bias divides the bias by the estimated true Q-value to visualize the scale of the bias in terms of the value function. We observe that the proposed SMSE loss can effectively reduce the bias in the action-value function by keeping the average bias close to O and providing competitive performance improvements. These findings indicate that SMSE better than the default MSE loss, and the improvement orthogonal to other methods proposed to address the estimation bias problem in RI. We are optimistic that further investigation into Asymmetric loss functions can benefit many algorithms and downstream applications that rely value functions for further optimization and planning.
19


Reviews:
[{"review_id": "wsVoUyorLB", "reviewer": "Reviewer_h4Ht", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use asymmetric Bellman losses for TD-learning in algorithms such as DDPG, SAC, etc. The goal is to resolve overestimation bias, and the idea of the paper is intuitive. They further propose an auto-tuning algorithm for the temperature hyperparameter in their algorithm.", "summary_of_the_review": "In summary, I think the empirical evaluation of the paper needs a lot more work. The paper needs to convince the reader why their method is significant, and why we should take note of it as a community, and this is where the paper falls short. I am opting for a reject score due to these reasons. Besides, it would also be good to see an analysis on why overestimation matters.\n\n___\n\n## After Rebuttal\n\nThanks for the responses! I am still not convinced about the significance in the long run, but I will move up my score to a 5, in light of the rebuttal.", "strength_and_weaknesses": "Strengths: the method does appear effective, in that it does reduce overestimation bias.\n\nWeakness:\n\n- it is unclear if the method really improves performance. In particular, the SAC performance with two critics is lower than typical SAC performance, sot he MuJoCo resutls are not trustworthy. For DQN results, DQN is now a 10 real old algorithm. Can you show that this helps with Rainbow?\n\n- It is not clear if fixing over-estimation should lead to better performance. Also, what about using other methods such as truncated distributional critics? How do they perform in comparison?\n\n- No clear significance: to be honest, I am not sure what the point of the result DDPG + SMSE == SAC is. In particular, SAC is so well adopted, that unless we improve over existing state of the art algorithms, or bring new insight to the table, it is not obvious why this result should be significant in the long run.\n\n- More experimental settings: it would be worth studying this in settings where over-estimation is likely to hurt more, like in offline RL, or in settings where we must take more gradient steps per environment step.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "rVstky2aif", "reviewer": "Reviewer_fW9A", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new method to better mitigate estimation bias. Essentially the method will allow weighted gradient updates when training the critics, and (depending on the temperature hyperparameter) give a higher weight to gradients that will lower Q value estimates and a lower weight to gradients that will increase Q value estimates. In this way a more fine-grained bias control can be achieved. A major benefit of this method is it lowers computation and memory consumption. Empirical results on Gym and some Atari tasks show it can outperform other baselines. Toy example and other empirical analysis and some theoretical support are also provided. ", "summary_of_the_review": "The proposed method is interesting, and a lot of technical details are provided which is great. My main concern is on the significance of the performance of the proposed method. All the methods compared are quite old, so it is unclear how the proposed method will perform against some of the current SOTA methods in bias reduction. ", "strength_and_weaknesses": "Strength: \n- Computation efficiency: The method imposes small computation and memory overhead. \n- Easy to apply: Proposed method can be applied to existing methods and does not require extensive modifications.\n- Some significant results: Results on 4 Atari tasks show significant performance gain over DQN and DDQN baselines\n- Technical details: many technical details are provided for better reproducibility, the computation comparison figure is great and there are studies on gradient clipping and the temparture. \n\nWeaknesses: \n- Significance of results: in Gym tasks, the proposed method can be stronger than other methods in some cases, but it is not that consistent, in some tasks, the performance gap is really small. In Atari tasks, there are only 4 tasks shown. It seems quite reliable that the proposed method can greatly improve DDPG, DQN, DDQN on given tasks. But these methods are a bit old and it is unclear how it compares to more recent methods? For example REDQ (Randomized ensembled double Q learning, ICLR 2021) on gym tasks and Rainbow variants on Atari. \n- Would like to see how your method compares to SOTA algorithms on gym, for example if you follow what's in the REDQ paper, use a higher update-to-data ratio, maybe your method will have a much better performance compared to competitors? If you can reach SOTA performance with much lower computation, then that will make the performance claim much stronger. \n- I believe you are tuning for each task. In that case why is it that DDPG+SMSE cannot beat DDPG in some tasks? Not sure if that really makes sense? The other thing is for the methods you are comparing to, are they also fine-tuned to each task individually? \n- Figure 1 the figure is OK but you might want simply make your discussions a bit more explicit (maybe just explicitly point out early in the paper that the proposed method can be seen as providing weighted update and will make Q values faster to go down and slower to go up and be explicit about the intuition you are trying to convey in the figure)\n- For figure 2, the plot is nice but the discussion on how you set up the toy example experiment is a bit vague, maybe you can rewrite some of the explanation to improve clarity. Is each curve in the figure showing what happens after many updates? And would be good to add a summary sentence in the end, just to make your insights more explicit for your reader. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pUORGFmwmw", "reviewer": "Reviewer_eb1X", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces an asymmetric loss called Softmax MSE to address the overestimation issue in value-based off-policy learning. Instead of taking a minimum of an ensemble of Q-functions, as was proposed in TD3, the authors propose penalizing overestimation in TD learning.", "summary_of_the_review": "Overall, this is an interesting paper, and I recommend it for acceptance. There are some minor flaws that I hope the authors will address in the next revision; however, these flaws do not affect my assessment of the submission.", "strength_and_weaknesses": "# Strength\n1. The idea is exciting and novel. I haven't seen prior work that mitigates the overestimation issue via an asymmetric loss instead of an ensemble.\n2. The experimental evaluation is adequate and demonstrates the advantage of the method.\n\n# Weaknesses\n1. TD3 natural decouples dynamics and model uncertainties. However, this method might also penalize good trajectories in the case of stochastic dynamics.\n2. Prior work [1] has demonstrated that overestimation bias might be domain-specific. While Clipped Double Q-learning introduced in TD3 helps in some instances, it also negatively affects results. The paper can be improved by extending the experimental evaluation to these tasks.\n\n\n[1] Tactical Optimism and Pessimism for Deep Reinforcement Learning\nTed Moskovitz, Jack Parker-Holder, Aldo Pacchiano, Michael Arbel, Michael I. Jordan", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "DZPfRGeN0_U", "reviewer": "Reviewer_a76y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new class of policy evaluation algroithms, called AsymQ, for a lightweight and effective method to control over-/under-estimation. Different from the previous algorithms that rely on value function ensemble, the key idea of AsymQ is to adopt softmax MSE (SMSE) rather than conventional symmetric loss functions based on a proxy of over-/under-estimation bias. This transforms the landscape to emphasize the policy evaluation where overestimation occurs. The automatic adjustment of softmax temperature is designed based on a pre-defined range of effective batch ratio (EBR). The proposed method is evaluated in MuJoCo continuous control tasks, showing the effect of AsymQ in controlling value function bias under different temperature, comparable performance of DDPG-SMSE to TD3/SAC and the time cost of AsymQ. The proposed method is also evaluated based in Atari and MinAtar, demonstrating the effectiveness when applied to DQN.", "summary_of_the_review": "According to my detailed review above, I think this paper is clearly below the acceptance threshold.", "strength_and_weaknesses": "$\\textbf{Strengths:}$\n+ The writing and the presentation of the proposed method is almost clear.\n+ The proposed method shows effective value function bias control without a value function ensemble.\n+ The proposed method is evaluated based on multiple algorithms and in the environments with continuous and discrete actions.\n+ In addition to performance comparsion, results from multiple aspects are provided, e.g., loss function landscape, value function bias comparison.\n\n&nbsp;\n\n$\\textbf{Weaknesses:}$\n- Closely related works [1-4] are not included in this paper. SUNRISE [1] also adopts an asymmetric loss function based on uncertianty quantification (which actually has an underlying connection to value function bias). [2-4] are SOTA bias control algorithms which achieve significant performance improvement over TD3 and SAC. At least, they should be included in this work and discussed.\n- Although the proposed method is clear, I have a concern on using 1-step td bootstrapping target as the proxy of ground true value. I know it is non-trivial to obtain a high-quality proxy and 1-step td bootstrapping target is indeed a convenient and conventional chocie (as also used in a few prior works).\n  - In my personal opinion, I do not consider that 1-step td bootstrapping target is a good proxy. As the authors also mention, “the constructed supervised target usually depends on bootstrapping as in Eq (4) and the loss is not the distance from the ground truth value function of the policy”.\n  - Especially, when applying SMSE to DQN, it becomes more confusing. This is because value-based RL algorithms like DQN, follow Value Iteration rather than Policy Evaluation. In another word, 1-step td bootstrapping target is a value estimate of the improved policy rather than the current value function. I think this may also be a potential reason to the non-robust performance of DQN-SMSE as mentioned by the authors.\n  - For another point, I have a little concern on the coupling of using $\\delta$ for both the softmax weights and the MSE loss calculation. Since the td error is used for a proxy of under-/over-estimation bias of a specific $Q(s,a)$, I think dependent estimation may be necessary for the proxy of under-/over-estimation bias (similar to the double sampling problem).\n- It is good to see the automatic temperature adjustment. However, the range of EBR needs to be predefined. The choice of the range needs to be considered in different domain. How different choices influence the learning performance is unknown. Moreover, it also introduces an additional hyperparameter $\\beta_{multiplier}$.\n- The proposed method DDPG-SMSE* performs infavorably to TD3/SAC-D in an overall view.\n\n&nbsp;\n\n\nReference:\n\n- [1] Kimin Lee, Michael Laskin, Aravind Srinivas, Pieter Abbeel. SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning. ICML 2021\n- [2] Xinyue Chen, Che Wang, Zijian Zhou, Keith W. Ross. Randomized Ensembled Double Q-Learning: Learning Fast Without a Model. ICLR 2021\n- [3] Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, Dmitry P. Vetrov. Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics. ICML 2020: 5556-5566\n- [4] Litian Liang, Yaosheng Xu, Stephen McAleer, Dailin Hu, Alexander Ihler, Pieter Abbeel, Roy Fox. Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networks. ICML 2022: 13285-13301\n\n\n&nbsp;\n\n$\\textbf{Questions:}$\n1) I notice that in Appendix B.5, the MC estimates of ground true value clip the trajectories at 500 steps. Can the authors justify this?\n2) Also in Appendix B.5 and Algorithm 2, the text says “10,000 state-action pairs that are uniformly sampled from the replay buffer” and the buffer B is collected by the policy to evaluate. So, the buffer is an on-policy buffer for the policy evaluate, right? It also means that each policy (checkpoint) evaluates its value function bias over its own on-policy state-action supports. Do I understand it right?\n3) According to Figure 3c,d and Figure 4c,d, it seems that the range of EBR [0.95, 0.98] results in a slight overestimation. Can the authors provide more explantion on this point?\n4) How many seeds are used for the results in Figure 4? It seems that the stds are large since only 0.25 std is shown.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject