Deep Reinforcement learning on Adaptive Pairwise Critic and Asymptotic Actor
Authors: Huihui Zhang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Maximum entropy deep reinforcement learning has displayed great potential on a range of challenging continuous tasks. The maximum entropy is able to encourage policy exploration, however, it has a tradeoff between the efficiency and stability, especially when employed on large-scale tasks with high state and action dimensionality.
Sometimes the temperature hyperparameter of maximum entropy term is limited to remain stable at the cost of slower and lower convergence.
Besides, the function approximation errors existing in actor-critic learning are known to induce estimation errors and suboptimal policies.
In this paper, we propose an algorithm based on adaptive pairwise critics, and adaptive asymptotic maximum entropy combined.
Specifically, we add a trainable state-dependent weight factor to build an adaptive pairwise target Q-value to serve as the surrogate policy objective.
Then we adopt a state-dependent adaptive temperature to smooth the entropy policy exploration, which introduces an asymptotic maximum entropy.
The adaptive pairwise critics can effectively improve the value estimation, preventing overestimation or underestimation errors. Meanwhile, the adaptive asymptotic entropy can adapt to the tradeoff between efficiency and stability, which provides more exploration and flexibility.
We evaluate our method on a set of Gym tasks, and the results show that the proposed algorithms have better performance than several baselines on continuous control.

Under review as a conference paper at ICLR 2023
DEEP REINFORCEMENT LEARNING ON ADAPTIVE PAIRWISE CRITIC AND ASYMPTOTIC ACTOR
Anonymous authors Paper under double-blind review
ABSTRACT
Maximum entropy deep reinforcement learning has displayed great potential on range of challenging continuous tasks. The maximum entropy is able to encourage policy exploration, however, it has a tradeoff between the efficiency and stability, especially when employed on large-scale tasks with high state and action dimen- sionality. Sometimes the temperature hyperparameter of maximum entropy term is limited to remain stable at the cost of slower and lower convergence. Besides, the function approximation errors existing in actor-critic learning are known induce estimation errors and suboptimal policies. In this paper, we propose an algorithm based on adaptive pairwise critics, and adaptive asymptotic maximum entropy combined. Specifically, we add trainable state-dependent weight factor to build an adaptive pairwise target Q-value to serve as the surrogate policy ob- jective. Then we adopt state-dependent adaptive temperature to smooth the en- tropy policy exploration, which introduces an asymptotic maximum entropy. The adaptive pairwise critics effectively improve the value estimation, preventing overestimation or underestimation errors. Meanwhile, the adaptive asymptotic en- tropy can adapt to the tradeoff' between efficiency and stability, which provides more exploration and flexibility. We evaluate our method on a set of Gym tasks, and the results show that the proposed algorithms have better performance than several baselines on continuous control.
INTRODUCTION
The task of deep reinforcement learning (DRL) is to learn good policies by optimizing a discount- ed cumulative reward through function approximation. In DRL, the maximization over all noisy Q-value estimates at every update tends to prefer inaccurate value approximation that outweighes valueTThrun& the true value Thrun Schwartz (193)), i.e., the overestimation. This error further accumulates and
broadcasts via bootstrapping of temporal difference learning SStttnn& & Barto (018), which estimates the value function using the value estimate of subsequent state. When function approximation is unavoidably adopted in the actor-critic setting continuous control, the estimation errors exag- gerated. These errors may cause suboptimal policies, divergence and instability. To some extent, the inaccurate estimation is unavoidable DRL because it is the basic trait for value-involyed DRL to use random variables as target values. On the one hand, these stochastic target values will introduce some estimation biases. On the other hand, even an unbiased estimate with high variance can still lead to future overestimation in local regions of state space, which in turn can negatively affect the global policy Fujimoo al. (2018). Therefore, diminishing the value variance without partiality can be an effective means to reduce estimation errors, no matter overestimation or underestimation. Tak- ing the twin delayed deep deterministic policy gradient (TD3) Fujimoto et al. (2018) for example, always selecting the lower value from pair of critics will induce an underestimation bias although it is beneficial for lower variance.
Several of recent works deal with errors like bootstrapping error caused by out-of-distribution (OOD) actions Kumar et al. (2019) 2020), and extrapolation error induced by the mismatch be- tween the distribution of buffer- buffer-sampled sampled data and true state-action visitation of the current policy Fujimoto et al. (2019). The authors Wu et al. (2019) address the distribution errors by extra value penalty or policy regularization.

---

Under review as a conference paper at ICLR 2023
Overestimation is another induced errors, which originally found in Q-learning algorithm by Watkins 1989), and was demonstrated in deep Q-network (DQN) Mnih et al. (2013) on discrete control. In recent years, overestimation is reported in function approximation of actor-critic methods on continuous control Fujimoto et al. (2018); Duan et al. (2021a). Although several algorithms are
created to address the overestimation errors Fujimoto al. (2018 2019); Kumar al. (2019); Wu iet al. (2019); Duan et al. (202Ia), the accuracy of function approximation is not flexibly touched since underestimation errors usually accompanies the correction to overestimation.
The paper has the following contributions, First, we propose the concept of adaptive pairwise critics, which connects a pair of critics using a trainable state-dependent weight factor, to combat estimation errors. Second, we propose the adaptive temperature which is also stateddepeneent so that the agent can freely explore with loose restriction on the selection of temperature hyperparameter. Based on this adaptive temperature, we organize term of asymptotic maximum entropy to optimize the policy. The asymptotic maximum entropy is combined with the adaptive pairwise crities to serve the target Q-value as well as the surrogate policy objective. Third, we present novel algorithm to tackle estimation errors and pursue effective and stable exploration. Finally, experimental evaluations are conducted to compare the proposed algorithm with several baselines in terms of sample complexity and stability.
2 RELATED WORK
In reinforcement learning, the agent needs to interact with the environment to collect enough knowl- edge for training. Without sufficient exploration, the collected data may be invalid for an optimal value. Therefore, reinforcement learning has to deal with the tradeoff between exploration and ex- ploitation. There several ways enhance exploration in deep reinforcement learning (DRL), one of which is the off-policy approach which takes full advantage of past experience from replay buffer instead of on-policy data Mnih et al. (2015). Another method adopts policy exploration to stimulate the agent's 's motivation for a better balance between exploration and exploitation Mnih et al. 20 HHarnoja et al. (201 8a). Among them, soft actor-critic (SAC) [Haarnoja et al.. (2018a) achieves good performance on set of continuous control tasks by adopting stochastic policies and maximum entropy. Stochastic policies generalize the policy improvement and introduce uncertainty into action decisions over deterministic counterparts Heess et al. (2015). and augmenting the reward return with an entropy maximization term encourages exploration, thus improving robustness and stability Ziebart et al. 2008); Ziebart 2010).
In recent years, many works have been proposed on top of SAC. The improvement of SAC can be realized by changing the rule of experience replay, for example, Wang & Ross (2019) samples more aggressively from recent experience while ordering the updates to ensure that updates from old data do not overwrite updates from new data, and Martin et al. (2021) relabels successful episodes as expert demonstrations for the agent to match. The distributional soft actor-critic (DSAC) Duan et al.) (2021b): Ren et al. 2220);; Ma et al. (2200) Duan et al. (2021c) combines the distributional return function within the maximum entropy to improve the estimation accuracy of the Q-value. It claims to prevent gradient explosion by truncating the difference between target and current return distri- butions, however, its assumptions of Gaussian distributions for random returns will induce more complexity and may not fit with the real distributions. [Akimov (2019): Hou et al. (2020) reparame- terize the reward representation and the policy, respectively, using neural network transformation composed of multivariate factorization, and Ward et al. (2019) constructs normalizing flow policies before applying the squashing function to improving exploration within the SAC framework.
It empirically shown that SAC is sensitive to the temperature hyperparameter. To provide flexi- bility for the choice of optimal temperature, SAC-v2 Haarnoja et al. (2018b) makes the first step to automatically tune the temperature hyperparameter by formulating constrained optimization prob- lem for the average entropy of policy. The dual to the constrained optimization will add an additional update procedure for the dual variable in determining the temperature. However, the assumption of convexity for theoretical convergence does not hold for neural networks, and the extra hy perparaaee- ter introduced by the transformation remains undetermined and needs more trials for generalization. Meta-SAC Wang & Ni (220)) uses metagradient along with novel meta objective to automatically tune the entropy temperature in SAC. It distinguishes metaparameters from the learnable parame- ters and hyperparameters, and uses some initial states train the meta temperature. However, due
2

---

Under review as a conference paper at ICLR 2023
to the limited data pool for the meta loss, the given experimental results show it does not perform better than SAC. Therefore, the auto adjustment of the temperature hyperparameter is still openly untouched for SAC.
The way to compute the target Q-value is an crucial design in DRL.
The strategies include delayed update Van Hasselt et al. (2016), soft updates Lillicrap et al. (2013); Haarnoja et al. (0188) and
sophisticated ensembles Fujimoto et al.] 2019); Kumar et al. (2199) of target Q-value. The sophis- ticated ensemble is some weighted mixture of the minimum and maximum among multiple learned Q-value functions, for example, TD3 adopts the minimum of pairwise critics, and bootstrapping error reduction (BEAR) Kumar et al.] (2019) increases the number of Q-functions to 4. Behavior regularized actor critic (BRAC) Wu et al. (2019) investigated these design choices and concluded that the number of Q-functions over only gives marginal improvement but significantly requires more computation cost. It is reported in WW et al. (2019) that the minimum of two Q-functions adopted in TD3 outweighes a weighed mixture of Q-values in terms of simplicity and efficiency, however, there is wide open unexplored area between them. How to design a mixture of Q-values is still largely left untouched.
3 BACKGROUND
We consider the infinite-horizon Markov Decision Process (MDP) in continuous action spaces, de- noted by the tuple p,, where S is the state space, AA is the action space, p(-18, a) is the transition probability of the next state s' E S conditioned on the current state s E S and action E A, and E S x is the reward which is the feedback from the environment of the current state and action a. The task of reinforcement learning (RL) is to learn an optimal policy that maximizes the reward return denoted by the expectation of discounted cumulative reward. DRL combines the neural networks with RL so that the reward return can be approximated by parameterized function, where the agent follows a behavior policy т to determine future rewards and next states. Let p(-1s, a) denotes the transition probability, then the surrogate function of the reward return can be selected as the action-value (Q-value) function with respect to the state-action pair in the form of
8 Σ 2'r((s,, ar) = 5, a0 = a
Qx a) = Bp
(1)
where r(s, a) is the immediate reward produced by the state-action pair, and y E (0.1) 1) is the discount horizon factor for future rewards. With the effect of behavior policy æ, p= (1xa (80; ao)
p(s a II. (ht----=((slss_1,.44 episode given the initial state-action pair (so, ao), and TT (8c41) indicates the probability for the agent to choose the action a++1 given the state 8:+1.
is the joint probability of all state-action pairs during an
Since the reward return has the property of Bellman equation, the temporal difference (TD) Tesauro (1995) is commonly used in the critic evaluation to minimize Bellman errors over sampled transition- (s, a, r, s'), which is given by E(,.a,r,"") [(r + 7Q'(s', x(s")) Q(s,, a}) Lillicrap et al. 2015 where Q' stands for the target Q-value. In off-policy algorithms using experience replay, (s, a, is the tuple stored in the replay buffer at every environment step, a is sampled from the experience pool, which is different from the on-policy next action (ss'). In the context, we use the term of 'iteration' to represent the index of updates. In the actor-critic paradigm, one iteration contains the evaluation step and the policy improvement step, which are used update Q-value function and then optimize the policy. After the minimization of Bellman errors, the policy improvement is per- formed by maximizing the expected return (()) = EE, [Q, (s, (s)]. J. In some algorithms. the policy regularization may be attached to the expected return to smooth training Kumar et al. (2019):[aque) al. (2019). These methods focus on constraining the policy gradient VoJ(B) to avoid gradient vanishing or exploding problems, which in turn reduces the estimation variance.
4 ADAPTIVE PAIRWISE CRITICS WITH ADAPTIVE ASYMPTOTIC ENTROPY
Value penalty policy regularization is common theme in DRL to improve stability. however, it tends to bring more hyperparameters for tuning, which will increase the difficulty for the designed algorithm to generalize more tasks. Therefore, it is important for reasonable auto-adjustment for
3

---

Under review as a conference paper at ICLR 2023
these hyperparameters. The approaches to these adaptations are varied. For example, PPO adapts the penalty coefficient by setting some threshold values for the KL divergence, SAC-v2 Haarnoja et al. (2018b) automatically tune the temperature hyperparameter by adding a constraint solved by related dual form, and Meta-SAC transforms the temperature hyperparameters into metaparameters.
Our work be started by addressing policy iteration method accompanying the adaptive pairwise critics and entropy estimation. We will first justify the adaptive pairwise critics and the adaptive asymptotic entropy, and verify the convergence of corresponding iterations, then organize the related algorithm with its usage of neural networks.
4.1 ADAPTIVE PAIRWISE CRITICS
The iteration of adaptive pairwise critic and adaptive asymptotic actor (APAA) is started by com- puting the revised target Q-value of rollout following policy ग,, which is combined with value penalty from the entropy exploration. Given the continuous MDP denoted by A, p, r), functions Q1 Q2 S x A + R can be the Q-values of two critics, then modified Bellman backup operator T is given by
Q" at) = (s1,, az) + 7 [Q(s++1, ((41)
where 8441 p(, an) and uc+1 (((186+1). and
(ss,, = Q(se, 0x) ((A(s)) + R.) log(T(a)(s:))
(3)
the APAA Q-value function, which can be obtained by repeatedly employing 7 for any policy n.
A(s,) ) is the adaptive random variable (ARV) dependent on the state st, S hs S is the asymptotic variable gradually increasing from 0 to 1 as the time step proceeds, and a is the fixed temperature hyperparameter. The sum of ARV and the asymptotic variable compose the adaptive asymptotic temperature for the entropy. The joint Q-value function Q is formularized as
(ss,, ..) (1 ((s:)) (Se, a²) + (2s))Q((ee))) at);
where 0 < r(.) I is the state-dependent adaptive random weight (ARW) to adjust the influence of two critics.
Lemma 1. Consider the sequence Q++1 = T*Qu, then given the condition that the Q-values are bounded, i.e. Q2(s, a)l < sc, a)) < oc, ass.)) E S A, the sequence Qx will converge to unique optimal value as k + xx.
The proof of LemmaBBcan be found in Appendix (aeendixAA((inn (in Supplementary Files), however, the sufficient condition is not always satisfied when the function approximation is applied. Since the state-action spaces are continuous and the transition probability is unknown in model-free DRL, the Q-value function cannot be formulated or tabulated by the state-action pairs, which means the function ap- proximation gives no absolute guarantee for the bounded Q-values. Therefore, instead of repeatedly applying (2) directly by equality, the practical evaluation step estimated by minimizing the ex- pected mean square error (MSE) between T* a) and Q1,++ (s, a) or Q2, i a). Once the expected MSE converges to zero, the two Q-value functions updated based on (2) will end up with little fluctuation around the same fixed point when the hyperparameters are chosen properly.
5 ADAPTIVE ASYMPTOTIC ENTROPY
When it comes to the policy improvement step, the purpose of APAA iteration contains two points, which aim improve the Q-value for each update while as projecting the policy onto normalized distribution. The policy update step is given by
neww
= arg max Exya. [Q(se, az) ((((())) + k,) log(T(ar(ss)))
r
= arg min Es, DKr. (T-111))) exp (a((s)) + ocks -
Q(sr.))
))
(5)
#E]l
4

---

Under review as a conference paper at ICLR 2023
where 8t ~ -., (-)(s)), (1 and the choice of policy is limited set of parameterized Gaussian distributions II for flexibility. With modest computation, the second equality holds, which can be found in Appendix B The KL divergence 1 shows that the improved policy is updated towards the distribution constituted by the exponential of the normalized Q-value function. The adaptive asymptotic temperature (se) he is not dependent on the action, and thus does not contribute to the policy gradient. However, it is still important to be chosen for a better insurance of the expected policy improvement.
In continuous control problems of model-free DRL, where the transition probability is unknown and the state and action spaces are both continuous, it is not possible to provides policy improvement at every state-action point over S x A, which we call as the absolute policy improvement in this paper. Therefore, we propose practical standard for the policy improvement, which is named as the expected policy improvement. It shows that the projected policy in (17) can produce higher updated Q-value with expression given by ①. and the result is organized Lemma
Lemma 2. Denote Nnew and aold as the policies before and after the update defined in respec- tively. Then the expected policy improvement, i.e., (xx,,))))xx(( (Q (St, at) - Quii (staa)) 2 0, can be guaranteed.
The proof of Lemma 4 can be found in Appendix C Besides projecting the policy into selected set
of distributions, (17 also maximizes the expectation of APAA Q-value function defined in (3) by choosing the specific adaptive asymptotic temperature A(sz) + ke, which is the key to the guarantee of the wanted expected policy improvement, shown in the second step of proving Lemma [ Fur- thermore, in discrete control problems, where the state-action spaces are both discrete and bounded, the absolute policy improvement can be realized by removing the expectation over 8 ~ S in (17).
The APAA iteration alternates between the policy evaluation and the expected policy improvement steps, and will converge to the optimal policy which provides higher expected Q-value than the other policies in II. The theorem describing the APAA iteration is organized in
Theorem 1. Let le be the learning rate at time step t, then given the condition that
0<4(x) S Σ 4(x)) 0o, E((()= oo w.p.l..
(6)
repeated application of policy evaluation and expected policy improvement will converge to op- timal policy x* E I such that ((er,a))SSxAA [Q (8t, az) - Qr at)l 2 O, Va E II.
Proof
See Appendix D
5.1 ALGORITHM OF ADAPTIVE PAIRWISE CRITICS WITH ADAPTIVE ASYMPTOTIC ENTROPY
We have discussed above the practical scenario of Theorem 2 in large continuous domains, which requires parameterized function approximations for both the Q-value function and the policy. To stabilize the training process, separated current and target networks are provided for both the Q- value function and the policy. Based on these parameterized networks and (2). the loss function for the update of critic parameters in the policy evaluation step be estimated by
[ ( Q(, Q(.,a))) -
I(w") B (ss..,rer)
where a' = no (6') is the action following the target policy parameterized by 0', and (s,a,r,8") 1',,88)) is tuple of history data sampled from the experience pool. And
Q' a²) = (1 - ( (s') a²) + T (s')
- -a(Ax (s') + log (a'1s')),
(8)
((x,a) (1 ()))),))))Q@ua( (s, + +Tr()))0
where w/, w², " and "² parameterize two critic networks and their target estimates, respectively. Besides, Ax is target ARV parameterized by X'. is is the asymptotic variable increasing from 0 to as the time step proceeds, the state-dependens ARW parameterized by f and p is clipped in [o. 1] to determine the influence of two Q-value functions, and mor (-Is') is the target policy distribution
5

---

Under review as a conference paper at ICLR 2023
the critic parameters be updated for each policy evaluation step. Then 7 can be optimized with stochastic gradient
conditioned the next state s'. By minimizing 7
Vii ((w") B (
())s,,)(((ss - --
(Q'(s', -
for iE (1,
(10)
It is noticeable that two extra networks has been added for ARV and ARW. however, avoid in- troducing extra more saddle point problems, we see their trainable parameters as part of the actor parameter, i.c., the actor parameter is composed of the policy parameter, the ARV parameter and the ARW parameter. Then the surrogate objective function update the current actor parameter (0, X, uu) in the expected policy improvement step (see Lemma 4 can be given by
J(0,A, (a) E. [Q(s, a) a(Ax(s) + k)) log(TT((ass))
(11)
where 8 comes from the tuple of history data, a (s) is the reparameterized action based on and the policy network parameterized by 0. Ax current ARV parameterized by A, I' is current ARW parameterized by μ., and To (-(+))) is the current policy distribution conditioned on the current
state By maximizing (II), the actor parameter can be updated for policy improvement each step. The gradient of [II is computed as
VoJ(0,A, = Es (0 (VoTo(a()(AA((s)) k.)
(12)
To(a|s)
@AJ(0. = EE, (
(13)
(n, A. = E. (})((s,))
(14)
Then the target parameters (wtt w2, 0', N, n²) are updated following the "soft" target updates Lill- icrap et al. (2015) by (w w M)), in the way of
w" 4+1 +- rul+1 (1 T)u", for ie (1,2)
0/ 70++1 (1 (1-T))0%
N44 + Th (1-T))1/
M441 f- 7/4+1 (1 T))p
(15)
where 0 < is the factor to control the speed of policy updates for the sake of small value error at each iteration, and 0 S TI < set as our application. organize above pro- cedures the adaptive pairwise critics with adaptive asymptotic entropy (APAA) algorithm, whose pseudocode described by Algorithm The algorithm alternates between running environment steps to collect experience and updating the network parameters using the stochastic gradients com- puted by the sampled batches from the experience pool. In (00).
(33) and (I4), the gradients in their expectation forms, however, practically they averaged the results of sampled
ples, which usually follow policies parameterized by different parameters off-policy methods. In some algorithms, one gradient step follows one or several environment steps stabilize the training process.
EXPERIMENTS
6.1 BENCHMARKS
The performance of our proposed method is compared with several prior model-free reinforcement learning algorithms in terms of the sample complexity and stability on set of gym continuous control tasks from the MuloCo suite Todorov al. (2012); Brockman al. 2016). Fig. T shows the illustrations of benchmarks adopted in this paper.
6.2 BASELINES
The adopted baselines include deep deterministic policy gradient (DDPG) Lillicrap et al.] (2015). TD3, SAC and BRAC. Before the existence of SAC, DDPG is regarded of the most efficient

---

Under review as a conference paper at ICLR 2023
Algorithm APAA Algorithm
1: Input: The update maximum time step T
2: Initialize parameters w + w w² +- wis +- 0g X Ao
0, X. Xy- p t 11 4: Initialize the learning rates les La for the critic and the actor, the time step 4- 0, the soft update hyperparameter т, the maximum time step I', the batch size B and the replay buffer D t- 0. do
3: Initialize target parameters w't <- w _ wi
5: while
6: Select action a; No, (at]sa)
7: Observe the reward and next state 8141,77 ~ p(88+((11s1 0c)
8: Store transition D <- {(sr; aes Te, St+1)
9:
Sample a batch of transitions for each time step do
(s, a, r, s') from D
10:
11:
w/+1 < wi - Le wt [(wf) for E 21 following Eq. 10)
12:
Oc+1 < 0t + la Vo, (O,, de. following Eq.
(2)
d++1 + d + la Vx, [(
J Ae: M4) following Eq.
14:
Me+1 se + la
(01, de, Me) following Eq.
15:
wh r+1 t Tw/+1 + (1 7) w for i E ) following Eq. 15
16:
end for
17:
81+1
18:
f t ++1
19:
end while
(a)
(b)
(c)
(d)
Figure 1: (a) Ant-v3; (b) Halfcheetah-v3; (c) 33pper-v3; (d) Walker2d-v3
off-policy DRL methods Duan et al. 2016), followed by TD3 as an extension. SAC has achieved model-free state-of-the-ar sample effciency multiple challenging continuous control domains Christodoulou (2019), and BRAC is can be seen variant of SAC by adopting extra policy regularization based on the KL divergence between policies before and after updating.
Our proposed algorithm shares the same set of hyperparameters with other baselines to keep fairness. The gaussian exploration noise with a fixed variance of 0.2 is added to the action at every time step, then the noisy action is clipped within the set boundary. With the discount horizon factor chosen as 0.99, algorithms including the proposed one, SAC and BRAC adopt the entropy term, which is computed by normal random policies, whose mean and variance parameterized by fully connected networks with two hidden layers, each of which has 256 units. Except that, both DDPG and TD3 use deterministic policies, also parameterized by fully connected networks with two hidden layers. We organize the network architectures and hyperparameters in Appendix [E] and respectively. The Adam optimizer Kingma & Ba (2014) used update the network parameters.
6.3 RESULTS
We train seeds for each algorithm to keep a fair comparison. After every 500 iterations (time steps), launch evaluation procedure, which averages 10 rollouts for test. The average reward of a test will be recorded at every evaluation procedure, and all tests throughout the time step seale give the result of each algorithm.
The average rewards of algorithms tested in chosen benchmarks are shown Fig. with 95% con- fidence interval (CI). From Figs. 5(a) 5(b) and 5(c) APAA over other baselines. In Hopper environment, since the converged value far lower than other benchmarks, the tolerance for the fluctuation around convergence is much lower, which causes
observe overwhelmed advantage of

---

Under review as a conference paper at ICLR 2023
5030
Ago
0000
Ngo
DDPG
DDPG
2000
1000
BRAC
BRAC
APAA
APA
2000
0.3
02
0.8
1.0
12
14
000
0.5
1.0
1.5
2.0
2.5
Time stpps
160
Thmle stap2
1e0
(a)
(b)
3600
8000
3000
Auaraga rooare
Zuaraga ranard
101
103
BRAC
SAC
2000
BRAC
SAC
1000
500
0.8
staps
stops
(c)
(d)
9000
4000
Ago
DDP3
3000
TD3
2000
APMA
0.2
0.5
0.5
0.55
10
1.2
Time steps
160
(c)
Figure 2: Average reward versus time step in (a) Ant-v3; (b) Halfchectah-v3; (c) Hopper-v3; (d) Walker2d-v3; (e) Humanoid-v3
the instability problems of tested baselines. However, APAA shows strong robustness and has con-
verged value up to 3500 compared with other baselines, as shown in 5(c) Fig. 5(d) APAA still shows better performance than that of other baselines, and converges around 5000. For Humanoid with high-dimensional action space, Fig. [(e) shows that APAA is much better than other baselines and can steadily converge 6000. Over all figures in Fig. 2 SAC and BRAC both have their up and downs, and DDPG gives the worst performance, considering the fact that the double critics are not employed in DDPG reduce potential overestimation. Due the stochastic property of random
8

---

Under review as a conference paper at ICLR 2023
3800
3000
Auarago F200833
Araraga Fanoor3
2000
4000
Aly
2000
AFKA
1500
APAA
2000
500
Iime stops
steps
(a)
(b)
300
4000
Aua rawaadd
Sagage rawaadd
8000
Alio
Alao
SAC
3000
SAC
2000
1000
0
0.5
3.4
30
10
12
144
Time
165
Time
165
(c)
(d)
Figure 3: Comparison between APAA and automating entropy adjustment SAC in (a) Ant-v3; (b) Hopper-v3; (c) Walker2d-v3; (d) Humanoid-v3
variables, the best performance cannot be ensured for every seed, which means the potential reduced stability and convergence (partly told by Fig. 2 are reasonable.
Since SAC has an variant working on automatic adjustment of the temperature hyperparameter Haarnoja al. (2018) 8b), use SAC-t represent it and compare its performance with APAA in Fig. 3 with 95% CI. SAC-t adds an extra hyperparameter as target entropy in exchange of the temperature, which may not lead to better performance because the target entropy cannot be generalized and also needs automatic tuming, as reported by Wu et al. (2019). According to Fig, 3 SAC-t fails to produce better performance than APAA given the choice of the target entropy 0.5, which implies the right way adjusting the temperature APAA.
Due to the page limit, make the comparison of value estimates in Appendix G
7 CONCLUSION
In this paper, we proposed state-dependent adaptive temperature to encourage policy exploration, which can strike a better balance between the efficiency and stability by introducing an asymptotic maximum entropy. Then the asymptotic maximum entropy is combined with the adaptive pairwise critics to benefit the policy evaluation and improvement steps. Based on the above two components, present APAA to gain better tradeoff between efficiency and stability. We evaluate method on a set of Gym tasks, and the results show that the proposed algorithms have better performance than several baselines continuous control.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dmitry Akimov. Distributed soft actor-critic with multivariate reward representation and knowledge distillation. arXiv preprint 191iv::911..3000 2019.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.013: 01146000001,,,... 2016.
Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprini 1rxv1:1907 2019.
Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng, Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE Transactions Neural Networks and Leurning Systems, 2021a.
Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE Transactions on Neural Networks and Learning Systems, 2021b.
Jingliang Duan, Yangang Ren, Fawang Zhang, Yang Guan, Dongjie Yu, Shengbo Eben Li, Bo Cheng, and Lin Zhao. Encoding distributional soft actor-critic for autonomous driving in multi-lane scenarios. arXiv preprint arXiv:2109.0555 2021c
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep rein- forcement learning for continuous control. In International conference machine learning, pp- 1329-1338. PMLR, 2016.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. 80:1587-1596, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. pp. 2052-2062, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. pp. 1861-1870, 8a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- 2018b.
cations. arXiv preprint 1rXiv:1812.05000
Nicolas Heess, Greg Wayne, David Silver, Timothy Lillicrap, Yuval Tassa, and Tom Erez. Learning continuous control policies by stochastic value gradients. arXiv preprint urXiv:1510.090 2015.
Zhimin Hou, Kuangen Zhang, Yi Wan, Dongyu Li, Chenglong Fu, and Haoyong Yu. Off-policy maximum entropy reinforcement learning: soft actor-critic with advantage weighted mixture pol- icy (sac-awmp). arXiv preprint arXiv:2002.02829 2020.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 1rXiv:1907.00456. 2019.
Diederík Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy 4- learning via bootstrapping emrrr reduction. Advances in Neural Information Processing Systems, 32, 2019.
Tucker,
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 04779, 2020.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: /0.. 2015.
10

---

Under review as a conference paper at ICLR 2023
Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Dsac: Distributional soft actor critic for sessiteni reinforcement learning. arXiv preprint arXiv:2004.15447, 2020.
Jesus Bujalance Martin, Raphaël Chekroun, and Fabien Moutarde. Learning from demonstrations with sacr2: Soft actor-critic with reward relabeling. arXiv preprint urXiv:2110.14466 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-553 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937, 2016.
Yangang Ren, Jingliang Duan, Shengbo Eben Li. Yang Guan, and Qi Sun. Improving generaliza- tion of reinforcement learning with minimax distributional soft actor-critic. In 2020 IEEE 23rd International Conference Intelligent Transportation Systems (ITSC), pp. 1 1-6. IEEE, 2020.
SATINDER SINGH, TOMMI JAAKKOLA, and MICHAEL L. LITTMAN. Convergence results for single-step on-policy reinforeement-larming algorithms. Machine Learning, 38(3);p.287- 308, 2000.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Gerald Tesauro. Temporal difference learning and td od-gammon. Communications of the ACM, 38(3): 58-68, 1995.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcemen- learning. Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum, 1993.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 8026-5033. IEEE, 2012.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Thirtieth AAAI conference on artificial intelligence, 2016.
Che Wang and Keith Ross. Boosting soft actor-critic: Emphasizing recent experience without for- getting the past. arXiv preprint arXiv: /906: 04009, 2019.
Yufei Wang and Tianwei Ni. Meta-sac: Auto-tune the entropy temperature of soft actor-critic via metagradient. arXiv preprint arXiv:2007.01932, 2020.
Patrick Nadeem Ward, Ariella Smofsky, and Avishek Joey Bose. Improving exploration in soft- actor-critic with normalizing flows policies. arXiv preprint arXiv:1906.02771. 2019.
Christopher John Corishh Hellaby Watkins. Learning from delayed rewards. 1989.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.113330. 2019.
Brian D Ziebart. Modeling purposeful adaptive behuvior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. Aaai, volume 8, pp. 1433-1438. Chicago, II, USA, 2008

---

Under review as a conference paper at ICLR 2023
A PROOF OF LEMMA 3
Lemma 3. Consider the sequence Qk+1 = Qu. then given the condition that the Q-values are bounded, i.e. Qi(s, [s) oc, [Q2(s, oc, V(s,a) 8 A, the sequence Qh will converge to a unique optimal value as + oc.
Proof
IT Q(s. -
= |Bect
(((4411
S I1 Mast1mppai 10((5++1, 0(+1) Q'(5++1:
Sy 8441 ~
max
((+1+1,0+11) (((++11;0++1)
=1 2o
(16)
where || coo means the max norm, and is short for p(-1st, az) and x(-(sz). Since the Q-values are assumed to be bounded, the adaptive pairwise critics Q also bounded, then the third inequality holds. We reach conclusion that ((s,, a.) E S A, (T6) holds, which can be rewritten max- norm contraction mapping as - T Q'Il (l S < Q'lloe According to the property of contraction operator, the sequence Q++1 T*Qu will converge to its fixed point.
B PROOF 17
Teew
arg max Eyy,a [Q(st, 0z) - a(((s:)) + + log(=(ar(ss)) HEll
Q(sr.)) (Q(( = arg min E,, DxL (T-11e))l exp (A((s)) + akis )) lEll
(17)
Proof
Tnew arg max E ( [Q(se, a(A(se) + ke) log log(T(ar(s:))) =IIl
Q(sc, ac)
= arg min E. S ((
[og((((a(<))) a(A(s;) + k,))
xell
Q at)
arg min E. R. xC]l
((aolst) [og(TT(a(st)) a(A(s)) + k2) ((sr.))
arg min E. DKL ((-1ss)]l exp a(A(st) + ke)
(18)
#ell
where the second equality holds because A(s:) and k2 are not dependent on a,, and DxL (-ll-) is the KL divergence.
C PROOF OF LEMMA
Lemma 4. Denote nee and Tata as the policies before and after the update defined in 177 respec- lively. Then the expected policy improvement, i.e., E(s,.,))SSAAl ( (St, at) Q Stg az)] 2 0, can guaranteed.
Proof
var(T nnew Quaie QTnen)
-E
[nT
moo Quees QFrenn) ) (IET Tue Qraid Qunee!)²
2
(19)
12

---

Under review as a conference paper at ICLR 2023
where var()) represents the variance. According to 19 we have
E Quaia Qxann 1)
E (T
Q Q Quann)
(20)
nnww) based on the expected MSE analyzed the para- graph after Lemma This constitutes the first step of the proof for the expected policy improvement, which can be written as
then T may will converge to EIQ Rnow
E (( (sc, [r)
-B(vaa))-Sxx (T mm Q (s:
-(xraa))-SxAA T Potld (s1,
B(v(a))-Sxx (rss az)l S (a 1 (9t+1, at+1)l
B(s.ai)-SxxA(((6) (s(c az)l 7. ( ( ($1+1, a++1)l
((e.,a))~SxA1000 (#t; at)];
(21)
where the second inequality holds because of the update rule following the first equality of 17 the third equality is the expected form of modified Bellman backup operator, the forth inequality holds because both the entropy and the adaptive asymptotic temperature are nonnegative, and the last equality variant of the bellman equation. Because of the unknown transition probability and continuous state-action spaces in continuous model-free DRL, the Q-value function is usually approximated by neural networks, which makes impossible to directly apply the bellman equation to every aate-action pair over S x A. Under the circumstance, the bellman equation only holds in statistical sense.
D PROOF OF THEOREM
Theorem 2. Let le be the learning rate at time step t, then given the condition that
054(x) S1, 44()))) 0o, ((x) < w.p.l.
(22)
repeated application of policy evaluation and expected policy improvement will converge to an op- timal policy x* E Ell such that E(er.a.)SSxAQ9.. (st, (s, az)] 0, VT E II.
Proof According Lemma 1 of (SINGH et al. 2000), the condition 22 make the expected
MSE of temporal difference (TD) converge to zero, which validates the policy evaluation of adaptive asymptotic
iteration and prove the first step of 21 to be true. With the monotonic increasing of the updated expected Q-value, the converged optimal policy will render (( (QQ (8t, ac) Qx (Sc. an) 2 O, VT E II.
E NETWORK ARCHITECTURE
We construct the critic network using a fully-connected MLP with two hidden layers. The input is composed of the state and action, outputting value representing the Q-value, The ReLU functions are adopted to activate the two hidden layers. The setting of policy network follows normal random distribution, whose expectation and variance are mully-connected networks fed only by the state. Both of them have two hidden layers activated by the ReL.U function. After the hidden layers, Tanh function and a Softplus function follows form the expectation and variance, respectively. With the expectation and variance, normal distribution policy. The network of state-dependent ARV and ARW I' are constructed similar to either the expectation and variance of policy network except replacing the last nonlinearity activation by Sigmoid function. The architecture of networks are plotted in Fig.
be achieved represent the random
The above mentioned network architecture is adopted for the random policy. For the algorithm using the deterministic policy, the critic constructed the same way. however, the actor network the same that of the expectation of normal random distribution.
13

---

Under review as a conference paper at ICLR 2023
Fully connacted Laver Bolaa
Fally connected layee
Teper
Bola
State Japer
Q
Action layez
Iepet lorer
Bein
Tocch
Ali
diatrsbution
State Tover
Fully conrected Layse
Pilly connettd loper
Bela
Selu
Suftplus
3/11
Pully
Trpes Toper
Jayes
Palia
Bell
Stace layer
A(s
Figure 4: Architecture networks.
F HYPERPARAMETERS
Table lists the common hyperparameters shared by all experiments and their respective settings. In this table, LRta means the learning rate the actor (includes lambda in our proposed algorithm), and LRtc means the learning rate of critics. 7_a and --c represent soft update hyperparameter of the actor and the critic, respectively, and 7-a = means we adopt immediate update for the actor. The symbol represents the variance of gaussian exploration noise, and is the fixed temperature hyperparameter, which applied in algorithms except DDPG and TD3, ar represents the Wight factor of KI. divergence for policy regularization applied in BRAC, and 3 is the asymptotic rise
14

---

Under review as a conference paper at ICLR 2023
Table List of hyperparameters
Hyperparameter
Value
Description
Algorithm applied
LR
0.0003
Learning rate of actor Learning rate of critic Soft update parameter of actor
All
LR_c
0.0003
All
7_a
All
0.005
Soft update parameter of critic Discount horizon factor
All
0.99
All
0.2
The variance of exploration noise
All
a
0.1
Fixed temperature
Except DDPG and TD3
Od
0.1
Wight factor of KL regularization
BRAC
B
0.9995
Asymptotic rise rate
APAA
H
0.5
Target entropy
SAC-t
Batch
256
Size of each mini-batch
All
Units
256
Hidden layer units
All
Memory
1000000
Size of replay buffer
All
Interval
500
Evaluation period
All
Test
10
Rollouts per evaluation
All
rate for he = 1 3r. Besides, we choose the target entropy H. as 0.5 for the automating entropy adjustment of SAC (SAC-t).
Moreover, Batch represents the size of mini-batches sampled for training, and M Memory is short for the size of replay buffer. The rest in Table Tare are the hyperparameters for the evaluation procedure, specifically, Interval means how many time steps between two successive evaluation procedures, and T Test means the number of rollouts run during each evaluation procedure.
G VALUE ESTIMATE
We plot the value estimate, approximated by the trained Q-value networks, over time steps to com- pare with the true value, which is represented by the discount return of rollout starting from 1000 random state-action pairs from the replay buffer. The discount return of a rollout is recorded every 500 time steps, which follows the updated policy at that time step and is different from the average return. The differences between the value estimates and the true values is illustrated in Fig. 5 From these figures,
observe that the algorithm without tuning the target Q-value (DDPG) suffers great overestimation, however, simply choosing the smaller Q-value from pair of critics (TD3 and SAC) will bring nonnegligible underestimation, instead. Since inaccurate value estimates will lead to poor policy updates, neither underestimation or overestimation is wanted. Dynamic adjustment of target Q-value used ARW of APAA provides a preference.
15

---

Under review as a conference paper at ICLR 2023
100
002
400
Ago
Ago
000
DOPO a buw
DDPO O hru
DOPG
TDO The
TDO
Discuut
30C
DDPG O mi
TDO
TD0 Thue
Tuu
Trua
SBG aol
BAG
APAA
ATMA Trus
APAA
APMA
0
0.0
Time shopo
Time shopo
(a)
(b)
602
960
250
DDPO
True
ad
G.
SAC Trus
SAC True
APAA Truo
APHA Truo
A/WA sal
A/WA sol
100
0.0
0.0
0.
1.0
1.4
0.0
1,4
(c)
(d)
100
Ago DDPG
2007
(c)
Figure 5: Comparison between the value estimate and the true value in (a) Ant-v3; (b) Halfcheetah-
v3; (c) Hopper-v3; (d) Walker2d-v3; (c) Humanoid-v3. 'Q True' means the true value and 'Q est' means the value estimate.
16


Reviews:
[{"review_id": "mUXgJPe_jV", "reviewer": "Reviewer_9zpF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper deals with two issues in deep reinforcement learning. One is to reduce the estimation bias by learning a state-dependent weighting function for two Q functions. The other is to tune a state-dependent coefficient of the entropy regularization term of the policy. The state-dependent weighting and coefficient functions are updated as part of the policy parameter. The proposed method is evaluated on the standard MuJoCo benchmarks, and the experimental results show that the proposed method outperforms the baselines. ", "summary_of_the_review": "Although this paper proposes a new idea, the experimental results do not support the main claim. Therefore, the ablation studies discussed above are needed to understand how the proposed method improves performance. ", "strength_and_weaknesses": "Strength\n1. The authors propose a state-dependent coefficient of the entropy regularization term that can be updated during learning. The update rule is different from SAC-v2.\n2. To reduce the estimation bias, multiple Q-value estimates are often used. The proposed method adaptively mixes two Q functions, where mixing weights are state-dependent and learnable. \n\nWeakness\n1. The proposed method does not reduce the estimation bias explicitly. Specifically, the estimation bias was not evaluated in the experiments, although it is one of the main paper's motivations. \n2. The state-dependent coefficient of the entropy term is not explained well. In particular, it is unclear why it is not given as a simple function \\alpha(s). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L_hsjo3rNzy", "reviewer": "Reviewer_gnMQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use adaptive pairwise critics and adaptive asymptotic maximum entropy to address the estimation errors.", "summary_of_the_review": "A deep and thorough analysis to support the authors' claims is lacking.", "strength_and_weaknesses": "Pros: (1) The paper writing is clear.\n\n(2) The experiments show the benefit of the proposed algorithm in several tasks.\n\nCons: (1) The main concern I have is why a per-state adaptive variable/weight benefits training. For example, for the state-dependent adaptive random weight that assigns weights to the two critics, is it actually mitigating the overestimation or something else? \n\n(2) The introduction mentioned bootstrapping error, extrapolation error, overestimation error, etc. I'm confused about how the proposed method addressed them. Analysis that supports the error-mitigation claim is lacking.\n\n(3) I'm also interested in which of the above errors are the contributing factors in policy degradation and which source of error the method addresses. I don't think a single design can address all of them at once since they are clearly not the same type of error.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Gga3oAFi-zS", "reviewer": "Reviewer_djw4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new algorithm over a baseline like SAC with 2 main modifications: a) adaptively learn a weight that balances between 2 critics and controls how the Q target value is computed b) adaptively learn a temperature for max entropy exploration. \n\nThe authors claim the proposed method can better balance the tradeoff between efficiency and stability. The proposed method is evaluated on 5 gym tasks and compared to a number of other baselines to show that the proposed method has better performance. Some theoretical results and some empirical analysis are provided. ", "summary_of_the_review": "The proposed method can be seen as somewhat novel and interesting, however, I have a lot of concerns on writing clarity, and on reliability of the comparison results. Additionally, the paper can benefit from more ablations, hyperparameter sensitivity study, and the claim on tradeoff should be backup up with more quantitative comparisons. \n\nA bit difficult to recommend the paper at its current state. ", "strength_and_weaknesses": "Strength:\n- technical details: 10 seeds for each experiment, and hyperparameter table in the appendix, it seems in most cases the same hyperparameter setting is used, which is good and makes results more reliable. \n- Novelty: adaptive bias control can be an interesting and somewhat novel method\n- Results: compared to baselines are somewhat good assume they are reliable\n- Simplicity: method is not too complicated\n\nWeaknesses: \n\n**Technical details**\n- With the additional parameters and the extra complexity, how does your method compare to others in terms of computation and parameter efficiency? \n- Although your adaptive parts are learned, there are some new hyperparameters, how hard it is to tune them? \n- You are combining two main modifications, might want to ablate them\n\n**Reliability of results and fairness of comparison**\n- Why does SAC get 2000 in HalfCheetah? SAC is a very popular baseline and should get at least about 12K score at 1e6. \n- For Figure 3, it seems a target entropy of 0.5 is chosen for SAC adaptive? Wouldn’t it reduce the performance of SAC adaptive if you arbitrarily set a temperature? What makes that a fair comparison? \n\n**related work**\n- “How to design a mixture of Q-values is still largely left untouched. ” not exactly sure if I agree with that. Figuring out how how get more accurate Q values is certainly important, but there are a number of other recent works that deal with this, in the paper there is no discussion on how your work relate to or has an advantage over these newer methods. (though I do acknowledge these works do not study adaptive bias control methods)\n- Randomized Ensembled Double Q-Learning: Learning Fast Without a Model (ICLR 2021)\n- Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (ICML 2020)\n- Dropout Q-Functions for Doubly Efficient Reinforcement Learning (ICLR 2022)\n\n**soundness of arguments**\n- It is unclear to me how the stability issue is evaluated. And why is there a tradeoff between efficiency and stability? Shouldn’t a more stable algorithm be also more efficient? The discussion on this point seems to be a bit vague, and there is no quantitative analysis on the results, for example, if it is about stable performance across seeds, you might want to measure that and show your method actually works better (has a better measured value) than other methods. If it is about more stable performance over test runs for a single training seed, you want to measure sth else. \n\n**writing and clarity**\n- Some minor typos or citation formatting issues, for example:\n- “the twin delayed deep deterministic policy gradient (TD3) Fujimoto et al. (2018) for example”\n“Several of recent works deal with errors like bootstrapping error caused by out-of-distribution (OOD) actions Kumar et al. (2019; 2020)” \n- Some parts of writing is not clear for example “In Hopper environment, since the converged value is far lower than other benchmarks, the tolerance for the fluctuation around convergence is much lower, which causes…” would be good to go through the paper carefully and rewrite some of the arguments for better clarity. \n- and there is a number of other issues please go through your paper and fix these writing issues. \n\n**novelty**\n- not exactly sure how novel the adaptive entropy part is... SAC adaptive already has some adaptiveness there\n\nOther questions:\n- is beta your only new hyperparameter? If so you might want to emphasize it\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7QJSWiw_ZWg", "reviewer": "Reviewer_6A7k", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies maximum entropy reinforcement learning in an actor-critic setting. Two improvements are proposed. First, a learnable state-dependent weighting between two critics is intended to balance over- and underestimation bias of the value estimates provided by the critic. Second, the temperature parameter of the entropy regularizing the actor is also made a learnable state-dependent variable in order to smooth the entropy policy exploration. The suggestions are incorporated into SAC and results are presented.", "summary_of_the_review": "The paper does not convince me why the proposed method makes conceptual sense. The evaluation is not thorough and hence also does not convince me that the proposed method is very useful.", "strength_and_weaknesses": "The paper is written clearly. However, the organization of the paper does not convince me. A lot of space is used to describe the base algorithm SAC and very little space to describe the suggested method. For example, equation 7 and especially equation 15 describing the soft update seem unnecessary to state in the paper. Instead, I suggest the authors include at least a short description of the function modeling the state-dependent factors in the main paper and not just in the appendix. Further, Figure 1 showing the mujoco environments is not needed in the main paper and Figures 2 and 3 could be made such that they use a lot less space by plotting the legend only once and rearranging the plots in one or two rows. This would allow the experiment about the value estimates to be in the main paper.\n\nConceptually, I do not understand and also can not see a given argument why updating the parameters of the state-dependent networks $\\lambda$ and $\\mu$ according to eq. 11(i.e. the actor loss)  makes sense.\n\nEvaluation:\nThe evaluation is poor. There is no reason why the proposed method APAA is compared with several baselines in Figure 2 and an extra Figure is used to compare it with SAC-t (SAC with automatic entropy tuning). This could and should all be in one Figure. Then suddenly HalfCheetah is not showing in the second Figure and judging from the performance of APAA on HalfCheetah in Figure 2 this is because it is far worse than SAC with entropy tuning. Further, the baselines for HalfCheetah in Figure 2) have clearly worse performance than what is otherwise reported in the literature.\nI do not see justifications for the claim that APAA has an 'overwhelmed advantage [...] over other baselines'.\n\nRelated work: There are some papers coming to my mind that are relevant but not mentioned. In [1] a parameter Is learned to penalize overestimation of the critics. [2] adapts a parameter online during training that balances over-and understimation of the critics. Other works use a (stochastic) weighting of critics to generate better value estimates [3, 4].\n\n\n[1] Cetin, Edoardo, and Oya Celiktutan. \"Learning pessimism for robust and efficient off-policy reinforcement learning.\" arXiv preprint arXiv:2110.03375 (2021).\n\n[2] Dorka, Nicolai, Joschka Boedecker, and Wolfram Burgard. \"Adaptively Calibrated Critic Estimates for Deep Reinforcement Learning.\" Deep RL Workshop NeurIPS 2021. 2021.\n\n[3] P. Lv, X. Wang, Y. Cheng, and Z. Duan. Stochastic double deep q-network. IEEE Access, 7:\n79446–79454, 2019. doi: 10.1109/ACCESS.2019.2922706.\n\n[4] Zongzhang Zhang, Zhiyuan Pan, and Mykel J. Kochenderfer. Weighted double q-learning. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI’17, pages 3455–3461. AAAI Press, 2017", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject