Revisiting Discrete Soft Actor-Critic
Authors: haibin zhou
Tong Wei
Zichuan Lin
junyou li
Junliang Xing
Yuanchun Shi
Li Shen
Chao Yu
Deheng Ye
Venue: Accepted by TMLR ()

Abstract: We study the adaption of Soft Actor-Critic (SAC), which is considered as a state-of-the-art reinforcement learning (RL) algorithm, from continuous action space to discrete action space. We revisit vanilla discrete SAC and provide an in-depth understanding of its Q value underestimation and performance instability issues when applied to discrete settings. We thereby propose Stable Discrete SAC (SDSAC), an algorithm that leverages entropy-penalty and double average Q-learning with Q-clip to address these issues. Extensive experiments on typical benchmarks with discrete action space, including Atari games and a large-scale MOBA game, show the efficacy of our proposed method. Our code is at: https://github.com/coldsummerday/SD-SAC.git.

Published in Transactions on Machine Learning Research (11/2024)
Revisiting Discrete Soft Actor-Critic
Haibin Zhou
husbinchoon@eennenllccom
Tencent Inc.
Tong Wei Tsinghua University
MM22@maiis.tsiinnhuuuuuu edu..c
Zichuan Lin
aichuamlkn@@eeecennl.comm
Tencent Inc.
Junyou Li Tencent Inc.
jungouul@@tenceel.ccom
Junliang Xing
jkring@tsinghuua.cd edu..cm
Tsinghua University
Yuanchun Shi Tsinghua University
shiyc@lsinghua.co
Li Shen Sun Yat-sen University
mathshendii@mmail.cco
Chao Yu Sun Yat-sen University
guchao3@@mill.syyee www.suedd..oom
Deheng Ye Tencedd
dericye
Reviewed on OpenReview:
https: // openreview. net/ forum? id= EUF2R6VBeU
Abstract
We study the adaptation of Soft Actor-Critic (SAC), which considered as state-of- the-art reinforcement learning algorithm, from continuous action space to discrete action space. We revisit vanilla discrete SAC, i.e., SAC for discrete action space, and provide in-depth understanding of its Q value underestination and performance instability issues when applied lo discrete sellings. We thereby propose Stable Discrete SAC (SD-SAC), an algorithm that leverages entropy-penalty and double average Q-learning with Q-clip to address these issues. Extensive experiments on typical benchmarks with discrete action space, including Atari games and a large-scale MOBA game, show the efficacy SD-SAC. Our code httpp://githhni com./colleuummerrrr//--AAAA....
1 Introduction
In the conventional model-free reinforcement learning (RI.) paradigm, an agent can be trained by learning
an approximator of action-value (Q) function (Mnih et al. 2015 Bellemare et al.) 2017). The class actor-
critic algorithms (Mnih al. 20166: Fujimoto value function. Motivated by maximum-entropy RL /iiebrr
2018 evaluates the policy function by approximating the
2008 Rawlik et al. 2012 Abdolmaleki al. 2018), soft actor-critic (SAC) (Haarnoja et al. 2018a) introduces action entropy in the framework of
* *Corresponding author.

---

Published in Transactions on Machine Learning Research (11/2024)
actor-critic to balance exploitation and exploration. It has achieved remarkable performance in range of environments with continuous action spaces (Haarnoja et al. 2018b), and is considered as the state-of-the-art algorithm for domains with continuous action space, e.g., Mujoco (Todorov et al. 2012).
However, while SAC solves problems with continuous action space, it cannot be directly applied to discrete domains since it relies on the reparameterivattinn of Gaussian policies to sample actions, which the action in discrete domains is categorical. Soft-DQN (Vieillard et al.] (020) provides a simple way to discretize SAC by adopting the maximum-cntropy RL to DQN (Mnih et al. 2013). However, Soft-DQN utilizes only Q-value parametrization to bypass the policy parameterization. Another discretization of the continuous action output and Q value in vanilla SAC is suggested by previous work (Christodoulou 2019) to adapt SAC to discrete domains, resulting in discrete SAC (DSAC). However, it is counter-intuitive that the empirical experiments in subsequent efforts (Xu et al.) 2021 indicate that discrete SAC performs poorly in discrete domains, e.g., Atari games. We believe that the idea of maximum entropy RI, applies to both discrete and continuous domains. However, extending the maximum-cntroe cottamp--nnttobseeddd SAC algorithm to discrete domains still lacks commonly accepted practice in the communily. Therefore, in this paper, similar the motivation of DDPG (deep deterministic policy gradient) (Lillicrap et al. 2016), which adapts DQN (deep Q networks) (Mnih et al. 2013) from discrete action space to continnous action space, we aim to optimize SAC algorithm for discrete domains.
Previous studies (Xu et al. 2021 Wang & Ni 2020 have analyzed the reasons for the performance disparity of SAC between continuous and discrete domains. Reviewing from the perspective of automating entropy adjustment, an unreasonable setting of target-entropy for temperature a may break the SAC value value-entroon entropy trade-off (Wang & Ni. 2020 Xu et al. 2021 Furthermore, the function approximation errors of Q-value are known to lead to estimation bias and hurt performance in actor-critic methods (Fujimoto et al. 2018). To avoid overestimation bias, both discrete SAC and continuous SAC resort to clipped double Q-learming (Fujimoto et al.) 2018) for aetor-critic algorithms. On the contrary, using the lower bound approximation to the critic can lead to underestimation bias, which makes the policy fall into pessimistic underexplored, as pointed by (Ciosek el al. 2019 Pan el al. 2020), mainly when the reward is sparse. However, existing works only focus on continuous domains (Ciosek et al. 2019 Pan et al. 2020), while SAC for discrete cases remains less explored.
In addition to the abovementioned issues, we conjecture that discrete SAC fails also due to the absence of policy update constraints. Intuitively, the unstable training causes shift in the Q function distribution and policy entropy, which generates a rapidly changing target for the critic network due to the soft Q-learning objective. Meanwhile, the critic network in SAC needs time to adapt to the oscillating target process, exacerbating policy instability.
To address the above challenges, we first design test cases to replicate the failure modes of vanilla discrete SAC, exposing its inherent weaknesses regarding training instability and Q-value underestimation. Then, accordingly, we propose Stable Discrete SAC (SD-SAC) to stabilize the training. We develop an entropy penalty on the policy optimization objective to constrain policy updates. We also develop double average Q-learning with Q-clip to confine the Q value within a reasonable range. We use Atari games (the default testbed for the RL algorithm for discrete action space) to verify the effectiveness of our optimizations. We also deploy our method to the Honor of Kings 1v1 game, a largo-scale MOBA game used extensively in recent RL advances (Ye et al. 2020b ca Stable Discrete SAC.
Wei et al. 2022). to demonstrate the scale-up capacity of our
To sum up, our contributions are:
We pinpoint two failure modes of discrete SAC, regarding training instability and underestimated Q values. We find that the underlying causes are the environment's deceptive rewards and SAC's double Q learning respectively.
To alleviate the training instability issue, we propose entropy- sarropy-penalty to constrain the policy update in discrete SAC.
To deal with the underestimation bias of Q value in discrete SAC, we propose double average Q- learning with Q-clip to estimate the state-action value.
2

---

Published in Transactions on Machine Learning Research (11/2024)
With the above contributions, we have obtained the Stable Discrete SAC (SD-SAC) algorithm. Extensive experiments on Atari games and a large-scale MOBA game show SD-SAC's superiority compared to baselines, with a 68% improvement of normalized scores in Atari and around 100% ELO increase in the Honor of Kings 1v1 game environment.
2 Related Work
Adaptation of Action Space. The most relevant works to this paper are vanilla discrete SAC (Christodoulon) 2019), TES-SAC (Xu et al. (021)) and Soft-DQN (Vieillard et al.) 2020). Discrete SAC replaces the Gaussian policy with categorical one and discretizes the Q-value output to adapt SAC from continuous to discrete action space. However, as will point out, direct discretization of SAC will have specific failure modes with poor performance. TES-SAC proposes new scheduling method for the target entropy parameters in discrete SAC. Soft-DQN has discretized SAC by adopting the maximum-entropp RL to DQN, utilizing only a Q value parametrization and directly applies a softmax operation to the Q-values to take action.
Q Estimation. Previous works (Fujimoto et al., 2018 Ciosck et al., 2019 Pan et al. 2020 Duan et al. 2021 have already expressed concerns about the estimation bias of Q value for SAC. SD3 (Pan et al, 2020 proposes to reduce the Kurtosis distribution of Q approximately by using the softmax operator on the original Q value output to reduce the overestimation bias. OAC (Ciosek et al. 2019) constrains the Q value approximation objective by calculating the upper and lower boundaries of two Q-networks. Distributional SAC DDaa et al., 2021) replaces the Q learning target. with the expected reward sum obtained from the current state to the end of the episode and uses a multi-frame estimates target to reduce overestimation. Maxmin Q-lcarning (Lan al. 2020) controls estimation bias by minimizing the complete ensemble in the larget. MME (Han & Sung 2021) exlends max-min operation to the entropy framework to adapt to SAC. REM (Agaraal et al.) 2020) ensembles Q-value estimations with a random convex combination to enhance generalization in the offline setting. REDQ (Chen et al. 2021b) reduces the estimation bias by minimizing random subset of Q-functions. AEQ (Gong et al. 2023 adjusts the estimation bias by using the mean of Q-functions minus their standard deviation. However, little research is on discrete settings. Our approach focuses on reducing the underestimation bias for the double Q-estimators to enhance exploration.
Performance Stability. Flow-SAC (Ward et al., 2019 applies a technique called normalizing flows policy contimuous SAC leading to the finer transformation that improves training stability when exploring complex states. However, applying normalizing flows to discrete domains will cause a degeneracy problem (Horval & Pfister 2021), making it difficult to transfer to discrete actions. SAC-AWMP (Hou et al. 2020 improves the stability of the final policy by using a weighted mixture to combine multiple policies. Based on this mothod, the cost of network parameters and inference speed is significantly inercased. ISAC (Bancrjec et al. 2022) increases SAC stability by mixing prioritized and ou-policy samples, enabling the actor to repeat learns states with drastic changes. Repeatedly learning priority samples, however, runs the risk of settling into a local optimum. By comparison, our method improves policy stability in case of drastic state changes with an entropy constraint.
3 Preliminaries
This section briefly overviews the symbol definitions of SAC for discrete action space. Following the maximum entropy framework, SAC adds an entropy term H((T(-1 s)) as regularization to the policy gradient objective:
n" = arginax
E az) + +HH(T(-)) |+))]]
(1)
seou
T
H(T(: 8)) = Σ
((a (ogg((al0)
E [- log ((a |s]
(2)
a-)((ss)
where π is a policy, 5 is the optimal policy, and a is the temperature parameter that determines the relative importance of the entropy term versus the reward r, thus controls the stochasticity of the optimal policy.
3

---

Published in Transactions on Machine Learning Research (11/2024)
Soft Bellman Backup The soft Q-function, parametrized by 8, is updated by reducing the soft Bellman error as described in the next subsection:
1 (r(s), a1) + (V(+)11) Qo(s), Jo(0) 2
(3)
where (s;) defines the soft state value function, which represents the expected reward estimate that policy obtains from the current state to the end of the trajectory.
((se) = (Qu az) - a log(T((; sx))].
(4)
Soft actor-critic minimizes soft Q-function with final soft Bellman error:
(Q( (( (Qo(s,, a.) (r(se, + + +. (
(5)
where D is a replay buffer replenished by rollouts of the policy и interacting with the environment. In the implementation, SAC (Harrnoja et al. 2018a) uses the minimum of two delayed-update target-critic network outputs as the soft bellman learning objective to reduce overestimation, The formula is expressed as
V(s++1) min En 1=12
log(T(ur+1 [844))],
(6)
where Qø; represents i-th target-critic network.
Policy Update Iteration The policy, parameterized by distills the softmax policy induced by the soft Q-function. The discrete SAC policy directly maximizes the probability of discrete actions, in contrast to the continuous SAC policy, which optimizes the two parameters of the Gaussian distribution. Then, the discrete SAC policy is updated by minimizing KT, divergence between the policy distribution and the soft Q-function.
2 (sc..))
argminDKL (Tosia (:1 st)l Z (st)
Thnew Sonid Ell
Note that the partition function 3 (s:) is normalization term that can be ignored since it does not affect the gradient concerning the new policy. The resulting optimization objective of the policy is as follows:
Ja (6) = (aaenDeaaeaaaaaaae [a log(Ts(aa) st)) ax)]]
Automating Entropy Adjustment The entropy parameter temperature a regulates the value-entropy balance in soft Q learning. The SAC paper proposes using the temperature Lagrange term to tune the temperature a automatically. The following equation can be regarded the optimization objective satisfying entropy constraint.
max Ep- Wo:T
Σ (((s) t=
s.t. (o log(TT((; ss))] 2 H. Vt.
(9)
where is the desired minimum expected entropy. Optimizing the Lagrangian term involves minimizing:
((a) ([()) log (a, se) -aH].
(10)
By setting a loose upper limit on the target entropy H. SAC achieves automatic adjustment of temperature Typically, the target entropy is set to 0.98 -log( Acctio) for discrete Siaccete/Christodoolloo 2019) and -dim(Actions) for continuous actions actions(Haarno) Haarnoja et al. 2018b).
4 Failure Modes of Vanilla Discrete SAC
We start by outlining the failure modes of the vanilla discrete SAC and then analyze under what circum- stances the standard choices of vanilla discrete SAC perform poorly.
4

---

Published in Transactions on Machine Learning Research (11/2024)
Lyres (lose life when collected, move horizontally in each round)
Objects (score when collected, move horizontally in each round)
Asterix (player controlled, can move horizontally and vertically)
300
(a) Gameplay screenshot of the Atari Game Asterix
(b) Deceptive rewards in Asterix
Gameplay screenshot of the Atari Game Asterix, including the player-controlled Asterix (yellow box), scoring objects (green box) and life-losing lyres (orange box) that appear rounds. Deceptive rewards appear in the early stage of game when there are only scoring objects.
Figure 1:
4.1 Unstable Coupling Training
The first failure mode comes from the instability caused by fluctuations in Q function distribution and policy entropy during training. The maximum entropy mechanism in SAC effectively balances exploration and exploitation. However, due to the existence of entropy term in the soft Bellman error, and the mechanism in discrete SAC that aligns the policy with the Q function, the policy update itcration (Eq. g) is strongly coupled with Q-learning iteration (Eq.
In environments with deceptive rewards (Hon et al. 2018), the agent can gain substantial returns in the early stages of training through short-term rewards, causing the Q value of specific actions risc rapidly and the Q function distribution to become sharper. The coupling learning paradigm of discrete SAC leads to sharper policy distribution and, thus, a decline in entropy. Consequently, the Q learning target becomes unstable, which can, in turn, deteriorate the policy learning. As result, the agent falls into local optima and struggles to discover alternative strategies with larger long-term payoffs. To illustrate this issue more concretely, we take the training process of discrete SAC in the Atari game Asterix as an example.
As shown Fig. [[a] the player controls Asterix, which can move horizontally and vertically. In each round, horizontally moving objects appear. Asterix scores points by collecting objects and loses life when collecting a lyre. In the early stage of the game, rounds often appear where there are only scoring objects and no life-losing lyres (Fig. [(b]], allowing the agent to score quickly by collecting objects, resulting in deceplive rewards. These rewards make the Q function sharper, thereby reducing the entropy of the policy, In Fig. [[a]] we sample a fixed set of states, and measure the variance of Q function across different actions for these states. We find that the Q function variance increases rapidly, indicating that the Q function becomes sharp quickly. Policy entropy also decreased during this period.
As the learning process continues, the policy entropy drops rapidly, and the action probabilities become deterministic quickly (Fig. ((c)) The agent can collect objects but struggles to avoid obstacles effectively. After the policy entropy reaches its lowest point at round million steps, neither the episode length (Fig. 2(d) nor the number of steps with rewards (Fig. [ increases significantly. At the same time, the drastic change of policy entropy misleads the learning process, and thus, both Q-value and policy fall into local optimum (Fig. ( and Fig. [(b).. J. Since both policy and Q-value converge to the local optimum, it becomes hard for the policy to explore efficiently in the later training stage. Even the policy entropy re-rises the later stage (Fig. [) the performance of policy does not improve anymore (Fig. (( Similar situations also occur in other Atari enviromments, and we provide more examples in Appendix A.6
To better understand why this undesirable behavior occurs, inspect the gradient of the soft Bellman object calculated by the formula,
(o) (: (r(s,, ar) + ((Qo (411) - a log( (t+1)))).
(11)
5

---

Published in Transactions on Machine Learning Research (11/2024)
O Finction Varienee
Pntimation
Policy Entropy
236
320
222-
4.84
3.5
022
102
Timesteps
Timenteps
3x2
(a) Q Function Variance
(b) Quvalue
(c) Entropy
Episode Length
Number of Steps with Rewards
Score
1400
3309
1280
2202
3600
3389
3380
00
022
0.5
3.0
30
62
2.5
0.0
28
3.0
02
04
65
33
20
(d) Episade Length
(e) Steps with Rewards
(f) Score
Figure 2: Measuring Q variance, estimation of Q-value, policy entropy, episode length, steps with rewards, and score on Atari Game Asterix with discrete SAC over 10 million timesteps.
As shown in Eq. 11 the improvement of (st,,, az) relies on the Q-estimation of the following states and policy entropy. However, a sharper Q function causes the drastically shifting entropy, incrcasing the uncortainty of gradient updates and misleading the learning of the Q-network. Since the soft Q-network induces the policy, the policy also become misleading and hurt performance. To mitigate this phenomenon, the key is to ensure the smoothness of policy entropy change to maintain stable training, In the next section, we will introduce how to constrain the policy's randomness to ensure smooth policy changes.
4.2 Pessimistic Exploration
The second failure mode comees from pessimistic exploration due the double Q-learning mechanism. To address the issue of overestimation DQN, double Q-learning was proposed. This approach mitigates the problem by employing two independent Q-networks, and using the minimum value between them as the final Q-value. The concept was initially introduced by Double DQN (Van Hasselt et al.) 2016) in the discrete domain. In the continuous domain, TD3 (Fujimoto et al.] 2018 and SAC (Haarnoja et al. 2018al also adopt clipped double Q-learning to mitigate overestimation, making it a favored technique acToss various reinforcement learning algorithms.
Fappricaal results demonstrate that the clipped double Q-learning trick can boost SAC performance continuous domains, but impact remains unclear discrcte domains. Therefore, we need revisit clipped double Q-learning for discrete SAC.
In experiments, in discrete domains, find that discrete SAC tends to suffer from underestimation bias instead of overestimation bias. This underestimation bias can cause pessimistic exploration, especially in the case of sparse reward. Here, we illustrate how the popularly used clipped double Q-learning trick causes underestimation bias and how the policy used with this trick tends to converge to suboptimal actions for discrete action spaces. Our work complements previous work with a more in-depth analysis of clipped double Q-larnnin. We demonstrate the existence of underestimation bias and then illustrate its impact on Atari games.

---

Published in Transactions on Machine Learning Research (11/2024)
Sttereta
2.0
0.25
6- dicueatt ABC singted
1040
12.0
1.00
1000
(a) Discrete SAC
(b) Single Q
(c) Score
Figure 3: The results Atari game Frostbite/Mas Pasman environment over 2/5 million time steps: a) Mcasuring Q-value estimates of discrete SAC; b) Measuring Q-value estimates of discrete SAC with single Q; e) Score comparison between discrete SAC and discrete SAC with single Q.
To analyze the estimated bias e, we introduce the mathematical expression of the soft stato-value function:
(ss)) = Earoa [Q(s; ax) - a log(T(a; st))].
(12)
where ((s,, ax) represents the true Q-value. In practice, SAC uses the clipped double Q-learning trick. The learning target of soft state-value function can be written as:
Vappoz (8t) = Engon min [Q0 (Se, a²) - a log(TTa;) st))] 1=1
(13)
where Qe: represents estimation of larget-critic nelworks parameterized by 0. The estimated bias for Q/ can be calculated as 6, Qoy (s. - Q(s, On the one hand, when e1 62 the clipped double Q-learning trick can help mitigate overestimation error due to the mán operation. On the other hand, when 6 62 < 0 or é1 < 0 < 62, the clipped double Q-learning trick will lead to underestimation (i.e., Vappox < V) and consequently result in pessimistic exploration (Pan et al. 2020:
Ciosek et al. 2019)
Does this theoretical underestimate occur in practice for discrete SAC and hurt the performance? We answer this question by showing the influence of the clipped double Q-learning trick for discrete SAC in Atari games, as shown Fig. g Here, we compare the true value the estimated value. The results are averaged over three independent experiments with different random seeds. We find that, in Fig, 3(a) the approximate values are lower than the true value over time, demonstrating the underestimation bias issue. At the same time, we also run experiments for discrete SAC with single Q (DSAC-S), which uses single Q-value for bootstrapping instead of clipped double Q-values. As shown in Fig. 3(b) without the clipped double Q-learning trick, the estimated value of DSAC-S is higher than the true value and thus has an overestimation bias. However, in Fig, 3(c) we discover that even though DSAC-S suffers from overestimation bias, performs much better than discrete SAC which adopts the clipped double Q-learning mechanism. This indicates that the clipped double Q-learning trick can lead pessimistic exploration issues and hurt the agent's is performance.

---

Published in Transactions on Machine Learning Research (11/2024)
5
Improvements of SAC Failure Modes
We provide two simple alternatives, which are the surrogate objective with entropy-penalty and double average Q-learning with Q-clip, to avoid the two failure modes of discrete SAC discussed in Section 4 Combining these two modifications, we propose stable discrete SAC (SD-SAC).
5.1 Entropy-Penalty
The drastic change of Q function distribution and entropy affects the optimization of the Q-value. Due to the mutual coupling of the Q function and policy training in discrete SAC, we optimize policy entropy to alleviate the unstable effect on training caused by sharp Q function distribution and rapid drop in entropy. Simply removing the entropy term will injure the exploration ability under the framework of maximum entropy RI. An intuitive solution is to introduce an entropy penalty in the objective of policy to avoid entropy chattering. We will introduce how to incorporate the entropy penalty in the learning process for the discrete SAC algorithu.
Recall the objective of policy in discrete SAC as in Eq. For a mini-batch transition data pair (st, at, Tri St+1) sampled from the replay buffer, we add an extra entropy term Hras to the transition tuple which reflects the randomness of policy (ie., (8t, de, 81+1, Hamu), where Votd denotes the policy used for data sampling, We calculate the entropy penalty by measuring the distance between Haas and Hy. Formally, the objective of the policy is as follows:
(Ø) = EA BaiNn (a log(To(as 8r)) - Qa a:)]
(14)
B E., Es (Ea | log(TT Em 1- log(()))))
2
where Eagsmaaae log(Toos)) represents policy entropy of Toota Barwn, (=lg(T)) represents policy entropy of Tæ, and B denotes a coefficient for the penalty term and is set to 0.5 in this paper. By constraining the policy objective with this penalty term, we increase the stability of the policy learning process.
Fig. Sshows the training curves to demonstrate how the entropy penalty mitigates the failure mode policy drastic change. In Fig. [(b] the entropy of discrete SAC (the blue curve) drops quickly, and the policy falls into a local optimum the early training stage. Later, the policy stops improving and even suffers from performance deterioration, as shown in the blue curves in Fig. [[c) and Fig. [(d)
On the contrary, our proposed method (i.e., discrete SAC with entropy-penalty) demonsttrats better stability than discrete SAC. As shown in Fig. 4(a] entropy penalty effectively constrains the sharpness of Q function, as a result, the policy changes smoothly during training (Fig. [(b]]_ Consequently, compared with discrote SAC, the policy in our approach can keep improving during the whole training stage and does not suffer from a performance drop at the later training stage (the red curves in Fig. 4(c) and Fig. [(d).
It is worth noting that, since the instability in training mainly manifests as the existence of policy entropy term in optimization, imposing constraints in the entropy space is more effective than constraints in the policy space. Other common methods, such as the KT. penalty, limit the magnitude. of policy updates and impose additional restrictions on policy updates. This is proved in experiments: KL penalty (the yellow curve) cannot effectively constrain the rise in Q variance (Fig, [(a] and the decrease in entropy (Fig. 4bb]. Consequently, the final Q-value and score of the KT, penalty are lower than those with the entropy penalty, with a difference of 12% and 23%, respectively.
The cntropy-penalty term ( log(TT temperature a, jointly regulates the exploration of policy. Different from other trust region methods such KL constraint (Schulman et al., 2015) or clipping surrogate objective (Schulman et al. 2017), our method penalizes the change of action entropy between old and new policies to address policy instability during training. By adding regularization in entropy space instead of policy space, our method can mitigate the drastic changes of policy entropy while maintaining the inherent exploratory ability of discrete SAC (as shown in Fig. [(b] the policy entropy changes smoothly. It keeps at a relatively high value encourage exploration).
logg=) in conjunction with the
8

---

Published in Transactions on Machine Learning Research (11/2024)
Function Variance
optropy
discrate SAC
2.2
0.4
discrete SAC with entrogy- peralty
discrete SAC
discrate SAC with Kipenally
discrete with entroey
1.9-
disorete SAC with kl-penaltty
0.3
Val
0.2
1.4-
1.2-
0.1
0.0
0.8
0.0
0.4
0.8
1.0
0.0
0.8
1.0 1e7
Timestaps
167
Timesteps
(a) Q Function Variance
(b) Entropy
Est Estimatiis ion of Q-value
Score
4000
algo
disertte SAC
di screte SAC with entropy penalty
3000
discrete SAC k Imperal by
Average Valum
Average
2000
algo
1000
discrete SAO with petao ty
discrete SAC with ki-penslty
0.0
0.2
0 4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Tinestops
167
Tinestops
1e7
(c) Q-value
(d) Score
Figure 4: Measuring Q function variance, policy action entropy, estimation of Q-value, and score on Atari game Asterix compared between discrete SAC, discrete SAC with KT-penalty and discrete SAC with entropy- penalty over 10 million time steps.
5.2 Double Average Q-learning with Q-clip
While several approaches Ciosek et al., 2019 Pan et al. 2020 have been proposed to reduce underestimation bias, they are not straightforward to be applied to discrete SAC due to the use of Gaussian distribution. In this section, we introduce a novel variant of double Q-learning to mitigate the underestimation bias for discrete SAC.
In practice, discrete SAC uses clipped double q-lcarning with pair of target critics (Qæ, Qo;), and the learning larget of these two critics is:
+ min ((s))). 1,1
(15)
When neural networks approximate the Q-function, there exists an unavoidable bias in the critics. Since policy is optimized concerning the low bound of double crities, for some states, we will have Qog (s, ()))) Qtrue Qo (s, TT()).. This is problematic because Qo; (s, (s)) will generally underestimate the true value, and this underostimated bias will be further exaggerated during the whole training phase, which results in pessimistic exploration.
To address this problem, we propose to mitigate the underestimation bias by replacing the min operator with ang operator. This results in taking the average between the two estimates, which we refer to as double average Qllearning:
y= v r avg(Qu; (5', ((())) Qa (s', m(s'))).
(16)
9

---

Published in Transactions on Machine Learning Research (11/2024)
By doing so, the other critic mitigate the underestimated bias of the lower bound of double critics. To improve the stability of the Q-learning process, inspired by value clipping in PPO (Schulman et al. 2017), we further add clip operator on the Bellman error to prevent drastic updates of the Q-network. The modified Bellman loss of Q-network is as follows:
Q0,- [(,,) max (Q0o.-)² (Qo; clip(Qe. -c,c)) - -)))
(17)
where Qs, represents the critic network's estimate, Qoy represents estimation of target-critic networks, and e is the hyperparameter denoting the clip range. This clipping operator prevents the Q-network from performing an incentive update beyond the clip range. In this way, the Q-learning process is more robust to the abrupt change in data distribution. Combining the clipping mechanism (Eq. 17) with double average Q-learning (Eq. 16 we refer to our proposed approach as double average Q-learning with Q-clip.
Fig. ddemonstrates the effectiveness our approach. We compare the discrete SAC and various ensemble Q- estimation methods, including Randomized Ensembled Double Q-Jearning (REDQ) Chen et al. (2021b) and Random Ensemble Mixture (REM) Agarwal et al. (2020), with our proposed method, SD-SAC. In Fig. 5(a)
the Q-value estimate of discrete SAC is underestimated than the true value. Therefore, the policy of discrete SAC suffers from pessimistic exploration and results in poor performance (purple curve in Fig. 5(e)) On the contrary, in Fig. 5(d) with double average Q-learning and Q-clip, the Q-value estimate eliminates underestimation bias and improves quickly at the early training stage. The improvement of Q-value carries over to the performance of policy. Consequently, our approach outperforms baseline discrete SAC by a large margin (Fig. 5(c)). The result also demonstrates that even though REDQ has less estimation bias Fig. 5(b] it still sulfers from underestinaatin bias, leading to suboptimal performance due to pessimistic exploration. Although REM addresses the underestimation issue in Fig. 5(c) the overestimation bias of REM significantly exceeds that of our proposed method, resulting in rapid decline in performance at 8 million steps. In Fig. 5(d) we also notice that the Q-value overestimates the true value during the carly training stage but finally converges to the true value after the training process. This encourages early exploration, which is consistent with the principle of optimism in the face of uncertainty (Kearns & Singh 2002).
5.3 Psudocode
Finally, we provide the pscudo code for SD-SAC (i.c., Stable Discrete SAC with entropy-penalty and double average Q-learning with Q-clip), as shown in Algorithm
6
Experiments
6.1 Experimental Setup
To evaluate our algorithm, we compare SD-SAC with most related basclines, i.c., discrete SAC (Christodoulou 2019), TES-SAC (Xu et al. 2021 Soft-DQN (Vieillard et al., 2020) and Rainbow et al.] 2018) which is widely accepted algorithm in the discrete domain. We measure their performance in 20 Atari games chosen as the same as (Christodoulou) (019) for a fair comparison. We evaluate for 10 a
episodes for every 50000 steps during training, and execute 3 random seeds for each algorithm for 10 million environment steps (or 40 million frames). For the baseline implementation of discrete-SAC, we use Tian- shou We find that Tianshou's implementation performs better than the original paper by Christodoulon (Christodoulou 2019), thus we use the default hyperparameters in Tianshou on all 20 games.
We start the game with up to 30 no-op actions, similar to (Mnih et al. 2013), to provide the agent with random starting position. To obtain summary statistics across games, following Hasselt (Van Hasselt et al.
2016), normalize the score for each game follows: Score normalized
Score agant
Score random Score random
Score human
10

---

Published in Transactions on Machine Learning Research (11/2024)
20.0
20.0
25.0
apece (disorate SAC)
SEDO
16.5
true Geiaorete 3NC
true idivorete RETG)
12.5
12.0
12.0
10.0-
00.
90.0
4.5-
5.0-
5.0
5.0
Appos ecrate
1000
0.0
0.0
0.2
0.6
0.0
0.8
0
0.0
0.2
0.6
0.6
0.8
0.0
0.2
0.6
Simactese
McPacmen
MyFormat
IoPooman
22.5
app0x screte SKC
15.0
15.0
15.0
12.6-
12.1
10.0-
10.C
7.6
2.5
7.6
2.9
2.5
apoo (ciscrete 290
1.5
eauce ldlacrate
Cinestses
1er
16
(a) Discrete SAC
(b) REDQ
(c) REM
Fraxtbite
Frosth ite
4002
15.0
REM
10.0
7.5-
5.0-
1002
Awwwmee
2.5-
0.2
1x7
McTacmAT
#chacran
17.9.
3000
15.0
12.64
5200
6.9
1000
Cinseeto
80
0.0
0.5
6.2
2.6
0.5
1.0
0.2
0.48
0.5
15
(d) Ours
(e)
Score
Figure 5: Measuring estimation of Q-value and score on Atari Game Frostbite/ Crostbite/MsPacmaa MsPacman environment compared between discrote SAC, discrete SAC with REDQ, discrete SAC with REM, and ours (SD-SAC) over million steps.
Table 1: Mcan and median normalized scores discrete SAC, TES-SAC, Rainbow, Soft-DQN and our method across all 20 Atari games at 1M and 10M steps.
Discrete SAC(IM)
TES-SAC(IM)
Soft-DQN(1M)
Our
Rainbow (10M)
Discrete SAC(10M)
Ous(10M)
Mean Median
0.5%
3.0%
41.7%
38.5%
187.4
151.4%
199.2%
220.0%
0.1%
2.1%
20.0%
11.1%
79.2 %
90.8%
107.7%
114.1%
11

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 1 SD-SAC: Stable Discrete SAC with entropy-penally and double average Q-learning with Q-clip
Input: 01, Φ
> Initial parameters
Output: 01. 12.02.0
>> Optimized parameters
Hyperparameters: 7, 7,B.CT
Initialise Qo : + -RAA,, Qø S RIA,,
Ts S - [0, 1] A
Initialise local networks b Initialise target networks Equalise target and local network weights Initialize empty replay buffer
Initialise Qo; : S + RI QØg S Rlaj
02 f 02 +
for each iteration do for each environment step do as ~ Ts (oz 8t) 8441 ~ (0t+1 81, a,)
b. Sample action from the policy Sample transition from the environment D. Calculate the entropy Hays of the current policy & Store the transition in the replay buffer
Hooi a-zo(+1s))
E
(- log (a(a 8t)]
DU f(st, ae, (st, at) stt1, Hmma))
end for for each gradient step do y~r ax) + y avg(Qa) (S (5++1, ((++1))), (044,,,, ((++1)))
b Double average Q-value
estimation
[(O.) ~ max (loo. - y)², (Qx: + clip(Qo. - QØ: -, c), - -))) for i (1,2) D. Clip the Q-value estimation from target critic network
0: < 0: do ((,(() for i e {1,2}
Update the Q-function parameters Calculate the entropy Ha of policy & >
Ha a-10(-10))
E log st]]
(Ø) EAED (BEL, logooso st)) Qo(sn, ae)ll + 8 1- (HHooe H₂)²
Φ Φ~ Ø- - do (()
D Update policy weights
a~a 1τ (a)
Update temperature
QØ + TQ. + (1 7)Qu; for i {1,2}
Update target network weights
end for
end for
Table 2: Raw scores across all 20 Atari games, For methods discrete SAC (1M) and TES-SAC(MM), the
scores come from the corresponding paper, and the NE means the score does not exist in the original paper.
Game
SAC (IM)
TESSSC(IM)
Soft-DQN(TM)
Ours(1M)
Rainhow 10M)
Discrote (10M)
Soft-DQN(1OM)
(10M)
Alien
216.90
685.93
726.33
981.67
1798.33
2717.67
2018.00
2158.33
Amidar
7.9
42.07
130.03
132.97
394.23
354.77
438.80
407.20
Assauls
350.0
7258.87
Asterix
272.0
378.5
676.67
733. 73
5853. 33
2860.00
3761.67
5993.33
BattleZone
5796
7933.33
24266.67
16850.00
24733.33
BeamRider
432.1
NE
3321 60
3468.60
3310.40
7160.60
7048.20
10506.60
155.83
CranyClimber
3668.7
4.0
25390.00
20753.33
30286668 67
126320.00
95156.87
140726.67
Enduro
0.8
54.23
0.93
1517.70
1326.77
1144.07
22466-40
Freeway
4.1
13.57
17.70
20.17
20.13
15.73
32.30
20.17
Frosthite
50.4
81.03
294.33
347 .00
4163.67
646.33
2859.00
4506.00 0000
Jemesbond
68.3
31.33
273.33
368.33
656.67
1396.67
965.00
2085.00
Kangaroo
29.3
307.33
160.00
1/00.K0
3716.67
2426.67
2703-33
5556.677
M&Pacnau
1526
3221.33
2356.33
Pong
-20.98
20.84
20.00
15.53
20.93
20.37
20.73
20.37
Qheet
74.93
1400.83
986.67
15290.17
12946.67
14203.33
15325.83
RondRummer
305.3
NF
5510.00
127913.3
45173.33
34043.22 33
33370100
432033333
Spocelusdders
160.8
NE
488.85
383.55
1330.50
458
816.00
586. 50
Seaquest
211.6
116.73
681.33
744.00
2106.33
1853.33
3438.67
2761.00
UpNDown
250.7
207.6
8727.33
8114.67
9110.00
17803-33
79313.00
61441.33
6.2 Overall Performance
Table provides overview of results and detailed results are presented in the Table and Appendix A.1 Since TES-SAC is not open-sourced and our re-implement algorithm following the paper underperforms the
hhtp:://gthhab..mm///h//mm/
12

---

Published in Transactions on Machine Learning Research (11/2024)
Antorix
Enduro
diacett 50
Seneity We's 5SP SAS
dlacreta 50
ponett WE's (D) SAG
straty ENC penaty SNE'S 10 540
diccnta stossie camality
2008
dlacnet dtossia puaally
Enc uusulity
500
-000-
1000
2000
1201
0.5
0.2
2.6
0.0
0.0
0.2
0.4
0.6
0.0
2.5
0.2
0.4
0.5
1.8
1.0
Kongaras
SANgANIt
porsityy MNE'S
4000
serate seality saerr 940
scrine Sed AnE'
serass Tac IEE's
3000
6000
48 serets THc
3000
ueetts tec seatty
2008
160
16
Figure 6: Scores of variant discrete SAC, which includes discrete SAC, discrete SAC with entropy-penalty, discrete SAC with double average Q learning with Q-clip,for Atari games Assault, Asterix, Enduro, Freeway, Kangaroo and Seaquest.
reported results, we adopt the normalized scores of discrete SAC and TES-SAC reported in the corresponding publication (Xu et al. 2021). When comparing our method to the discrete SAC and TES-SAC, mean normalized scores increase by 38% and 35.5%, respectively. And our method improves the median normalized scores by 10.7% and 9.0% while compared with discrete SAC and TES-SAC.
To verify the elfect of a longer training process, lable 1 also compares discrete SAC, Rainbow, Soft-DQN, and our method performance on 10 million steps. Compared with discrete SAC, our method has improved the normalized scores by 68. 68.6% and 23.3% on mcan and modian, respectively. Additionally, our proposod method outperformed Rainbow by 32.6% on the mean and by 34.9% on the median. Better Q-estimation and steady policy updates are responsible for the performance increase in average scores. The experimental results demonstrate that benefiting from the deterministic greedy policy and entropy regularization in the evaluation step, Soft-DQN's performance improves rapidly in the early stages and achieves the best results at 1 million steps. However, due to the early convergence of the deterministic greedy policy, Soft-DQN's performance stagnates after million steps, as seen in Fig. [ Our method outperforms Soft-DQN in the final 10 million steps by 20.8% on average and 6.4% on median, due to the training stability brought by entropy penalty and the optimistic exploration altered by the double avg-Q with Q-clip.
6.3 Ablation Study
Fig. G shows the learning curves for 6 environments. Entropy-penalty (purple curve) increases performance compared to the discrete SAC in each of the six environments and even incrcases 2x scores Assault. This shows that discrete SAC can perform excellently after removing unstable training. Except for Asterix, the alternative choice of clipped double Q-learning, which is double average Q learning with Q-clip (yellow curve), also shows some improvement compared to the discrete SAC in 5 environments. Additional improvements can be derived when the combination of both alternative design choices is used simultaneously.
To evaluate the influence of hyperparameter tuning, we also conducted a comprehensive hyperparameter analysis. By experimenting with different a and learning rate the discrete SAC algorithm, we idcntify the performance upper bound of discrete SAC. The results show that, under varions 3 values, SD-SAC consistently outperforms this upper bound, demonstrating that entropy penalty serves as better and more
13

---

Published in Transactions on Machine Learning Research (11/2024)
balanced constraint. This further confirms that SD-SAC significantly achieve stable training process. We present the experiment details and results in Appendix 1.2.1
6.4 Qualitative Analysis
Seaquest
olgo diserate oure
3000
Returaa
2000
0.0
0.2
0.8
1.0
Timestops
(a) Loss Surfaces
(b) Scores
Figure 7: The loss surfaces of discrete SAC and our method on Atari game Seaquest with trained weights at million, 5 million and 10 million steps.
Fig. 7 shows loss surfaces of the discrete SAC and our method by using the visualization method proposed in [Li el al. 2018 Ota el al. 2021) with the loss of TD error of Q functions. According to the sharpness/ flat in these two sub-figures, our method has a nearly convex surface, while discrete SAC has a more complex loss surface. The surface of our method has fewer saddle points than the discrete SAC, which further shows that it can be more smoothly optimized during the training process.
Battle-ELO
Discrete SAC
1000
810
870
800
736
600
557
TIME
(a) Honor of Kings
(b) ELO Scores
Figure 8: a) A screenshot of the Honor of Kings 1v1 game. b) The ELO scores, compared with discrete SAC and our method, were tested for three snapshots of 24, 36, and 48 hours during training.
6.5 Case Study using Honor of Kings
We further deploy our method into Honor of Kings 1v1, commercial game in the industry, investigate the scale-up ability of proposed SD-SAC algorithm.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Honor of Kings is popular MOBA (Multiplayer Online Battle Arena) game and good testbed for RL research Ye et al. 2020b Chen et al. 2021a Wei et al. 2022 The game descriptions are in (Ye et al. 2020c a). In our experiments, we use the one mode (Ivl solo), with both sides being the same
hcro: Diao Chan. The state of the game represented by feature vectors, as reported in (Ye et al. 2020c
Wei et al.] 2022). The action space is discrete; i.e,, we discretize the direction of movement and skill, same to (Ye et al.] a)) The goal of the game is to destroy the opponent's 's turrets and base crystals while protecting its own. We use the ELO rating system (Elo Sloan, 1978), which calculate scores from the win rate, to measure the ability of two agents. A detailed introduction of the ELO system is presented in Appendix A.5
We selected three snapshots of 24, 36, and 48 hours during the training process, resulting in 6 agents (SD- SAC-24h, SD-SAC-36h, SD-SAC-48h, DSAC-24h, DSAC-36h, DSAC-48h). We conducted 48 one-on-one matches for each agent, resulting in a total of 720 matches and thus serving as the basis of ELO calculation.
The results are shown in Fig. [ Throughout the entire training period, our method outperforms discrete SAC by significant margin, which indicates method's efficiency in large-scale cases. Specifically, SD- SAC-48h achieved 35 wins, 7 draws, and 6 losses, with a win rate of 72.92%. The agent also exhibits higher skill hit rate, higher Kill/Death ratio and better turret-dashing ability.
7 Conclusions and Future Work
Many algorithmic design choices in reinforcement learning are limited to the regime of the chosen benchmark lasks. We highlight that soft actor-critic (SAC), that widely accepted design choices in continuous action space do not necessarily generalize to new discrete environments. We conduct failure mode analysis and obtain two main insights: 1) duc to the deceptive reward, the unstable coupling update of policy and Q function will further disturb training; 2) the underestimattin bias caused by double Q-learning results the agent's 's pessimistic exploration and inefficient sample usage. We thereby propose two alternative design choices for SAC: entropy-penalty and double-average Q-Jearning with Q-clip, resulting in a new algorithm, called SD-SAC. Experiments show that our alternative design choices increase the training stability and Q-value estimation accuracy, which ultimately improves overall performance. In addition, we also apply our method to the large-scale MOBA game Honor of Kings 1v1 to show the scalability of our optimizations.
Finally, the success obscures certain flaws, one which is that our improved discrete SAC still performs poorly in instances involving long-term decision One possible reason is that SAC can not accurately estimate the future only by rewarding the current frame. In order to accomplish long-term choices with SAC, our next study will concentrate on improving the usage of the incentive signal across the whole episode.
References
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Ried- miller. Maximum posteriori policy optimisation. arXiv preprint arXiv:1806.0692) 2018.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforce- ment learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 104-114. PMLR, 2020. URL http://prooceedings mlr httpr//cnnns...ppprrrss/////aaaa preesss/1119//aaavaa20....
Chayan Banerjee, Zhiyong Chen, and Nasimul Noman. Improved soft actor-critic: Mixing prioritized off- policy samples with on-policy experiences. IEEE Transactions on Neural Networks and Learning Systems. 2022.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pp. 449-458. PMLR, 2017.
htttps://githubb.com// http:/hhhbb.ccc//ttnnttttteeeeee
15

---

Published in Transactions on Machine Learning Research (11/2024)
Sheng Chen, Menghui Zhu, Deheng Ye, Weinan Zhang, Qiang Fu, and Wei Yang. Which heroes to pick? learning to draft in moba games with neural networks and tree search. TEEE Transactions on Games, 13 (4):410 421, 2021a.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without model. In International Conference on Learning Representations, 2021b. URL
https://openreeview.. wwwpeeeewwwweeeefrreeeeeeee wott:forrrr/ddd-oe22222eeo
Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint arXiv:1910.07207, 2019.
Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019.
Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distributional soft actor-critie: Off-policy reinforcement learning for addressing value estimation errors. IEEE transactions on neural networks and learning systems, 2021.
Arpad E Elo and Sam Sloan. The rating of chessplayers: Past and present. (No Title), 1978.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Xiaoyu Gong, Shuai Lü, Jiayu Yu, Shong Zhu, and Zongze Li. Adaptive estimation q-learning with uncer- tainty and familiarity, In Proceedings of the Thirty-Secons International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China, pp. 3750 37500 ijeai.org, 2023. doi: 1.221963/jjeai.22222 417. URL https: http:///a.............////.aa... 2023/411
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement lcarning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint
2018b.
Seungyul Han and Youngehul Sung. A max-min entropy framework for reinforcement learn- ing. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jen- nifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 25732 25745, 2021. URL httts:://ppoeeeee http::////cccecnss... neuriienrpppp www.eee.....aaappppepeeeee cc/pappr//02//naat 27b76edf 71b76off770022677777777a448 -tt.aaasssssdd--ssaaa....... Abstract. html
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement lcarning. In Thirty-scoond AAAI conference on artificial intelligence, 2018.
Zhang-Wo Wei Hong, Tzu-Yun Shann, Shih-Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Yi Lec. Diversity- driven exploration strategy for deep reinforcement learning. Advances in neural information processing systems, 31, 2018.
Christian Horvat and Jean-Pascal Pfister. Denoising normalizing flow. Advances in Neural Information Processing Systems, 34:9099 9111, 2021.
Zhimin Hou, Kuangen Zhang, Yi Wan, Dongyu Li, Chenglong Fu, and Haoyong Yu. Off-policy maximum entropy reinforcement learning: Soft actor-critic with advantage weighted mixture policy (sac-awmp). arXiv preprint arXiv:2002.02880 2020.
Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 49(2):209 202,,22222222 2002.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Qingfeng Lan, Yangehen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estima- tion bias of q-learning. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. Oeen 2020. URL httrs://openrevieee.nntt//eee
BkgOu3Etwr
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Contimuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL
02971
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Anlonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/ 5602,0,,, 2013. URL
http://arxiv wwwaxiii.oo//aaaaea aaaaaba/119 13122160165
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through doep reinforcement learning. nature, 518(7540):5299-555 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mchdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928-1937. PMLR, 2016.
Kei Ota, Devesh K Jha, and Asako Kanezaki. Training larger networks for deep reinforcement learning. arXiv preprint arXiv:2102.07920, 2021.
Ling Pan, Qingpeng Cai, and Longbo Huang. Softmax deep double deterministic policy gradients. Advances in Neural Information Processing Systems, 33:11767-11777, 2020.
Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and reinforcement
learning by approximate inference. Proceedings of Robotics: Science and Systems VIII, 2012.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Rescarch, pp. 1889 1897, Lille, France, 07 09 Jul 2015. PMLR. URL https://proccedingg.. mlr. wwtppp/re//acaapapp//ee httpa///33//cchllmaa33 httl
John Schulman, Filip Wolski, Prafulla Dharival, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv: AiXi:1700.00344 06347, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference intelligent robots and systems, pp. 5026 5033. IEEE, 2012.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Nino Vieillard, Olivier Pictquin, and Matthicu Geist. Munchausen reinforcement learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-FForinna Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020. httphhttss://poceedteiie settpe/..caaaee////aaa////2222ce htw.://ooo..0e-aaaatttttttt.....
Yufei Wang and Tianwei Ni. Meta-sac: Auto-tune the entropy temperature of soft actor-critic via metagra- dient. In Proceedings of the International Conference on Machine Learning workshop, 2020.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Patrick Nadeem Ward, Ariella Smofsky, and Avishek Joey Bose. Improving exploration in soft-actor-critic with normalizing flows policies. In Proceedings of the International Conference on Machine Learning workshop, 2019.
Hua Wei, Jingxiao Chen, Xiyang Ji, Hongyang Qin, Minwen Deng, Siqin Li, Liang Wang, Weinan Zhang, Yong Yu, Liu Linc, Lanxiao Huang, Dcheng Ye, QIANG FU, and Yang Wei. Honor of kings arena: environment for generalization in competitive reinforcement learning. In Thirly-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openreviiet net/
formm21d++76666LE0333
Yaosheng Xu, Dailin Hu, Litian Liang, Stephen McAleer, Picter Abbeel, and Roy Fox. Target entropy annealing for discrete soft actor-critie. Advances in Neural Information Processing Systems workshop, 2021.
Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, et al. Towards playing full moba games with deep reinforcement learning. Advances in Neural Information Processing Systems, 33:621 632, 2020a.
Deheng Ye, Guibin Chen, Peilin Zhao, Fuhao Qiu, Bo Yuan, Wen Zhang, Sheng Chen, Mingfei Sun, Xiaoqian Li, Siqin Li, et al. Supervised learning achieves human-level performance in moba games: A case study of honor of kings. IEEE Transactions on Neural Networks and Learning Systems, 2020b.
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojic Yang, Xipeng Wu, Qingwei Guo, al. Mastering complex control moba games with deep reinforcement learning. Tn Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6672 6679, 2020c.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximun entropy inverse reinforeement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, II. USA, 2008.
18

---

Published in Transactions on Machine Learning Research (11/2024)
A
Additional Details and Experiment Results
A.1
Detailed Experiment Results on 20 Atari Game Environments
In Figure
wc present the learning curves of all 20 experiments.
Al ten
Am dbr
Asc-u
502
8000
aggo
7500
402
inbon saft
cift
Raturne
Datume
500
nonete
102
eoft
1nT
Timenbupa
1
Timesttpps
167
Asterix
Batt leZone
Deaniii der
6000
Bara
ulee
20202
Verro
80000
3000
discrete $AG
20002
sosoree SRG
inbon
inbro soft-dan
di sonete SAG
4000
cefitddion
15002
se Firddan
40000
10002
5002
0.0
0.7
D1
0.5
1AT
Tiwcskopa
Timestops
CrazyOl imbor
Enduro
170002
Alo
inbrom
1500
durs
80302
soroto $AC
9000
relrious
soft-dan
40003
500
20002
0.0
0.2
0.A
0.
0.0
0.3
0.6
0.0
0.7
0.
0.5
Tiecrkosa
Timestccp
Freeeay
FrestDi to
Janesbond
4032
20
25-
Algs
3002
Inbon
1000
U-dan
sure
Necumm
2002
9200
500
Imartaps
107
Tineshupt
1imenhape
167
19

---

Published in Transactions on Machine Learning Research (11/2024)
Kangaroo
Pong
10000
Rigo ouTS
soft
2502
Datume
0000
2002
1892
alga
gn
oura
1.11
wara
2000
discrate
sott
L.22
den
0.0
0.2
0
0.6
1.0
0.0
0.2
0.8
0.0
0.2
0.6
0,6
0.8
limestape
Obart
RoadRunner
Spacellvader Invaders
1400
11300
40332
10000
50032
800
600
400
200
Serquest
UpMicen
80002
AlEs
Alga
4000
disereil 312
inbbe
50002
inlle
seftt-don dog
sort
20002
1000
01
0.0
0.6
Timessips
167
Tiaestose
147
Figure 8: Learning curves for discrete SAC, Rainbow, Soft-DQN, and ours, for each game. Every curve is smoothed with moving average of 10 improve readability.
A.2 Further Comparison of SD-SAC and DSAC
Beyond the comparison of Figure we further measure SD-SAC and DSAC in terms of episode length and number of steps with rewards in Figure g After 2e6 steps, the algorithm with entropy penalty demonstrates significant. longer episode lengths and more reward sleps compared to discrete SAC. This indicates that the entropy penalty helps the agent learn both scoring and avoidance skills, leading to continued performance improvement.
A.3
Result Curves of Individual Runs
For clearer demonstration, we show the results of individual run curves for figures during our major analysis in Section and 5.
A.3.1
Comparison Between True Values and Estimate Values
We plot Figure 3 by individual runs Figure 10 The results show that for cach individual soed, discrete
SAC consistently suffers from an underestimation problem, while using single Q leads to an overestimation
issue.
20

---

Published in Transactions on Machine Learning Research (11/2024)
Episode Length
Number Steps with Rewards
1600
60
50
1200
Averaged valuee
1000
40
Averaged
30-
600
20
10-
discrete SAC
discrete SAC
discrete SAC with entropy penalty
discrete SAC with entropy penalty
0.2
10
0.2
0.4
Timestaps
Timesteps
Episode Length
(v) Number of Steps with Rewards
Figure 9: Comparison of SD-SAC and DSAC in terms of episode length and number of steps with rewards in Asterix.
Froatbiie
Freathi to
Frovch
Kermprenata the
discrate with L.send Eiscreto with sinpts 3. reed
142
162-
Eiscreta aned 354
10.55
10.0
WiPcoman
RaPaern
RoParmnn
17.04
2006
15.0
(400p 14
12.5-
2000
12.0
disrrete strnc end weet
10.
1000
dinirete arri sved discrate SAC vest 6137
(a) Discrste SAC
(b) Single
(c) Score
Figure 10: The results of Atari game Frostbite/ MsPacman environment over 2/5 million time steps: a) Measuring Q-value estimates of discrete SAC; b) Measuring Q-value estimates of discrete SAC with single Q; Score comparison between discrete SAC and discrete SAC with single Q
A.3.2
Comparison Between Different Policy Constraints
We provide Figure by individual runs in Figure 11 The results show that entropy penalty enables stable training and better performance across all seeds.
A.3.3
Comparison Between Different Q-value Estimation Methods
We provide Figure by individual runs in Figure 12 Our approach demonstrates effectiveness in alleviating underestimation and reduce bias in all individual
21

---

Published in Transactions on Machine Learning Research (11/2024)
Funtion Yar
ontropy
discrete 560 seed_1235 1234
2.7
discrete 5áG seed_
3.35
disorate 340 seed 4321
disorate 540 seed 4321
diserate SK2 234d_2341
disorate 840 2344_2341
0.32
1.3
ercrapy pana
Asaraee Value
ipponn aoee_1224
1ipponaly asss_1224
0.20
<iupena seed_
Irpone fly sone
1.4
E-paral sees 2301
0.15
0.2
00-
0.8
1.0
0.0
0.8
Timescape
Tx7
llaectaps
147
(a)
Function Variance
Entropy
Esi Entinotion of drvalue
Score
4000
12244.
3900
diserrtee
2002 of ertereerpeee perto
entroyy para
decrage Va
2002
K1 sirpan to 1254
entropy perna
dicorate SAC
2002 "ppana 0000 4021
plppan 2001
endrepy penalty by 2194
1502
entrooy speality need
ontoreymmennlly 2861
1002
<irpens ity seee 1254
Imppon sond .221
Iimenn sose 2041
Tiessiopa
Tiecstopo
Q-value
(d) Score
Figure 11: Measuring function variance, policy action entropy, estimation of Q-value, and Atari game Asterix compared between discrete SAC, discrete SAC with KT--penalty and discrete SAC with entropy- penalty over 10 million time steps.
Table Hyperparameter for Discrete SAC and SD-SAC
Hyperparameter
Discrete SAC
SD-SAC
learning rate optimizer
10-3
10-5
Adam
Adam
mini-batch size
64
64
discount (7)
0.99
0.99
buffer size hidden layers
105
105
2
2
hidden units per layer
512
512
larget smoothing coefficient (r)
0.005
0.005
Learning iterations per round
1 0.05
alpha
0.05
n-step B
3
False
0.5
False
0.5
A.4 Hyperparameter Used in SD-SAC
Please refer to Table 3
22

---

Published in Transactions on Machine Learning Research (11/2024)
Frastbise
Frostbite
upprex_
Glisenete
12-
apprex.
(dtecrate
ceorex 23c1
(ilseente
RE90
true, 1254
(s isserets 54C
R200
ou disccesee atc
True 2321
(Sieerate SAC
0700)
ex 2021
(diserete sto
lecertte 542
m000)
2041
Ggiveress 9Mo
Iecraae
1254 (dteerace stc
6321 Officrent M00
5-
0.6
0.0
0.2
1.0
Cimestion
18
Averrge Value
10-
sorste RiDG)
asprex
screte SAC
seorex
screte BD0
asproi_ 2341
sorete SAC)
SAC)
6-
oporoi 2041
61 sorete SAC REDG)
SAC
4321
(siocrete SAC)
4329_4321
fdiscrete BAC
2-
true 2341
Isiecrate SKG)
true. 2341
(clecrate SAC REEA)
2-
(a) Discrete SAC
(b) REDQ
Frosthite
Frostbite
30-
20.0
1224
(dizcrata KMM
25.
Galsereto REM
(diecrate
AM
17.5
15 0-
20
Mecres telue
2311
6m10
12.5
10.0
7.6
aopron Joural
(san))
expros, 2341
Joural
true, 1234 Soars)
2.6
(sura)
2341
1:0
1x7
MuPacman
McPacran
18-
17.5
16-
15.0
14-
Nerrge Vaha
12.5
Awaraa
10.0
10-
seprex: 1234
Celacrate SAC 1E50
apprex: 1254 (ourd)
4321
(éiourete
REM
7.5-
4321
(ours)
approx, (siscrete
5.0
approx_ 2341
(ours)
true. 1234 (d) screte 5AG 1930
true. 1234
(ouri)
true_502 4321 (di senete BAG REO
true_400 .921
(surs)
(surs)
6.0
0.0
0.2
0.4
0.6
0.a
1.0
0.3
0.3
0.4
0.6
0.8
1.0
muctope
107
tinectops
107
REM
(d) Ours
Figure 12: Measuring estimation of Q-value on Atari Game Frostbite/ MsPacman environment, compared between discrete SAC, discrete SAC with REDQ, discrete SAC with REM, and SD-SAC (discrete SAC with double average Q-learning with Q-clip) over 10 million steps.
23

---

Published in Transactions on Machine Learning Research (11/2024)
A.5 Introduction of the ELO System
The ELO rating system (Elo & Sloan 1978 is widely-used mechanism for assessing the relative skill levels of players agents, commonly applied in chess, competitive games, and other ad adversarial environments. In our study, the ELO ratings of agents are calculated through the following process:
1. Each agent is assigned an initial ELO rating Rease.
2. Before agent A competes against agent B, the expected score for each agent is calculated based their current ELO ratings: EA (+10((g-nA)//n
3. The ELO ratings are updated based on the outcome of the match between A and B, where SA is the actual score and K' is a constant: R'A = RA + K -SA-E) - -A)
4. Through multiple matches among various agents, the FL. ratings are adjusted according to the results of these matches. The final ELO ratings reflect each agent's 's relative strength compared to the others.
A.6 Other Atari Environments for Unstable Coupling Training of Discrete SAC
We conduct cross-validation in other Atari environments, as presented in Fig. 14 16
The result show that
in other environments with deceptive rewards, the rapid decrease in policy entropy due to larget Q variance similarly affects training.
A.6.1
Games with Deceptive Rewards
000126
180
(a) Assault
(b) Jameshond
(c) MsPacman
Figure 13: Three examples of Atari game environments with deceptive rewards.
We take the Atari games Assault, Jamesbond and MsPacman as examples to further illustrate the manifes- fation and impact of deceptive rewards in game environments. (Fig. 13
In the game Assault, mothership releases different kinds of aliens. They move along the screen, with the bottom-most alien firing various types of weapons. The player controls cannon that can shoot bullets horizontally or vertically to attack the aliens and fireballs they shot. Hitting an alien scores points, while being hit or cannon overheating results in a loss of life.
In Jamesbond, the player controls craft that needs to complete various mission achieve final victory. In the first mission, the player must navigate through desert with craters, acoid overhead satellite scans and helicopter bombings, and score points by hitting diamonds through fixed-angle shooting.
24

---

Published in Transactions on Machine Learning Research (11/2024)
As for MsPacman, the player controls Pacman, who scores points by cating dots in maze while avoiding floating ghosts. When Pacman eats energy pill, she can attack the ghosts to gain higher scores.
In all three environments, the agent can quickly gain deceptive rewards through short-term payofis. For Assault and Jamesbond, all points come from shooting actions that hit specific targets, while avoiding obstacles can prevent the loss of life but does not bring clear rewards. Thus, agents often excel at shooting but struggle with dodging. In MsPacman, the numerous dots in the maze provide many rewards for the agent's movement. As result, the agent finds it difficult to learn advances strategies such avoiding ghosts and picking up energy pills to attack ghosts. The presence of deceptive rewards leads to the training process stuck in local optima, making it challenging to explore better, long-term strategies.
A.6.2 Plots Training Process
We present the training process in the three aforementioned environments with deceptive rewards in Fig. 16 It can be observed that in each case, deceptive rewards cause rapid increase in Q variance and decrease in policy entropy, leading the training process to fall into local optima.
q Function Verience
Intimecion
Policy Entruyy
5.37
0.91
0.33
0.32
16.
0.51
0.91
01
29
3.0
Tirettps
(a) Q Function Variance
(b) Q-value
(c) Entropy
Edisade Length
Number of Steas w Rewards
Score
1200
237
6030
4390
5000
030
2006
52-
3800
00
02
65
3.8
10
03
0.4
0.5
10
8.00
0.25
3.50
0.75
3.75
230
(d) Episode Length
(c) Steps with Rewards
(f)
Score
Figure 14: Plots of Q function variance, estimation of Q-value, policy action entropy, episode length, number of steps with rewards and score on Atari Game Assault environment with discrete SAC over 10 million time steps.
A.6.3 Comparison of Different Algorithms on Additional Atari Environments
Here in Figure 17 we provide comparative performance curves of DSAC, DSAC with entropy penalty, and DSAC with KL penalty in three additional Atari game environments: Assault, Jamesbond, and MsPacman. As shown in the results, the entropy penalty consistently offers the best carly-stage regulation of entropy changes across all three environments. This regulation helps prevent the agent from falling into local optimum during the learning process, thereby improving the final score performance.
25

---

Published in Transactions on Machine Learning Research (11/2024)
Q Function Ve lence
Extimesion of
Polac Eniweyy
0.30
244
0.34
0.32
24
0.00
15
14.
100
100
(a) Q Function Variance
(b) Q-value
(c) Entropy
Edizodi Length
Number of Steod th Rewards
Score
3000
2330
1508
2000
1008
2500
3890
C5
1,2
3.0
Tirnstogle
(d) Episode Length
Steps with Rewards
(f) Score
Figure 15: Plots of Q function variance, estimation of Q-value, policy action entropy, episode length, number of steps with rewards and score on Atari Game Jamesbond environment with discrete SAC over 10 million time steps.
Q Function Veriance
Istimiiion of Wil
Poliey Entrepy:
14.
15.0
1.25
1.90
0.5
0.50
0.2
0.00
2.5
-0.23
600
0.6
1,0
0.
(a) Q Function Variance
(b) Q Q-value
(c) Entropy
10:31
Frieode Length
Number of Steas on Rewards
Srore
3330
3800
2008
1000
102
(d) Episode Length
(c) Steps with Rewards
(f) Score
Figure 16: Plots of Q function variance, estimation of Q-value, policy action entropy, episode length, number of steps with rewards and score on Atari Game MsPacman environment with discrete SAC over 10 million time steps.
26

---

Published in Transactions on Machine Learning Research (11/2024)
1.4
discrata
discrete 5NG
discrete with eetrio/seraity
2.4-
1.75
discrete 5N0 with eppeaalyy
dissreto S4d with ligwaalls
dincrete 5M0 with aippouiity
2.2
1.35
1.4
8MC discrate SI peaaity
0.75
1.4-
0.30
0.0
0.0
0.2
0.6
Time steps
187
Tisestape
Timeetuss
147
Estivation of D-value
Estinetion of G-valuv
Estinuaiion of Q-vallus
Valee
Awrre Value
10-
diseroto MA
disceate 316
5-
clso diserete AMO
dlecrete SAB with entripy perality
discrate SAE with entresy ranaa
discreta SNG wich entr pesalty discrete SNG wich alpprniity
dlacrete 545 wi tranalltyy
discrate 5AC with kI kirpena Iay
0.0
0.2
4.4
6'6
1.5
0.0
6.2
0
0.6
0.6
0.0
0.2
9.6 Timestess
0.6
0.8
1.0
Scora
Score
2000
4000
No0rap TAlle
Iuorect Valm
2000
1550
5000
1000
1000
discrete eytroea-meeall
Alp-
0.0
0.0
0.2
0.0
1.0
107
Timestuse
(a) Assault
(b) Jameshond
(c) MsPacman
Figure 17: Measuring Q function variance, policy action entropy, estimation of Q-value, and score on Atari game Assault, Jamesbond and MsPacman, comparing between discrete SAC, discrete SAC with KL-penalty and discrete SAC with entropy-penalty over 10 million time steps.
B Further Analysis
B.1 SAC Training Pattern MuJoCo
We only observe the failure modes in discrete SAC. The reason SAC does not exhibit these failure modes in continuous environments is twofold. First, SAC employs the reparameterization trick, fitting actions with a Gaussian distribution, allowing it to adapt to deceptive rewards without sacrificing policy diversity. Second, in contimous environments, actions that deviate slightly from the best response may have minimal impact on the outcome, whereas in discrete settings, different actions can have entirely distinct meanings. Therefore, our analysis primarily focuses on the challenges SAC faces in discrete cnvironments.
To validate this point, test SAC on three tasks of the MuJoCo environment. Results in Figure 18 indicate that in MuJoCo, the SAC algorithm does not encounter local optimum issues; policy entropy changes are minimal and gradual while the scores stadily increase. This suggests that SAC does not face the problems described in the paper when applied to inntin lasks.
B.2
Hyperparameter Analysis
B.2.1
Different Hyperparameter Choices of DSAC
Our design method incorporates two hyperparameters, i.c., entropy-pent penoatty coefficient and Q-clip range c. compares various entropy-penalty coefficient, B and Q-elip range e values. The constraint, proportion
Fig. 19
27

---

Published in Transactions on Machine Learning Research (11/2024)
Entropy
5-
401
2-6
0.
05
03
251
Score
Score
7004
14800
10900
2003
4890
1000
0%
52
10e
Thrastope
Temndops
(a) Ant
(b) HalfChectah
(c) Walker2d
Figure 18: The results of SAC in the MuJoCo environment.
Seaquest
Seaquest
7000
algo
algo
6=0.1
4000
a=2
6000
=00.2 2
6=1
8-0.5
0=0.5
5000
3000
Returns
4000
3000
2000
1000
0.0
0.2
0. 4
0.6
0.8
1,0 1e7
0.0
0. 2
0.
0.
1.0
Timesteps
Timesteps
1e7
(a) Entropy-penalty B
(b) Q-clip
Figure 19: Scores on Seaquest: a) variants entropy-penalty coelficient 3 with 0.1, 0.2, 0.5 and 1. b) variants Q-clip e with 0.5, 1, and 5.
of policy change is dotermined by the entropy-penalty coefficient 3. Intuitively, an excessive penalty term will lead to policy under-optimizatton. We experiment with different in {0.1, 0.2, 0.5, 1}. We find that = 0.5 can effectively limit entropy randomness while improving performance. The Q-clip constrains different ranges of Q value range c, and experiments with different ranges in {0.5, 1, 2, 5) show that 0.5 is reasonable constraint value.
B.2.2 Different Choices of Clip Ratio
In Figure 20 compare the clip ratio and final scores of different in Q-clip.
28

---

Published in Transactions on Machine Learning Research (11/2024)
Seaquest
4000
0.0 2
3000
6F1
discrete SAG (base inne
2000
1000
0.2
0.4
0.6
0.8
1.0
Timestaps
1e7
(a) Clip-Ratio
(b) Score
Figure 20:
Mcasuring clip-ratio and score on Atari Game Seaquest environment with our method over 10 million time sleps using variants Q-clip e with 0.1, 0.2, 0.5, 0.8 and 1.0
B.2.3 Various Learning Rates for Discrete SAC
We introduce various learning rates for experiments on Asterix using vanilla discrete SAC in Fig. 21 An excessively high learning rate leads to carly convergence of entropy, while excessively low learning rate results in insulficient optimization. The experiments show that the entropy instability issue of discrete SAC is not caused by inappropriate learning rate scttings.
Est Enatiatoon of Grva
Score
2.0
I--8-0
le-2
Valle
2000
letter?
1.0-
1
3000
0.5
64
0.2
(a) Entropy
(b) Q- Value
(c) Score
Figure 21: Measuring policy action entropy, estimation of Q-value and score on Atari Game Asterix envi- ronment with discrcte SAC over 10 million time steps using different lcarning rates.
B.2.4
Different Choices of Temperature a in Discrete SAC
In Figure (2(a) we compare scores on Asterix by discrete SAC using variants a with 0.01, 0.025, 0.05, 0.075 and 0.1 over 10 million time steps.
B.2.5 Comparison Across SD-SAC and DSAC
We first determine discrete SAC's best combination of a (Fig. 22(a)) and learning rate (Fig. 21 Then, we compare the scores between SD-SAC with different 8, and DSAC with this combination (a = 0.5; Ir Io-5) in Figure 2b( The result shows that across various values, SD-SAC consistently outperforms DSAC. This indicates that the entropy penalty is better and more balanced constraint than merely limiting
the extent of policy updates.
29

---

Published in Transactions on Machine Learning Research (11/2024)
Score
Score
3000
3500
3000
2000
Average value
Ner Vatue
1500
2000
beta-0.1
elole:0 66
1000
beta=0.2
elolend 61
beta-1
000
atptect D/s
elotend 005
500
beta-0.5 (ours)
alphe:0
(base)
0.0
0.2
0,4
0.6
1.0
1x7
Timesteps
(a) Discrete SAC Different
(b) SD-SAC with Various 3 and Learning Rate Combination.
DSAC with the Best a
Figure 22: Comparison of different hyperparameters, including a, learning rate and B.
B.3 Computation Overhead
We test the computational speed machine equipped with an Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz with 24 cores and single Tesla T44 GPU. The unit "it/s" represents the number of steps interacting with the environment per second. Detailed data are shown in the Table f below. The results demonstrate that our method has a 10.86% reduction(265.c (265.41->236.55) in speed compared to the vanilla discrete SAC, while maintaining the same parameter size.
Table 4: Computational speed our method and discrete SAC.
algorithm
speed
discrete SAC
265.41it/s
discrete SAC + entropy-penalty
146.83it/s(-18.555
discrete SAC + avg-q + q-clip
250.27it/s(-15.14)
SD-SAC
236.58it/s(-28.88)
B.4 Cosine Similarity Comparison
We visualize the changes in cosine similarity between adjacent stales before and after incorporating the entropy penalty in the DSAC algorithm in Figure 23 The results indicate that, following the addition of the entropy penalty, state transitions exhibit smaller and more stable changes. This observation further substantiates that the entropy penalty contributes to more stable policy updates, thereby enhancing the overall performance of the algorithm.
30

---

Published in Transactions on Machine Learning Research (11/2024)
Casim Simi i by
0.1
Average Value
0. 3.1455
0.104
0.135
algo
0.00
disorete SAC with penaliiy
0.0
0.
0.
0.6
0.
1.0
1e7
(a) State Similarity
Figure 23: Mcasuring cosine similarity of states on Atari game Asterix compared between discrete SAC and discrete SAC with caadroyy-penalee 10 million time steps.
31


Reviews:
[{"review_id": "AgqiAniSL1", "reviewer": "Reviewer_roTL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Paper is well structured and clearly written\n- Nice investigation of the vanilla SAC shortfalls\n- New method is interesting and empirical results look promising\n\nWeaknesses:\n- Following up on the comments in the previous round of review, I believe the investigation of the 2 issues outlined in section 4 could still benefit from more detailing and testing, e.g. in a different environment that Asterix", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new improvement on Discrete Soft Actor-Critic (SAC) algorithms. After a detailed study of the vanilla SAC and outlining two main issues with it - training instability and pessimistic exploration - the paper presents its solution to these problems based on an entropy penalty term and a variant of double Q-learning. It also provides an extensive empirical study of the performance in different game environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses section.\n\nAlso noticed a few typos:\n- \"Adaptation\" in the abstract\n- H(\\pi(. | s_t)) in the definition of SAC (p3 section 3)", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "N5Ioyq7VbP", "reviewer": "Reviewer_4k4v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[STRENGTHS]:\n- The paper studies what I take to be an important question: SAC has become one of the standard algorithms of the RL community, and this paper gives a careful examination of how it has been adapted to handle discrete actions. The two failure-modes identified are convincing, and they naturally give rise to two modifications that underly SDSAC. In this sense, I believe the approach could be readily adopted by many experimentalists looking to use SAC in discrete domains.\n- The experiments are broad and deep. There are significant baselines, domains, and analysis, and the evidence presented is convincing.\n- The paper is well organized and does a good job of walking through the narrative that (1) SAC should be adapted to discrete action domains, (2) but it suffers from some issues, so (3) we need some fixes. I am really sympathetic to the story, and find the main contributions to be valuable and I believe will have an audience in the TMLR community.\n\n[WEAKNESSES]\n- There are aspects of the writing that could be improved along two axes. First, some aspects of the introduced notation and math are slightly unclear. I include some suggestions in the \"Requested changes\" section below, but I encourage the authors to take a careful pass through any introduced equations and ensure that all notation has been properly and consistently introduced. Second, parts of the paper are hard to read, such as the brightly colored blue blocks of text. This is an easy fix: I suggest removing all of the blue and orange colors from the text blocks.\n- Related to colors: I found the colored series in the plots quite difficult to read. It is in part due to the fact that the figures are so small: can you make them larger? However, the actual colors themselves are extremely similar (at least to my eye), so I had a hard time differentiating between the blue and light purple, for instance.\n\nOverall I believe this paper presents a valuable contribution: the main claims are supported by evidence, and there will be an audience for the work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper takes a careful look at how we might adapt Soft Actor Critic (SAC) to discrete domains. First, the paper explores the standard methods of discretization, and highlights two failure modes of this standard approach that result in training instability and underestimated Q-values. The paper finds that \"deceptive rewards\" and the use of a double Q-learning estimator contribute to these failure modes. In light of this, the work then proposes a new variant of discrete SAC that is designed to explicitly overcome these two issues. First, to account for training instability, an entropy penalty is introduced. Second, to account for the underestimation issues, an averaged variant of double Q-learning is used. Together, these two pieces combine to form Stable Discrete SAC (SD-SAC), a new variant of SAC. The remainder of the paper contains extensive experiments that evaluate SD-SAC in a wide variety of domains, including Atari and large-scale MOBA.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Writing Suggestions:\n\nHigh-level:\n- I suggest removing all colors from the text throughout the paper. I found the blue and oranger text made the paper much harder to read.\n- Move Algorithm 1 pseudocode to Section 5.\n- Make figures larger.\n- Take a careful pass through your equations and notation, with some of the below suggestions in mind.\n\nLow-level, section by section:\n\nIntro:\n- \"achieve exploit-explore trade-off\" --> \"balance exploitation and exploration\"\n- \"suggested by (Christodoulou, 2019)\" --> \"suggested by Christodoulou (2019)\"\n- \"resulting in the discrete version of SAC, denoted as discrete SAC (DSAC)\" --> \" resulting in discrete SAC (DSAC)\"\n- \"fails also include the absence of policy update constraints\" --> \"fails also due to the absence of policy update constraints\"\n- \"we propose Stable Discrete SAC\" --> \"we propose Stable Discrete SAC (SD-SAC)\"\n\n Preliminaries:\n- If the action space is discrete, then the Shannon entropy of $\\pi(\\cdot \\mid s)$ would be better expressed as a sum, rather than an integral. \n- \"bellman\" --> \"Bellman\"\n- Equation 1 uses a finite horizon, undiscounted expected (regularized) return. But later, discounts are used. It would be helpful to clearly establish your setting, first. \n- Terms are bolded unexpectedly (Equation 9), can you clarify why this is the case in the text?\n\nFailure Modes\n- \"with the Q-learning iteration\" --> \"with Q-learning iteration\"\n- Since your new method builds on the double Q-learning trick, it might be valuable to include slightly more detail about what it is (Section 4.2).\n- Can you move Algorithm 1 to Section 5?\n\nCase Study in Honor of Kings\n- Since this is only a few paragraphs, I believe you can make this a subsection of Section 6 rather than its own section.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OwVn6Xzucm", "reviewer": "Reviewer_vJSy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper does not clearly explain how the failure modes of discrete SAC are specific to discrete SAC. Section 4.1 also fails to provide convincing evidence that \"unstable coupling training\" (awkward wording, by the way) is indeed a cause of discrete SAC's poor performance in some cases. Issues with the case study in Section 4.1:\n1. In the Asterix example, how are the rewards \"deceptive\" and why would quickly learning to pick up all items from the early rounds be a problem? If the policy learns to pick up all items and applies this strategy in later rounds, it will quickly get data showing that collecting a lyre is actually bad and will ideally adapt to this new data quickly. This behavior is the usual progression for optimistic exploration, so why would it be a problem?\n\n    The data in Figure 2 also does not seem to match the analysis in the main text. The episode length increase ~40% after two million steps and the number of steps with rewards increases ~27%. Both of these increases mostly happen within the next 3 million steps. These increases are both substantial compared to the shown variability, which directly contradicts the main text.\n2. The text notes that the policies become deterministic \"quickly\" (I suppose the authors mean within two million steps). Since the authors have chosen the minimum entropy and the learning rate for $\\alpha$ tuning, is this not simply a signal that the minimum entropy is too low or the $\\alpha$ learning rate is poorly tuned for this task? Figure 15 in the appendix show that the single shared learning rate has a huge impact on entropy and overall performance.\n3. The text claims that the policy and Q-value converge to local optimums, and then immediately contradicts this statement by noting that the policy re-rises in the later stage of learning. If the policy is at a local optimum, its entropy could not change. The Q-value curve also suggests that the critic network is still changing at the end of the run. Given a comment in the appendix, these lines are probably also smoothed, is that correct? If so, the networks are actually changing even more than the plots suggest. Is there evidence for this claim of convergence?\n\nIn Section 4.2 how are the true values computed for Figure 3 and why is there confidence shading? Are the true values not exact? I think the approximate and true values for each individual run should be compared together so it is unclear how to interpret these plots. Plotting the error rather than the values could alleviate this issue.\n\nI am unconvinced that all of discrete SAC's tuning options were explored before introducing another loss component and hyperparameter. Figure 15 confirms that the shared learning rate has a massive impact on entropy and overall performance. What if the $\\alpha$ learning rate is tuned separately? What if $\\alpha$ is set to a fixed value, or if that value is annealed over time? There should be a relationship between the learning rate of $\\alpha$ and the value of $\\beta$ and I would want to see that explored. Since $\\alpha$ is the temperature parameter, there is a formal relationship between $\\alpha$ and how far the policy update step can move the policy from the previous policy (and therefore how much the entropy can change) in terms of the KL divergence, which we can see in the analysis of mirror descent with a KL divergence regularizer. Does the method proposed by this paper merely trade away having to carefully tune $\\alpha$ in return for having to carefully tune $\\beta$?\n\nA general concern about the experimental results presented in this paper is that the results are very noisy and variable for only performing three repetitions. In many of the figures, I am uncertain that the observed patterns are actually reliable. Part of the difficulty is that the figures plot confidence as shaded regions (though exactly what these shaded regions is never explained, as far as I can tell). Three runs is not enough to show uncertainty as confidence intervals. Confidence intervals require a variance estimate, which almost always requires more than three runs to be accurate. Please plot each individual run instead, at least in the appendix. For the main plot figures when there are too many competing methods to show all three runs from each, plot the \"median run\", that is, find the runs for a given method with the largest and smallest final or average returns (or both if they are different and each analysis provides different insights) and exclude them. Then plot only the remaining run.\n\nThe Q-clipping that this paper presents is unsafe from a learning and optimization perspective. The clipping not only prevents incentivizing the critic network from making predictions more than $c$ away from those of the target network, it zeros out gradients whenever the critic network makes predictions $c$ away from those of the target network. That means that if $Q \\ge Q' + c$ or $Q \\le Q' - c$ on all the data points, $Q$ is a local minimum. While the network may not be incentivized to reach such a state, if it does reach such a state, all learning in the critic would stop until the target network is updated and the predictions would likely be bad.\n\nDiscrete SAC + penalty + avg-q appears to perform much worse than Discrete SAC + avg-q, but this failure is not addressed. Why does this failure occur?\n\nI do not understand how Figure 7 shows an improvement over discrete SAC. Most of the plots look qualitatively similar to me. The main difference I see is that the plot after ten million steps is more shallow that discrete SAC's but why is that better? All of them look nearly convex and both methods look to me like they have a comparable number of saddle-points and are of similar \"complexity\".\n\nThe Honor of Kings case study is poorly developed and has multiple issues.\n1. Saying that Honor of Kings is \"the world's most popular MOBA\" is an unnecessarily hyperbolic claim.\n2. Stating that Honor of Kings is a \"popular testbed for RL research\" when all three citations for this claim share many of the same authors is disingenuous.\n3. This paper proposes that discrete SAC specifically suffers from issues that do not befall SAC in continuous action domains. If that is true, why discretize the direction of movement and skill? Honor of Kings seems like an odd choice to illustrate improvements to discrete SAC when its natural action representation is at least partially continuous.\n4. How should we interpret the ELO scores? How were they computed? Who were the opponents? How many matches were played? What does 1343 ELO play look like compared to 430 in this game?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to improve the performance of the soft actor critic (SAC) approach in discrete action tasks. The authors propose some hypotheses about the issues holding back the discrete SAC algorithm and present experimental metrics as part of this discussion. They then suggest three algorithmic modifications: an entropy change penalty, averaging two critic networks, and clipping critic network predictions during critic training so they are not too far from target critic network predictions. The paper presents experimental results in Atari games and ELO scores for an application of their method in a a multi-player online battle arena game.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A less critic change that would strengthen the work is to perform more repetitions to ensure that observations are more signal than noise.\n\n### Critical Changes\n\n1. Improve the analysis of failure modes and explain why they are specific to discrete SAC.\n2. Discuss and analyze the relationship between $\\alpha$, the $\\alpha$ learning rate, and $\\beta$. Show why the entropy change loss and $\\beta$ are indeed necessary.\n3. Improve plots showing values from individual runs, comparing median runs, and analyze variability more transparently by comparing the min, max, and median runs.\n4. Address the optimization difficulties that Q-clipping introduces, either by modifying the approach to be sound or by showing thoroughly that the lack of soundness is necessary.\n5. Improve the experiments section to clarify the points I mentioned in the previous section.\n6. Address questions and concerns about the Honor of Kings case study.\n7. Proofread the paper and edit wording. There are a number of awkward wordings in the paper where it is difficult to understand what is being said.", "broader_impact_concerns": "I am not familiar with the game Honor of Kings so perhaps the community and stakeholders have ways to mitigate cheating with autonomous agents but a possible ethical concern is that people could use released code or re-implement methods from this paper to build policies for Honor and Kings, passing the play of those policies off as themselves.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall I think the paper makes a decent contribution to RL research. The two proposed tricks perform well under the present settings and may be used more broadly in future work. The paper improves over the previous TMLR submission and gave more convincing explanations to the observed phenomena than the previous version. 

That said, there are many remaining concerns from the reviewer which the authors must address in their final version:
* More careful statistical significance analysis of the results, such as plotting existing runs with individual runs shown (with matching seeds for different algorithms) similar to Figure 10, or (optionally) running more seeds
* Details about the Honor of Kings experiment mentioned in the rebuttal
* (Optionally) more comprehensive tuning of $(\alpha, \beta)$ in the entropy regularization, and justifications of the choices made in the main text experiments
Decision: Accept (poster)