Addressing High-dimensional Continuous Action Space via Decomposed Discrete Policy-Critic
Authors: Yechen Zhang
Jian Sun
Gang Wang
Jie Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
continuous control
actor-critic
decomposed policy
discretized action

Abstract: Reinforcement learning (RL) methods for discrete action spaces like DQNs are being widely used in tasks such as Atari games. However, they encounter difficulties when addressing continuous control tasks, since discretizing continuous action space incurs the curse-of-dimensionality. To tackle continuous control tasks via discretized actions, we propose a decomposed discrete policy-critic (D2PC) architecture, which was inspired by multi-agent RL (MARL) and associates with each action dimension a discrete policy, while leveraging a single critic network to provide a shared evaluation. Building on D2PC, we advocate soft stochastic D2PC (SD2PC)  and deterministic D2PC (D3PC) methods with a discrete stochastic or deterministic policy, which show comparable or  superior training performances relative to even continuous actor-critic methods. Additionally, we design a mechanism that allows D3PC to interact with continuous actor-critic methods, contributing to the Q-policy-critic (QPC) algorithm, which inherits the training efficiency of discrete RL and the near-optimal final performance of continuous RL algorithms. Substantial experimental results on several continuous benchmark tasks validate our claims.

Under review as a conference paper at ICLR 2023
ADDRESSING HIGH-DIMENSIONAL CONTINUOUS AC- TION SPACE VIA DECOMPOSED DISCRETE POLICY- CRITIC
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning (RI.) methods for discrete action spaces like DQNs being widely used in tasks such as Atari games. However, they encounter diffi- culties when addressing continuous control tasks, since discretizing continuous action space incurs the curse-of -inee--f-dmmessiinaltty. To tackle continuous control tasks via discretized actions, we propose a decomposed discrete policy-critic (D2PC) architecture, which was inspired by multi-agent RL (MARL) and associates with each action dimension a discrete policy, while leveraging a single critic network to provide a shared evaluation. Building on D2PC, we advocate soft stochastic D2PC (SD2PC) and deterministic D2PC (D3PC) methods with a discrete stochastic or deterministic policy, which show comparable superior training performances rel- ative to even continuous actor-critic methods. Additionally, we design a mechanism that allows D3PC interact with continuous actor-critic methods, contributing to the Q-policy-critic (QPC) algorithm, which inherits the training efficiency discrete RL and the near-optimal final performance of continuous RL algorithms. Substantial experimental results on several continuous benchmark tasks validate our claims.
INTRODUCTION
Reinforcement learning Sutton Barto (2018) a class of machine learning methods that train models using interactions with the environment. RI. can be divided into discrete RL with a discrete action space that contains limited actions, and continuous RL with a continuous action space having infinite actions. Actor-critic Konda & Tsitsiklis (1999) structure that uses policy network to learn a state-action value function, is commonly used to address continuous RL tasks such as continuous control, but it is demonstrated to be fragile and sensitive hyperparameters Haarnoja al. (2018a).
Employing discrete RL algorithms for continuous control via action discretization is a feasible way to improve the training efficiency Pazis & Lagoudakis (2009), since discrete RI. algorithms including Q-learning Watkins & Dayan (992) and deep Q network (DQN) et al.) (2013) 2015) have low complexity and exhibit stable training behavior. But for continuous control tasks with high-dimensional continuous action spaces Levine Leaces et al. (2016), naive action discretization would lead the dimension explosion problem. For example, if the continuous action space contains M dimensions and each dimension is discretized to N fractions, there will be N actions for the discrete RL algorithm.
As a remedy, a possible way to address this problem is to discretize independently each action dimension of the high-dimensiooal continuous action space, so that the total number of discrete actions MN grows linear instead of exponential in dimension M. Nonetheless, this procedure may lead to the low sample-efficiency problem, because one can only take on-policy iterations to collaboratively learn the relationship between different action dimensions' policies. To integrate the discrete action space and off- pol-co iterations a continuous RL algorithm, we propose a decomposed discrete policy-critic (D2PC) architecture inspired by multi-agent actor-critic methods, by viewing each action dimension as an independent agent and assigning a uni diidimmssiooal policy which is optimized by referencing centralized critic network.

---

Under review as a conference paper at ICLR 2023
Based D2PC, we propose two algorithms which can effectively address continuous control tasks via discrete policy and experience replay. The first algorithm is soft D2PC (SD2PC), which takes maximum-entropp value iterations and optimizes softmax stochastic policies for each action dimension; and the second called determined D2PC (D3PCC) which assigns an independent Q function for each action dimension fitting the critic's value function by supervised learning. Experiments show that the two algorithms exhibit high training efficiencies and markedly outperform state-of-the-rr actor-critic algorithms like twin delayed deep deterministic policy gradients (TD3) Fujimoto et al. (2018) or soft actor-critic (SAC) Haarnoja et al. (20
Though algorithms with D2PC trains fast, we observe from experiments that they may be limited in uchieving the best performance finally, namely getting stuck in locally optimal solutions. Since D2PC and standard actor-critic methods share the same critic structure, this motivates us to combine the best of the two methods, by developing the Q-policy-crite diserete-continuuus hybrid policy, co-trains continuous actor along with D2PC, and dynamically exploits the discrete and continuous actors improve the critic network. Substantial numerical tests over continuous control tasks demonstrate that QPC achieves significant improvement in convergence, stability, and rewards compared with both D3PC and continuous actor-critic methods.
(QPC) algorithm. Our QPC uses
2 RELATED WORK
As pointed out in Tavakoli et al. (2018) it is possible for discrete RL algorithms to address continuous action spaces, by regarding the multi-dimensional continuous action space as fully cooperative multi- agent reinforcement learning between dimensions, that is, each action dimension agent. Several attempts have been made by splitting a multi-dimensional continuous policy into single-dimensional discrete policies. For example, Metz et al. (2017) proposed a next-step prediction model to learn each dimension's 's discrete Q value sequentially; Tang & Agrawal (2020) integrated dimeenniiil discrete policies with ordinal parameterization to encode the natural ordering between discrete actions; see also Jaškowski et al.] (2018) and Andrychowicz et al. (2020). It is worth remarking that these methods either cannot exploit experience replay cannot deal with nigh-dimenssonal continuous action spaces, rendering them typically less effective than actor-critic methods for continuous control.
independent
Our proposed D2PC structure, in some way, can be seen as a variant of multi-agent actor-critic, which relies on a centralized critic network to optimize each action dimension's 's discrete policy. Since the seminal contribution of deterministic policy gradients Silver et al. (2014) in 2014, off- policy actor-critic methods have arguably become the most effective way dealing with continuous control tasks, thanks to their appropriate structure to deal with continuous actions and higher data efficiency compared with on-policy algorithms, e.g.. proximal policy optimization (PPO, [Schulman al. (2015)), trust-region policy optimization Schulman et al. 2015); Wu et al. (2017), among many others. Off-policy actor-critic methods can be grouped into two categories: deterministic algorithms including e.g., the celebrated DDPG Lillicrp et al. (2015). D4PG Barth-Maron et al. (2018). TD3, and stochastic algorithms such as SQL Haarnoja et al. (211)). and SAC Haarnoja et al. (2018a). DDPG uses the Bellman equation [Bellman (1966) with the temporal difference method to iteratively update the critic network, in which the loss function is given by
(9(02) = (())s()))((()))))))))))))))) - (re+ +Q(((1 1(:(1+
(1)
where s(t) represents the state, a(t) the action, rt the reward, and 7 the discounting factor; 0g and & denote the parameters of the critic network and the target critic network; 0₂ and 6,r denote the parameters of the current and the target deterministic policy networks. The critic network DDPG provides evaluations of the actions to optimize the policy of the actor network by minimizing the following loss function
J((,))) = - ((((())) ((s)))))
(2)
For SAC with a stochastic policy, they developed soft RL, which introduces entropy into value and policy iterations. SAC's critic network is trained using optimizing the following loss function
(a((z) = (0(s(),a(1)00) - -(r+ + >7((s( ()) + =aH)]
(3)
2

---

Under review as a conference paper at ICLR 2023
Derenacn Poliy NNtt
Action Pemmuutions
Actios
Sbacs
PaRay Nazt:
Action sosoutiionn
Acd onG
Ditarnico
Dimention 2
Decommposod
D Dimerdoon M
Cimernzzon M
minesoon
Choice
(a) Distributed discrete policy network
(b) Decomposed discrete policy network
Figure 1: Network comparison between distributed and decomposed discrete policy networks.
where the entropy H := log (aa)s((t + 1); 0m) with a' ~ <aaa s + 0,) is sampled from the stochastic policy æ, and a is a "temperature hyperparameter' which controls the exploration rate of the algorithm and can be automatically turned Haarnoja al. (2018b). SAC's policy loss function is given by
Js (0+) = - Q(((()((()))))))))()))))))))QQ 02))
where H = log n (a) (()) 0x) with a (a]s(((); 02). The loss function of our SD2PC algorithm is based on SAC, which generalized its value and policy iterations to decomposed, fully cooperative multi-agent settings.
Although our D2PC structure is inspired by the framework of multi-agent actor-critic RL, we make considerable simplifications to reduce the computational complexity. previous works like MADDPG Lowe et al.] (2017). they set an actor and critic for each agent, or COMA Foerster et al. (2018) which uses recurrent neural networks such as long-short term memory (L Hochreiter & Schmidhuber (1997) or gated recurrent units (GRUs) Chung et al. (2014) to encode the partially observed states. Our framework can be viewed as dealing with observable states and fully cooperative capitalize single critic for all action dimensions instead of one for
agents. Therefore, each and every action dimension. Imitating the multi-agent actor-critic RL, we allocate a policy network (actor) for each action dimension like the structure in Figure (T.). This structure, however, requires to optimize M sets of policy network parameters for task, which still incurs relatively high computational complexity. To further reduce the complexity, we advocate a centralized decomposed discrete policy network shown in Figure (Tp), which can output all the action-dimensional agents' policies simultaneously, markedly improving the algorithm's 's efficiency.
DECOMPOSED DISCRETE POLICY-CRITIC STRUCTURE
In this work, we focus addressing continuous control tasks by discrete actions, where continuous action space A == {a E RM) with M dimensions is discretized into countable discrete actions. Without loss of generality, we assume that each continuous action is restricted am E [-1, 1], so that one can discretize each continuous action dimension m into N fractions. The resulting discrete action space Ad for the m-th dimension denoted by
m = (am.1: : 2,221 N for m m [M] == (1.2....,,A) M
(5)
where ad represents the n-th discrete action corresponding to the action dimension The discretization above operates on a single action dimension. The discretized action space for A is obtained by combining all possible M combinations of the discrete actions, namely, A := {am. [e[ where [N] == 21, N). Instead of setting up an output neuron for each element in Ad, we construct for each dimension an independent discrete policy covering all the discrete actions in A
In previous works, there are mainly two types of methods which are compatible with experience replay, that is. DQN and actor-critic methods. We first consider using DQN to train a deterministic policy for each action dimension, given its simple structure and low computational overhead. Nevertheless, found is problematic to utilize DQN for each action dimension while using experience replay. The reason is as follows. Consider the Markov decision process (MDP) over the discretized action space Ad, described as AS, p. r} in which S is the state space, p is the transition probability function, and is the reward function. In RL. algorithms using TD(O) and experience replay, the experiences
3

---

Under review as a conference paper at ICLR 2023
(s(t),a(t) T2, s(t 1)} obey the transition probability distribution function p(s(t+ a(t)), so they can be judiciously exploited to train the state-action value function. When updating each action dimension's 's policy independently, the MDP will degenerate into single-action MDP {S, AS Pm r with action space Am As for the remaining dimensions, their polices can be viewed as fixed since independent updates are assumed across dimensions. Hence, the remaining policies can be accounted for by defining a marginal transition probability function as follows
Pm (s(t + ())s(t), Om am))))))))))))))))))))))) (v,,.()) (e.,..o)(p((((ttttt 1))s(t), (lms((t,,mmtt)) (t),), am (t)) where (t) [a (t), (t). ...., am- (t), am++ (() with at(t) (a)((())))))..
(6)
(t)
In order to accommodate the technique of experience replay in our single-dimension action's update, the experiences which used to replay must obey the transition probability function pm- Nonetheless, the actions am of experiences in the replay buffer are not sampled by the current policy Tm (t)s(t))) but by some xa (am (t)) so the experiences sampled in the past do not obey the transition probability function pm- That is, one cannot readily use TD(O) along with experience replay to train the rrr-actioo-ddmmemsn
policy.
Concerning the issues above, we turn to the actor-critic structure with experience replay, by means of adopting a centralized critic network (ss(t,, a(t); (Q) to evaluate all the actions and distributed policy networks Fm (cm (t)): f.... )) M instead of mnc-dimensional value functions for each action dimension. The critic network's 's value function can be updated by using experience replay, as it provides critic values for all M action dimensions and its updating process based on the MDP instead of independent single-action MDPs. As to the policy networks associated with action dimensions, they can be updated by using policy gradients with critic values provided by the centralized critic network. Each action dimension's 's policy loss can be given by
J(O=...) ) = E= (um ((); Q(s(()((()))))))))()))))))))QQ am (1), Um (1);
M where, with slight abuse of notation, we write a(t) {am (am(())) (t) m=1 {am Tm (t)) yielding Q(s(t), am (t), 7. (t); 02) ) - Q(s(t), a()) 00) for any m E [M]. Overall, corresponding to equa- tion 7 the considered network structure with centralized critic and distributed policies is described in Figure a.). However, this structure has limitation, which has M groups of policy network parameters [ A leading high computational complexity. In order to improve the train- ing efficiency, we design decomposed global policy network π { mE[M] S - R M parameterized by 0.. Rather than outputting each action dimension's 's policy by multiple individual networks, the decomposed policy network outputs a policy matrix. Each row of the matrix stands for a discretized action dimension's 's policy distribution. The decomposed discrete policy network's structure is shown in Figure I bb).
The structure of the decomposed discrete policy network allows us to update its parameters simply by the stochastic policy gradient method, which samples action a(t) and updates the policy of all the M action dimensions simultaneously as follows
am (t) Tm fmm (t))s(t); 0.), where ä(t) ((),))..,,,(t)),
(8)
0x ー 0x + +(((((),
II m=1
Tm (ām (a)((())))())))))))))))))))))))) (t))s(t):0.))
where A >> 0 is the learning rate.
4 STOCHASTIC SOFT DECOMPOSED DISCRETE POLICY-CRITIC
Due the high variance of stochastically sampled critic values and the lack of exploration strategies, updating the discrete policy with the stochastic policy gradients in equation 9 may not be effective. To value
improve the training efficiency of D2PC, we consider introducing soft RL and dimension expectation technique into D2PC, which lead to our effective SD2PC algorithm. In the following, we begin by using a soft Q function with entropy H := - -> log Rm (#m (t + 1)]s(t + (C) generated by the decomposed discrete policy, to update the D2PC's s critic network, leading to a value loss function similar to that of the SAC algorithm in equation 3
(Q())) = (Q((((). - (ra + * 1), (1:(1+1):0)) 1):0;) + +0H1]²
(10)
4

---

Under review as a conference paper at ICLR 2023
Frobaailty by of et on dimernienn
Pokky Malta
Action Destousoons
Tons
Σ
Occomposce Net
Contrilzed Net
State-Acte Action
olcbal loa (10g)
Dimeenann 8f
Random
siggeevation
Figure 2: The computation flow of evaluating the loss function (0-1())) in SD2PC, while all
J m.n (0z)'s I's can be evaluated simultaneously in minibatch to improve computational efficiency.
where = ....... (t + 1)} is global action sampled by the target policies {ām (t (t+ 1) ~J (am (t (1))s(t)); [m([ sharing single target policy network 8' As far as the policy update, we employ the loss function proposed in [Christodoulou (2019) for discrete SAC methods based on policy gradient updates
M
J" (=,(0)) = N
log Tm (im (t))s(();;))) Q()() (())0()((.0)0-00000000)))))0)))
(1I)
m=
practice however, it impossible to evaluate the expectation involving high-dimensional random actions in equation 11] while requires us cover the entire global discrete action space Ad.. On the other hand, we simply draw one action ā(t) and rely on the instantancous loss to approximate the expected one J (0=), the resulting stochastic policy gradients will certainly have high variance. Nonetheless, we observe that the expectation of the loss function per dimension m can be efficiently evaluated in closed-form. Therefore, mitigate the high variance of stochastic sampling as well the high computational complexity of covering Add, we compute a surrogate of equation 11 by changing the order of the expectation and summation, resulting sum of expectations across dimensions, for which one only need to cover each Ar efficiently
M
(0,)
Σ m=
Ean (1)mmm) a log nm (ām (t))s(t):0,)) (o, (t)))))
M
(12)
The key idea behind equation 12 is to regard each action dimension as an independent agent, in that way we can first sample global action @()). then cover each and every action dimension's action space Add to obtain the averaged sample loss. Mcanwhile, the remaining dimensions' actions
collectively denoted by (t) are viewed deterministic and replaced with their sampled values
when generating the Q value ((s(t), (tm(t), 7. (t)}: for ām Further, equation 12 can be equivalently rewritten follows
N
(²
1
Σ
mm (0x)
(13)
M
m=
ā.. (t) J (f.) = ( log Ti (a (t)) (1/M)Q(s(t)) ( (2)]
The evaluation of each J m.n (6,) is illustrated in Figure where m = = 1. For each J m.n (0z), one needs propagate over the critic network once. Fortunately, the loss evaluations can be performed in parallel across both dimensions and discrete actions process. addition, the discrete policy of SD2PC can also be updated by minimizing a policy loss based on KL divergence, which is different from equation 13 but shows the same effect. We presents the details of our KL divergence policy loss SD2PC Appendix A.5 which can an optional substitute of equation 13
dimension, thus accelerating the training
In terms of the exploration strategy for SD2PC, we implement the automatic temperature adjustment strategy, which presented in the second version of SAC in Haarnoja et al. (2018b). With prescribed target entropy H. SD2PC can automatically adjust the temperature hyperparameter a to control the algorithm's exploration rate. When H > H. i.e., the exploration rate excessive, SD2PC
5

---

Under review as a conference paper at ICLR 2023
will decrease a, and increase otherwise, by means of minimizing the following temperature loss function using gradient descent
M
J(a)
Σ M
1
Bam (t)mrmm
log Tm (mm(mm)))))))()))))))))))))))))) a
(14)
m=1
Similarly to equation
12 the expectation can be approximately evaluated using
M
[[-o7 - a J(o) M
1
Σ
Σ
(a m.n sst)) log n. (c
(15)
We would like to remark that the target entropy H in SD2PC is set for each action dimension's policy, while SAC sets a target entropy for the global policy. That is, one can interpret that Sd2PC scale-free hyperparameter H. which does not vary with the number of action dimensions M in changing tasks. Nevertheless, in discrete action settings, the proper target entropy H. is relevant to the quantity N of discretized actions, which renders H hard adjust. To address this issue, propose an action probability transfer trick, by transferring the discrete probabilities into probability densities. With this trick, in all the baseline environments considered in Section 7 details about the trick are provided in Appendix F.2
we set H. 1.5.. More
DETERMINISTIC DECOMPOSED DISCRETE POLICY-CRITIC
Although the proposed SD2PC method in Section 4 can tackle continuous control tasks via discrete stochastic policies, it is problematic to address continuous action space via discrete deterministic policies. When dealing with discrete deterministic settings, nether can one update the policy by optimizing the policy distribution nor adjust the action by deterministic policy gradients like DDPG Lillicrap et al. (2015), because the policy distribution is deterministic, and the discrete actions are fixed during discretization.
In order to address continuous RL via discrete deterministic policies, we further put forward decomposed deep Q network to replace the decomposed stochastic policy network in D2PC. Instead of outputting policy distribution matrix, the decomposed deep Q network outputs Q value matrix, in which each row of the matrix stands for the discrete Q values Qa of all the discrete actions associated with one action dimension. Let us use Qu to denote the state-action value of s(t) and discrete action d mn while the other dimensions' actions generated by the current policy
8o
Qa(s(t), an Epen Mm
Σxrr k==
where ō(t + k) = (tmmt + +)), (t (t+ k))
(16)
Tm (1) a
Hm (1 + k) = um (s(1 k)) for k >
ãm (t + k) = Fm + k)) for k 2
s(t k) p( ss( + k 1), a(t + 1))
in which Pmm represents the m-th action dimension's 's greedy policy, which relies on the Q value matrix generated by the decomposed deep Q network, i.c., by selecting the action having the maximum Qa value in the m-row of Q value matrix follows
am (1) Mm (s(()) arg max Qa(s((), am..). ad E.Ad.
(17)
Let us write ja(((()) {ur (s(1)) ...,)) PM (s( = (sm(((()) pm (s(()) to be the global policy. If the policy Fi (s(t + k)) is fixed, then (s)s(t), a ((s(t), to am (t)}), where ā. (t) = T. (s(( + k:)). So, for decomposed deep Q network with parameters 0g, we can update its deterministic greedy policy through minimizing the difference between each discrete action's Q value in the matrix and the corresponding Q value by the critic network
Jdl m. (0x) = (Ra(s(1). 04) Q(s((), ( (a F (s()))))))))))))))
(18)

---

Under review as a conference paper at ICLR 2023
Decrnee Vakue of efa
Diornee Direndion
M
Σ
Decemposed Dcop
Centrolaedd Not
glcoal loss MMP
Dimens
org max
sigle acton
Figure 3: The computation flow of evaluating the partial loss function (0,) in D3PC. Similarly, all Jd (0d)'s ss can be obtained in parallel using mini-batcher m.n
The flow for computing the partial loss J m.n in equation 18 is demonstrated in Figure 10 for m 1. Similarly to SI22PC, for every action in the discrete action space, compute local loss function that contributes to the global loss as follows
M Σ
(0,)
Σ
Jál m.n (0a).
(19)
=1=I ==
The update of the critic network can be done using TD(0) learning. which is similar to equation 3 DDPG, but the target actions are generated by our proposed decomposed discrete policy. We also deploy target policy network parameterized by 0'2 to stabilize the target policy. The loss function for D3PC's critic can given as follows
(Q ((s((t) - (ri 1Q((t 1)) ():
(20)
For D3PC, design a novel e-Gaussian exploration strategy Th (a()))))(()) 0a) to promote explo- ration over the action space. Our e-Gaussian strategy first relies on e-greedy to select a discrete action and subsequently maps it back to the continuous action space, which, combined with a Gaussian exploration noise (0000 a²), is used to interact with the environment. We compared different explo- ration strategies Appendix [.3 which show that our e-Gaussian outperforms both the e-greedy and the plain-vanilla Gaussian exploration strategies. In D3PC, we also employ soft target updates and TD3's double critic Network/Fujimot et al. (2018). The pseudo-code of the D3PC algorithm is provided in Appendix B
6 ACTOR-CRITHE WITH DISCRETE- HYBRID POLICY
In our experiments, we found that D3PC outperforms in training efficiency, but may sacrifice the final performance when comparing with continuous actor-critic algorithmss, such as TD3, in baseline environments like Humanoid-v2. This observation has motivated us to combine the best of the two worlds, namely the training efficiency of discrete RL and the outstanding final performance of continuous RL algorithms. To this aim, we realize that D3PC and DDPG are only different in their policy networks, with their critic networks being the same. It is thus natural question whether we can use the D3PC's critic network to train a continuous actor, or use the DDPG's critic network to train a decomposed discrete Q network?
Concerning the fact that we can update an actor if there is a well-trained critic network, we propose two conceptual algorithms as a manner explore the common behind D3PC and DDPG. The first algorithm, which we call 1.D3PCCCA, trains an additional continuous actor network along with D3PC using the same critic network, but the new actor does not contribute to the D3PC's value iterations; and, the second algorithm is 2. 2DDD-D that adds decomposed discrete network in TD3 (a variant of DDPG), and trains the Q network using TD3's critic free from TD3's value iterations. Further details can be found in Appendix C Experimental results of the two algorithms several benchmark tasks are depicted in Figure Obviously, D3PC-CA's continuous policy performs much better than TD3-DQ in terms of final reward performances.
According to the conclusions above. we consider improving D3PC's performance by intertwining with continuous policy. Practically, we can start by using D3PC at the beginning of the training
7

---

Under review as a conference paper at ICLR 2023
Figure 4: Total averaged rewards of DCQ-AC and TD3-CQ over 5 seeds.
Discrete Q Values Dimension 1
Value Matric
Discrete Actions
States
6
Aimeniio
Target Crii
Dacomposed Daap Net
mension M
Weighted Average
2ng max
6g'
State-action value V
Continuous Actor
Target CFRi Net
Continuzus Actions
Critic's updates QPC, whose target Q value is the weighted average the Q values generated by the decomposed Q network and by the policy network.
Figure 5:
process, then gradually shifting to an algorithm hybridizing D3PC's and TD3's policies, which is expected to inherit the initial training efficiency of D3PC well the final performance of TD3. Following this idea, we develop the Q-policy-critic (QPC) algorithm, which is composed of three networks, with Q representing a decomposed deep Q network ((s(t); 6a). Prepresenting a continuous policy network ye (s(t); 0p). and C representing critic network ((s(t),(()) 0Q). While training QPC, the continuous policy Op is updated using deterministic policy gradients in equation decomposed Q network updated by equation T9 As for the critic's 's update, it performs TD(0) iterations with TD target obtained by averaging two target Q values: Q' = Q(s(t + ((((t + 1). ()): ()) with target action generated by the Q network's policy, and Ø = Q(s(t +
2 and
+ 1). 1),6)));00)) ())))) whose target action is generated by the continuous target actor. The way of computing the TD target in QPC is summarized in Figure 5 with the following value loss function
O'pp = 3Q, + (1 31-3)
(21a)
(9()2) Q(s(()) - (rr + [(ap)]
(21b)
where [0, 1] hyperparameter balaneing between the continuous RL and the discrete RL. Typically, one can start the training by setting / = for the first T3 = 1 x 10s steps to fully exploit the training efficiency of D3PC. Afterwards, a decay rate of 3_
- 5 x 10-6 6 is used, until minimum of 8.ii 0.5 is reached and kept fixed. Additionally, QPC hybrid policy to interact with the environment, which is found by averaging the discrete action and the continuous action as follows
discrete-comtinuous
a(t) Bu(s(tt))))))))) 02) + (1 ())
(22)
EXPERIMENTS
The objective of our experimental evaluation is to understand how effective and efficient our proposed discrete RL methods can handle challenging continuous control tasks from the OpenAl gym Brock man et al. (2016) benchmark suite based on MuJoCo Todorov et al. (2012) environments, including Hopper-v2, Walker2d-v2, Ant-v2, Halfcheetah-v2. Humanoid- v2, and InvertedDoubleeendduum
Details of the baseline environments provided in Appendix D
8

---

Under review as a conference paper at ICLR 2023
Figure 6: Training curves for the proposed algorithms, and the baselines on benchmark tasks, averaged over random seeds.
We compared our SD2PC, D3PC, and QPC algorithm to the most commonly used off- off-pollcy actor- critic algorithms, including SAC SAC Haarnoja et al. (201 8b). TD3 Fujimoto et al. (2018), and DDPG Lillicrap al. (201)). For the three baseline algorithms, we used their author-provided implementa- tions with the original hyperparameters, except for TD3 with Gaussian exploration noise 0.05 in Humanoid-v2 to avoid training failures. All of the tasks have continuous actions clipped to [-1,]], 1]. while we discretize each action dimension into N 20 actions. We set the target entropy hyperparameter H 11.5 5 for SD2PC, and QPC. Hyperparameters for the algorithms and additional network details are listed in Appendices A and C All the algorithms are realized in PyTorch Paszke al. (2019), and evaluated
0.05 for the exploration strategies in D3PC and
PC with one NVIDIA GPU. For each task in our experiments, we train the algorithms with Adam Kingma & Ba (2014) optimizers, and report experimental results averaged over random seeds. During the we evaluate performances of the algorithms over five episodes by turning off the exploration strategy.
training process, every 5,000 steps (except for 500 for Inverted
The experimental results are shown in Figure in which the total average reward curves are smoothed for visual clarity. Surprisingly, the proposed methods using discrete policies exhibit similar and oftentimes competitive training performance relative to the baseline continuous RL methods in all
the benchmark environments. In contrast, as observed Tavakoli et al. (2018), existing discrete RL methods cannot even compete with the DDPG method Lillicrap et al. (015)) To the best of our knowledge, for the first time, our discrete RL methods can achieve improved performance over continuous RL methods in such challenging tasks. However, D3PC may have drawbacks in final performance, especially when dealing with complicated tasks such as Ant-v2 and Humanoid. As remedy for D3PC, QPC fixes the drawback by mixing the discrete and continuous methods. Additionally, we made some ablation studies on the exploration policies and hyperparameters for our algorithms. Due space limitations, these results deferred to Appendix D
8 CONCLUSIONS
This paper has considered addressing nigh-dimensional continuous RL tasks using discrete policies. Toward this objective, we put forth decomposed discrete policy-critic framework. building on which three algorithms are developed: SD2PC for discrete stochastic policies, D3PC for deterministic discrete policies, and QPC for discrete-continuous hybrid policies. Substantial experimental results on benchmark tasks corroborate the efficiency of the proposed discrete RL algorithms relative to competing continuous RL alternatives including SAC, TD3, and DDPG. Despite their promising performance, we point out several directions in which the proposed methods could be improved. For example, generating the discrete action's 's Q values through linear interpolation instead of the critic network can reduce the computational complexity, and incorporating strategies such including the prioritized experience replay (Schaul et al. (2013), distributional value functions [Bellemare et al. (2017), and multi-step temporal -iifeeeeeeeeeeeeee could be effective.

---

Under review as a conference paper at ICLR 2023
REFERENCES
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-200 2020.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. Arxiv Preprint 1804.088 2018.
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pp. 449 458. PMLR, 2017.
Richard Bellman. Dynamic programming. Science, 153(3731):34 37, 1966.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. Arxiv Preprint Arxiv: 2016.
Petros Christodoulou. Soft actor-critic for discrete action settings. Arxiv Preprint Arxiv: 1910:0090007207, 2019.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. Arxiv preprint Arxiv: 441.:3/4 2014.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference Machine Learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnjja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference Machine Learning, pp. 1861-1870. PMLR, 2018a. 8a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. Arxiv Preprint 112.:5181,.05505, 2018b.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, 1997.
Wojciech Jaškowski, Odd Rune Lykkebø, Nihat Engin Toklu, Florian Trifterer, Zdeněk Buk, Jan Koutník, and Faustino Gomez. Reinforcement learning to run... fast. In The NIPS' 17 Competition: Building Intelligent Systems, pp. 155-167. Springer, 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. Arxiv Preprint A412 6980, 2014.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in Neural Information Processing Systems, 12, 1999.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):13341333. 2016.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. Arxiv preprint Arxiv: 1509. Arxiv:1509.022 0099002991..... 2015.
10

---

Under review as a conference paper at ICLR 2023
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed coppeative-compettinee environments. Advances in Neural Nnformation Processing Systems, 30, 2017.
Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete sequential prediction of continuous actions for deep rl. Arxiv Preprint Arxiv: / 705.05035, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. Arxiv Preprint Arxiv:/312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533-333, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perfoperformnce deep learning library. Advances Neural Information Processing Systems, 32, 2019.
Jason Pazis and Michail G Lagoudakis. Binary action search for learning continuous- control policies. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 793-800, 2009.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. Arxiv Preprint Arxiv:1511.055 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. International Conference on Machine Learning, pp. 1889-1897. PMLR, 2015.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, pp. 387-395. PMLR, 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5981-5988, 2020.
Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action branching architectures for deep rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, IEEE, 2012.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine Learning, 83):279-292, 1992.
Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. Advances in Neural Information Processing Systems, 30, 2017.
11


Reviews:
[{"review_id": "QpTSkdKTI1K", "reviewer": "Reviewer_zaix", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Reinforcement Learning algorithms for continuous actions based on per-dimension discretization and a centralized critic network. The aim of the algorithms is to allow for sample-efficient discretized RL, without the curse of dimensionality it incurs. The work is inspired from multi-agent systems, in which a centralized critic (that observes everybody's action) is used to train the separate actors of the agents.\n\nThe proposed algorithms outperform strong baselines on challenging environments.", "summary_of_the_review": "Very interesting idea, but whose presentation is difficult to follow. The interesting idea really deserves to be published at a top conference. But the presentation really makes this paper difficult to follow (and thus not to accept). I'm currently recommending rejection, but an updated version of the paper with a clearer path towards QPC would allow me to recommend acceptance.", "strength_and_weaknesses": "Strengths:\n\n- The paper presents many novel ideas, propose algorithms that appear sound, and exhibits strong empirical evidence that the proposed algorithms work and outperform strong baselines. This is clearly a paper that teaches something to the readers and is interesting to read.\n\nWeaknesses:\n\n- The paper is quite difficult to read and to understand. It is possible to understand the contributions, but there are so many things in this paper (the stochastic algorithm for instance, while QPC is built on the deterministic algorithm) that understanding what is the contribution and what is the path towards it is difficult.\n\nFrom my understanding, SD2PC is only used to later build D3PC, and hence is part of the explanation, not one of the contributions. I would therefore maybe move its description in an appendix, or better present it as the general idea: from a state, we compute a per-action-dimension discrete action, that is then merged into a full action and used to train the critic. Then we move to D3PC in which the per-action-dimension actions are generated by taking the argmax over Q-Values.\n\nThe same applies to QPC, that seems to be the one contribution of this paper, and whose general idea is to mix two deterministic actors: one based on discrete actions, one on continuous actions. Both actors are trained in parallel from a centralized critic, and which actor we use for training the critic (and action execution) changes over time, to privilegiate the discrete one in early stages of learning, then progressively move to the continuous one.\n\nI think that mentioning (in the authors's word and with higher accuracy) what I explain above, in the introduction or even the abstract, would greatly help the readers understand what this paper is about, and navigate it. It would also explain from the beginning something that nows appears a bit too suddenly: the \"let's now use DDPG's critic network with our actor and some other continuous actor\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NgxqQxxr26", "reviewer": "Reviewer_hBbu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes several methods for addressing continuous control problems by using discrete-action methods, in such a manner that they scale gracefully with increasing action dimensionality. ", "summary_of_the_review": "**Minor Questions:**\n\n- In Par 1 of Sec 2, it is mentioned that previous approaches, e.g., Tavakoli et al. (2018) or Metz et al. (2017) are not compatible with experience replay or cannot deal with high-dimensional tasks. From the experiments of both these papers, I see that they both experiment with the Humanoid task (the highest dimensional task in your paper) and they both use experience replay. Can you comment in what specific context you mean they are incompatible?\n\n- What is the difference between Fig. 1(a) and 1(b)? From what I understand, 1(a) has no shared parameters, while 1(b) has all shared parameters except for the output linear layer, correct?\n\n\n**Minor:**\n\n- `\\citet` and `\\citep` are used mistakenly throughout the paper. E.g. \"*Since the seminal contribution of deterministic policy gradients Silver et al. (2014) in 2014* [...]\", here you need to display *(Silver et al., 2014)* by using `\\citep`.", "strength_and_weaknesses": "**Strengths:**\n- The paper focuses on an interesting problem. \n\n- The illustrations really help understand the ideas.\n\n- I think this paper would become a valuable resource as a one-stop coverage of all existing and potentially some novel algorithmic combinations for addressing continuous control problems through discretization (assuming all missing related works get incorporated appropriately and the missing discussions get added in great details).\n\n\n**Weaknesses:**\n- The argument for QL/DQN with Experience Replay not being compatible with factored action-space representations (discussed in Sec 3) is well known (especially in the multiagent RL literature). Yet, Tavakoli et al. (2018) have shown that this combination can still perform competitively with DDPG, even significantly outperforming it in the high-dimensional task of Gym's Humanoid. But unfortunately, despite citing said paper, relevant discussions of it (or the details of the approach) are missing in the paper. In fact, using a similar parameter-sharing scheme in the way of Fig 1(b) was used in the approach proposed in the said paper, and also they used a centralized state-value critic (as in dueling networks) to make the training more centralized. \n\n- How does the approach in Sec 3 differ from what was proposed in Tang & Agrawal (2020)? It's basically the result of the policy gradient theorem, used with a factored policy representation (using the independence of policies' assumption). \n\n- In fact, a COMA-style PG update would have lower variance without any bias wrt. the method proposed in Sec 3 (see [1]).\n\n- Important/critical related works are missing such as [2,3,4,1], and some other important related works have been included but very shallowly such as Metz et al. (2017), Tavakoli et al. (2018), Tang & Agrawal (2020).  \n\n- How does the approach of Sec 4 differ in principle from the SAC variations in Ref. [4]? They used distributions over two and three subactions per action-dimension, but the same update rule can be used to train with higher number of subactions per action dimension.\n\n- Comparison with the Amortized Q-learning algorithm of Ref. [3] is necessary in my view. Also, a detailed discussion of the difference of said approach wrt. those presented in this paper would be important.    \n\n- Comparison to a well-tuned agent from Tavakoli et al. (2018) and HGQN (r=1) from Ref. [2] would be necessary. Approach of Sec 5 is similar to Branching DQN by Tavakoli et al. (2018), with the difference that the target is a joint-action Q-estimator. Whether such a join-action Q-estimator would work better or not needs to be empirically evaluated. To compare, you'd have to use a similar architecture to that used in your paper, spend the same strategy to tune its hyperparameters to see whether there is a clear advantage.   \n\n- Claims such as \"*To the best of our knowledge, for the first time, our discrete RL methods can achieve improved performance\nover continuous RL methods in such challenging tasks.*\" clearly illustrate that the authors should closely read the suggested related works.  \n\n\n**References:**\n\n[1] Wu et al. (2018) \"Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines.\" ICLR. \n\n[2] Tavakoli et al. (2021) \"Learning to represent action values as a hypergraph on the action vertices.\" ICLR.\n\n[3] Van de Wiele et al. (2020) \"Q-Learning in enormous action spaces via amortized approximate maximization.\" arXiv.\n\n[4] Seyde et al. (2021) \"Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies.\" NeurIPS.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LONYLgcA3dd", "reviewer": "Reviewer_AHHq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes three algorithms for finding optimal policies in environment with high-dimensional continuous controls based on decomposing the policy into independent, but cooperating policies for each action dimension, discretizing uniformly each action, and finding its optimal policy efficiently. Empirical evaluation on popular benchmarks in OpenAI Gym show faster reward improvement than well established baseline algorithms for continuous-action RL, such as SAC, DDPG, and TD3.   ", "summary_of_the_review": "It appears that the three proposed algorithms have computational advantages with respect to established methods for continuous RL, such as SAC, DDPG, and TD3, so it is worth including them in the toolbox of RL practitioners. I could not find any deep insights about why these advantages are observed - intuitively, optimizing each action dimension independently could be quite suboptimal, and I wonder why this is not hampering these algorithms more. The presentation and citation style can be improved a bit. ", "strength_and_weaknesses": "The main strength of the paper is the computational effectiveness of the proposed algorithms on well-known benchmark problems. One weakness is the need to quantize the actions, possibly leading to suboptimality of the learned policy. The third algorithm, QPC, addresses this limitation, but the empirical results show that it is not necessarily the one that always achieves the highest asymptotic reward.\n\nMinor typos and inaccuracies:\nP.1: The terms \"limited actions\" and \"infinite actions\" are not precisely defined, did the authors mean discrete and continuous actions?\nP.2: \"algorithms with D2PC trains fast\" -> \"algorithms with D2PC train fast\"\nP.3: \"automatically turned\" -> \"automatically tuned\"?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "06WIFyoff3M", "reviewer": "Reviewer_yKnh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to discretize the continuous action space and model each dimension independently. Authors claim this idea is motivated by multi-agent RL and each agent corresponds to a dimension of the high-dimensional action space. Three variations of algorithms motivated by the idea are proposed that are SD2PC, D3PC and QPC. SD2PC is based on SAC, which is an actor-critic algorithm with policies modelled explicitly. D3PC is based on DQN which only models Q values therefore the policy is deterministic if one greedily selects the action with the highest Q value. QPC mixes continuous and discrete components, which leads to better performance.\nThe empirical evaluations on gym-locomotion control tasks show that discretized agents can be as good as methods that are directly operating on continuous space and the mixed agent can perform slightly better than both continuous and discrete variation of agents.", "summary_of_the_review": "I think the paper does have some potential. I guess the authors want to tell a story like Q-MIX that simple methods with seemingly strong assumptions work surprisingly well. But I believe the current quality of the paper didn't match the ICLR standard, especially the write up of the paper should be improved in future submission.", "strength_and_weaknesses": "Strength\n* Discretization and independent modelling allows discrete approaches to be applied continuous task without too much efforts. \n* The empirical results are somewhat surprising since a potential weakness of such independent modelling is just about whether it can be scaled up to high dimensionality.\n\nWeakness\n* The experiments didn't answer any concrete research questions. The paper proposed a new way of discrete action modelling but how does this compare to previous discretization/modelling methods? What if you still model action dimensions autoregressively or jointly? Are they going to be slow? Or maybe they can even perform worse (it will be surprising if it's indeed the case)? If the main claim is your discretized methods perform better than continuous agents, the experiment should provide more insights about why that's the case because it's not intuitive.\n* The presentation of this work is quite confusing. Authors claim the method is motivated by multi-agent RL, which is an unnecessary detour. As the authors mentioned, action discretization itself is not a novel idea. The problem is how we can model the joint distribution of the discretized dimensions. One advantage of discretization is we can easily model multiple modes of the action distribution while it also blows up the size of the action space. One way to avoid the blowup is to model action dimensions auto-regressively (Metz et al). This paper basically says that just assuming each dimension is independent also gives a decent performance.\n* This paper didn't clearly describe the relationships with prior works. It simply says\n> It is worth remarking that these methods either cannot exploit experience replay or cannot deal with high-dimensional continuous action spaces, rendering them typically less effective than actor-critic methods for continuous control\n\nThis is confusing because I'm not sure why discretization and action modelling is tied to replay buffers. Aren't these just some orthogonal design choices?\n\n* The write-up of the methodologies reads like a technical report that simply describes the development of the whole project. For example, there are a lot of sentences like:\n\n> We first consider using DQN to train a deterministic policy for each action dimension, given its simple structure and low computational overhead. Nevertheless, we found it is problematic to utilize DQN for each action dimension while using experience replay.\n\nIn the methodology section, readers expect a structured description of the final version of your algorithm. If other natural design choices are important for the story, it would be better to discuss them in the ablation.\n\n* Tons of low-level details are coupled with high-level ideas throughout the whole paper which is distracting.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-d7nyIts4b", "reviewer": "Reviewer_zjS6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose three new RL algorithms for tasks with continuous action space. The first two algorithms involve the core idea of discretizing the action space in a scalable manner where each action dimension is discretized independently and the action selection is independent as well. Such discretization allows the authors to convert the original problem with high-dimensional action space into a multi-agent RL problem with different discrete policy for each action dimension and a shared critic for all action dimensions. Although these  two algorithms are fast in the beginning of training, they can lead to worse final performance due to discretization. To address the sub-optimal final performance issue, the author propose the final algorithm (QPC) that use a combination of discrete policies and a continuous policy throughout the training. This final algorithm enjoys the benefits of RL with discrete actions while matching the final performance of prior continuous action space RL algorithms.", "summary_of_the_review": "While I do find the proposed methods to be conceptually interesting and that it could potentially be scaled up to higher dimensional problems, they do occur to me right now as complicated methods with little to no improvements over existing methods. As the current state of the paper, I would not recommend acceptance.\n\nIt is possible that the methods are able to perform much better than baselines in higher dimensional problems (right now the environment with the largest action space is only 17-dimensional), and showing such results could greatly strengthen the paper especially around the point that the discretization proposed in the paper is more scalable.", "strength_and_weaknesses": "*Strength*:\n\nThe paper is generally well-written and easy to follow. \n\n*Weaknesses*:\n\nI found the empirical results not strong enough to support the main claims of the paper (that their RL algorithms can match and improve over continuous actor-critic methods). First of all, I have doubts over the validity of the baseline performance reported in Figure 6 (see more details below) which makes it hard to judge how much benefits that discretizing actions could actually bring. In addition, the QPC was motivated to make its final performance  better by transitioning from discrete policies to a continuous policy; however, I did not find statistical significant evidence that this is true in Figure 6 (QPC works very similar to SD2PC, D3PC on all tasks).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject