Neural Discrete Reinforcement Learning
Authors: Yazhe Niu
Yuan Pu
Chuming Li
Zhenjie Yang
Hongsheng Li
Yu Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Reinforcement Learning
Representation Learning
Action Space

Abstract: Designing effective action spaces for complex environments is a fundamental and challenging problem in reinforcement learning (RL). 
Some recent works have revealed that naive RL algorithms utilizing well-designed handcrafted discrete action spaces can achieve promising results even when dealing with high-dimensional continuous or hybrid decision-making problems. However, elaborately designing such action spaces requires comprehensive domain knowledge.
In this paper, we systemically analyze the advantages of discretization for different action spaces and then propose a unified framework, Neural Discrete Reinforcement Learning (NDRL), to automatically learn how to effectively discretize almost arbitrary action spaces.
Specifically, we propose the Action Discretization Variational AutoEncoder (AD-VAE), an action representation learning method that can learn compact latent action spaces while maintain the essential properties of original environments, such as boundary actions and the relationship between different action dimensions. Moreover, we uncover a key issue that parallel optimization of the AD-VAE and online RL agents is often unstable. To address it, we further design several techniques to adapt RL agents to learned action representations, including latent action remapping and ensemble Q-learning. Quantitative experiments and visualization results demonstrate the efficiency and stability of our proposed framework for complex action spaces in various environments. 

Under review as a conference paper at ICLR 2023
NEURAL DISCRETE REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Designing effective action spaces for complex environments fundamental and challenging problem in reinforcement learning (RL). Some recent works have revealed that naive RL algorithms utilizing well-designed handcrafted dis- crete action spaces can achieve promising results even when dealing with high- hybrid decision-making problems. However, elabo-
dimensional continuous rately designing such action spaces requires comprehensive domain knowledge. In this paper, we systemically analyze the advantages of discretization for differ- ent action spaces and then propose a unified framework, Neural Discrete Rein- forcement Learning (NDRL), to automatically learn how to effectively discretize almost arbitrary action spaces. Specifically, we propose the Action Discretization Variational AutoEncoder (AD-VAE)
action representation learning method that can learn compact latent action spaces while maintain the essential properties of original environments, such as boundary actions and the relationship between different action dimensions. Moreover, we uncover a key issue that parallel opti- mization of the AD- VAE and online RI agents is often unstable. To address it, we further design several techniques to adapt RL agents to learned action represen- tations, including latent action remapping and ensemble Q-learning. Quantitative experiments and visualization results demonstrate the efficiency and stability of our proposed framework for complex action spaces in various environments.
INTRODUCTION
Recent advances in Reinforcement Learning have yielded many promising research achievements Vinyals et al.] (2019); |Benner et al. (209);; |Schrittiiesrr et al. (209)) However, the complexity of action spaces still prevents us from directly utilizing advanced RI. algorithms to real-world sce-
narios, such as high-dimensional continuous control in robot manipulation Lillicrap et al. (2016)
and structured hybrid action Decision-makinn
in strategy games Kanervisto et al.] (2022). Complex action spaces lead to extensive challenges in designs of policy optimization Xiong et al. (2018b),
efficiency of exploration Seyde al. (202Tb) and behaviour stability of learned agents Bester et al. (2019)
To handle these issues, some existing work first elaborately design particular reinforcement learning methods in original complex action spaces. Specifically, deterministic policy gradient methods Lil- licrap et al. (2016); Fujimoto al. (2018) are designed to handle continuous control problems. And Xiong et al. (2018)) |1 Fan et al. (2019)) propose some techniques to extract the relationship between different action dimensions, which is important in hybrid action spaces. However, these designs of- ten suffer from low exploration efficiency and unstable training due to the infinite action spaces and interference between different sub- actions Bester et al. (2019), respectively. Action space shaping Kanervisto et al. (2020) is another way to tackle these problems. Particularly, many RL applications Kanervisto et al. (022);
; Wei et al. (2022) design specific action discretization mechanisms to sim- plify the decision makinn spaces, leading to the promising performance improvement, but requires intensive investigations about the corresponding environments. Moreover, the combination of many manually discretized sub-action sub-actions will result in the exponential explosion of action numbers, which is incompatible with large action spaces. Recently, some works propose to learn abstract action representations to boost RL training. HyAR et al. (202T) designs a special training scheme with VAE Kingma & Welling (2014) to map the original hybrid action space to a continuous latent action space. Some other methods Dadashi al. (2022; Shafiullah et al. (2022) ; [Jian al. (2022) build prior sets of discrete actions to from expert demonstrations, and then deploy RI. agents on this fixed discrete action sets. To preserve the necessary attributes of environments, all the above discretiza-

---

Unified action discretization provides powerful and general approach to dealing with reinforcement learning in complex action spaces. is equivalent to split the entire pipeline into two parts: (I), representation learning and (2). decision-making. The former focus on intrinsic properties and data distributions of the action space, then transform various Under review as a conference paper at ICLR 2023
tion techniques require related domain knowledge to discard redundant information about actions, which means that they are unsuitable for different environments with arbitrary action spaces.
In this paper, we focus on how to learn a unified discrete action representations from scratch without any domain knowledge. Based on previous analysis and our investigations (as shown in Figure D. we summarize the following advantages of discretization for the complexity of the action space:
action spaces into standard discrete action sets, while the latter only needs to solve core decision- decision-making problems.
Effective discretization can improve sample efficiency by reducing the overhead in repeat- ing sub-optimal, useless, and semantically similar actions. RL agent can just explore and exploit the necessary subsets of the original action space during training.
Then, we introduce Neural Discrete Reinforcement Learning (NDRL) framework. Specifically, inspired by VQ-VAE van den Oord et al. (2017), we propose a action representation method called (AD-VAE) to learn latent discrete action space from
Action Discretization Variational the original environment, and conduct RL on the learned space utilizing any classical RL techniques about the discrete action. It is essential to capture the intrinsic properties of the original action space, which is beneficial to learn a compact latent action space while keeping necessary information of the original action space. Therefore, we design utilize graph neural network (Kipf & Welling 2016) and soft-argmax operation pperaaiio[Luvizon Luvizon et al. (2019) to improve the capability of AD-VA VAE for the relationships between different action dimensions and boundary action values. Furthermore, we find a core issue of parallel optimization of AD-VAE and RL agents: the online updates of AD-VAE may lead to semantic changes of latent actions (i.e. the non-stationary of decision spaces), resulting in severe data staleness and Q-value over-estimation. To solve this problem, we introduce action remapping and ensemble Q-learning. Concretely, we apply the classic DQN as an instance to our framework, named Action Discretization Q-learning (ADQ), which can be deployed for most complex action spaces. Compared with pioneer works (Chandak et al.) 2019a Zhou al. 2020 Dadashi al. 2022), to our best knowledge, our proposed framework is the first online RL paradigm capable of employing in discrete action spaces learned from different continuous and hybrid decision-making environments.
conditonnd action encoder and decoder, and
To demonstrate the efficiency and stability of our NDRL framework and AD- DDEE method, we eval- uate it the classic continuous control benchmark MuJoCo Todorov et al. (012)) showing that ADQ can achieve excellent performance operating in high-dimensional continuous space even with small number of actions. To evaluate the generality, we test it on the hybrid action environments Gym Hybrid (hoomshirtz] (202I), HardMove from HyAR and GoBigger Zhang (02I). The results show that ADQ outperforms current sfate-ofttheaar hybrid action algorithms in both sample effi- ciency and final performance. Besides, we also conduct a series of ablation study experiments and interpret more details about NDRL by visualization on the latent space.
2 RELATED WORK
Discretization and continuity are like the relationship between 0 and I in the binary world. All things. including time and space, continuous, but for the convenience of cognition, we will discretize all of them. Only then can we have measures, such as the concepts of hours, minutes and meters. In RL, learning directly on high-dimensional continuous action space may present difficulties in exploration due to the uncountable set of actions. In addition, (Bjorck) et al. 202T) argues that the nonlinear function saturation caused by unstable network parameter- ization will cause the well-known high variance problem. The most straightforward solution is discretization, however, this usually suffers from the curse of dimensionality. To alleviate this prob- lem, many assumptions about the action space have been proposed. For example, (Tang & Agrawal, 2020) verifies the feasibility of discretizing the action space in on-policy optimization by utilizing the factorized distribution across action dimensions. In (Dadashi et al.) (022), the authors proposed to circumvent the curse of dimensionality problem by learning a set of plausible discrete actions from expert demonstrations. We argue that this algorithm can naturally be seen as a special case of
Action Discretization
2

---

Under review as a conference paper at ICLR 2023
our NDRL framework. (Seyde et al. 202 explored the effect of extreme actions continuous action control, which also inspired the design of AD-VAE for maintaining boundary actions.
Hybrid Action Space Many real-world problems may have hybrid action spaces. For example, in the GoBigger game, we need to select an action type first, and then give its corresponding con- tinuous control arguments. The simplest idea is map it onto unified homogeneous action space, like discretizing continuous actions or making discrete actions continuous, however this may cre- ate scalability issues with the curse of dimensionality. Going step further, recent work proposes various hand-designed network structures to learn directly on the original hybrid action space. For instance, Parameterized Action DDPG (Hausknecht & Stone 2016) uses a modified DDPG actor- critic structure and HPPO (Fan et al. 2019a) proposes different types of heads for different action types. PDQN (Xiong et al.) 20188) and MPDQN (Bester et al. 2019) use hybrid structure of DQN and DDPG, explicitly modeling the dependencies between continuous and discrete sub-actions.
Action Representation Learning The concept of the latent space is widely used in various ele- ments reinforcement learning, such as latent state and dynamics. But in action space, (Chandak) et al.] 2019b) proposes action representation learning in large action space, leveraging the structure in the space of actions and showing its importance for enhancing generalization over large action sets in real-world large-scale applications. (Li et al. (021)) propose Hybrid Action Representation (HyAR) to learn a compact and decodable latent representation space for the original hybrid action space. HyAR constructs the latent space and embeds the dependence between discrete action and continuous arguments via an embedding table and conditional Variational Auto-Encoder (VAE).
BACKGROUND
Markov Decision Process In reinforcement learning, model decision- problem Markov Decision Process (MDP) M=(S, P. R, 7, Po). where S and A represent the state space and the action space, P is the transition function: S x A + S, R is the expected reward fune- tion: S x A + R. y E [0, is the discounted factor, and po is the initial state distribution. The objective of RL is to learn a policy ((x) Ro P.R IDEoo 7're), where the expectation taken with respect to the trajectory distri- bution induced by and environment dynamics.
S A to maximize the expected discounted return
Hybrid Action Space In decision- maainn problems, at each time t, the agent receives a state and carries out an action, which can be divided into 3 types: discrete, continuous and hybrid action. Here we give a general formalization. A hybrid action a contains N decision nodes (sub-actions). At each decision node i and time step the agent needs to give proto-action az,; with two attributes including type and range of values; the type of value at indicates whether it is continuous or discrete, and the range of value dt,i.range indicates its corresponding executable action set. Thus, we use an ordered tuple like = (at.1, 02.2, .... (t.v) to describe these basic nodes. Furthermore, the relations between decision nodes can be defined by adjacency matrix A,.: in graph theory. called action relation matrix. The value of the elements in the matrix is {0. 1}. If the element Ar,, in row i and column j is equal to 1, it means that there is directed edge from decision node i to j. If equal O, there is no dependency between them. In many real-world problems, the dependencies between the proto-action always are invariant, that is, the action relation matrix is independent of l. Generally, hybrid action space can be defined as a tuple:
A = (far,i.type Oannee E [1,..., NJ},
(1)
The Parameterized Action Space defined in (Masson et al. 2016) is special instance of our defi- nition, specifically, which is equivalent to action a containing 2 decision nodes and dguu,tye = 0, ar,,.ranee = K, Qt 1.tpe 1, dt.1.range X. There is only a directed edge from decision node to decision node 2. formally, the adjacency matrix A, is:
( 0 ! ))
Action Transformed MDP Here we augment a MDP with action transformation, which can be defined as M=(S, P, R, 7,, Po. T), where T denotes the transformation operator on action space, such as action discretization. Denote the transformed action as k. we can describe T aa:
T k: = T(s,a) a = T=,
The other elements are consistent with the original definition of MDP. Through this transformation, we can learn an RL agent more efficiently on this new, often reduced, latent action space.
3

---

Under review as a conference paper at ICLR 2023
Revrr Myj w18
MaieZurrde (11) v.777
MunaDraffted (0 1m
1.0
DON expert cluster
200
Noop Action Ratio
0.8
Retum
-100
0.2
DGN experi cluster
-20
0.0
0M
0.25M
0.5M
0.75M
1M 1.55M 1.5M
0M
0.26M
0.5M
1.25M 1.5M
Env Steps
Env Steps
Figure 1: Top: Visualization analysis about different action spaces in the same LunarLander (Brock- man et al. (016)) environment. ExperrCluster is discrete action obtained by clustering TD3 expert data). HandCrafted is obtained by the threshold of spaceship engine. Vertical engine only enables when bigger than O; And if is smaller than -0.5, the left booster will fire, and if is bigger than (left) Episode return of three algorithms on LunarLander: TD3 (original continuous action space), DQN expert cluster, ADQ (discrete action learned by AAE from scratch): ; (right) The ratio of some semantically same actions (no operation), more no-op actions means greater redundancy during training, i.e., lower is better.
0.5, the right booster will fire. Bottom:
Vector Quantised Variational AutoEncoders Motivated by vector quantization (VQ) and Vari- ational AutoEncoder, VQ-VAE van den Oord et al. (2017)) is designed to learn a discrete latent representations to represent the original data distribution (e.g., text, image) in unsupervised man- ner. VQ-VAE mainly comprises of an encoder cg. an decoder dg. and a learnable code table Ve- K-
The learnable code table maintains a set of embeddings {e
Firstly, the encoder takes the data input, and outputs embedding vector f(x). Then using the embedding vector = to query the nearest (usually in Euclidean distance) code vector ad the code table and outputs an latent index simultaneously. Thirdly, the decoder uses the code vector d as its input to produce reconstructions é. The whole objective is to minimize the following loss function
L C Lal0 + ||sg (cs -
201 [ee(z)) - sg[2"]11 -g[22"11
(4)
d = where k = argmin, ||a egl12
(5)
Where sg(.) is the stop gradient function. The first term of loss function is to reconstruct error in certain distance metric, the second item and the third term embedding loss and commitment loss respectively. Please refer to van den Oord et al. (2017) for more details.
NEURAL DISCRETE REINFORCEMENT LEARNING
4.1 MOTIVATION
First, to motivate our proposed framework, we further discuss the advantages of reinforcement learn- ing in discrete action spaces from the following 3 aspects: unity, efficiency and stability.

---

Under review as a conference paper at ICLR 2023
4.1.1 UNITY
In practice, researchers need to first transform the original MDP form. Due to the different types of target action spaces, it is inevitable to utilize different techniques for the corresponding action spaces, which brings non-negligible learning and tuning costs beyond core RL optimization. But when we dive deeper into this problem, we find there are obvious redundancies in most complex action space, e.g., only few discrete actions/samples can perform well in multi-dimensional continuous control [Syyde et al. (2021b); Hubert et al. (2021). In Figure [(a] we also illustrate the entropy of different action spaces show the effectiveness of discretization. Therefore, learning compact action representations instead of repeating some dirty work in the action space is natural and powerful choice. Moreover, the decoupling of action representation learning and RL allows researchers to concentrate on only one of the topics.
problem into a standard
4.1.2 EFFICIENCY
Furthermore, we verify the efficiency of action discretization in online RL training. As shown in Figure [[a] we find the well-designed discrete action space shows lower entropy and more compact representation. We also conduct a simple experiment to test the performance and the ratio of useless action like excessive no operation in [[b)] Based on these observations, we design AD-VAE to automatically learn the latent discrete action space from the original action space. On one hand, this model can learn approximate the necessary parts and ignore meaningless parts of the original space, which is beneficial to improve exploration efficiency. On the other hand, some works Jiang ctt al. (2022) show that the marginal distribution of each action dimension is often multi-modal, thus using a discrete categorical distribution is more suitable than the simple regression in TD3.
4.1.3 STABILITY
However, previous methods Dadashi et al. (2022); Jiang al. (002) only succeed in deploying action discretization on imitation learning and offline RL settings, which means that it needs to first learn latent space and then apply decision-making algorithms on the fixed discrete action space. We tried to directly utilize AAE on online settings but obtain unstable episode returns. Compared to training on the frozen discrete action space, we find some abnormal indicators including unusually high Q-values and obvious fluctuations in gradient scale and variance. Due to parallel optimization about ADE and RL agents, it would be a hazardous non-stationary MDP if the latent action space changes too much. To figure out this problem, we propose action remapping and ensemble Q-learning. Together with AD-VAE, these techniques form the entire NDRL framework.
4.2
NDRL FRAMEWORK
4.2.1
OVERVIEW
Motivated by the above analysis, we propose framework that combines online RL training with learnable action discretization on complex action spaces, named Neural Discrete Reinforcement Learning (NDRL). At high-level, this framework is "meta-algorithm" that splits decision-making in arbitrary complex action spaces into two parts: the part of representation learning mapping the original action space new discrete action space and the reinforcement learning part built on learned discrete representations. Based on this design, any reinforcement learning designed for discrete space (e.g. DQN, PPO) can be potentially applied different complex action spaces. The overview of NDRL framework is described Figure 1, and we will introduce the data collecting phase and network training phase respectively:
This phase describes how to collect data used in the training of AD-VAB and RL agents. Given the current state s;, RL agents first select corresponding discrete latent action k then utilizing AD-VAE decoder to transform back the original action space. Moreover, NDRL also deploys some extra randomization operations (e.g. epsilon greedy in DQN) on the decoded action to maintain sufficient exploration. The final action a, interacts with the environment and it returns reward r1 and next state $1+1. All necessary data will be packed into transition and put into buffers.
Collecting Phase:
The training phase is to execute two training pipelines with different data buffers in parallel. (1) For action representation learning, NDRL follows the main training scheme of VQ- VAE to reconstruct primitive actions with state conditions. (2) For RI. training, first remaps the
Training Phase:
5

---

Under review as a conference paper at ICLR 2023
Neural Diserete Reinforcement Learning Framework
Fty 8543
Infercace
AD-VAE Deceder
#t
Agent
Collertag Plane
Diernee
R
AB AD-VAE Cade Tabls
Hylell
wow co cocee co Action Representation Learning
AD-VAE Encodler
Yeuking
Code Tible
AD-VAR Decoder
åg
Thaining
Action Remapping
Normal RL Training
(4t, Hee 1
AD-VAF Encoder
Code Table
Lou Panto Rest
Agent
Discretization (AD-VAE)
RL Agent
Overview of NDRL framework. Collect Phase: Given the current state 8t. the RL agent gives latent discrete action k;, and utilizes AD-VAE code table and decoder to obtain the raw action then interacts with the environment. Train Phase: First remap the sampled transition (st, Ft, St+1) with latest AD- VAE encoder and table kts then deploy the normal RL training. Note the VA- is also trained with state condition in parallel. Black round means continuous action space, while grey round means discrete, the whole tree structure means hybrid action.
Figure 2:
original action in the sampled transition with AD-VAE encoder obtain the latest latent action kr. then conduct normal policy optimization on transformed data.
Besides. for the better initialization of off- off-policy RL algorithms, we also design a pretrain stage the beginning of the entire algorithm. The full pseudo-code of NDRL is provided in Algorithm 1. If there are some expert demonstrations, promising set of discrete action candidates can be learned from it. Otherwise, we can collect data with random policy to train the AD-VAE and learn some basic properties of the original action space for subsequent parallel learning. After pretrain stage, data collecting and two parts of training can be executed asynchronously. so the computational cost of NDRL can be easily optimized and show the same efficiency as other methods.
4.2.2 AD-VAE
In this section, first analyze the problem of directly using the original VQ- VAE, then introduce the specific design about our proposed Action Discretization Variational AutoEncoders (AD-VAE),
Modeling Intrinsic Properties of Action Spaces online RL training, there are both necessary and redundant subsets of the original action space. For instance, (Scyde et al. (022lb) pointed out that in some continuous action tasks, the optimal action may be at some extreme boundary values (e.g. -1 and 1), which is common phenomenon in several physical simulation environments. Also, BBstter et al. (2019) revealed that effective combinations between different action parts is significant for the optimization in hybrid action space. Therefore, it is wise to pay more attention to those actions that are more beneficial the optimal policy, and ignore some useless even harmful actions. Otherwise, trivial action reconstruction with the same weights and distance metrics can only obtain some over-smooth actions. Besides, we also can take advantages of the value function to focus on those actions with higher future return, saving the cost of agent exploration and exploitation.
Information Completion in AD-VAE We first introduce the technique of information completion in AD- VAE. In some complex environments, the set of optimal actions could be large and vary in different training stages. If we reconstruct actions without state information, AD-VAE must maintain large discrete action sets and RL agents needs learn decisions on great number of actions, which can easily make it overwhelming and cause training instability. On the contrary, properly use of ctate-conditioned input in both encoder and decoder of AD-VAE can increase the representational

---

Under review as a conference paper at ICLR 2023
power and diversity, and reduce the burden of action reconstruction and RL training. Moreover, the relationship between different action parts, i.e. the action relation matrix mentioned in Section 3 can assist AD-VAE to learn more efficiently and reasonably, so we can represent this information as the connections of nodes with a graph neural network. Besides, AD-VAE not designed to learn the entire action space but to properly model the subsets required by current RI optimization, thus we customize sampling method and training scheme for AD-VAE, including sampling a mixture of stale data in replay buffer and latest collected data as training mini-batch, validating the reconstruction error of actions to indicate the update frequency.
Continuous Action Regression in AD-VAE Another important intrinsic property of original ac- tion spaces is some special action values, such as the extreme actions mentioned in [Seyde et al. (021b) or thresholds of engine dynamics in LunarLander. It is critical for action representation networks to restore these continuous value accurately. Therefore, in AD-VAE, we adopt the soft- argmax operation to automatically reconstruct these special actions. Assuming the range of original action is Amin, A. and it is divided into N bins average, the predicted action is:
N
à
8j *
(8:), 83 Amin + * mAmaa - Amin)/N
(6)
1=
When evaluation, we directly output the corresponding support value if the probability of the support greater than threshold (e.g. 0.9). In some environments like Hopper/Halfchettah, we find this design can help a simple DQN agent achieve a comparable performance with TD3.
Other parts of VAE follow the design of VQ-VAE, the whole training procedure is to minimize the loss function described in Equation
4.2.3 ADAPTING RL TO LATENT ACTION SPACES
In this section, we continue analyze why and how to adapt online RL to latent discrete action spaces, and then illustrate an instance our NDRL framework on the value-based RL algorithm DQN, Action Discretization Q-learning (ADQ).
Semantic Inconsistency In online RL, Double DQN (van Hasselt et al. 2016) pointed out that Q-value over-estimation problems caused by the function approximation eπтоr and the max operator in the bootstrap target often lead performance deterioration. Furthermore, this problem may be exacerbated in NDRL. On one hand, latent action stored in replay buffer will be stale and biased due to updates of AD-VAE. On the other hand, since AD-VAE is dynamically and simultaneously up- dated together with RL agents, for a particular latent action, the corresponding action in the original action space could often change, which is more likely lead to the overestimation of Q-value.
Latent Action Remapping Similar the reanalyze operation in MuZero [Schrittwieser et al. (2019), we design a latent action remapping operation to solve the problem of stale data. In the collected mini-batch {st, ae. kelld re, $141,, the latent action is determined by the old version of AD-VAE. When updating RL agents, we remap the original action to the corresponding latent ac- tion via the latest action encoder eg: k2- = (st, at), and then executes RL training on the remapped samples {st, sg[kpee)
Tt, St41} (sg means the stop gradient operation).
Ensemble Q-learning To further alleviate the Semantic Inconsistency problem, inspired by pre- vious work Anschel et al. (2017); An et al. (202I). we propose an ensemble Q-learning method for more stable Q-value updates for latent action space, reducing the uncertainty of approximation error and over-estimation, which greatly improves the stability of parallel optimization of AD-VAL and RL agents. Specifically, we utilize shared state encoder and N ensemble Q-value heads, i.e., the penultimate layer of the Q network is connected to N linear layers and outputs N Q-value, then we adjust the update equation of Double DQN as follows:
L, [Q(st aei 0x) Ira +
0z)]
1
at+1 ar arggaar
(8)
N
Where La means the loss function of i-th Q head. Detailed experiments can be found in Section 5.3
7

---

Under review as a conference paper at ICLR 2023
Hopparv0
Humanddood
Movergu
Golleger
Figure 3: Training curves of ADQ against other baseline algorithms in environments with complex action spaces. Top: In four continuous action environments of MuJoCo, ADQ significantly outper- forms DQN with manually discretized action space, and is basically comparable to the classic TD3 algorithm. Bottom: In four hybrid action environments in Gym-Hybrid, HardMove and GoBigger, ADQ outperforms the baseline MPDQN and HPPO in both performance and stability. Curves and shadings denote the mean and standard deviation over seeds.
030
300
-000 -0.3 M0 675 130 ab -05 d dde eis sio 6i5 109
Figure 4: Visualization of the latent action space of LunarLander games. Details is in 5.2
5 EXPERIMENTS
For the evaluation of our NDRL framework, ask and answer the following questions: 1) Is it efficient and stable to employ online RL training on discretization action spaces for different decision problems, especially in high-dimensional continuous and hybrid action spaces? (Section [.T]: 2) How do we interpret the training of the latent action space? Can we further verify some observations mentioned in introduction parts? (Section 5.2): 3) How do various designs improve the NDRL framework, including AD-VAE and other RL adaption techniques? (Section 5.3)
5.1
MAIN RESULTS
In this section, we investigate the performance and efficiency of NDRL in various continuous and hy- brid action environments against previous algorithms designed specifically for these action spaces. Firstly, we evaluate our methods on MuJoCo, a classic continuous control benchmark. including two high-dimensional continuous domains (Ant and Humanoid with 8 and 17 dimensions respec- tively). Note we also add a few redundant dimensions in original action spaces. We set up two comparison groups for ADQ, one is the popular continuous action space algorithm TD3, and the other is naive DQN deployed in the manually discretized action space, i.c., equally dividing the original continuous action into bins at each dimension and using their Cartesian product to obtain handcrafted discrete actions. In Figure 3 ADQ can acquire comparable results to TD3 and show a obvious improvement over naive DQN in all four domains on the top. Secondly, we leverage Gym- Hybrid, HardMove and GoBigger to verify the effectiveness of ADQ more complex hybrid action spaces. This requires to deal with the relationship between different actions parts (e.g., the value of action arguments depend on the choice of action type) and more complex environment dynamics (GoBigger). At the bottom of Figure 3, we compare ADQ with two types of hybrid action space al- gorithms, MPDQN and HPPO, suggesting that ADQ can automatically learn the intrinsic properties of discrete action type and continuous action arguments, and performs excellent performance and solid stability. All the detailed settings shown in Appendix ?? and ?? respectively.
8

---

Under review as a conference paper at ICLR 2023
HalfCheetah-v3
HardMove-vO-n10
10000
2.0
8020
1.5
1.0
Retur
8000
Return
0.5
4020
0.0
2020
-0.5
ADG NJ PETTAS
AC0 monim
-1.0
0.5M
0M
1.514
Env Steps
Env Steps
Figure 5: Ablating results of ADQ in two types of environments (continuous and hybrid) over 5 seeds. When remove any one of the proposed techniques, the performance of ADQ will drop significantly, which verifies the effectiveness of our proposed techniques. ADQ w/o AR is especially important for improving ADQ performance in environments such as HalfCheetah-v3.
5.2 VISUALIZATION ANALYSIS OF LATENT ACTION SPACE
Furthermore, we demonstrate the learned latent action representation in Lunarl.ander environment for 2-dimensional continuous control. Figure first shows the status of the spaceship when it is about to land (left), i.e., launching both horizontal and vertical engines to control speed and position, then uniformly samples points in original continuous space and transforms them with AD-VAE encoder find their nearest discrete indexes (middle). Also, directly send the corresponding embeddings in code table to decoder to acquire their counterparts in the raw space (right). We can observe that the learned latent space is similar the intrinsic mechanisms of this environment, highlighted by red line: only using one discrete action to represent less important actions in this state like no-op, mapping several different actions the bottom right corner.
first
5.3 ABLATION STUDIES
We also empirically evaluate the specific impacts of our proposed AD-VAE and RL adaption tech- niques on two example environments respectively: HalfCheetah--3 (continuous), and HardMove- v0-n10 (hybrid). The ablation results are shown in Figure Concretely, we have the following five ablation variants in total, and their brief descriptions are as follows:
ADQ w/o remapping: ADQ variant that does not remap latent actions during RL training.
ADQ w/o ensemble: A variant of ADQ that doesn't 't utilizing the Ensemble L-Lernnig technique.
ADQ w/o warmup: ADQ variant that starts training without any warmup pretraining.
ADQ w/o AR: ADQ variant with traditional VQ-VAE reconstruction head.
ADQ w/o AR + hand-crafted action: Built on ADQ w/o AR, this variant adds manually selected boundary actions (i.c. the Bernoulli extreme actions) to latent discrete action spaces for RL training.
Figure 5 show that when we remove either of the proposed techniques, the performance of ADQ drops significantly in both two environments, verifying the effectiveness of our proposed techniques. Due to the semantic inconsistency problem, the ADQ w/o ensemble agent suffers from severe over- estimation issues and finally show poor performance. The ADQ w/o remapping agent meets the same problem, but in some cases the over-estimation problem can be partially alleviated by Ensem- ble Q-learning technique. The ADQ w/o warmup agent shows much slow learning progress due lack of good starting points. Note that in environments with boundary optimal actions such as HalfCheetah-v3. ADQ w/o AR is especially important for improving ADQ performance, even het- ter than the variant using extra hand-crafted actions. Other ablation results like pretraining on expert demonstrations and the sensity of paramerrs can also be found in Appendix.
CONCLUSIONS AND LIMITATIONS
Starting from comprehensive analysis for action discretization, we introduce general and efficient paradigm named Neural Discrete Reinforcement Learning, including our proposed AD-VAE and RL adaption techniques. We empirically evaluate the efficiency and stability of our framework. Although our method achieve superior performance in different benchmark environments, there still some challenging action spaces multi-agent games, such variable-length actions in episodes. Besides, combining latent discrete actions with MCTS is also a valuable attempt. We will continue pursue ultimate solution for action space shaping in future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based of- fline reinforcement learning with diversified q-ensemble. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Ad- vances Neural Information Processing Systems 34: Annual Conference on Neural In- formation Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp- 7436 7447, 2021. URL https:: newsineurips. wttpppppppppp//eepeepppp Abstrc-bbsttttt..... html
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqu: Variance reduction and stabilization for deep reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 176-185. PMLR, 2017. URL /tttt::/ppooceeeeeeeeeetoo
mlr. www.ll..eesss/lll 1ttpss//777/mmsshe117....... html
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. CoRR, abs/1912.06680, 2019. URL http://arxiv http/aawwi..ggaaaaa/ wttp://aaag/....////./
Craig J. Bester, Steven D. James, and George Dimitri Konidaris. Multi q-networks for deep reinforcement learning with parameterised action spaces. CoRR, abs/1905.04388 2019. URL
htttp://arxiv.orge abs/ 1905.19..00 04388
Johan Bjorck, Carla P Gomes, and Kilian Q Weinberger. Is high variance unavoidable in rl? case study in continuous control. arXiv preprint arXiv:21 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Y. Chandak, G. Theocharous, J. Kostas, S. M. Jordan, and P. S. Thomas. Learning action represen- tations for reinforcement learning. In ICML, volume 97, pp. 941-950, 2019a.
Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip Thomas. Learning action representations for reinforcement learning, In International conference on machine learn- ing, pp. 941-950. PMLR, 2019b.
Robert Dadashi, Léonard Hussenot, Damien Vincent, Sertan Girgin, Anton Raichuk, Matthieu Geist, and Olivier Pietquin. Continuous control with action quantization from demonstrations. In Ka- malika Chaudhuri, Stefanie Jegelka. Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 4537-4557. PMLR. 17-23 Jul 2022. URL https: /tpp:c//proceedii ngs . ml wttpgss.w..........e.
da/daaashiia22... htm
Z. Fan, Su, W. Zhang, and Y. Yu. Hybrid actor-critic reinforcement learning in parameterized action space. In IJCAI, pp. 2279-2285, 2019a.
Zhou Fan, Rui Su, Weinan Zhang, and Yong Yu. Hybrid actor-critic reinforcement learning in parameterized action space. In Sarit Kraus (ed.), Proceedings of the Twenty- Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 2279 2285. ijcai.org, 2019b. doi: 224966/jjcai22019333 URL htttss://oi....o 24963/1jca1 20199/9/311221111
S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In /CML, volume 80, pp. 1592-1591, 2018.
M. Hausknecht and P. Stone. Deep reinforcement learning in parameterized action space. ICLR, 2016.
10

---

Under review as a conference paper at ICLR 2023
Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Re- search, pp. 4476 4486. PMLR, 2021. URL
ngs. ml ptw.s//es39
bubert2la.her htm
Zhengyao Jiang. Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefen- stette. and Yuandong Tian. Efficient planning in a compact latent action space. CoRR, doi: 10448500/rXi.2200..0222
abs/2208. 2022.
URL https: /oii. org/1C.
48550/arXiv 2208 10291
Anssi Kanervisto, Christian Scheller, and Ville Hautamäki. Action space shaping in deep rein- forcement learning. In IEFE Conference Games, CoG 2020, Osaka, Japan, August 24- 27, 2020, pp. 479-484668888 486. IEEE, 2020. doi: 10. 00.1099CC044335522000222121 //doi. //doi.org//i 10. 11119//044335555 2020. 9231687
URL https:
Anssi Kanervisto, Stephanie Milani, Karolis Ramanauskas, Nicholay Topin, Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, Wei Yang, Weijun Hong, Zhongyue Huang, Haicheng Chen, Guangjun Zeng, Yue Lin, Vincent Micheli, Eloi Alonso, François Fleuret, Alexander Nikulin, Yury Belousov, Oleg Svidchenko, and Aleksei Shpilman. Minerl diamond 2021 com- petition: Overview, results, and lessons learned. CoRR, abs/2202.10583, 2022. URL https: //axiiv.rg/abb..
10583
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL
org/
abs 0oss/1312. 023.1..60 6114
Thomas N Kipf and Max Welling. Sumi-sueervised classification with graph convolutional net- works. arXiv preprint arXiv.1609.02907, 2016.
Boyan Li, Hongyao Tang, Yan Zheng, Jianye Hao, Pengyi Li, Zhen Wang, Zhaopeng Meng, and Li Wang. Hyar: Addressing discrete-contimuuus action reinforcement learning via hybrid action representation. arXiv preprint 1tt.::220990549900. 2021.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. (RLhttp: // www.iii.rroo abs/ 1509 02971
Diogo C. Luvizon, Hedi Tabia, and David Picard. Human pose regression by combining indirect part detection and contextual information. Comput. Graph., 85:15 22, 2019. doi: 10.1016/jjcag 2019.09.002. URLhttps: http://oooooo///////////o
cag 2019. 09.0000.
W. Masson, P. Ranchod, and G. D. Konidaris. Reinforcement learning with parameterized actions. In AAAI, pp. 1934-1940, 2016.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel. Timothy P. Lillicrap, and David Silver. Mastering atari, go. chess and shogi by planning with a learned model. CoRR, abs/1911.08265, 2019, URL http://arxiv www.aaiwww..i.iaaaaoooaaaooooore abs/ 1911. 08265
Tim Seyde, Igor Gilitschenski, Wilko Schwarting, Bartolomeo Stellato, Martin Riedmiller, Markus Wulfmeier, and Daniela Rus. Is bang-bang control all you need? solving continuous control with bernoulli policies. Advances in Neural Information Processing Systems, 14:27209-22222) 2021a. a.
Tim Seyde, Igor Gilitschenski, Wilko Schwarting, Bartolomeo Stellato, Martin A. Ried- miller, Markus Wulfmeier, and Daniela Rus. Is bang-bang control all you need? solv- ing continuous control with bernoulli policies. In Marc" Aurelio Ranzato, Alina Beygelz- imer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances
11

---

Under review as a conference paper at ICLR 2023
in Neural Information Processing Systems 34: Annual Conference Neural Informa-
tion Processing Systems 2021, NewrlPS 2021, December 6-14, 2021, virtual, pp. 27209- 27221, 2021b. URL https: htttpps://prrcceeddnng hww.:eeeedd.in.................. cC htt/paper/222111 /hash/
e4 #46bbe1 F0c0oe0000000000c0oooo (atpedd5dd222000beee 92cb0eb- Abst
Nur Muhammad (Mahi) Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Be-
havior transformers: Cloning k modes with one stone. CoRR, abs/2206. 1 11251, 2022. doi:
10.485500/rXivv2206. 11251. URL.tttps://doi.rrgggp wttt:///......//........o 2206. 11251
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization.
In Proceedings of the aaai conference artificial intelligence, volume 34, pp. 5981-5988, 2020.
thomashirtz.
2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033.
IEEE, 2012. doi: 10.1 2ttp:/RROS...22..63388
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete repre- sentation learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 30: Annual Conference on Neural Infor- mation Processing Systems 2017. December 4.9, 2017, Long Beach, CA, USA, pp- 6306 6315, 2017. URL 7.590a17.e2a00000e2e333333233.29 wbt. html
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Dale Schuurmans and Michael P. Wellman (eds.), Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pp. 2094 2100. AAAI Press, 2016. URL http::/www. aaai. wwwp:osgg/oooooooooooooooo wwwpoossinnnnneeeeeeeeeeeeeeeeee php/AAAI/ AAAI/ heAT1//paper/vieww/ 12389
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Çaglar Gülçehre. Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps. and David Silver. Grandmaster level in starcraft II using multi-agent reinforce- ment learning. Nat., 575(7782) 2019. doi: 10.1038//41588-001 1724-2. URL
/ttt:::/doo..rrr 01tp://000000/1111011011--111110 1724--
Hua Wei, Jingxiao Chen, Xiyang Ji, Hongyang Qin, Minwen Deng, Sigin Li, Liang Wang, Weinan Zhang, Yong Yu, Lin Liu, Lanxiao Huang, Deheng Ye, Qiang Fu, and Wei Yang. Honor of kings arena: an environment for generalization in competitive reinforcement learning. In P Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2022.
Xiong, Q. Wang, Z. Yang, P. Sun, L. Han, Y. Zheng, H. Fu, Zhang, Liu, and H. Liu. Parametrized deep n-nttworks learning: Reinforcement learning with discrete-continuous hybrid action space. CoRR, 110.091,,,,,94, 2018a
Jiechao Xiong, Qing Wang, Zhuoran Yang, Peng Sun, Lei Han, Yang Zheng, Haobo Fu, Tong Zhang, Ji Liu, and Han Liu. Parametrized deep g-networks learning: Reinforcement learning with discrete-conniimuuus hybrid action space. CoRR, abs/1810006394, 2018b. URL http://
arxiv. www.v/.org//bb/1110... 10tp://1110000001111 06394
Ming Zhang. Gobigger: A scalable platform for cooperative-competiiiee multi-agent reinforcement learning. httt::///g...aao..ommm///ddadad/
ctt
2021.
W. Zhou, S. Bajracharya, and D. Held. PLAS: latent action space for offline reinforcement learning.
CoRR, abs/2011.00211 2020.
12


Reviews:
[{"review_id": "xTfbZ4Jm5i", "reviewer": "Reviewer_qLKy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a framework for the adaptive discretization of continuous-action spaces in an online manner, by employing a two-phase learning structure: (1) latent action space learning; (2) RL on the learned action space. ", "summary_of_the_review": "See my comments above.\n\nOverall, the idea is quite appealing. But to see its benefits, we'd need more experiments. ", "strength_and_weaknesses": "**Strengths:**\n\n- The target problem is very important: To have a unified approach for dealing with continuous-action problems by adaptively discretizing them and applying discrete-action RL methods and importantly in an online manner (not from offline or expert data).\n\n- The approach is reasonable but requires (somewhat justifiably) a few patches (e.g. ensemble learning to deal with overestimation over a non-stationary action space) \n\n\n**Weaknesses:**\n\n- The paper is somewhat poorly written with an unclear flow at times and missing a good positioning/elaboration by connecting to many related works to better. \n\n- The size of the latent action spaces is limited to K, which is a priori determined. This is not a major issue but nonetheless could introduce statistical bias in performance. In fact, this bias could be potentially stronger than naive discretization (equidistance discretization). In naive discretization, the number of subactions per action dimension is fixed (say 5 subactions per dimension). Therefore, the size of the effective action space ends up increasing with increasing action dimensionality (i.e. size would be $5^N$ where $N$ is # of action dimensions). But the NDRL framework assumes a fixed effective action space size with $K$ latent actions, independent of the number of action dimensions. This can be manually adjusted in principle, but the underlying RL method should be equipped with a capacity to scale with action dimensionality as well. For instance, standard DQN doesn't have such a capacity, but Sequential DQN [1], Branching DQN [2], Amortized Q-Learning [3], and HGQN-r1 (DQN + action hypergraph networks) [4] have such capacity. \n\n- Comparison with a representative subset of scalable algorithms (such as HGQN-r1) using the naive equidistant discretization scheme would be critical to support the argument that naive discretization is not performant. Comparison with DQN (which doesn't have a generalization capacity over combinatorial action spaces) is not useful in determining whether adaptive discretization is critical over naive discretization.\n\n- Comparison with a bang-bang or bang-off-bang discretization is needed to ensure the domains in experiments indeed require more complex learned action spaces.\n\n- Analysis of what those learned latent actions correspond to would be needed. E.g., similar to that in Fig 7 of Tavakoli et al. (2021), can you show that in Walker2D, the latent actions are able to capture a relationship between the two hip joints? \n\n**References:**\n\n[1] Metz et al. (2017) *Discrete sequential prediction of continuous actions for deep reinforcement learning*. arXiv.\n\n[2] Tavakoli et al. (2018) *Action branching architectures for deep reinforcement learning*. AAAI.\n\n[3] Van de Wiele (2020) *Q-learning in enormous action spaces via amortized approximate maximization*. arXiv.\n\n[4] Tavakoli et al. (2021) *Learning to Represent Action Values as a Hypergraph on the Action Vertices*. ICLR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "tJotfi9srm2", "reviewer": "Reviewer_f3BE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose NDRL, which is a class of methods to automatically discretize action spaces.  They identify issues with prior work and with their methods, and propose improvements to address these issues.  They analyze their methods empirically.", "summary_of_the_review": "The authors study an important and interesting problem; however, the clarity of the paper could be improved.  There are also some empirical problems.  However, the empirical contribution overall was strong.\n", "strength_and_weaknesses": "**Edits:**\n- introduction, “prior sets of discrete actions to from expert demonstrations”: grammar issue?\n- introduction: “utilize graph neural network” should be “utilize a graph neural network”\n- introduction: “and soft-argmax operation” should be “and a soft-argmax operation”\n- “Generally, hybrid action space can be defined as a tuple:” add “a” before a “hybrid”\n- “Then using the embedding vector z^e…”: grammar, what noun is doing the “using”?\n- “The full pseudo-code of NDRL is provided in Algorithm 1”: this is in the supplementary material, so the reference to the pseudo-code should say “in supplementary material Section …”, to avoid making the reader look futilely around the main body of the paper for the algorithm.\n- bottom of page 8: “All the detailed settings are shown in Appendix ?? and ?? respectively.”\n\n**Strengths:**\n- The ablation studies are nicely done and convincing.\n- The authors appear to have chosen their hyperparameters in a principled manner: based on prior work, with little (potentially-unfair-to-baselines) tuning.  When they did tune, they explored those hyperparameters thoroughly, which further strengthens this work (Section A.5).\n\n**Small clarity weaknesses:**\n- Section 3, Hybrid Action Space, first ~5 sentences (through “to describe these basic nodes.”): This is a little hard to follow.  Upon several rereads of this part, I think I fully understand, but a more rigorous approach to defining this notation would make this part of the paper clearer and easier to read.\n- psi, phi, and several other symbols were not defined.  Don’t make the reader guess or infer these definitions.\n\n**Larger weaknesses (clarity and other):**\n- Section 3, Vector Quantised Variational AutoEncoders: I found this difficult to read.  See the question below and the relevant edit above for ideas to make this part clearer.\n- ”In Figure 1(a), we also illustrate the entropy of different action spaces to show the effectiveness of discretization”: A minor issue is there is no 1a subfigure.  A more significant issue: is the entropy being shown, as indicated in the quote above?  How so?  Is H entropy?  Was this defined?  What is x in this figure?  It also seems to be undefined.\n- On a similar note, Figure 1 and its caption are very confusing.  In addition to the issues mentioned above:\n    - x and y are undefined. (The x here does not seem to be the same as the also-undefined x which appears in the top part of the figure)\n    - The sentences “HandCrafted is obtained…right booster will fire” do not make sense to me.  (These sentences seem to be describing a hand-crafted policy, but that does not seem to fit this context, which is talking about action spaces.)\n- Section 4.2.1: The paper completely lost me here; the clarity needs improvement.  Some examples of things I am confused about from this section:\n    - Comparing Figure 2 to Algorithm 1, the phases and terminology are different.  Is the “Collecting Phase” the same thing as “Stage 1”?  If so, another issue is that Figure 2 implies that data is only collected at this stage, but Algorithm 1 talks about using that data to train during this stage.\n    - Figure 2: Shouldn’t there be an arrow from the k vector to the code table?\n    - It is not clear (from the figure, the caption, or the text) where the collected data is going or how it is being used in the training phase (or even if it is being used in the training phase).\n    - “All necessary data will be packed into a transition and put into buffers.”  I am not sure what a transition means in this context (maybe a {s, a, s’, r} tuple?), or where the buffers go, or how the data is subdivided into different buffers\n    - It’s not clear to me how the decoder, the hybrid action space, and the selected action (a_t) interact.  (Although this could be related to the clarity issues from Section 3, rather than a problem with Section 4.2.1.) \n    - Does the collecting phase happen once (many episodes of data), and then the training phase happen after?  Or does the algorithm loop back and forth between the phases? (And if so, how often?  Once per episodes?  Once per timestep?  Some other interval?)  The answers to these questions are not clear from Figure 2 and Section 4.2.1.  Also, while Algorithm 1 may give answers to these questions, the relationships between the components of Algorithm 1 and the components of Figure 2 is not clear (discussed more above).\n- Claim: “ADQ significantly outperforms DQN with manually discretized action space”. In RL, 5 seeds is not usually sufficient to make a claim like this, due to the large variance between runs.  This claim is not convincing, nor are several other claims in the paper based on a similarly-small number of seeds.\n\n**Question:** “the decoder uses the code vector e^d”: should this be z^d?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YXQHQswrWx", "reviewer": "Reviewer_zzhQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an action representation learning method ( AD-VAE)  to learn compact latent action spaces to discretize the action spaces for reinforcement learning training. Furthermore, the paper proposes a few techniques (latent action remapping, ensemble) to mitigate the instability of AD-VAE while training with RL algorithms. The method is demonstrated with a Q-learning-based algorithm. The methods are evaluated on several continuous and hybrid action-space reinforcement learning tasks.", "summary_of_the_review": "Important algorithm details are missing, which hampers the evaluation of the paper's contribution. In addition, the evaluation lacks important experiments that can justify the proposed methods' effectiveness. Overall, the paper needs more coherency in writing. In many places, the motivation and the methodology are tangled, eventually hampering understanding of the methodological contribution.", "strength_and_weaknesses": "Strength:\nThe problem of action discretization seems interesting as it allows the reuse of effective discrete action-based RL algorithms to continuous action tasks.\nThe choice of evaluation on high-dimension continuous tasks seems interesting.\n\nWeakness (see details comments):\nImportant algorithm details need to be included, which hampers the evaluation of the paper's contribution.\nLack of evaluation to justify the effectiveness of the proposed method.\nOverall, the paper needs more coherency in writing. In many places, the motivation and the methodology are tangled. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Vioox9BjhKB", "reviewer": "Reviewer_qGjk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes using a VQ-VAE to map continuous actions to a discrete latent space. The paper then applies RL training on top of the discrete latent action space.", "summary_of_the_review": "I do not recommend acceptance due to the low quality of the writing and presentation (for example, as explained by my comments regarding figure 1).", "strength_and_weaknesses": "Strength \n\nThe proposed idea is intuitive and novel as far as I know,\n\nWeakness\n\nThe writing at the beginning of the paper is quite clear, but the quality of writing deteriorates quickly as the paper progresses. I list a few examples below:\n\n- what is the takeaways for Figure 1? The figure seems hastily prepared, and the caption does not fully explain what the readers should take away from the figure. Section 4.1.2 says that Figure 1 shows that well-designed action space leads to smaller entropy, but afaik, figure 1 does not discuss this at all.\n\n- the writing in 4.1.1 is too informal. What is dirty work in raw action space?\n\n- figure 3 is too small and I can not read the legend at all.\n\nAlso, if the proposed method is equivalent to TD3, then why do we need to use the method at all, since TD3 is relatively simple?\n\nThe paper also misses comparison to obvious baselines, such as https://arxiv.org/abs/1705.05035", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject