HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation
Authors: Boyan Li
Hongyao Tang
YAN ZHENG
Jianye HAO
Pengyi Li
Zhen Wang
Zhaopeng Meng
LI Wang
Venue: ICLR 2022 Poster (2022)

Abstract: Discrete-continuous hybrid action space is a natural setting in many practical problems, such as robot control and game AI. However, most previous Reinforcement Learning (RL) works only demonstrate the success in controlling with either discrete or continuous action space, while seldom take into account the hybrid action space. One naive way to address hybrid action RL is to convert the hybrid action space into a unified homogeneous action space by discretization or continualization, so that conventional RL algorithms can be applied. However, this ignores the underlying structure of hybrid action space and also induces the scalability issue and additional approximation difficulties, thus leading to degenerated results. In this paper, we propose Hybrid Action Representation (HyAR) to learn a compact and decodable latent representation space for the original hybrid action space. HyAR constructs the latent space and embeds the dependence between discrete action and continuous parameter via an embedding table and conditional Variantional Auto-Encoder (VAE). To further improve the effectiveness, the action representation is trained to be semantically smooth through unsupervised environmental dynamics prediction. Finally, the agent then learns its policy with conventional DRL algorithms in the learned representation space and interacts with the environment by decoding the hybrid action embeddings to the original action space. We evaluate HyAR in a variety of environments with discrete-continuous action space. The results demonstrate the superiority of HyAR when compared with previous baselines, especially for high-dimensional action spaces.

Published as a conference paper at ICLR 2022
HYAR: ADDRESSING
ACTION REINFORCEMENT LEARNING VIA HYBRID ACTION REPRESENTATION
Boyan Li Hongyao Tang" Yan Zheng'!, "aieny Hao'l, Pengyi Li', Zhen Wang"), Zhaopeng Meng', Li Wang'
'College of Intelligence and Computing, Tianjin University,
School of Artificial Intelligence, Optics and Electronics (iOPEN) and School of Cybersecurity, Northwestern Polytechnical University
ABSTRACT
Discrete-continuous hybrid action space is a natural setting in many practical problems, such as robot control and game AI. However, most previous Reinforce- ment Learning (RL) works only demonstrate the success in controlling with either discrete or continuous action space, while seldom take into account the hybrid action space. One naive way address hybrid action RL is to convert the hybrid action space into a unified homogeneous action space by discretization or con- tinualization, that conventional RL algorithms can be applied. However, this ignores the underlying structure of hybrid action space and also induces the scala- bility issue and additional approximation difficulties, thus leading to degenerated results. In this paper, we propose Hybrid Action Representation (HyAR, R) to learn compact and decodable latent representation space for the original hybrid action space. HyAR constructs the latent space and embeds the dependence between discrete action and continuous parameter via embedding table and conditional Variational Auto-Encoder (VAE). To further improve the effectiveness, the action representation is trained to be semantically smooth through unsupervised environ- mental dynamics prediction. Finally, the agent then learns its policy with con- ventional DRL. algorithms in the learned representation space and interacts with the environment by decoding the hybrid action embeddings to the original action space. We evaluate HyAR in variety of environments with duscrete-continuon action space. The results demonstrate the superiority of HyAR when compared with previous baselines, especially for nigh-dimensional action spaces.
INTRODUCTION
Deep Reinforcement learning (DRL) has recently shown a great success in a variety of decision- making problems that involve controls with either discrete actions, such as Go (Silver et 2016) and Atari (Mnih et al.) 2015). or continuous actions, such as robot control (Schulman et al. 2015 Lillicrap al. 2015). However, in contrast these two kinds of homogeneous action space, many real-world scenarios requires more complex controls with discrete-continune
hybrid action space, e.g., Robot Soccer (Masson et al.. 20166 and Games (Xiong et al., (0188). For example, in robot soccer, the agent not only needs to choose whether to shoot or pass the ball (i.e., discrete actions) but also the associated angle and force (i.e., continuous parameters). Such hybrid action is also called parameterized action in some previous works (Hausknecht & Stone, 2016) MMasaarlli et al.) 2020). Unfortunately, most conventional RL algorithms cannot deal with such heterogeneous action space directly, thus preventing the application of RL. in these kinds of practical problems.
To deal with hybrid action space, the most straightforward approach is to convert the heterogeneous space into homogeneous one through discretization or continualization. However, it is apparent
Equul contribution.
Corresponding authors: Yan Zheng (yanzheng @ @tw..edu.cn) Jianye Hao (jianye. hao (iw.euuuuuu.uun)) and Zhen Wang(w-zhen @.w.u.cdu.cc)

---

Published as a conference paper at ICLR 2022
that discretizing continuous parameter space suffers from the scalability issue due to the exponen- tially exploring number of discretized actions; while casting all discrete actions into a continuous dimension produces a piecewise-function action subspace, resulting in additional difficulties in ap- proximation and generalization. To overcome these problems, a few recent works propose specific policy structures to learn DRL. policies over the original hybrid action space directly. Parameterized Action DDPG (PADDPG) (Hausknecht & Stone, 2016) makes use of a DDPG (Lillicrap et al. 2015) structure where the actor is modified to output unified continuous vector as the concatenation of values for all discrete actions and all corresponding continuous parameters. By contrast, Hybrid PPO (HPPO) (Fan et al. 2019) uses multiple policy heads consisting of one for discrete actions and the others for corresponding continuous parameter of each discrete action separately, similar idea is also adopted in (Peng & Tsuruoka 2019). These methods are convenient to imple- ment and are demonstrated effective in simple environments with low-dimensional hybrid action space. However, PADDPG and HPPO neglect the dependence between discrete and continuous components of hybrid actions, thus can be problematic since the dependence is vital to identifying the optimal hybrid actions in general. Besides, the modeling of all continuous parameter dimen- sions all the time introduces redundancy in computation and policy learning, and may also have the scalability issue when the hybrid action space becomes nigh-dimensional.
very
To model the dependence, Parameterized DQN (PDQN) (Xiong et al., (018) pro- poses a hybrid structure of DQN (Mnih) et al. 2015) and DDPG. The discrete pol- icy is represented by a DQN which ad-
Algorithm
Scalability
Stationarity
Dependence
Latent
PADDPG
x
HPPO
X
PDQN
X
X
ditionally takes as input all the continu- ous parameters output by the DDPG ac- tor; while the DQN also serves the critic of DDPG. Some variants of such a hybrid structure are later proposed in (Delalleau eet al. 2019 Ma et al. 2021 Bester et al.]
HHQN
X
HyAR (Ours)
Table 1: A comparison algorithmic properties of ex- isting methods for discrete-continous hybrid action RL.
2019) Due to the DDPG actor's 's modeling of all parameters, PDQN also have the redundancy and potential scalablity issue. In an upside-down way, Hicrarchical Hybrid Q-Network (HHQN) (Fu cet al. 2019) models the dependent hybrid-action policy with a two-level hicrarchical structure. The high level is for the discrete policy and the selected discrete action serves as the condition (in analogy special to subgoal) which the low-level continuous policy conditions on. This can be viewed
two-agent cooperative game where the high level and low level learn to coordinate at the optimal hybrid actions. Although the hierarchical structure (Wei et al., 2018) seems to be natural, it suffers from the high-level stationnana caused by off-policy learning dynamics (Wang et al., 2020), i.e., a discrete action can no longer induce the same transition in historical experiences due to the change of the low-level policy. In contrast, PADDPG, HPPO and PDQN are stationary in this sense since they all learn an overall value function and policy thanks to their special structures, which is analogous to learning joint policy in the two-agent game. All the above works focus on policy learning over original hybrid action space. As summarized in Table [] none of them is able to offer three desired properties, i.e., scalability, stationarity and action dependence, at the same time.
In this paper, we propose novel framework for hy- brid action RL, called Hybrid Action Representation (HyAR), to achieve all three properties in Table [ A conceptual overview of HyAR is shown in Fig. The main idea is to construct a unified and decodable representation space for original discrete-continuous hybrid actions, among which the agent learns a la- tent policy. Then, the selected latent action is de- coded back to the original hybrid action space so as to interact with the environment. HyAR is inspired by recent advances in Representation Learning DRL. Action representation learning has shown the potentials in boosting learning performance (Whit-) ney ct al. 2020), reducing large discrete action space (Chandak et al. 2019). improving generalization in offline RL. (Zhou et al.) 2020) and so on. Different from these works, to the best knowledge, we are the
Latent Policy
Decode
Lateat Action
Latent Space
State Roward
Env
Original Highii
Aritee
Diserete
Continancas
An overview of HyAR. Agent learns latent policy in the latent represen- tation space of Soscrete-continuuue actions. The selected latent action is then decoded into the original space so as to interact with the environment.
Figure 1:
2

---

Published as a conference paper at ICLR 2022
first propose representation learning for discrete-continuout hybrid actions, which consist of het- erogeneous and dependent action components.
In HyAR, we maintain continuous vector for each discrete action in a learnable embedding table; then a conditional Variational Auto-encoder (VAE) (Kingma & Welling 2014) that conditions on the state and the embedding of discrete action is used to construct the latent representation space for the associated continuous parameters. Different from HHQN, the conditional VAE models and embeds the dependence in an implicit fashion. The learned representation space are compact and thus scalable, while also provides convenient decoding by nearest-neighbor lookup of the embed- ding table and the VAE decoder. Moreover, we utilize the unsupervised environmental dynamics to learn dynamics predictive hybrid action representation. Such a representation space can be seman- tically smooth, i.e., hybrid action representations that are close the space have similar influence on environmental dynamics, thus benefits hybrid action RL in the representation space. With the constructed action representation space, we use TD3 algorithm (Fujimoto et al., (2188) for the latent policy learning. To ensure the effectiveness, we further propose two mechanisms: latent space straint and representation shift correction to deal with unreliable latent representations and outdated off- oiflpolicy action representation experiences respectively. In experiments, evaluate HyAR in a few representative environments with hybrid action space, as well as several new and more challenging benchmarks.
Our main contributions are summarized below:
We propose novel and generic framework for discrete-continuour hybrid action RL by leveraging representation learning of hybrid action space for the first time.
We propose an unsupervised method of learning a compact and decodable representation space for discret-ccontnuouu hybrid actions, along with two mechanisms to improve the effectiveness of latent policy learning.
. Our algorithm consistently outperforms prior algorithms in representative hybrid-action benchmarks, especially demonstrating significant superiority when the hybrid action space becomes larger. For reproducibility, codes are provided in the supplementary material.
2 PRELIMINARIES
Markov Decision Process Consider a standard Markov Decision Process (MDP) (8, A, P, R. 7, T), defined with a state set S, an action set A. transition function P ; 8 AxxxS - R. reward function R: : S A + R. discounted factor 7 E [0, 1) and horizon T. The agent interacts with the MDP by performing its policy S - A. The objective of an RL agent is to optimize
its policy maximize the expected discounted cumulative reward J(xx) E.
t=o E ?'rs]. where 80 ~ Pa (5o) the initial state distribution, ag ~ π (st), 8141 ~ P (8t+1 8t, 02) and r; = R at).
The state-action value function Q" defined as Q3 (s, a) E Σ 7rr 80 8, Ag =
Parameterized Action MDP In this paper, we focus on a Parameterized Action Markov Decision Process (PAMDP) (S, H.P. R., 7, T) (Masson et al. 2016). PAMDP is an extension of stardard MDP with discrete-continuous hybrid action space 7t detied as:
H [(ki, xx) I E As for all k E K.,
where IC = (1..... K} is the discrete action set, Xi is the corresponding continuous parameter set for each k E IC. We call any pair of k æy as hybrid action and also call H. as hybrid action space for short in this paper. In turn, we have state transition function P S x H x - R, reward function R: : S x H + IR, agent's 's policy S + H and hybrid-action value function Q= k, 2x).
Conventional RL RL algorithms are not compatible with hybrid action space H. Typical policy represen- tations such as Multinomial distribution or Gaussian distribution can not model the heterogeneous components among the hybrid action. Implicit policies derived by action value functions, often adopted in value-based algorithms, also fail due to intractable maximization over infinite hybrid ac- tions. In addition, there exists the dependence between discrete actions and continuous parameters, as a discrete action k: determines the valid parameter space X₂ associated with it. In other words, the same parameter paired with different discrete actions can be significantly different in semantics. This indicate that in principle an optimal hybrid-action policy can not determine the continuous parameters beforehand the discrete action is selected.
3

---

Published as a conference paper at ICLR 2022
Reinforemmnt Leaming Action Representation
Hybdi Action Representation Model
Bepreeraton Dennding
Odigiaal
Endodiing Tade
Dnsoder
Decoda
(k)
Invinommett
Autrenrader
Predivive
Repessoraioo Trcskiy
lange Facoppros JVAM) Dement
Figure 2: Illustrations of: (leff) the framework DRL with HyAR; and (right) overall workflow of hybrid action representation model, consisting of an embedding table and a conditional VAE.
3 HYBRID ACTION REPRESENTATION (HYAR)
As mentioned in previous sections, it is non-trivial for an RI agent to learn with discrete-continee hybrid action space efficiently due to the heterogeneity and action dependence. Naive solutions by converting the hybrid action space into either discrete degenerated performance due the scalability issue and additional approximation difficulty. Previ- ous efforts concentrate on proposing specific policy structures (Hausknecht & Stone [016]] Fu et al.)
continuous action space can result in
2019) that are feasible to learn hybrid-action policies directly over original hybrid action space. However, these methods fail in providing the three desired properties: scalability, stationarity and action dependence simultancously (See Tab. I
Inspired by recent advances in Representation Learning for RL (Whitey et al., 2020 Chandak et al., (2019), we propose Hybrid Action Representation (HyAR), novel framework that converts the original hybrid-acue aybrid-action policy learning into a continuous policy learning problem among the la- tent action representation space. The intuition behind HyAR is that discrete action and continuous paramcter are heterogeneous in their original representations but they jointly influence the environ- ment; thus we can assume that hybrid actions lie on homogeneous manifold that is closely related to environmental dynamics semantics. the following, we introduce an unsupervised approach of constructing a compact and decodable latent representation space to approximate such a manifold.
3.1 DEPENDENCE-AWARE ENCODING AND DECODING
A desired latent representation space for hybrid actions should take the dependence between the two heterogeneous components into account. Moreover, we need the representation space be decod- able, i.e., the latent actions selected by latent policy can be mapped back to the original hybrid actions so as to interact with the environment. To this end, we propose dependence-awee ewweee encoding and decoding of hybrid action. The overall workflow is depicted in the right of Fig. 2 We establish an embedding table Eg R&xd; xdi with learnable parameter Ç represent the K discrete actions, where each row ec.& = Ec(A) (k) (with being the row index) is a d dy-dimensional continuous vector for the discrete action k. Then, we use a conditional Variational Auto-Encoder (VAE) (Kingma & Welling. 2014) to construct the latent representation space for the continuous parameters. In spe- cific, for hybrid action k, r and state s, the encoder 4a(2 2K, 8, eç.)) parameterized by & takes and the embedding Cç.A condition, and maps F& into the latent variable d With the same condition, the decoder Pu: (ãx 2,8, eç.x) parameterized by th then reconstructs the continuous parameter ž from In principle, the conditional VAE can be trained by maximizing the variational lower bound (Kingma & Welling 2014).
More concretely, we adopt Gaussian latent distribution M(un, ox) for qa(= I,, 8,,,,) where May Oz are the mean and standard deviation outputted by the encoder. For any latent variable sample 2 ~ (p2, Jz), the decoder decodes it deterministicly, i.e., H = (2 e,.))
Io: fy (z, s, eç,)), where fyg is transformation network and Iv. is fully-connected layer for reconstruction. We use & = (0, 16, to denote the parameters of both the decoder network and prediction network (introduced later Sec. 3.2p as well as the transformation network they share. With a batch of states and original hybrid actions from buffer D. we train the embedding table Egg and the conditional VAE 9o. Pv together by minimizing the loss function LYAE below:
L 1. C) B...
-x 1/ + DxI. (go( vs, ec. ())]
(2)
4

---

Published as a conference paper at ICLR 2022
where the first term is the Ly-norm square reconstruction error and the second term is the Kullback- Leibler divergence DxL between the variational posterior of latent representation = and the standard Gaussian prior. Note Zig is differentiable with respect to v/. Ç and & through reparameterization trick (Kingma & Welling 2014).
The embedding table and conditional VAE VAE jointly construct a compact and decodable hybrid action representation space (E tda) for hybrid actions. We highlight that this is often much smaller than the joint action space IR < i l&Kel considered in previous works (e.g., PADDPG, PDQN and HPPO), especially when K or Σn [& is large. In this sense, HyAR is expected to be more scalable when compared in Tab. [ Moreover, the conditional VAE embeds the dependence of continuous parameter on corresponding discrete action in the latent space; and allows to avoid the redundancy of outputting all continuous parameters at any time (i.e., RE Ael). This resembles the conditional structure adopted by HHQN (Fu et al. 2019) while HyAR is free of the non-stationary issue thanks to learning a single policy in the hybrid representation space.
For any latent variables E Ih and 2n E R they can be decoded into hybrid action k. 2y conve- niently by nearest-neighbor lookup of the embedding table along with the VAE decoder. Formally, we summarize the encoding and decoding process below:
Encoding: eg.k = Ek), 2z ~ q IKk,,,
for s, k, Ik k == gE(e) arg mingee lleç, - ell2, I S, for 8, Zx
8, E,G,k)
(3)
Decoding:
3.2 DYNAMICS PREDICTIVE REPRESENTATION
In the above, we introduce how to construct a compact and decodable latent representation space for original hybrid actions. However, the representation space learned by pure reconstruction of VAE may be pathological in the sense that it is not discriminative to how hybrid actions have different influence on the environment, similarly studied in (Grosnit et al.) (02T). Therefore, such a represen- tation space may be ineffective when involved in the learning of RL. policy and value functions, as these functions highly depends on the knowledge of environmental dynamics. To this end, we make full use of environmental dynamies and propose an unsupervised learning loss based on state dynamics prediction to further refine the hybrid action representation.
Intuitively, the dynamies predictive representation learned is semantically smooth. In other words, hybrid action representations that are closer in the space reflects similar influence on environmental dynamics of their corresponding original hybrid actions. Therefore, in principle such representa- tion space can be superior in the approximation and generalization of RL policy and value functions, than that learned purely from VAE reconstruction. The benefits of dynamics predictive representa-
tion are also demonstrated in (Whimey et al. 2020 (Schwarzer et al. 2020).
As shown in the right of Fig. 2 HyAR adopts subnetwork h.s, that is cascaded after the transfor- mation network før of the conditional VAE decoder (called cascaded structure or cascaded head below) to produce the prediction of the state residual of transition dynamics. For any transition sam- ple (s, k, 2k, s"), the state residual is denoted by 82,0' = s. With some abuse of notation (i.e., Pv = hyz o f here), the prediction J... is produced as follows, which completes Eq. 3
Prediction:
80.0/ Pz( 8, EÇ,A) for s, e, Zx
(4)
Then we minimize the L2-norm square prediction error:
LDyn (1), 1b, () By [1ō
(5)
Our structure choice of cascaded prediction head is inspired by (Azabou et al. 2021). The reason behind this is that dynamics prediction could be more complex than continuous action reconstruc- tion, thus usual parallel heads for both reconstruction and the state residual prediction followed by the same latent features may have interference in optimizing individual objectives and hinder the learning of the shared representation.
So far, we derive the ultimate training loss for hybrid action representation as follows:
(6) where is hyper-parameter that weights the dynamics predictive representation loss. Note that the ultimate loss depends on reward-agnostic data of environmental dynamics, which is dense and
LIyAR (b) vi, () Lvvr(0) v. () + Dyn v2, (),
usually more convenient to obtain (Stooke et al. 2021 Yarats et al. 2021 Erraqabi al. 2021)
5

---

Published as a conference paper at ICLR 2022
Algorithm 1: HyAR-TD3
Initialize actor Tue and critic networks Q+ Qn, with random parameters w, 01, 02
Initialize discrete action embedding table Eg and conditional VAE go, Pe with random parameters Ç, $
Prepare replay buffer '
repeat Stage
Update C and é, db using samples
see Eq.
until reaching maximun warm-up training times;
repeat Stage
for 1 <- T
// sclect latent actions in representation space
10
= (8) + ce. with 6, <> (0,
11
// decode into original hybrid actions
k (e), (= fç,+)
see Eq.
3
13
Execute (ks 24). observe r. and new state
Store {s,
e, Za, s'} in D
15
Sample bani-
experience from
16
Update Qo Qa,
see Eq. > see Eq.
17
Update Tua with policy gradient
18
repeat
Update and d, using samples in D until reaching maximun representation training times;
see Eq
20
21
until reaching maximun total enviromment steps,
4 DRL WITH HYBRID ACTION REPRESENTATION
In previous section, we introduce the construction compact, decodable and semantically smooth hybrid action representation space. As the conceptual overview in Fig. [ the next thing is to learn latent RL policy the representation space. In principle, our framework algorithm-agnossic and any RL algorithms for continuous control can be used for implementation. In this paper, we adopt model-free DRL algorithm TD3 (Fujimoto al., (018)) for demonstration. Though there remains the chance to build world model based on hybrid action representation, we leave the study on model-based RL with HyAR for future work.
TD3 is popular deterministiccpolliy Actor-Critic algorithm which is widely demonstrated to be ef- fective in continuous control. As illustrated in the left of Fig. with the learned hybrid action representation space, the actor network parameterizes a latent policy Tu with parameter w that out- puts the latent action vector, i.e., 2 (s) where Rd1, 2g E Rd2, The latent action be decoded according to Eq. [3] and obtain the corresponding hybrid action k, The double critic networks Q0 Qo, take as input the latent action approximate sybrid-action value function QTa, i... Q8. e, 2z) xe QT (s, k xR). With a buffer collected transition sample (s, e, 2z, r, s'),, the critics are trained by Clipped Double Q-Learning, with the loss function below for 1,2:
LCDO Es,e,tin (w (s,e, where y +-++o min Qa (8', ()()) 1=1.2
where 0 @ are the target network parameters. The actor (latent policy) updated with Deter- ministic Policy Gradient Silver et al. 2014) as follows:
VCJ( = E. [V xu (s) Qo, (s, (s))
Algorithm ddescribes the pseudo-code of HyAR-TD: containing two major stages: warm-up stage and training stage. In the warm-up stage, the hybrid action representation models are pre-trained using prepared replay buffer D (line 4-6). The parameters the embedding table and conditional VAE is updated by minimizing the VAE and dynamics prediction loss. Note that the proposed algorithm has no requirement on how the buffer D prepared and here we simply use random policy for the environment interaction and data generation by default. In the learning stage, given environment state, the latent policy outputs a latent action perturbed by a Gaussian exploration noise, with some abuse of notions e, (line 10). The latent action is decoded into original hybrid action sample is stored in the replay buffer (line 12-14). Then, the latent policy learning is preformed using the data sampled from D (line 15-17). It is worth noting that the action representation model
interact with the environment, after which the collected transition

---

Published as a conference paper at ICLR 2022
updated concurrently in the training stage make continual adjustment the change of data distribution (line 19-21).
Outlier
Before
After
One significant distinction of DRL with HyAR described above compared with conventional DRL. is that, the hybrid ac- tion representation space is learned from finite samples that are drawn from mov- ing data distribution. The induced un- reliability and shift of learned represen- tations can severely cripple the perfor- mance of learned latent policy they are not carefully handled. Hence, we propose two mechanisms to deal with the above two considerations detailed below.
Latent Space
Boundary
(a) Representation unreliability
(b) Representation shift
Figure 3: Illustrations of representation unreliability and representation shift. Dots denote the hybrid action rep- resentations selected by policy (red) and known by en- coder (blue). Gray lines form the areas, among which representations can be well decoded and estimated.
Latent Space Constraint (LSC) As the latent representation space is constructed by finite hybrid action samples, some areas in the latent space can be highly unreliable in decoding as well as Q- value estimation. Similar evidences are also founded in (Zhou et al.) 2020; Notin et al.] [021]. In Tig. the latent action representations inside the boundary can be well decoded and estimated the values, while the outliers cannot. Once the latent policy outputs outliers, which can be common in the early learning stage, the unreliability can quickly deteriorate the policy and lead bad results. Therefore, we propose to constrain the action representation space the latent policy inside a reasonable area adaptively. In specific, re-scale each dimension of the output of latent policy (i.e.. [-1, 1] +d2 by tanh activation) to a bounded range [bjower. bupper]. For a number of s, k, Ikk collected previously, the bounds blowers bupper are obtained by calculating the e-percentage central range where c E [0, 100]. We empirically demonstrate the importance of LSC. See more details in
Appendix A & 3
Representation Shift Correction (RSC) As in Algorithm 1 the hybrid action representation space is continuously optimized along with RL process. Thus, the representation distribution of original hybrid actions in the latent space can shift after a certain learning interval (Igl et al.) 2020). Fig. b)) illustrates the shift (denoted by different shapes). This negatively influences the value function learning since the outdated latent action representation no longer reflects the same transi- tion present. To handle this, propose representation relabeling mechanism. In specific, for each mini-batch training in Eq we check the semantic validity hybrid action representations in current representation space and relabel the invalid ones with the latest representations. In this way, the policy learning always performed on latest representations, so that the issue of representation shift can be alleviated. Empirically evaluations demonstrate the superiority of relabeling techniques in achieving a better performance with lower variance. See more details in Appendix A &
EXPERIMENTS
We evaluate HyAR in various hybrid action environments against representative prior algorithms. Then, detailed ablation study is conducted to verify the contribution of each component in HyAR. Moreover, we provide visual analysis for better understandings of HyAR.
5.1 EXPERIMENT SETUPS
Benchmarks Fig. visualizes the evaluation benchmarks, including the Platform and Goal from
Masson et al. 2016). Catch Point from (Fan et al. (219)), and a newly designed Hard Move specific to the evaluation in larger hybrid action space. We also build complex version of Goal, called Hard Goal. All benchmarks have hybrid actions and require the agent to select reasonable actions to complete the task. See complete description of benchmarks in Appendix B.I
Baselines Four state-of-the-art approaches are selected as baselines: HPPO (Fan et al. 2019 PDQN (Xiong al. 2018), PADDPG (Hausknecht & Stone 2016), HHQN (Fu al. 2019). In addition, for a comprehensive study, we extend the baselines which consists of DDPG to their TD3 variants, denoted by PDQN-TD3, PATD3, HHQN-TD3. Last, HyAR-DDPG and HyAR-
7

---

Published as a conference paper at ICLR 2022
Tuge aren
Carreat antion nene
Actiatens
ADNY
cald thtees
(a) Plzeform
(6) Goal
(c) Catch Point
(d) Hand Meve
Figure 4: Benchmarks with discrete-cominnuus actions: (a) the agent selects a discrete action (run, hop, leap) and the corresponding continuous parameter (horizontal displacement) reach the goal; (b) The agent selects a discrete strategy (move, shoot) and the continuous 2-D coordinate to score; (c) The agent selects a discrete action (move, catch) and the continuous parameter (direction) grab the target point; (d) The agent has n equally spaced actuators. It can choose whether each actuator should be on off (thus 2n combination in total) and determine the corresponding continuous parameter for each actuator (moving distance) to reach the target area.
HPPO
PADDPG
PIQN
HBQQ
HyAR-DDPG
PATD3
PDQN-TD3 HHQN-IDA
HyAR-TD3
ENV
PPO-based
DDPG-busd
ID3-based
Gool
0.0 + 0.0
0.55 +000
0.70-+D00 0.07
0.0+BD
0.53+0.02
DDD+000
0.71+0.10
0.0+0.0
0,7%-0.63
Hard Goal
0.0 1 0.0
0.0 1.00
0.0.1.00
0.010.0
0.30.10.8
0.1110.05
0.06.10.07
0.0110.01
0.6010.07
0.4610.25
0.87:10.06
0.9410.10
0.9310.03
0.6210.233
0.9810.01
0.95
0.8220010
0.8920.07
Hard Move 4)
0.09 1: 0.02
0.03 1.000
0.69 1.077
0.3910.14
0.91100.3
0.66.10.1
0.8510.10
0.5210.17
0,93.00.2
Hard Move (n- 6)
0.05 + 0.00
0.04
+ 0.01
0.41
= 0.06
0.32=0.17
0:91+0004
0.04=0002
0.74+00.08
0.29++0.13
0.92-0.64
Hard Move 8)
0.04 4: 0.01
0.06
= 0.03
0.04
= 0.00
0.05200.22
0.85:20.06
0.06::0.02
0.05::0.01
0.05+00.02
0.89:0001
Hanl Move (n m 0.05 + 0.01
0.04 + 0.01
0.06 + 0.02
0.04+0.01
0.82:-0066
0.07-0022
0.05+0.02
0.05+0.02
0.75+0005
Table 2: Comparisons of the baselines regarding the average performance at the end of training process with the corresponding standard deviation over 5 runs. Values in bold indicate the best results in each environment.
TD3 to denote our implementations of DRL with HyAR based on DDPG and TD3. For a fair comparison, the network architecture (i.e., DDPG and TD3) used in associated baselines are the same. For all experiments, use a random strategy to interact with the environment for 5000 episodes during the warm-up stage. For each experiment, run trials and report the average results. Complete details of setups provided in Appendix
give each baseline the same training budget. For algorithms,,
5.2 PERFORMANCE EVALUATION
To conduct comprehensive comparison, all baselines implemented based on either DDPG or TD3 are reported. To counteract implementation bias, codes of PADDPG, PDQN, and HHQN are directly adopted from prior works. Comparisons in terms of the averaged results are summarized in Tab, where bold numbers indicate the best result. Overall, we have the following findings. HyAR-TD. and HyAR-DDPG show the better results and lower variance than the others. Moreover, the advan- tage of HyAR is more obvious in environments in larger hybrid action space (e.g.. Hard Goal & Hard Move). Taking Hard Move for example, the action space grows exponentially, the performance of HyAR is steady and barely degrades, while the others deteriorate rapidly. Similar results can be found in Goal and Hard Goal environments. This is due to the superiority of HyAR of utilizing the hybrid action representation space, among which the latent policy can be learned based on compact semantics. These results not only reveal the efffectiveness of HyAR in achieving better performance, but also the scalability and generalization.
In almost all environments, HyAR outperforms other baselines for both the DDPG-based and TD3- based cases. The exceptions are in Goal and Platform environments, where PDQN performs slightly better than HyAR- We hypothesize that this is because the hybrid action space of these two environments is relatively small. For such environments, the learned latent action space could be sparse and noisy, which in turn degrades the performance. One evidence is that the conservative (underestimation) nature in TD3 could compensate and alleviates this issue, achieving significant improvements (HyAR-TD3 v.s. HyAR-DDPG). Fig. 5 renders the learning curves, where Hy AR- TD3 outperforms other baselines in both the final performance and learning speed across all environ- ments. Similar results are observed in DDPG-based comparisons and can be found in Appendix In addition, HyAR-TDS shows good generalization across environments, while the others more or
8

---

Published as a conference paper at ICLR 2022
ДИРРО
PDON-TD3
PATD3
HHQN-TD3
HyAR-TDS (Chus)
Average Episodic
(4)
Hard Gool
Platform
Catch
(0) Hard Move (n-4)
Hard Move (1-66)
Hard Move (r=S)
(b) Hard Move (a-10)
Figure 5: Comparisons of algorithms in different environments. The x- and y-axis denote the envi- ronment steps (x109) and average episode reward over recent 100 episodes. The curve and shade denote the mean and standard deviation over 5 runs.
less fail some environments (e.g., HPPO, PATD3, and HHQN-TTD3 fail in Fig. a) and PDQN- TD3 fails in Fig. [])).. Moreover, when environments become complex (Fig. Je--)), HyAR-TD3 still achieves steady and better performance, particularly demonstrating the effectiveness of HyAR in high-dimensional hybrid action space.
5.3 ABLATION STUDY AND VISUAL ANALYSIS
We further evaluate the contribution of the major components in HyAR: the two mechanisms for latent policy learning, i.e., latent space constraint (LSC) and representation shift correction (RSC), and the dynamics predictive representation loss. We briefly conclude our results as follows. For LSC, properly constraining the output space of the latent policy is critical to performance; otherwise, both loose and conservative constraints dramatically lead performance degradation. RSC and dynamics predictive representation loss show similar efficacy: they improve both learning speed and convergence results, additionally with lower variance. Such superiority is more significant in the environment when hybrid actions are more semantically different (e.g., Goal). We also conduct ablation studies on other factors along with hyperparameter analysis. See complete details and ablation results in Appendix C.2 &
Finally, adopt t-SNE (Maaten & Hin- ton 2008) to visualize the learned hybrid action representations, i.e., (e, 2₂), in 2D plane. We color each action based on its impact on the environment i.c., 8, As shown in Fig. (6] observe that actions with a similar impact on the environment relatively closer in the latent space. This demonstrates the dynamics predic- tive representation loss is helpful for deriv- ing dynamics-awer aware representation for fur- ther improving the learning performance, efficacy, and stability (see results in Ap- pendix & C.4
(a) Goal
(b) Hard Move (n =
Figure 2D (-SNE visualizations of learned represen- tation for original hybrid actions, colored by ID t-SNE of the corresponding environmental impact.
CONCLUSION
In this paper, we propose Hybrid Action Representation (HyAR) for DRL agents to efficiently learn with coscrete-onninuuus hybrid action space. HyAR use an unsupervised method derive a com- pact and decodable representation space for discrete-comtimuous hybrid actions. HyAR can be easily extended with modern DRL methods leverage additional advantages. Our experiments demon- strate the superiority of HyAR regarding performance, learning speed and robustness in most hybrid action environment, especially in high-dimensional action spaces.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The work is supported by the National Science Fund for Distinguished Young Scholars (Grant No.: 62025602), the National Natural Science Foundation of China (Grant Nos.: 11931015, 62106172), the XPLORER PRIZE, and the New Generation of Artificial Intelligence Science and Technology Major Project of Tianjin (Grant No.: 19ZXZNGX00010).
REFERENCES
M. Azabou, M. G. Azar, R. Liu, C. Lin, E. C. Johnson, K. B. Nair, M. D., K. B. Hengen, G. Roncal, M. V., and E. Dyer. Mine your own view: Self-supervised learning through across-sample prediction. CoRR, abs/2102.10106, 2021.
C. J. Bester, S. D. James, and G. D. Konidaris. Multi-pas pasti-passssas q-networks for deep reinforcement learning with parameterised action spaces. CoRR, abs/1905.00 00.888488. 2019.
Y. Chandak, G. Theocharous, J. Kostas, S. M. Jordan, and P. S. Thomas. Learning action represen- tations for reinforcement learning. In ICML, volume 97, pp. 941-950, 2019.
O. Delalleau, M. Peter, E. Alonso, and A. Logut. Discrete and continuous action representation for practical RL in video games. CoRR, abs/1912.11077, 2019.
A. Erraqabi, M. Zhao, M. C. Machado, Y. Bengio, S. Sukhbaatar, L. Denoyer, and A. Lazaric. Exploration-drive drriav representation learning in reinforcement learning. In Unsupervised Reinforce- ment Learning workshop on ICML, 2021.
Z. Fan, R. Su, W. Zhang, and Y. Yu. Hybrid actor-critic reinforcement learning in parameterized action space. In IJCAI, pp. 2279-2285, 2019.
H. Fu, H. Tang, J. Hao, Z. Lei, Y. Chen, and C. Fan. Deep multi-agent reinforcement learning with discrete- discrete-continuous continuous
hybrid action spaces. In IJCAI, pp. 3329-2335, 2019.
S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation enror in actor-critic methods. In JCML, volume 80, pp. 1591, 2018
A. Grosnit, R. Tutunov, A. Maraval, R. Griffiths, A. Cowen-Rivers, L. Yang, L. Zhu, W. Lyu, Z. Chen, J. Wang, J. Peters, and H. Bou-Ammar. High-dimensional bayesian optimisation with variational autoencoders and deep metric learning. CoRR, abs/2106.03309 2021.
X. Hao, W. Wang, H. Mao, Y. Yang, D. Li, Y. Zheng, Z. Wang, and J. Hao. Api: Boosting multi- agent reinforcement learning via agent-permuuation-nnvee 2022.
networks. CoRR, abs/2203.05285,
M. Hausknecht and P. Stone. Deep reinforcement learning in parameterized action space. ICLR, 2016.
M. Igl, G. Farquhar, J. Luketina, W. Bochmer, and S. Whiteson. The impact of non-stationarity on generalisation in deep reinforcement learning. CoRR, abs/2006.05826, 2020.
D. Kingma and M. Welling. Auto-encoding variational Bayes. ICLR, 2014.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
P. Lillicrap, Hunt, A. Pritzel. N. Heess, Erez, Tassa, D. Silver, and D. Wierstra. Contin- uous control with deep reinforcement learning. In ICLR, 2015.
Ma. S. Yao, G. Chen, J. Song, and Ji. Distributed reinforcement learning with self-play in parameterized action space. In 2021 IEEE International Conference on Systems, pp. 1178-1185, 2021.
L. V. D. Maaten and G. E. Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605, 2008.
10

---

Published as a conference paper at ICLR 2022
S. Massaroli, M. Poli, S. Bakhtiyarov, A. Yamashita, H. Asama, and J. Park. Neural ordinary differential equation value networks for parametrized action spaces. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
W. Masson, P. Ranchod, and G. D. Konidaris. Reinforcement learning with parameterized actions. In AAAI, pp. 1934-1940, 2016.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforce- ment learning. Nature, 518(7540):529 533, 2015.
M. Neunert, A. Abdolmaleki, M. Wulfmeier, T. Lampe, J. T. Springenberg, R. Hafner, Romano, J. Buchli, N. Heess, and M. A. Riedmiller. Continuous-diserete reinforcement learning for hybrid control in robotics. CoRL, volume 100, pp. 735-751, 2019.
P. Notin, J. M. Hernández-Lobato, and Y. Gal. Improving black-box optimization in VAE latent space using decoder uncertainty. CoRR, abs/2107.00096 2021.
L. Peng and Y. Tsuruoka. Improving action branching for deep reinforcement learning with a multi- dimensional hybrid action space. In The 24th Game Programming Workshop, 2019.
Schulman, S. Levine, P. Abbeel, M. I. Jordan, and Moritz. Trust region policy optimization. In ICML, pp. 1889-1897, 2015.
J. Schulman, P. Moritz, S. Levine, M. I. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. ICLR, 2016.
M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. C. Courville, and P. Bachman. Data-efficient reinforcement learning with momentum predictive representations. CoRR, abs/2007.05929) 2020.
R. Shen, Zheng, J. Hao, Z. Meng, Y. Chen, C. Fan, and Y. Liu. Generating behavior-diverse game ais with evolutionary multi-objective deep reinforcement learning. In IJCAI, pp. 3371-3377, 2020.
D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. A. Riedmiller. Deterministic policy gradient algorithms. In ICML, pp. 387-395, 2014.
D. Silver, A. Huang, J. Maddison, A. Guez, L. Sifre, G. Driessche, Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever,
T. P. Lillierap, M. Leach, K. Kavukeuoglu, T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 2016.
A. Stooke, K. Lee, P. Abbeel, and M. Laskin. Decoupling representation learning from reinforce- ment learning. ICML., volume 139, pp. 9870-9879, 2021.
Sun, Y. Zheng, J. Hao, Z. Meng, and Liu. Continuous multiagent control using collective behavior entropy for large-scale home energy management. In AAAI, pp. 922-929, 2020.
H. Tang, Z. Meng, Chen, Chen, C. Chen, Y. Yang, L. Zhang, W. Liu, and J. Hao. Foresee then evaluate: Decomposing value estimation with latent future prediction. In AAAI, pp. 2021.
R. Wang, R. Yu, B. An, and Z. Rabinovich. Phrr: Interactive influence-based hierarchical reinforce- ment learning. In IJCAI, pp. 3131-3138, 2020.
E. Wei, D. Wicke, and Luke, Hierarchical approaches for reinforcement learning in parameterized action space. In 2018 AAAI Spring Symposia, 2018.
W. F. Whitney, R. Agarwal, K. Cho, and A. Gupta. Dynamics- aware embeddings. In ICLR, 2020.
Xiong, Q. Wang, Z. Yang, P. Sun, L. Han, Y. Zheng, H. Fu, T. Zhang, Liu, and H. Liu. Parametrized deep genttworks learning: Reinforcement learning with discrete-continuous hybrid action space. CoRR, abs/ abs/1181, 2018.

---

Published as a conference paper at ICLR 2022
D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Reinforcement learning with prototypical represen- tations. ICML, volume 139, pp. 11920-11931, 2021.
Y. Zheng, Z. Meng. Hao, Z. Zhang, T. Yang, and C. Fan. A deep bayesian policy reuse approach against non-stationary agents. In NeurIPS, pp. 962-972, 2018.
Y. Zheng, X. Xic, T. Su, L. Ma, J. Hao, Z. Meng. Y. Liu, R. Shen, Y. Chen, and C. Fan. Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. In ASE, pp. 772-784, 2019.
Y. Zheng. J. Hao, Z. Zhang, Z. Meng, T. Yang, Y. Li, and C. Fan. Efficient policy detecting and reusing for non non-staiionarity in markov games. AAMAS, 35(1):1-29, 2021a.
Y. Zheng, Y. Liu, X. Xie, Y. Liu, L. Ma, J. Hao, and Liu. Automatic web testing using curiosity- driven reinforcement learning, In ICSE, pp. 423-435, 2021b.
Y. Zheng, Z. Yan, K. Chen, J. Sun, Y. Xu, and Y. Liu. Vulnerability assessment of deep reinforcement learning models for power system topology optimization. IEEE Transactions on Smart Grid, 12 (4):3613-3623, 2021c.
W. Zhou, S. Bajracharya, and D. Held. PLAS: latent action space for offline reinforcement learning. CoRR, abs/2011.07213 2020.
12

---

Published as a conference paper at ICLR 2022
A DETAILED FOR LATENT SPACE CONSTRAINT (LSC) AND REPRESENTATION SHIFT CORRECTION(RSC)
Latent Space Constraint (ISSC) As we can see in Fig. aa), the latent action representations inside the boundary can be well decoded and estimated the values, while the outliers cannot. Therefore, the most critical problem for latent space constraint (LSCC) is find reasonable latent space boundary- Simply re-scale policy's outputs in a fixed bounded area b. b] could lose some important infor- mation and make the latent space unstable (Zhou et al.) 2020: Notin et al. 2021) We propose mechanism to constrain the action representation space of the latent policy inside a reasonable area adaptively. In specific, we re-scale each dimension of the output of latent policy (i.e., [-1, 1]4: + by tanh activation) to a bounded range [brower, bupper] At intervals (actually concurrent with the up- dates of the hybrid action representation models), we first sample M transitions 8, ki, 2g from buffer, then we obtain the corresponding latent action representations with current representation models. In this way, we will get M different latent variable values in each dimension. We sort the latent variable of each dimension and calculate the c-percentage central range, i.e., let the quantile and 1- 2 quantile of the range to be blower and bupper of the current latent variable. We called as latent select range where c E [0, 100]. With the decrease of c, the constrained latent action representation space becomes smaller. The experiment on the value of latent select range is in Appendix
Representation Shift Correction (RSC) Since the hybrid action representation space is continu- ously optimized along with the RL learning, the representation distribution of original hybrid actions in the latent space can shift after certain learning interval (Igl et al., 2020). Fig. 3b) visualizes the shifting (denoted by different shapes). This negatively influences the value function learning since the outdated latent action representation no longer reflects the same transition at present. To handle this, we propose representation relabeling mechanism. In specific, we feed the batch of stored original hybrid actions to our representation models to obtain the latest latent representations, for each mini-batch training in E For latent discrete action e, if can not be mapped to the corresponding original action k in the latest embedding table, we will relabel e through looking up the table with stored original discrete action k i.e., < cç. + (0, 0.1). The purpose of adding noise /O, 0.1) is to ensure the diversity of the relabeled action representations. For latent contin-
uous action Zz, we first obtain å..d' through the latest decoder (22, 8, s,,x))) Then we verify if [|ō..// - Se.e 113 > du (threshold value & is set to be 4 Lyya where Lyya is the moving empiri- cal loss), i.e., the case that indicates that the historical representations has no longer semantically consistent (with respect to environmental dynamics) under current representation models. Then 2n will be relabeled by the latest latent representations z ~ qa Tk; 8, ec,)). In this way, the policy learning is always performed on latest representations, so that the issue of representation distribution shift can be effectively alleviated. The experiment on relabeling techniques is in Appendix C.3
Limitation and Potential Improvements of LSC and RSC The ideas of LSC and RSC are gen- eral. In our work, both LSC and RSC are easy to implement and empirically demonstrated to be effective. We also consider that there remains some room further improve these two mechanisms in some more challenging cases. For LSC, one potential drawback may be that the central range constraint on per dimension adopted in work, is suboptimal to the case if the distribution of learned action representation of original hybrid actions is multi-modal. In such cases, the central range constraint may be less effective and can be further improved. For RSC, it requires additional memory and computation cost. However, in our experiments, such memory and computation cost is relatively negligible. The dimensionality of each original hybrid action sample is relatively small compared to the state sample stored; the additional wall-clock time cost of HyAR (with RSC) to HyAR without RSC is about 6% (tested in Goal).
B EXPERIMENTAL DETAILS
B.
SETUPS
Our codes are implemented with Python 3.7.9 and Torch 1.7.1. All experiments were run on a single NVIDIA GeForce GTX 2080TI GPU. Each single training trial ranges from 4 hours to 10 hours, depending on the algorithms and environments. For more details of our code can refer to the HyAR, zip in the supplementary results.
13

---

Published as a conference paper at ICLR 2022
Layer
Actor Network (iT(s)
Critic Network (Q(s, a) V(s))
Fully Connected
(state dim, 256)
(state dim R laxel, 256)
(state dim + R [&4 256) or (state dim, 256)
Activation
ReLU
ReLU
Fully Connected Activation
(256, 256)
(256, 256) ReLU
ReLU
Fully Connected
(256, IR* and (256, Ex lXl))
(256, 1) or (256, RR
or (256, R2, (Ka))
Activation
Tanh
None
Table Network structures for the actor network and the critic network (Q-network or V-network).
Benchmark Environments We conduct our experiments on several hybrid action environments and detailed experiment description is below.
Platform (Masson et al. 2016): The agent need reach the final goal while avoiding the enemy or falling into the gap. The agent need to select the discrete action (run, hop. leap)
and determine the corresponding continuous action (horizontal displacement) simultane- ously to complete the task. The horizon of an episode is 20.
Goal (Masson et al., 2016); The agent shoots the ball into the gate to win. Three types of hybrid actions are available to the agent including kick-to(xxy), shoot- sooot--oalllef((h) shoot- goal-right(h). The continuous action parameters position (x, y) and position (h) along the goal line are quit different. Furthermore, We built a complex version of the goal environ- ment, called Hard Goal. We redefined the shot-goal action and split it into parameter- ized actions by dividing the goal line equidistantly. The continuous action parameters of each shot action will be mapped to a region in the goal line. The horizon of an episode is 50.
Catch Point (Fan al. 2019 The agent should catch the target point (orange) in limited opportunity (10 chances). There are two hybrid actions move and catch. Move is param- eterized by continuous action value which is directional variable and catch is to try to catch the target point. The horizon of an episode is 20.
Hard Move (designed by us): The agent needs to control equally spaced actuators to reach target area (orange). Agent can choose whether each actuator should be on or off. Thus, the size of the action set is exponential in the number of actuators that is 2 Each actuator controls the moving distance in its own direction. n controls the scale of the action space. As increases, the dimension of the action will increase. The horizon of episode is 25.
B.2 NETWORK STRUCTURE
Our PATD3 is implemented with reference to github. from/sfujim/DD implemented with reference to
(TD3 source-code).
PADDPG and PDQN For a fair comparison, all the baseline methods
https: / cwm/ Soycraig/MP-DDa have the same network structure (except for the specific components each algorithm) HyAR-TD3 implementation. For PDQN, PADDPG, we introduce a Passthrough Layer (Masson al., 2016) to the actor networks to initialise their action-paramett combination of state variables. HPPO paper does not provide open source-code and thus we implemented it by ourselves according to the guidance provided in their paper. For HPPO, the discrete actor and continuous actor do not share parameters (better than share parameters in our experiments).
policies to the same linear
As shown Tab, 3 we use two-layer feed-forward neural network of 256 and 256 hidden units with ReLU activation (except for the output layer) for the actor network for all algorithms. For PADDPG, PDQN and HIQN, the critic denotes the Q-network. For HPPO, the critic denotes the V-network. Some algorithms (PATD3, PADDPG, HHQN) output two heads at the last layer of the actor network, one for discrete action and another for continuous action parameters.
14

---

Published as a conference paper at ICLR 2022
The structure of HyAR is shown in Tab. 4 We introduced element- wise product operation (Tag ct al., 2021) and cascaded head structure (Azabou et al. 2021) to our HyAR model. More details
about their effects are in Appendix
C.3
Model Component
Layer (Name)
Structure
Discrete Action Embodding Table Eig
Parameterized Táble
Conditional Encoder Network 9o (a 8,CA)
Fully Connected (encoding)
(R*,, 256)
Fully Connected (condition)
(state dim + _ 256)
Element-wisc Product
ReLU (encoding) ReLU(condition)
Fully Connected
256)
Activation
ReI.U
Fully Connected (mean)
(256, TR4x)
Activation
None
Fully Connected (log.std) Activation
(256, I
None
Conditional Decoder & Prediction Network Pa 8ape Rgg 8,0,,)
Fully Connected (latent)
<R 256)
Fully Connected (condition)
(state dim + R 256)
Element-wise Product
ReLU(decoding) ReLLU(onndiiinn)
Fully Connected
2566
Activation
RcL.U
Fully Connected (reconstruction)
(256, RR*))
Activation
None
Fully Connected
(256, 256)
Áctivation
ReLJU
Fully Connected (prediction) Activation
(256, state dim)
Nome
Table 4: Network structures for the hybrid action representation (HyAR) including, the discrete action embedding table and the conditional VAE.
B.3 HYPERPARAMETER
For all our experiments, we use the raw state and reward from the environment and no normalization or scaling are used. No regularization is used for the actor and the critic in all algorithms. An exploration noise sampled from (0, 0.1) (Fujimoto et al. 2018 added all baseline methods when select action. The discounted factor is 0.99 and we use Adam Optimizer (Kingma & Ba 2015) for all algorithms. Tab. shows the common hyperparamters of algorithms used in all our experiments.
Hyperparanntter
HPPO
PADDPG
PDON
HHQN
PAIDS
PDQN-TD3
HHQN-TDA TD3
HyAR-DDIG DDPG
HYAR TD3
Actor Learning Rate
1-10-t
1-10-t
1-10-t
1-10-1
3-10-1
3-10-4
3-10-1
1-10
3-10-4
Clitic Leurning Rate
1.10-
1.10
1-10-5
1-10-3
3.10
3-10
3-10
1-10
3-10
Representation Model Leaming Rate
1-10"
Factor
Optimizer
Target Acter Updite
1-10-3
1-10-3
1-10-3
5:10-1
5:10
1-10-1
Target Critic Update Rate
1-10-9
1-10-3
1-30-3
5:10-1
5.10
5:10
5:10
5.10
Exploration Policy
A't0, 0.1)
A10,0.1
N/10 0.1)
A/10,0.1
2/10,00
A/10,.11 0.10
Batch Size
128
128
128
128
128
128
128
128
128
Ruffer Size
10'
10
10
10
10P
10a
10
10"
10"
Actor Epoch
Caitic Epoch
Table 5: A comparison of common hyperparameter choices of algorithms, We use to denote the 'not applicable' situation.
B.4 ADDITIONAL IMPLEMENTATION DETAILS
Training setup: For HPPO, the actor network and the critic network are updated every 2 and 10 episodes respectively for all environment. The clip range of HPPO algorithm is set to 0.2 and we use GAE (Schulman al. 2016) for stable policy gradient. For DDPG-based, the actor network and the critic network is updated every 1 environment step. For TD3-based, the critic network is updated every 1 environment step and the actor network updated every 2 environment step.
The discrete action embedding table is initialized randomly by drawing each dimension from the uniform distribution ((-1,1) before representation pre-training. The latent action dim (discrete or
15

---

Published as a conference paper at ICLR 2022
Algorithm 2: HyAR-DDPG
Initialize actor Tu, and critic networks Qo with random parameters w. and the corresponding target network parameters w, 0
Initialize discrete action embedding table Eg and conditional VAE To. Py with random parameters Ç, 6, v
Prepare replay buffer ?
repeat Stage
Update and d using samples in D until reaching maximun warm-up training times;
Eq. 6
repeat Stage
2
for f- / do
// select latent actions in representation space
10
c, 2z = (s) + ce, with se ~> (0, o)
11
// decode into original hybrid actions
12
k = 95 (e), 24 Pa (a Eç,A)
> see Eq.
Execute (k, observe r, and state s'
14
Store {s, k, Th, e, Za, s in DD
15
Sample mini-batch B of N experience from D
Update critic by minimizing empirical loss (() N EB (v - Qols, zx))? where
y = r - QQa (s', (x)))
17
Update actor by the deterministic policy gradient
(w)(w) N ".-EE.. E.EB (V) [
18
19
repeat
Update ( and éi vi using samples in D
Eq 6
21
until reaching maximun representation training times; until reaching maximum total enviromment steps;
22
continuous latent action) default value is 6. We the KL. weight in representation loss TANE 0.5 and dynamics predictive representation loss weight 3 as 10 (default). More details about dynamics predictive representation loss weight are in Appendix C22
For the warm-up stage, we run 5000 episodes (please refer to Tab. 7 for the corresponding environ- ment steps in different environments) for experience collection. We then pre-train the hybrid action representation model (discrete action embedding table and conditional VAE) for 5000 batches with batch size 64, after which we start the training of the latent policy. The representation models (the embedding table and conditional VAE) are trained every episodes for batches with batch size 64 for the rest of RL training. See Appendix(Dfe different choices of warm-up training and subsequent training of the hybrid action representation.
B.5 DDPG-BASED HYAR ALGORITIIM
Additionally, we implemented HyAR with DDPG (Lilicraap et al.) 2015), called Hy-R-DDPG. The pseudo-code of complete algorithm is shown in Algorithm Results of DDPG-based experimental comparisons can be found in Appendix C.I
C COMPLETE LEARNING CURVES AND ADDITIONAL EXPERIMENTS
C.1 LEARNING CURVES FOR DDPG-BASED COMPARISONS
Fig. visualizes the learning curves of DDPG- based comparisons, where HyAR-DDPG outperforms other baselines in both the final performance and learning speed in most environments. Besides the learning speed, HyAR-DDAR environments. When the environments become complex (shown in Fig. e--))) HR-DDDD still achieves steady and better performance than the others, particularly demonstrating the effectiveness and generalization of HyAR in high-dimensional hybrid action spaces.
also achieves the best generalization as HyAR-TD3 across different
16

---

Published as a conference paper at ICLR 2022
PCON
Hing
Mynvicrd
2005
FOOM
1106
(812)
(a) Goal
(b) Hard Goal
(c) Platform
(d) Catch Point
mass
mi
FOOM
Shee Gett
Time
The Crts
(c) Hard Move (n = 4)
(1) Hard Move (n 6 (g) Hard Move (n = 8) (h) Hard Move (n 10)
Figure 7: DDPG-based comparisons related baselines different environments. The x- and y- axis denote the environment steps (x 100) and average episode reward over recent 100 episodes. The results are averaged using& 5 runs, while the solid line and shaded represent the mean value and a standard deviation respectively.
C.2 LEARNING CURVES FOR THE DYNAMICS PREDICTIVE REPRESENTATION
Fig. shows the learning curves HyAR-TD3 with dynamics predictive representation loss (Fig. a-b)) and the influence of dynamics predictive representation loss weight / on algorithm performance (Fig. (()).. We can easily find that the representation learned by dynamics predictive representation loss is better than without dynamics predictive representation loss. For the weight of dynamics predictive representation loss, we search the candidate set {0.1, 1, 5, 10, 20). The results show that the performance of the algorithm gradually improves with the increase of weight 3, reaches the best when 8 10 and then goes down as further increase of We can conclude that the dynamics predictive representation loss is helpful for deriving an muvironment-awarce sentation for further improving the learning performance, efficacy, and stability. More experiments on representation visualization are in Appendix C.4
repre-
40.
Aud Eoooce Bronrt
50
20
100
9ca
MyAhTTS
0.
10 65
1.
2x
33
(e5)
10
Thn Shipe (145)
(a) Goal
(b) Hard Move (n 8)
(c) Goal
Figure 8: Learning curves for variants of dynamics predictive representation in HyAR. (a) and (b) show the comparison between WyARTTD3 with and without dynamics prediction auxiliary loss; (c) shows the effects of the balancing weight 9 in Eq/o) The results averaged using 5 runs, while the solid line and shaded represent the mean value and standard deviation respectively.
C.3 LEARNING CURVES AND TABLE FOR THE RESULTS IN ABLATION STUDY
As briefly discussed in Sec. 5.3 we conduct detailed ablation and parameter analysis experiments on the key components of the algorithm, including:
element-wise product (Tang et al., 2021) (v.s. concat) operation;
cascaded head (Azabou al.
2021 (v.s. parallel head) structure;
17

---

Published as a conference paper at ICLR 2022
latent select range {80,90 90, 96, 100), for the latent space constraint (LSC) mechanism;
action representation relabeling, corresponding to representation shift correction (RSC);
latent action dim d1 d2 E {3, 6, 12);
Fig. 9 shows the learning curves of HyAR-TD3 and its variants for ablation studies, corresponding to the results in Tab. 6
observe that element-wise product achieves better performance than concatenation (Fig. e))). As similarly discovered in (Tang al., (02T), we hypothesize that the explicit relation between the condition and representation imposed by element wise product forces the conditional VAE to learn more effective hidden features. Second, the significance of cascaded head demon- strated by its superior performance over parallel head (Fig. ((,e)) which means cascaded head can better output two different features. Third, representation relabeling shows an apparent improve- ment (Fig. (), which show that representation shift leads to data invalidation in the experience buffer which will affect RL training. Fourth, a reasonable latent select range plays an important role in algorithm learning (Fig. ((c.g)) Only constrain the action representation space of the latent policy inside reasonable area (both large and small will fail), can the algorithm learn effectively and reliably. These experimental results supports our analysis above.
First,
We also analyse the influence of latent action dim d1, d2 for RL (Fig. (d.h)). In the low-dimensional hybrid action environment, we should choose a moderate value (c.g.. 6). While for high-dimensional environment, larger value may be better (e.g., 12). The insight behind is that the proper dimension- ality of latent action representation may be comparable (or more compact) to the dimensionality of state (ranging from 4 to 17 dimensions in different environments in our experiments). This is because the latent action representation should reflect the semantics of original hybrid action, i.e., the state residual.
096-22
(246622 Sea7 mage 1EP%)
(paa41 Jear on E
WASTTT Jemee deve TD)
Teation
(a) Operator Structure (b) Representation relabeling (c) Latent select range
(d) Latent action
5,0 122
2000
MARTDE Sole deve
1996-03 Po0
1964-10 MoNNDDe)
1are 3030
(bwa672 Sear
(c) Operator Structure Representation relabeling (g) Latent select range
(h) Latent action dim
Figure 9: Learning curves of ablation studies for HyAR (i.e., element-wiss + cascaded head
representation relabeling + latent select range = 96% latent action dim = 6) corresponding to Tab. 6 From top bottom is Goal and Hard Move (n = 8) environment. The shaded region denotes standard deviation of average evaluation over 5 runs.
C.4 REPRESENTATION VISUAL ANALYSIS
In order to further analyze the hybrid action representation, we visualize the learned hybrid action representations. Fig. 10 and Fig. I shows the t-SNE visualization for HyAR in Goal and Hard Move (n = 8) environment.
As we can see from Fig. 10 we adopt t-SNE to cluster the latent continuous actions, i.e., (2i)), outputted by the latent policy, and color each action based on latent discrete actions i.e., (e). We conclude that latent continuous actions can be clustered by latent discrete actions, but there multiple modes in the global range. Our dependence-aware representation model makes good use of this relationship that the choice of continuous action parameters is depend discrete actions.
18

---

Published as a conference paper at ICLR 2022
Resul
Eloss Wist Pood.
Cencal.
Ciacaded
Pandlel Lazent Select Range
Laiera Action Dim
Reisheling Dynumio Predictive
Resals (Gral)
Reulls (Hand Movel
95%
0,78 eJa
0.59 +: 0.01
0,56
0.08
100%
0,62 0.11
0,782 0.13
907
6
0.61 0.04
0.78 0DX
80S
0.08 0.17
0.36 0.12
0:5k+
0.05
Table 6:
Ablation of our algorithm across each contribution Goal and Hard Move (n = 8). Results are average success rates the end of training process over 5 runs. = corresponds standard deviation. The corresponding episode reward learning curves are shown in Fig.
For the dynamics predictive representation loss, we adopt t-SNE cluster the latent actions, i.c., (e, Zg), outputted by the latent policy, and color each action based on its impact on the environment (i.e., ). As shown in Fig. 11, observe that actions with similar impact the environment are relatively closer in the latent space. This demonstrates the dynamics predictive representation loss is helpful for deriving environment-awarenes. representation for further improving the learn- ing performance, efficacy, and stability.
320
200
10
(a) Goal
(b) Hard Move (n = 8)
Figure 10: t-SNE visualization diagram of continuous action embedding žns color coded by discrete
action embedding The continuous actions related to the same discrete actions are mapped to the similar regions of the representation space.
308
-3
-20
(a) Goal
(b) Hard Move (n = 8)
Figure 11: t-SNE visualization diagram of hybrid action embedding pair (e, Ey), color coded by 5e.s, The hybrid actions with a similar impact on the environment are relatively closer in the latent space.
19

---

Published as a conference paper at ICLR 2022
D ADDITIONAL EXPERIMENTS
In this section, we conduct some additional experimental results for a further study of HyAR from different perspectives:
We provide the exact number of samples used in the warm-up stage (i.e., stage 1 in Al- gorithm | in each environment in Tab. 7 The numbers of warm-up environment steps are about 5% 10% of the total environment steps in our original experiments.
Moreover, we also conduct some experiments to further reduce the number of samples used in the warm-up stage (at most 80% off). See the colored results in Tab. 7 HyAR can achieve comparable performance with < 3% samples of the total environment steps.
We also provide Fig. T2 for another view of the learning curves where the number of sam- ples used in the warm-up stage is also counted for HyAR.
We conduct additional experiments to compare HyAR-TD3 and HyAR-TD3 with fixed hybrid action representation space trained in the warm-up stage in the environments Plat- form, Goal, Hard Goal, Catch Point and Hard Move (n = 8). The results are provided in Fig. 13 demonstrating the necessity of subsequent training of the representation trained in the warm-up stage.
We further conduct experiments to study the effects of different choices of training fre- quency of subsequent representation training, in the environments Goal, Hard Goal. The results are provided in Fig. [3] demonstrating a moderate training frequency works best.
Environment
Numher of Warm-up Fav. Steps
Number Total Env. Steps
original
Goal
20000 (0.067)0.78)
5000 (0.017 (0.7)
300000
Hard Goal
20000 (0.067|0.60)
5000 (0.017 00.55)
300000
Platform
10000 (0.95)
5000 (0.025 (0.66)
200000
Catch Point
100000 (0.1 (0.90)
20000 (0.02
1000000
Hard Move (n=4)
100000 (0.1
20000 (0.02 (0.91)
1000000
Hard Move (n=6)
100000 (0.1 (0.92) 20000 (0.02 0.99)
1000000
Hard Move (n=8)
100000 (0. [0.89) 20000 (0.02 (0.8)
1000000
Hard Move (n=10)
100000 (0.1 (0.75) 20000 (0.02 (0.70)
1000000
Table 7: The exact number of samples used in warm-up stage training in different environments. The column of 'original' denotes what is done in our experiments; the column of 'new" denotes ad- ditional experiments we conduct with fewer warm-up samples (and proportionally fewer warm-up training). For each entry z (z), is the number of samples (environment steps), y denotes the per- namber of warm-up environment steps
centage number of total environment steps during the training process and denotes the corresponding performance of HyAR-TD3 as evaluated in Tab. 2
Conclusion: The numbers of warm-up environment steps are about 5% 10% of the total environment steps in our original experiments. The number of warm- up enviromment steps can be further reduced by at most 80% off (thus leading to 3% of the total environment steps) while comparable performance of our algorithm remains.
E ADDITIONAL. DISCUSSION ON DISCRETE-CONTINUON HYBRID ACTION SPACE
To the best of our knowledge, most RL problems with discrete-continuon hybrid action spaces can be formulated by Parameterized Action MDP (PAMDP) described in Sec. 2 These hybrid action spaces can be roughly divided into two categories: structured hybrid action space (mainly considered in our paper) and ntr-structured hybrid action space. In structurdd hybrid action space, there is no dependence among different parts of hybrid action. For a mixed type of the above two, we may conclude it in the case of structured hybrid action space.
To be specific, learning with non-structured hybrid action spaces can be viewed as a special case of PAMDP. It can be similarly formulated with the definition of discrete action space IC (p.s., this can be joint discrete action space when there are multiple discrete dimensions) and the definition of
20

---

Published as a conference paper at ICLR 2022
10/0
2008 041
(a) goal
(b) Hard Goal
(c) platform
(d) catch point
roce-13
(c) Hard Move (n=4)
(f) Hard Move (n=6)
Hard Move (n=8) (h) Hard Move (n=10)
Another view of the learning curves shown in Fig. where the number of samples used warm-up training (i.e., stage in Algorithm [ also counted for HyAR. Comparisons of algorithms in different environments. The x- and y-axis denote the environment steps and average episode reward over recent 100 episodes. The results are averaged using 5 runs, while the solid line and shaded represent the mean value and standard deviation respectively.
Figure 12:
30
Uniised HyAR
Fxed HyAR
Aog
-10
Unfived HyAR Read HyAR
335
0.5
1.8
2,5
30
300 7.55 0.55
0.75 3.00 125
155
135 2:5
(a) Goal
(b) Hard Goal
-10
Speecce
0.6
-32
Unrinze MyA Fieed Hydi
Unlaud MyAh
Urfised MyAh HyAR
Maed ligh
02
Time Steze (145)
Time stase (145)
Time Staps (245)
(c) Platform
(d) Catch Point
(e) Hard Move(n=8)
Figure 13: Comparison between HyAR-TD3 (denoted by Unfixed HyAR) and HyAR-TD3 with fixed hybrid action representation space trained in warm-up stage (denoted by Fixed HyAR) in dif- ferent environments. The x- and y-axis denote the environment steps and average episode reward over recent 100 episodes. The results are averaged using 5 runs, while the solid line and shaded represent the mean value and standard deviation respectively. Conclusion: Unfixed HyAR outper- forms Fixed HyAR across all the environments; while Fixed HyAR performs very poorly Goal and Hard Goal. We conjecture that in these environments, random policy is quite limited in collect- ing effective and meaning hybrid actions these environments thus the learned fixed representation space is not able to support the emergence of effective latent policy.
uniform continuous action space X' rather than X₂ since there is structural dependence in this case. A few examples and analogies can be found in Robotic control (Neunert et al.) 2019) where several gears switchers need be selected in addition other continuous control parameters,
21

---

Published as a conference paper at ICLR 2022
40
30
Avg Episoue Retwrrd
30-
Episade Reward
20
Fieed
Fixed
10
10
150gp
1/2cpp
120cp
2/10pp
N10ep
5/10ep
10/10ep
10/10e>
0.75
1.00
1.75
2.00
Steps
(a) Goal
(b) Hard Goal
Figure 14: The effects of different choices of training frequency of subsequent representation train- ing of HyAR-TDS in the environments Goal, Hard Goal. For example, 1/10ep denotes that the hybrid action representation is trained every 10 episodes for 1 batches with batch size 64. The black dashed lines denote the convergence results of Fixed HyAR in Figure [3] The x- and y-axis denote the environment steps and average episode reward over recent 100 episodes. The results are aver- aged using runs, while the solid line and shaded represent the mean value and standard deviation respectively. Conclusion: all configurations of subsequent training outperform Fixed HyAR; while a moderate training frequency works best.
several dimensions of continuous parameters are discretized. In this view, our algorithm proposed under PAMDP is applicable both structured and non-structured hybrid action spaces.
However, contrtttocotinuuuuu hybrid action spaces are not well studied in DRL. Most existing works evaluate proposed algorithms in relatively low-dimensional and simple environments. Such environ- ments may reflect some significant nature discrete-continunn hybrid action control problem yet far away from the scale and complexity of real-world scenarios, e.g., more complex robots with natural discrete and (dependent) continuous control (beyond the basic robot locomotion in like pop- ular MuJoCo RL suite), RL-bbased software testing (Zheng et al. 202 2019), game Al generation (Shen et al. 2020), recommendation systems (e.g., e-shop, music, news), and industrial production (e.g., production process control & smart grid control) (Sun et al. 2020 Zheng et al.) (021c). We believe that the emergence of such environments will beneñit the community of discrete-contimuous hybrid action RL study. To evaluate and develop our algorithm on multiagent scenarios (Hao et al.,
2022 Zheng al. 2021a| 2018) with more complex and practical hybrid action spaces one of our main future directions.
22


Reviews:
[{"review_id": "w9tzUXNXB5A", "reviewer": "Reviewer_9YW6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Novel approach to handle mixed discrete-continuous action spaces\n- Complete evaluation with comparison to multiple baselines\n\nWeaknesses:\n- The paper focuses on a specific type of mixed discrete-continuous action space. While this setup is common in robotics and other ML problems, other setups with mixed spaces exist, for example, robots with continuous and discrete joints (e.g. a binary “close/open” gripper action). To avoid confusion I would make this very clear in the intro/abstract\n- Understanding the contribution of LSC and RSC is critical. I would include more about it in the main text\n- It would be interesting to apply it to more complex robotics problems (e.g. robot control) to see the limits of the solution\n", "summary_of_the_paper": "The paper presents an approach to handle action spaces with discrete selections and continuous parameters within each selection. This is a common setup in robotics, where agents need to select a skill to apply, and then the parameters to instantiate that skill (e.g. MOVE skill and where to move). The paper proposes an embedding space where the discrete component is mapped through a nearest neighbor selection, and the continuous component is mapped with a decoder conditioned on the embedding of the discrete component. The work includes additional modifications to improve policy learning: first, a method to select actions only in the region confidently represented in the mapped action space, and second, a method to remap the actions every time the mapping changes during training to keep the experience replay buffer “valid”. The method to handle heterogeneous discrete-continuous action spaces and the two extensions for RL training are evaluated on several simple interactive tasks that were used before in related work papers proposing methods for mixed discrete-continuous action spaces. The results indicate an improvement over existing methods.", "summary_of_the_review": "I would recommend accepting the paper for the conference. The text can be streamlined (there is a couple of typos) but the results are pretty solid, the problem is relevant and the idea is novel. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZBlSnNH0WN3", "reviewer": "Reviewer_TSgb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- The paper is overall well-written and easy to follow. The being studied problem and mathematics are clearly defined.\n\n- The proposed method is well motivated and well designed. In particular, two mechanisms are proposed (LSC, RSC) to alleviate the corresponding issues, which I feel important and should be described in more details in the main texts. \n\n- The experimental results are convicinble and comprehensive. Abalation studies is well-conducted. Implemetation details are clearly explained. \n\nCons:\n\n- It seems HyAR is relatively sensitive to the choice of hyper-parameters (Table 6). It remains unclear how to select good hyper-parameters rather than grid/random search.\n\n- For the representation shift problem, although the proposed RSC mechanism showed empirical performance gain, I feel that the problem is not \"resolved\", but only \"alleviated\". Therefore, to say HyAR is fully stationary is a bit overclaimed in my opinion (Table 1).\n\nMinor problems:\n\n- In the definition of Q function in 2.1: \\gamma^l  --> \\gamma^t\n\n- VAE reference: bayes -> {B}ayes, similar to t-SNE, etc.\n", "summary_of_the_paper": "The authors proposed a novel framework for hybrid (discrete and continuous) action RL, called Hybrid Action Representation (HyAR). The main idea is to take advantage of representation learning in Deep RL to encode hybrid action in a conbtinuous latent vector space. Then any RL algorithm for continuous action space can be used to train the agent. Importantly, to mitigate the practical problems of unreliability and representation shift of the learned action encoding, the paper proposed two techniques LSC and RSC, which were empirically shown to be effective.\n\nExperiments in various hybrid action tasks demonstrated that the proposed HyAR method contribute to significant performance gain to the baseline TD3 and DDPG models, as well as previous methods. The authors further performaned an ablation investigation to justify their claims. Moreover, the t-SNE visualization of learned actions representation shows an relatively interpretable latent encoding.", "summary_of_the_review": "The paper proposed a method to solve hybrid action control problems by encoding action to a continuous latent space via representation learning. The motivation is sound and the writting is clear. The technical contribution is significant. The experiments are comprehensive and support the paper's major claims. Overall, I recommend acceptance.\n\n================== post-rebuttal ==================\n\nAlthough I believe this work has some limitations that can be improved in the future, I feel the contributions of the current paper are already significant and I vote for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "DZrCxzx9wq", "reviewer": "Reviewer_JLuH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The concept of latent actions is very interesting. In the latent space, both discrete and continuous actions in the original space are represented by continuous vectors. This is based on the prior work on action representation learning for RL, but this work uses it to solve the difficulty of handling hybrid actions.\n\nIn addition, the decoder predicts the difference between current and next states (s and s'). This may be a small addition, but it looks reasonable to make the representations discriminative. This can be done in an unsupervised manner, so it doesn't introduce any additional cost of labeling.\n\nThis is how to deal with the hybrid actions, and it can be used in any RL methods by replacing actions with the proposed latent actions. This means that the method has a variety of potential applications.\n\nA concern is that among the four factors in Table 1, stationarity is not explicitly discussed in the paper.\n", "summary_of_the_paper": "This paper proposes to handle both continuous and discrete actions in reinforcement learning. Typical MDP suggests actions in either (not both) continuous or discrete action spaces, however in more general cases hybrid action spaces are needed. The proposed method solves this problem by using the latent space and the latent action. Instead of using raw actions, which are continuous and discrete, actions are embedded into a latent space, in which all are continuous. By decoding the latent action to actions in the original hybrid space, the whole process can be seamlessly integrated into any RL algorithms.\nExperiments show that the proposed method method combined with TD3 and DDPG performs significantly better.\n", "summary_of_the_review": "Overall the paper is well organized and written, and the provided code ensures the reproducibility. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4kC5EixIH42", "reviewer": "Reviewer_Am1g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- The paper is well written and easy to follow. The problem at hand is very interesting and the proposed solution meaningful. Some works, like AlphaStar had already tackled hybrid action spaces (although little detail is given in the corresponding paper), but their solution was very ad-hoc to the StarCraft problem, while the authors of this paper propose a very generic solution that could be applied to any (? I am not 100% sure about this, cf my question later) hybrid-action space problem.\n- The experimental evaluation of the method is very complete, the ablations are convincing and enlightening the importance of the different components of the method.\n- The authors made the effort to implement “enhanced” versions of the baselines (td3 based rather and ddpg based), which makes the comparisons fairer.\n\n\n**Weaknesses**\n- Method:\n    - Is the method applicable to any hybrid action spaces? Is there not a underlying assumption that the continuous part of the action depends on the discrete part but not the other way around? Could you think about such examples?\n    - For the sake of mathematical correctness, I think the function predicting $\\hat{\\delta}$ should not be named the same as the decoder as they are different, although some parameters are shared. If you want to mathematically underline the fact that some parameters are shared, you could write $g \\circ f$ and $h \\circ f$. \n    - The paragraph starting with “Our cascaded structure” is very hand-wavy and unclear to me. As far I understand, $L_{Dyn}$ is just an auxiliary loss helping the learning of a better representation. Am I correct? If yes, I encourage the authors to simplify this paragraph. If not, I would encourage the authors to explain why and enhance the explanations given in this paragraph. \n    - “Reward agnostic data of environmental dynamics which is easy to obtain”. I argue this statement is too strong. In particular, for a lot of environments, random behaviors lead to an extremely small coverage of the state space.\n    - I would like the authors to discuss the potential drawbacks of the two main tricks: LSC and RSC. For example, 1) are there environments/setups for which they would be limiting. 2) RSC requires more memory because you have to store both the original action and the embedded one in the replay buffer, and it needs more compute to calculate several times the embedding of a given action. I think it would be valuable for the work if the authors are more open on the limitations of the method.\n\n- Related Work\n    - I think one related work that would be worth discussing and maybe include in the baselines is Neunert et.al. (https://arxiv.org/abs/2001.00449).\n    - The absence of the “Related Work” section does help positioning the paper with respect to the literature.\n\n- Experiments\n    - The experiments, although convincing, are all made in pretty low dimensional state and action spaces. It would be of course an additional strength to validate the method in a more complicated environment.\n    - Although interesting ablations are done, I would have liked two more. 1) What happens if the embeddings are fixed after pretraining? 2)     - What happens if you can’t pretrain (because no data is available at this point)?\n    - I encourage the authors not to talk in terms of “learning steps” or “episodes” because these are hyperparameter-dependent (e.g. batch size dependent) and environment dependent. I argue they should use in x-axis the number of environment steps as is done for most online methods.\n    - HyAR has access to more data than the baselines (the 5000 episodes used in the pretraining are not used by baselines…). Could you try to use them for the baselines too? For example filling the replay buffer with them? Otherwise I don’t find the comparison fair.\n    - I would like to see a paragraph (in Appendix) discussing how hyperparameters were tuned. In particular, I think it is important to understand how much effort was put into tuning the method as well as the baselines.\nThe values of the most important hyperparameters should be explicit in the main text like the choice of $d_1$ and $d_2$.\n\n- Writing\n    - The writing is mostly clear (except this paragraph I mentioned earlier). Yet, there are a few typos, e.g. ”Variantional” in the abstract. “embeds the the dependence” in the introduction. “$\\gamma^l$” in 2.1. \n", "summary_of_the_paper": "- The authors are interested in the problem of Reinforcement Learning with hybrid action spaces. More precisely, MDPs with “structured” action spaces for which an action corresponds to the choice of a discrete value and a continuous vector. One important point is that the “meaning” of the continuous vector is dependent on the choice of the discrete action.\nIn order to tackle this issue, the authors propose to 1) embed the discrete action using an embedding table into a vector space of dimension $d_1$. 2) Use a conditional VAE to embed the continuous action in a vector space of dimension $d_2$. 3) Train a continuous-action RL algorithm in the $d_1+d_2$ latent space. 4) Decode the action back to the original action-space to pass it to the environment.\n\n- The authors also propose a number of tricks to enhance the performance of this embedding: 1) an auxiliary loss predicting environment dynamics. 2) a rescaling of the policy to avoid actions that would not be correctly decoded. 3) A relabeling technique to ensure that off-policy-ness added to the learning of the action embeddings does not hurt the learning.\n", "summary_of_the_review": "This is a very interesting paper that could be excellent with a bit more transparency in the method’s limitations as well as a bit more care in the experimental validation. I would increase my score if authors can provide details on the limitations of the method and tackle the major points I mention in the \"experiment\" part.\n\n================== POST REBUTTAL 1 ================\n\nI would like to thank the reviewers for the detailed answers and the revision of the manuscript.\n\nThe additional data provided by the authors in Table 7 and Figure 12 provide convincing proofs of the method's efficiency that were lacking (or at least that we could doubt) before the revision.\nThe additional ablation in Figure 13 brings nice insights on the methods. I even personnally would prefer to see it in the main text in place of the t-SNE visualizations as I find it more informative (but it is also fine like this!).\n\nInsisting on these minor points:\nI would rather not have footnote 2 and be more precise in the text.\nI would also detail in the HP section of the appendix HOW hyperparemeters were selected and not only which ones were selected (even if the sentence just says that not much tuning was done, I think it is important.)\n\nI am now recommending acceptance and raising my score to a 6 (it could be a 7 if it existed).\nThe work would be a clear 8 with an additional experiment on a more challenging and less toy-ish environment.\nYet, I understand it can be left for future work. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new approach to solve mixed discrete-continuous action RL problems, based on embedding actions into a latent space so that standard continuous control algorithms (like TD3) can be applied. Experiments over standard discrete-continuous benchmarks demonstrate the superiority of the proposed approach vs. existing baselines.

There is overall a strong consensus of all reviewers towards acceptance, especially after the discussion period where the authors were able to submit several revisions addressing most of the questions and concerns raised in the original reviews, in particular w.r.t. the quality and relevance of the results.

I believe this submission could be improved along two axes though:

1. As pointed out by some reviewers, the current environments are somewhat simple. A more realistic robotic task for instance could be a good fit for such an algorithm. That being said, as the authors pointed out, this may require custom development due to the lack of existing public environment with the proper setup.

2. As a reviewer mentioned, there is no "Related Work" section, and although previous work is discussed in the Introduction, I consider that it remains limited, and more previous work should have been discussed. Here are some pointers regarding relevant work I am aware of:
- Hierarchical Approaches for Reinforcement Learning in Parameterized Action Space (https://arxiv.org/abs/1810.09656)
- Neural Ordinary Differential Equation Value Networks for Parametrized Action Spaces (https://openreview.net/forum?id=8WKd467B8H)
- Improving Action Branching for Deep Reinforcement Learning with A Multi-dimensional Hybrid Action Space (https://ipsj.ixsq.nii.ac.jp/ej/index.php?action=pages_view_main&active_action=repository_action_common_download&item_id=199976&item_no=1&attribute_id=1&file_no=1&page_id=13&block_id=8)
- Distributed Reinforcement Learning with Self-Play in Parameterized Action Space (https://cgdsss.github.io/pdf/SMC21_0324_MS.pdf)
- Discrete and Continuous Action Representation for Practical RL in Video Games (https://arxiv.org/abs/1912.11077)
- Multi-Pass Q-Networks for Deep Reinforcement Learning with Parameterised Action Spaces (https://arxiv.org/abs/1905.04388)

In particular, I believe the last one (MP-DQN) should have been one of the baselines, since it is supposed to be an improvement over the P-DQN algorithm (that is one of the baselines used here). I encourage the authors to try and incorporate it for the final version (at the very least, it should be cited).

In spite of these concerns, I still recommend acceptance since the combination of action space embedding with mixed discrete-continuous actions is novel and non-trivial, and the empirical validation is convincing enough in its current state.
Decision: Accept (Poster)