Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents an approach to handle action spaces with discrete selections and continuous parameters within each selection. This is a common setup in robotics, where agents need to select a skill to apply, and then the parameters to instantiate that skill (e.g. MOVE skill and where to move). The paper proposes an embedding space where the discrete component is mapped through a nearest neighbor selection, and the continuous component is mapped with a decoder conditioned on the embedding of the discrete component. The work includes additional modifications to improve policy learning: first, a method to select actions only in the region confidently represented in the mapped action space, and second, a method to remap the actions every time the mapping changes during training to keep the experience replay buffer “valid”. The method to handle heterogeneous discrete-continuous action spaces and the two extensions for RL training are evaluated on several simple interactive tasks that were used before in related work papers proposing methods for mixed discrete-continuous action spaces. The results indicate an improvement over existing methods.
main_review: Strengths:
- Novel approach to handle mixed discrete-continuous action spaces
- Complete evaluation with comparison to multiple baselines

Weaknesses:
- The paper focuses on a specific type of mixed discrete-continuous action space. While this setup is common in robotics and other ML problems, other setups with mixed spaces exist, for example, robots with continuous and discrete joints (e.g. a binary “close/open” gripper action). To avoid confusion I would make this very clear in the intro/abstract
- Understanding the contribution of LSC and RSC is critical. I would include more about it in the main text
- It would be interesting to apply it to more complex robotics problems (e.g. robot control) to see the limits of the solution

summary_of_the_review: I would recommend accepting the paper for the conference. The text can be streamlined (there is a couple of typos) but the results are pretty solid, the problem is relevant and the idea is novel.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors proposed a novel framework for hybrid (discrete and continuous) action RL, called Hybrid Action Representation (HyAR). The main idea is to take advantage of representation learning in Deep RL to encode hybrid action in a conbtinuous latent vector space. Then any RL algorithm for continuous action space can be used to train the agent. Importantly, to mitigate the practical problems of unreliability and representation shift of the learned action encoding, the paper proposed two techniques LSC and RSC, which were empirically shown to be effective.

Experiments in various hybrid action tasks demonstrated that the proposed HyAR method contribute to significant performance gain to the baseline TD3 and DDPG models, as well as previous methods. The authors further performaned an ablation investigation to justify their claims. Moreover, the t-SNE visualization of learned actions representation shows an relatively interpretable latent encoding.
main_review: Pros:

- The paper is overall well-written and easy to follow. The being studied problem and mathematics are clearly defined.

- The proposed method is well motivated and well designed. In particular, two mechanisms are proposed (LSC, RSC) to alleviate the corresponding issues, which I feel important and should be described in more details in the main texts. 

- The experimental results are convicinble and comprehensive. Abalation studies is well-conducted. Implemetation details are clearly explained. 

Cons:

- It seems HyAR is relatively sensitive to the choice of hyper-parameters (Table 6). It remains unclear how to select good hyper-parameters rather than grid/random search.

- For the representation shift problem, although the proposed RSC mechanism showed empirical performance gain, I feel that the problem is not "resolved", but only "alleviated". Therefore, to say HyAR is fully stationary is a bit overclaimed in my opinion (Table 1).

Minor problems:

- In the definition of Q function in 2.1: \gamma^l  --> \gamma^t

- VAE reference: bayes -> {B}ayes, similar to t-SNE, etc.

summary_of_the_review: The paper proposed a method to solve hybrid action control problems by encoding action to a continuous latent space via representation learning. The motivation is sound and the writting is clear. The technical contribution is significant. The experiments are comprehensive and support the paper's major claims. Overall, I recommend acceptance.

================== post-rebuttal ==================

Although I believe this work has some limitations that can be improved in the future, I feel the contributions of the current paper are already significant and I vote for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - The authors are interested in the problem of Reinforcement Learning with hybrid action spaces. More precisely, MDPs with “structured” action spaces for which an action corresponds to the choice of a discrete value and a continuous vector. One important point is that the “meaning” of the continuous vector is dependent on the choice of the discrete action.
In order to tackle this issue, the authors propose to 1) embed the discrete action using an embedding table into a vector space of dimension $d_1$. 2) Use a conditional VAE to embed the continuous action in a vector space of dimension $d_2$. 3) Train a continuous-action RL algorithm in the $d_1+d_2$ latent space. 4) Decode the action back to the original action-space to pass it to the environment.

- The authors also propose a number of tricks to enhance the performance of this embedding: 1) an auxiliary loss predicting environment dynamics. 2) a rescaling of the policy to avoid actions that would not be correctly decoded. 3) A relabeling technique to ensure that off-policy-ness added to the learning of the action embeddings does not hurt the learning.

main_review: **Strengths**

- The paper is well written and easy to follow. The problem at hand is very interesting and the proposed solution meaningful. Some works, like AlphaStar had already tackled hybrid action spaces (although little detail is given in the corresponding paper), but their solution was very ad-hoc to the StarCraft problem, while the authors of this paper propose a very generic solution that could be applied to any (? I am not 100% sure about this, cf my question later) hybrid-action space problem.
- The experimental evaluation of the method is very complete, the ablations are convincing and enlightening the importance of the different components of the method.
- The authors made the effort to implement “enhanced” versions of the baselines (td3 based rather and ddpg based), which makes the comparisons fairer.


**Weaknesses**
- Method:
    - Is the method applicable to any hybrid action spaces? Is there not a underlying assumption that the continuous part of the action depends on the discrete part but not the other way around? Could you think about such examples?
    - For the sake of mathematical correctness, I think the function predicting $\hat{\delta}$ should not be named the same as the decoder as they are different, although some parameters are shared. If you want to mathematically underline the fact that some parameters are shared, you could write $g \circ f$ and $h \circ f$. 
    - The paragraph starting with “Our cascaded structure” is very hand-wavy and unclear to me. As far I understand, $L_{Dyn}$ is just an auxiliary loss helping the learning of a better representation. Am I correct? If yes, I encourage the authors to simplify this paragraph. If not, I would encourage the authors to explain why and enhance the explanations given in this paragraph. 
    - “Reward agnostic data of environmental dynamics which is easy to obtain”. I argue this statement is too strong. In particular, for a lot of environments, random behaviors lead to an extremely small coverage of the state space.
    - I would like the authors to discuss the potential drawbacks of the two main tricks: LSC and RSC. For example, 1) are there environments/setups for which they would be limiting. 2) RSC requires more memory because you have to store both the original action and the embedded one in the replay buffer, and it needs more compute to calculate several times the embedding of a given action. I think it would be valuable for the work if the authors are more open on the limitations of the method.

- Related Work
    - I think one related work that would be worth discussing and maybe include in the baselines is Neunert et.al. (https://arxiv.org/abs/2001.00449).
    - The absence of the “Related Work” section does help positioning the paper with respect to the literature.

- Experiments
    - The experiments, although convincing, are all made in pretty low dimensional state and action spaces. It would be of course an additional strength to validate the method in a more complicated environment.
    - Although interesting ablations are done, I would have liked two more. 1) What happens if the embeddings are fixed after pretraining? 2)     - What happens if you can’t pretrain (because no data is available at this point)?
    - I encourage the authors not to talk in terms of “learning steps” or “episodes” because these are hyperparameter-dependent (e.g. batch size dependent) and environment dependent. I argue they should use in x-axis the number of environment steps as is done for most online methods.
    - HyAR has access to more data than the baselines (the 5000 episodes used in the pretraining are not used by baselines…). Could you try to use them for the baselines too? For example filling the replay buffer with them? Otherwise I don’t find the comparison fair.
    - I would like to see a paragraph (in Appendix) discussing how hyperparameters were tuned. In particular, I think it is important to understand how much effort was put into tuning the method as well as the baselines.
The values of the most important hyperparameters should be explicit in the main text like the choice of $d_1$ and $d_2$.

- Writing
    - The writing is mostly clear (except this paragraph I mentioned earlier). Yet, there are a few typos, e.g. ”Variantional” in the abstract. “embeds the the dependence” in the introduction. “$\gamma^l$” in 2.1. 

summary_of_the_review: This is a very interesting paper that could be excellent with a bit more transparency in the method’s limitations as well as a bit more care in the experimental validation. I would increase my score if authors can provide details on the limitations of the method and tackle the major points I mention in the "experiment" part.

================== POST REBUTTAL 1 ================

I would like to thank the reviewers for the detailed answers and the revision of the manuscript.

The additional data provided by the authors in Table 7 and Figure 12 provide convincing proofs of the method's efficiency that were lacking (or at least that we could doubt) before the revision.
The additional ablation in Figure 13 brings nice insights on the methods. I even personnally would prefer to see it in the main text in place of the t-SNE visualizations as I find it more informative (but it is also fine like this!).

Insisting on these minor points:
I would rather not have footnote 2 and be more precise in the text.
I would also detail in the HP section of the appendix HOW hyperparemeters were selected and not only which ones were selected (even if the sentence just says that not much tuning was done, I think it is important.)

I am now recommending acceptance and raising my score to a 6 (it could be a 7 if it existed).
The work would be a clear 8 with an additional experiment on a more challenging and less toy-ish environment.
Yet, I understand it can be left for future work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to handle both continuous and discrete actions in reinforcement learning. Typical MDP suggests actions in either (not both) continuous or discrete action spaces, however in more general cases hybrid action spaces are needed. The proposed method solves this problem by using the latent space and the latent action. Instead of using raw actions, which are continuous and discrete, actions are embedded into a latent space, in which all are continuous. By decoding the latent action to actions in the original hybrid space, the whole process can be seamlessly integrated into any RL algorithms.
Experiments show that the proposed method method combined with TD3 and DDPG performs significantly better.

main_review: The concept of latent actions is very interesting. In the latent space, both discrete and continuous actions in the original space are represented by continuous vectors. This is based on the prior work on action representation learning for RL, but this work uses it to solve the difficulty of handling hybrid actions.

In addition, the decoder predicts the difference between current and next states (s and s'). This may be a small addition, but it looks reasonable to make the representations discriminative. This can be done in an unsupervised manner, so it doesn't introduce any additional cost of labeling.

This is how to deal with the hybrid actions, and it can be used in any RL methods by replacing actions with the proposed latent actions. This means that the method has a variety of potential applications.

A concern is that among the four factors in Table 1, stationarity is not explicitly discussed in the paper.

summary_of_the_review: Overall the paper is well organized and written, and the provided code ensures the reproducibility.
review_title: null

Meta-review: This paper proposes a new approach to solve mixed discrete-continuous action RL problems, based on embedding actions into a latent space so that standard continuous control algorithms (like TD3) can be applied. Experiments over standard discrete-continuous benchmarks demonstrate the superiority of the proposed approach vs. existing baselines.

There is overall a strong consensus of all reviewers towards acceptance, especially after the discussion period where the authors were able to submit several revisions addressing most of the questions and concerns raised in the original reviews, in particular w.r.t. the quality and relevance of the results.

I believe this submission could be improved along two axes though:

1. As pointed out by some reviewers, the current environments are somewhat simple. A more realistic robotic task for instance could be a good fit for such an algorithm. That being said, as the authors pointed out, this may require custom development due to the lack of existing public environment with the proper setup.

2. As a reviewer mentioned, there is no "Related Work" section, and although previous work is discussed in the Introduction, I consider that it remains limited, and more previous work should have been discussed. Here are some pointers regarding relevant work I am aware of:
- Hierarchical Approaches for Reinforcement Learning in Parameterized Action Space (https://arxiv.org/abs/1810.09656)
- Neural Ordinary Differential Equation Value Networks for Parametrized Action Spaces (https://openreview.net/forum?id=8WKd467B8H)
- Improving Action Branching for Deep Reinforcement Learning with A Multi-dimensional Hybrid Action Space (https://ipsj.ixsq.nii.ac.jp/ej/index.php?action=pages_view_main&active_action=repository_action_common_download&item_id=199976&item_no=1&attribute_id=1&file_no=1&page_id=13&block_id=8)
- Distributed Reinforcement Learning with Self-Play in Parameterized Action Space (https://cgdsss.github.io/pdf/SMC21_0324_MS.pdf)
- Discrete and Continuous Action Representation for Practical RL in Video Games (https://arxiv.org/abs/1912.11077)
- Multi-Pass Q-Networks for Deep Reinforcement Learning with Parameterised Action Spaces (https://arxiv.org/abs/1905.04388)

In particular, I believe the last one (MP-DQN) should have been one of the baselines, since it is supposed to be an improvement over the P-DQN algorithm (that is one of the baselines used here). I encourage the authors to try and incorporate it for the final version (at the very least, it should be cited).

In spite of these concerns, I still recommend acceptance since the combination of action space embedding with mixed discrete-continuous actions is novel and non-trivial, and the empirical validation is convincing enough in its current state.
Decision: Accept (Poster)