Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose NDRL, which is a class of methods to automatically discretize action spaces.  They identify issues with prior work and with their methods, and propose improvements to address these issues.  They analyze their methods empirically.
strength_and_weaknesses: **Edits:**
- introduction, “prior sets of discrete actions to from expert demonstrations”: grammar issue?
- introduction: “utilize graph neural network” should be “utilize a graph neural network”
- introduction: “and soft-argmax operation” should be “and a soft-argmax operation”
- “Generally, hybrid action space can be defined as a tuple:” add “a” before a “hybrid”
- “Then using the embedding vector z^e…”: grammar, what noun is doing the “using”?
- “The full pseudo-code of NDRL is provided in Algorithm 1”: this is in the supplementary material, so the reference to the pseudo-code should say “in supplementary material Section …”, to avoid making the reader look futilely around the main body of the paper for the algorithm.
- bottom of page 8: “All the detailed settings are shown in Appendix ?? and ?? respectively.”

**Strengths:**
- The ablation studies are nicely done and convincing.
- The authors appear to have chosen their hyperparameters in a principled manner: based on prior work, with little (potentially-unfair-to-baselines) tuning.  When they did tune, they explored those hyperparameters thoroughly, which further strengthens this work (Section A.5).

**Small clarity weaknesses:**
- Section 3, Hybrid Action Space, first ~5 sentences (through “to describe these basic nodes.”): This is a little hard to follow.  Upon several rereads of this part, I think I fully understand, but a more rigorous approach to defining this notation would make this part of the paper clearer and easier to read.
- psi, phi, and several other symbols were not defined.  Don’t make the reader guess or infer these definitions.

**Larger weaknesses (clarity and other):**
- Section 3, Vector Quantised Variational AutoEncoders: I found this difficult to read.  See the question below and the relevant edit above for ideas to make this part clearer.
- ”In Figure 1(a), we also illustrate the entropy of different action spaces to show the effectiveness of discretization”: A minor issue is there is no 1a subfigure.  A more significant issue: is the entropy being shown, as indicated in the quote above?  How so?  Is H entropy?  Was this defined?  What is x in this figure?  It also seems to be undefined.
- On a similar note, Figure 1 and its caption are very confusing.  In addition to the issues mentioned above:
    - x and y are undefined. (The x here does not seem to be the same as the also-undefined x which appears in the top part of the figure)
    - The sentences “HandCrafted is obtained…right booster will fire” do not make sense to me.  (These sentences seem to be describing a hand-crafted policy, but that does not seem to fit this context, which is talking about action spaces.)
- Section 4.2.1: The paper completely lost me here; the clarity needs improvement.  Some examples of things I am confused about from this section:
    - Comparing Figure 2 to Algorithm 1, the phases and terminology are different.  Is the “Collecting Phase” the same thing as “Stage 1”?  If so, another issue is that Figure 2 implies that data is only collected at this stage, but Algorithm 1 talks about using that data to train during this stage.
    - Figure 2: Shouldn’t there be an arrow from the k vector to the code table?
    - It is not clear (from the figure, the caption, or the text) where the collected data is going or how it is being used in the training phase (or even if it is being used in the training phase).
    - “All necessary data will be packed into a transition and put into buffers.”  I am not sure what a transition means in this context (maybe a {s, a, s’, r} tuple?), or where the buffers go, or how the data is subdivided into different buffers
    - It’s not clear to me how the decoder, the hybrid action space, and the selected action (a_t) interact.  (Although this could be related to the clarity issues from Section 3, rather than a problem with Section 4.2.1.) 
    - Does the collecting phase happen once (many episodes of data), and then the training phase happen after?  Or does the algorithm loop back and forth between the phases? (And if so, how often?  Once per episodes?  Once per timestep?  Some other interval?)  The answers to these questions are not clear from Figure 2 and Section 4.2.1.  Also, while Algorithm 1 may give answers to these questions, the relationships between the components of Algorithm 1 and the components of Figure 2 is not clear (discussed more above).
- Claim: “ADQ significantly outperforms DQN with manually discretized action space”. In RL, 5 seeds is not usually sufficient to make a claim like this, due to the large variance between runs.  This claim is not convincing, nor are several other claims in the paper based on a similarly-small number of seeds.

**Question:** “the decoder uses the code vector e^d”: should this be z^d?

clarity,_quality,_novelty_and_reproducibility: While the authors’ ideas make sense at a high level, I struggled to follow the details of their proposed algorithms (see specific clarity issues above).  This negatively affected the quality and reproducibility of the paper in my view.

summary_of_the_review: The authors study an important and interesting problem; however, the clarity of the paper could be improved.  There are also some empirical problems.  However, the empirical contribution overall was strong.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes using a VQ-VAE to map continuous actions to a discrete latent space. The paper then applies RL training on top of the discrete latent action space.
strength_and_weaknesses: Strength 

The proposed idea is intuitive and novel as far as I know,

Weakness

The writing at the beginning of the paper is quite clear, but the quality of writing deteriorates quickly as the paper progresses. I list a few examples below:

- what is the takeaways for Figure 1? The figure seems hastily prepared, and the caption does not fully explain what the readers should take away from the figure. Section 4.1.2 says that Figure 1 shows that well-designed action space leads to smaller entropy, but afaik, figure 1 does not discuss this at all.

- the writing in 4.1.1 is too informal. What is dirty work in raw action space?

- figure 3 is too small and I can not read the legend at all.

Also, if the proposed method is equivalent to TD3, then why do we need to use the method at all, since TD3 is relatively simple?

The paper also misses comparison to obvious baselines, such as https://arxiv.org/abs/1705.05035
clarity,_quality,_novelty_and_reproducibility: As I discussed before, the paper writing is clear in the beginning, but the quality decreases quickly in the body of the paper. The method is novel as far as I know, the motivation for using the method is not clear, and the paper also fails to compare to reasonable baselines. 
summary_of_the_review: I do not recommend acceptance due to the low quality of the writing and presentation (for example, as explained by my comments regarding figure 1).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a framework for the adaptive discretization of continuous-action spaces in an online manner, by employing a two-phase learning structure: (1) latent action space learning; (2) RL on the learned action space. 
strength_and_weaknesses: **Strengths:**

- The target problem is very important: To have a unified approach for dealing with continuous-action problems by adaptively discretizing them and applying discrete-action RL methods and importantly in an online manner (not from offline or expert data).

- The approach is reasonable but requires (somewhat justifiably) a few patches (e.g. ensemble learning to deal with overestimation over a non-stationary action space) 


**Weaknesses:**

- The paper is somewhat poorly written with an unclear flow at times and missing a good positioning/elaboration by connecting to many related works to better. 

- The size of the latent action spaces is limited to K, which is a priori determined. This is not a major issue but nonetheless could introduce statistical bias in performance. In fact, this bias could be potentially stronger than naive discretization (equidistance discretization). In naive discretization, the number of subactions per action dimension is fixed (say 5 subactions per dimension). Therefore, the size of the effective action space ends up increasing with increasing action dimensionality (i.e. size would be $5^N$ where $N$ is # of action dimensions). But the NDRL framework assumes a fixed effective action space size with $K$ latent actions, independent of the number of action dimensions. This can be manually adjusted in principle, but the underlying RL method should be equipped with a capacity to scale with action dimensionality as well. For instance, standard DQN doesn't have such a capacity, but Sequential DQN [1], Branching DQN [2], Amortized Q-Learning [3], and HGQN-r1 (DQN + action hypergraph networks) [4] have such capacity. 

- Comparison with a representative subset of scalable algorithms (such as HGQN-r1) using the naive equidistant discretization scheme would be critical to support the argument that naive discretization is not performant. Comparison with DQN (which doesn't have a generalization capacity over combinatorial action spaces) is not useful in determining whether adaptive discretization is critical over naive discretization.

- Comparison with a bang-bang or bang-off-bang discretization is needed to ensure the domains in experiments indeed require more complex learned action spaces.

- Analysis of what those learned latent actions correspond to would be needed. E.g., similar to that in Fig 7 of Tavakoli et al. (2021), can you show that in Walker2D, the latent actions are able to capture a relationship between the two hip joints? 

**References:**

[1] Metz et al. (2017) *Discrete sequential prediction of continuous actions for deep reinforcement learning*. arXiv.

[2] Tavakoli et al. (2018) *Action branching architectures for deep reinforcement learning*. AAAI.

[3] Van de Wiele (2020) *Q-learning in enormous action spaces via amortized approximate maximization*. arXiv.

[4] Tavakoli et al. (2021) *Learning to Represent Action Values as a Hypergraph on the Action Vertices*. ICLR.
clarity,_quality,_novelty_and_reproducibility: - The approach is novel and in my view quite justifiable. However, I think the experiments leave a lot to be desired. I think that the experiments do not necessarily depict an advantage for their adaptive discretization method over a naive discretization scheme (see my reasons in the above section).      

- There are missing references that would help depict a better picture of what subproblems the method is addressing. For instance, the problem of learning about synergies between different action dimensions has been studied by Tavakoli et al. (2021) [4], but the paper does not discuss this problem in detail and how not capturing such relationships could bias performance. I know that the authors refer to it but, in my opinion, to a reader unfamiliar with such problems it will not fully be clear exactly what the authors are referring to.

- Relationship/connection to *Action Hypergraph Networks* [4] is not discussed; e.g., "Therefore, we design a state-conditioned action encoder and decoder, and utilize graph neural network (Kipf & Welling, 2016) and soft-argmax operation Luvizon et al. (2019) to improve the capability of AD-VAE for the relationships between different action dimensions and boundary action values." Here, what is the role of GNNs in capturing such relationships? Isn't that similar to that captured by *action hypergraph networks*? And if they are related, why a graph topology should be preferred over a hypergraph formulation? A graph captures pairwise interactions/relationships, but what if a relationship exists on a higher-order combination (where a hypergraph could capture)?

- What do you mean by hybrid action spaces? I often see this term used to refer to action spaces that have discrete and continuous components. 
Statements like this are unclear to me: "*Xiong et al. (2018b); Fan et al. (2019b) propose some techniques to extract the relationship between different action dimensions, which is important in hybrid action spaces.*" This statement depends on your definition of hybrid action spaces. But based on what I understand of the term, this statement is invalid: the relationships are important regardless of whether the action space is hybrid or not.

- Branching DQN (BDQN) [2] reaches >3000 performance on Humanoid-v3 in 3M environment steps using only naive uniform-interval discretization, outperforming the reported performance for the proposed NDRL method with a learned discretization (refer to results here: [link](https://github.com/atavakol/action-hypergraph-networks/blob/main/data/images/physical_results.png)). Also, in Ant-v3, HGQN-r1 outperforms the proposed approach with a naive discretization, and BDQN is somewhat on par with the proposed approach again using the naive discretization.  
Not comparing to such agents has in my view led to a misinterpretation of the results of the Neural Discretization method.
Note that the aforementioned methods of BDQN and HGQN-r1 are super basic additions to DQN, without any need for such complex additions!

- I couldn't find any experiments for understanding the value of the size of latent action space (size of $K$) on performance. This is quite important in my view. On this line, seeing similar plots as that of Fig 4 (middle) but for a varying $K$ would be interesting.

- **Question:** Can there be a comparison between the optimal Q-function of LunarLander and the learned latent discrete-action partitions? This could help us understand: (1) How close to an optimal discretization are we getting based on what is our important objective (to learn the $Q^*$); (2) What would be the difference in learned discretizations if the system didn't capture inter-dimensional relationships?; (3) What aspects of the inter-dimensional relationships is the system potentially missing right now? 

- **Question:** If the AD-VAE training occurs before RL training, wouldn't this make RL training experience transitions invalid (as the underlying action that generated the experience would be different from the believed action by the agent)?


**Minor:**
You have used `citet` instead of `citep`. This really affects readability. E.g., "*[...] such as high-dimensional continuous control in robot manipulation Lillicrap et al. (2016) and [...]*". Here you need to use `citep` so you get *(Lillicrap et al., 2016)*. 

summary_of_the_review: See my comments above.

Overall, the idea is quite appealing. But to see its benefits, we'd need more experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an action representation learning method ( AD-VAE)  to learn compact latent action spaces to discretize the action spaces for reinforcement learning training. Furthermore, the paper proposes a few techniques (latent action remapping, ensemble) to mitigate the instability of AD-VAE while training with RL algorithms. The method is demonstrated with a Q-learning-based algorithm. The methods are evaluated on several continuous and hybrid action-space reinforcement learning tasks.
strength_and_weaknesses: Strength:
The problem of action discretization seems interesting as it allows the reuse of effective discrete action-based RL algorithms to continuous action tasks.
The choice of evaluation on high-dimension continuous tasks seems interesting.

Weakness (see details comments):
Important algorithm details need to be included, which hampers the evaluation of the paper's contribution.
Lack of evaluation to justify the effectiveness of the proposed method.
Overall, the paper needs more coherency in writing. In many places, the motivation and the methodology are tangled. 

clarity,_quality,_novelty_and_reproducibility: Detailed review:

The paper is somewhat clear, and the proposed method seems novel. However, in its current form, the quality of the method is hard to justify due to the missing details.

How is the TD3 ExpertCluster obtained (Figure 1)? It needs to be clarified why TD3 is chosen as the algorithm for continuous action space. In the context of DQN, DDPG seems a reasonable choice to show how the task can be solved using the continuous version.

It is unclear to me what is the main contribution of the paper. Is it a framework (NDRL) or a method AD-VAE for action representation (discretization?), or a better algorithm (ADQ) for the continuous high-dimensional task? 

If it’s a framework, then at least two algorithms need to be demonstrated to show that the framework is effective. As it is now, I can see DQN-based algorithm is incorporated.

For the action representation or discretization, an equivalent algorithm should be compared. For example, the proposed ADQ is compared with discrete-action DQN and continuous-action TD3. However, it is unclear if the action discretization is helpful from the comparison of ADQ vs. TD3. A reasonable comparison would use DDPG, a continuous action version of the DQN. 

A suggestion to the author(s) is to use PPO in evaluating the action discretization. PPO can be applied to discrete and continuous action spaces with minimal algorithmic changes.
In performance, the action representation should help in getting better performance compared to continuous action. However, if the performance remains the same, then that conflicts with the motivation of the paper, that is:
“However, the complexity of action spaces still prevents us from directly utilizing advanced RL algorithms to real-world scenarios, such as high-dimensional continuous control in robot manipulation.” 
Why do we still need discretization if we have a better continuous-action algorithm?


I am a bit confused with the use of Discrete Action Space (k^0, ..,k^{K-1}) in Figure 2. My understanding is that this is used for the remapping step in RL training. So how do you get this Discrete action space for a continuous control task? 
Algorithm 1 is mentioned, but the details discussion is missing. How are the k_t and a_t calculated in Algorithm 1 (Appendix) lines 18 and 20?

summary_of_the_review: Important algorithm details are missing, which hampers the evaluation of the paper's contribution. In addition, the evaluation lacks important experiments that can justify the proposed methods' effectiveness. Overall, the paper needs more coherency in writing. In many places, the motivation and the methodology are tangled, eventually hampering understanding of the methodological contribution.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the topic of building a unified action representations consisting of mixed continuous and discrete action spaces. Traditionally these hybrid action spaces required domain knowledge and hacks but the authors investigate a VQ-VAE based approach to represent any action type via a vector quantized encoder-decoder style architecture -- bringing all actions into a unified representation.

Since the semantics of action is learned over the course of training, the algorithm operates in two iterative phases: (1) learn the latent action space and then (2) use RL on top of the learned action space.

I think this is largely an open problem in RL, especially when its applied to the real world where the action spaces are large and hybrid. And this paper formulates the problem in an interesting manner that has the potential to scale well for many real world problems. 

However, the general consensus is that the paper needs more work before it is ready for a publication. Even after the rebuttal process, there were major questions left about clarity of the paper, proper baseline comparisons and explaining all the design choices made in experiments. I believe that this paper is too early in its life cycle and I would urge the authors to resubmit this paper with all the feedback received in this process -- I do believe that this is a promising solution to an important problem.
justification_for_why_not_higher_score: There were major questions left about clarity of the paper, proper baseline comparisons and explaining all the design choices made in experiments. 

I believe many questions are left unanswered. For instance, details about the training setting, architecture and baselines (e.g. MuJoCo-Gym-v3's Humanoid task referred to by one of the reviewers and its comparison to BDQ). The paper needs to be stronger on all fronts (clarity, experiments, writing, explaining decisions) before it is ready for publication.
Decision: Reject