Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors propose a new method for online reinforcement learning with continuous action spaces.  This method uses a VQ-VAE to learn a state-dependent discretization of the action space.  They then show that this discretization provides significant benefits to existing offline reinforcement learning techniques.
review: Strengths:
- The paper presents a novel solution to a challenging and practically useful problem
- The paper is clearly written and easy to understand.
- The method appears to provide substantial improvements over existing approaches.

Weaknesses:
- No D4RL experiments in the main paper.  The authors devote space to showing images from the D4RL benchmark (Figure 3) but not their actual results, which are instead in the appendix. 
- The average improvement in each category in D4RL hides an interesting observation that deserves more attention: SAQ appears to slightly harm performance for many individual tasks, but drastically improves performance in  others.  The locomotion tasks especially exhibit this phenomenon, where for most tasks, performance is actually harmed by SAQ, even though the average scores are propped up by very large gains in a few specific tasks.  A more thorough analysis here would be extremely useful.  In these tasks where SAQ hurts performance, what is the failure mode?  Are the continuous actions in successful rollouts not covered by the discretization, or is something else happening?  This may require more space than could fit in the main paper, but would be really great supplemental analysis.
- A comparision against AQUADEM's log-sum-exp discretization technique seems important here.  Even though, AQUADEM is designed for demonstrations+RL and SAQ is designed for offline RL, they both share the technique of pretraining a state-dependent discretization, and so it would be extremely useful to know which discretization technique works better here.

Typos:
- "re-visit once we have full results" at the end of the 1st paragraph of 5.2

Post rebuttal: I am keeping weak-accept at this time.  The other reviewers found some inconsistencies in the experiments, which the authors attempted to correct.  It seems like the others disagree about whether or not these issues were properly addressed.  The issues of the other reviewers are somewhat concerning, but to me there are enough interesting components and ideas here  that I don't think it's worth moving my score down.  I generally buy Aebd's argument in response to g7YW: "I think that your robomimic concerns are orthogonal to the main point of the paper. In my mind, the point of the paper is to compare base algorithms (e.g. CQL, BC, IQL) with exactly the same algorithms plus SAQ. While they could tune hyperparameters for each domain like robomimic, it seems like a cleaner comparison between standard CQL and CQL+SAQ to use the simpler hyperparameter tuning strategy used in this paper. Moreover, if the hyperparameter tuning were that extensive, it would be the same for CQL+SAQ."
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper demonstrates new practical tools for addressing important problems in the learning and robotics communities.  If I were starting a new project in this area in the near future, I would want to read this paper.

Post Rebuttal: see notes in main review above.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This proposes that offline RL methods like CQL, IQL, and BRAC employ a sampling step to optimize the constraints when the action space is continuous, making their performance poor. Thus, this paper discretizes the action space through state-conditioned reconstruction of action using a VQ-VAE (thus ensuring a small action space). The discretized versions of the above algorithms perform better than their continuous counterparts.
review: Overall, I liked the paper as it is interesting, clearly written, and shows significant improvements over the baselines. But, it is missing some analyses to understand the core issue in the continuous action spaces and why the discretization is performing much better.

# Strengths
- Clearly written, well-explained and the transformation of continuous to discrete versions of the algorithms is clear.
- The idea of using state-conditioned VQ-VAE to discretize the action space is quite interesting and it's nice to see it works well too in practice.
- The results often show a significant gain over the compared continuous action space counterparts. The results seem very strong.


# Weaknesses
My main concern is that the paper is **missing qualitative and quantitative investigation of the contributions of this paper**: 1. the use of VQ-VAE to discretize and 2. discrete v/s continuous action space in offline RL:

## VQ-VAE for discretization
Some implementation details and crucial analyses are missing:
- I wonder how difficult or unstable the training of VQ-VAE was, and is there any plan to open-source the code? What was the size of the codebook? Could the authors provide more training details, especially those pertaining to this paper's contributions.
- Is the discrete codebook sufficiently able to recover all the action reconstructions? What is the performance of the action reconstruction?
- What exactly is the role of the codebook and the VQ-VAE decoder? One trivial solution is that the VQ-VAE decoder learns to disregard the codebook and simply learns a BC policy $f(a | s, e_k) \sim f_{BC}(a | s)$. Since, the decoder essentially does BC and the other offline-RL methods now use this decoder implicitly, it is possible that their performance improves more on expert dataset, because of this inherent bias towards BC, by assuming that the data is good for imitation. I am not at all claiming that this happens, but understanding what is the learned discrete action space, codebook, and VQ-VAE encoder-decoder seems important to justifying where the performance gains come from.
- **Other discretization approaches**: The paper mentions several action discretization works [21-28] which either assume independence of action dimensions or model the action space autoregressively, and the claim is that naive discretization can result in exponential blowup or imprecise actions. While this makes sense intuitively, since the task is offline RL, the challenge of exploration is not there. This means that there is a chance that even with such a discrete action space, learning is feasible and at least easier than the continuous action space counterpart — for instance, if the discretized action space is quite large, then the sampling versions of the offline RL algorithms can still be applied. Nevertheless, since VQ-VAE as the discretization scheme is one of the core contributions, it is important to compare against alternate action discretization schemes to disentangle the roles of discretization scheme and the question of discrete v/s continuous offline RL optimization. I do appreciate the ablation of unconditioned VQ-VAE, but that only explores one style of discretization: reconstruction with VQ-VAE. It would be nice to see the effect of some other methods such as [23, 24, 25].

## Discrete v/s continuous action space for offline RL
It is great to see the improvement with the discretization, but the current experiments do not convincingly show that the benefit is actually because of replacing the approximate calculations with exact versions, or due to other artifacts of discrete action spaces, such as the ease of multi-modal modeling of data.
- Especially, the result of BC v/s SAQ-BC hints at the fact that there are major improvements even in the absence of any approximation made in the continuous action space.
	+ It's not even clear that multimodality is the core reason of improvement, because the SAQ-BC v/s BC gains are more significant in expert datasets (especially, halfcheetah and hopper) than medium datasets. It is unclear to me why SAQ-BC (92.63) would be so much better than BC (28.49) when the dataset is expert.
- In Fig 2, at around 1200 grad steps in continuous CQL achieves optimal performance despite the estimated R($\theta$) already diverging from the exact value. What explains this success despite the divergence? It would hint that even continuous action space offline RL can reach optimal performance despite the approximations over the constraints.
	+ Also, if instability only arises post optimality, then wouldn't early stopping be a simple solution to continuous CQL's issues?
	+ Can the authors provide the learning curves for all the results computed to observe the learning dynamics? Since the key claim is the poor optimization due to approximations in continuous action spaces, it is important to observe the performance over the course of training.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, I liked the paper as it is interesting, clearly written, and shows significant improvements over the baselines. But, it is missing some analyses to understand the core issue in the continuous action spaces and why the discretization is performing much better. The current hypothesis that continuous action space offline RL suffers due to approximations in the estimation of the constraints is not empirically justified well. In this state, I cannot recommend acceptance despite the strong results the paper reports. I would be more than happy to reconsider my position if provided with more information such as implementation details, training curves, hyperparameter tuning approach, the author's answers, and analysis experiments.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper presents state-conditioned action quantization (SAQ) a method for discretizing action spaces with a state-conditioned VQ-VAE. The method can be added on top of any standard offline RL algorithm to simplify the action space. Experiments are presented on a variety of base algorithms and tasks from D4RL and robomimic that show substantial improvements in performance by providing a simpler action space with a strong prior for actions in the dataset.
review: ## Strengths

1. SAQ is a simple and general method that can be applied on top of any base algorithm for offline RL. This makes its applicability potentially large and impactful.  

2. The empirical results are clear and show substantial improvements in performance. These improvements seem to hold across most environments, but are especially strong on high dimensional tasks with more realistic demonstration data like robomimic, underscoring the potential importance for the CoRL audience. 

## Weaknesses

1. The idea is not entirely original since Dadashi et al (cited in the paper as [29]) propose a very similar method for online RL. However, the paper provides this reference and builds on the idea to extend it into offline RL with a VQ-VAE.

2. Section 4.3 provides an explanation of why SAQ is helpful on top of CQL, but leaves a few lingering questions. (1) While it is clear that CQL makes an approximation with continuous actions, this explanation does not as cleanly extend to IQL or BC, but these methods also benefit from SAQ. Thus, it seems that this explanation may not be complete. (2) It is unclear how the exact CQL penalty is calculated. Is it just by taking many more samples so the estimate concentrates around the true value?
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper presents a simple method that provides clear cut improvements over baseline methods and seems to be most effective on tasks that are most applicable to robotics. I think the paper is a good contribution and should be accepted.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents a method to learn a quantized action representation for use in continuous action offline reinforcement learning approaches. The method is to train a state-conditioned VQ-VAE to learn a set of discrete latent actions, which can be reconstructed by a state-conditioned decoder. Several offline RL methods are then adapted to use discrete actions instead of continuous actions. The action representation and discrete variants of the offline RL methods are evaluated against their continuous counterparts on two benchmarks - D4RL and robomimic, and shown to increase performance.
review: **Strengths**

This paper presents an interesting approach to learn action discretization and presents useful empirical evidence showing that the discrete offline RL method variants outperform their continuous counterparts, sometimes significantly. Furthermore, this is shown across 3 different methods (CQL, IQL, and BRAC). As an added benefit, some of the modifications to the offline RL methods seem to simplify their implementation and allows for computing quantities exactly, due to the use of discrete actions. 

**Weaknesses**

There are some issues with the experimental evaluation and the writing and clarity. Comments related to the experiments follow:

- The paper only compares the proposed method against the continuous action variants of the offline RL methods. However, there is limited analysis on whether the choice of VQ-VAE is important for the method to work. The VQ-VAE appears to be a crucial choice compared to other ways to discretize the actions, but that isn't really shown. The appendix does have some experiments showing that the state-conditioning for the VQ-VAE is important, and that the codebook size does not matter much. However, there are no comparisons to other discretization methods. Some other choices of discretization include the AQuaDem paper (https://arxiv.org/abs/2110.10149), which is highly related, and acknowledged in the related work section. It seems as though their discretization learning method could be applied to offline RL (even though they apply it to online RL). Other alternatives include using a categorical VAE with gumbel-softmax trick (https://arxiv.org/abs/1611.01144), and potentially learning a state-conditioned Gaussian Mixture Model (as in the robomimic paper, https://arxiv.org/pdf/2108.03298.pdf) and using the mode centers.

- Some of the robomimic results are questionable. Looking at Table 1 in the robomimic paper (https://arxiv.org/pdf/2108.03298.pdf), CQL obtains 92.7 and 38.0 respectively on the Lift (PH) and Can (PH) datasets, while the implementation in this paper achieves 64.2 and 19.6. The BC results in this paper are also significantly lower than the ones reported in the robomimic paper (e.g. Lift 100 vs. 59.5, Can 95.3 vs. 31.7, Square 78.7 vs. 19.3, Transport 17.3 vs. 0.3, and Tool Hang 29.3 vs. 1.87). Given that the implementation is open-source, it is hard to understand why these results are so much lower. 

Some of the writing should be edited to improve clarity. Comments related to writing and clarify follow:

- The preliminaries section could be shortened - there are many details for CQL, BRAC, and IQL presented, and they are not all relevant to understand the core contribution of the method (using VQ-VAE to learn action discretization). The changes to each method are also highlighted in section 4.2 in a somewhat self-contained manner. It could be useful to make more explicit connections to the preliminaries section or just present all of the relevant details directly in section 4.2 and omit the sections in the preliminaries on these methods.

- Section 4.1 could be explained a little more clearly and precisely. For example, what is meant by single dimensional latent code? It most likely corresponds to K vectors in the codebook with D=1 (using the notation from the preliminaries section). The preliminaries section seems to have redundant information (for example equation 6 vs. equation 7 in this section). It would be better to make explicit connections to the preliminaries section to identify how the general VQ-VAE formulation was modified. This seems to be the core part of the method, so making it extremely clear is very important.

- The writing in Section 4.2 could also highlight the advantages of the use of discrete actions instead of continuous actions a little more. For example, it wasn't immediately clear that the CQL conservatism loss becomes an exact sum instead of an approximate sample-based integral. This would also help motivate the use of discrete actions over continuous actions more.

- Related to the point above, it might also help to provide some more intuition about the state-conditioned VQ-VAE. While the number of discrete latent actions is fixed, the interpretation of these actions depends on the state. In this sense, these latent codes could probably be thought of as modes of a conditional state-action distribution, similar to a Gaussian Mixture Model (GMM) or a categorical VAE. Given this choice, it seems a natural point of comparison could be to a GMM or categorical VAE (as mentioned in a comment above on adding more comparisons).

- It would be nice to see more useful analysis in section 5.1 and section 5.2 of the results in Table 1 and 2, with references to particular comparisons in the tables. For example, which improvements were the most significant and which were the least? Was this to be expected?

- More details on the specific implementation details should be provided. For example, what kinds of networks and hyperparameters were used? Did the BC baseline use a simple L2 loss, and how else did it differ from the one in the robomimic paper? How was the SAQ-BC model trained? Were the actions labeled into discrete codes and used to train a network to map states to discrete codes via a cross-entropy loss?

Some typos and potential mistakes:
- line 79 "actin"
- line 259 "re-visit once we have full results"
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Currently, the issues with the experimental evaluation in the paper outweigh the positives. To summarize, the issues include not showing the value of the VQ-VAE compared to other discretization schemes and the poor results on the robomimic benchmark compared to the ones reported there.

Post-rebuttal: After the author rebuttal and the discussion, I have decided to raise my score to weak accept.
review_title: null

Meta-review: This paper proposes action-discretization based on VQ-VAE to improve offline RL algorithms. The proposed method was combined with offline RL methods such as IQL and CQL, and the experimental results show that the proposed action discretization improves the performance of offline RL methods.

While the reviewers pointed out several unclear points, the authors addressed them in the rebuttal. Based on the overall assessment, AE recommends the acceptance of the paper. 
However, Reviewer Pinn pointed out that the authors do not provide the learning curves of the proposed method. Please consider adding some learning curves and further analyzing what happens in the proposed method.
Decision: Accept (Poster)