Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning
Authors: Jianlan Luo
Perry Dong
Jeffrey Wu
Aviral Kumar
Xinyang Geng
Sergey Levine
Venue: CoRL 2023 Poster (2023)
Keywords: Offline Reinforcement Learning
Discretization
Robot Skill Learning

Abstract: The offline reinforcement learning (RL) paradigm provides a general recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. While policy constraints, conservatism, and other methods for mitigating distributional shifts have made offline reinforcement learning more effective, the continuous action setting often necessitates various approximations for applying these techniques. Many of these challenges are greatly alleviated in discrete action settings, where offline RL constraints and regularizers can often be computed more precisely or even exactly. In this paper, we propose an adaptive scheme for action quantization. We use a VQ-VAE to learn state- conditioned action quantization, avoiding the exponential blowup that comes with naïve discretization of the action space. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme. We further validate our approach on a set of challenging long-horizon complex robotic manipulation tasks in the Robomimic environment, where our discretized offline RL algorithms are able to improve upon their continuous counterparts by 2-3x. Our project page is at saqrl.github.io

Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning
Jianlan Luo Perry Dong Jeffrey Wu Aviral Kumar Xinyang Geng Sergey Levine University of California, Berkeley
The offline reinforcement learning (RL) paradigm provides general recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. While policy constraints, conservatism, and other methods for mitigating distributional shifts have made offline reinforcement learning more effective, the continuous action setting often necessitates various approximations for applying these techniques. Many of these challenges greatly alleviated in discrete action settings, where offline RI. constraints and regularizers can often be computed more precisely or even exactly. In this paper, we propose an adaptive scheme for action quantization. We use a VQ-VAE to learn state- conditioned action quantization, avoiding the exponential blowup that comes with naïve discretization the action space. We show that several state-of tatte-of-thh-aae the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme. We further validate approach on set of challenging long- nonghooizzon complex robotic manipulation tasks in the Robomimic environment, where our discretized offline RL algorithms are able to improve upon their continuous counterparts by 2-3x. Our project page is at saqrl. -iggittii github. io
Abstract:
Keywords: Offline Reinforcement Learning, Discretization
Introduction
Offline reinforcement learning (RL) aims to learn policies from static databases of previously collected experience. As opposed to pure imitation, offline RI. holds the promise to transform logged datasets into policies that perform better than the behavior policy that collected the dataset by maximizing task rewards. The key challenge in offline RL is the overestimation of the value actions that were not seen the dataset, which destabilizes training and leads to policies that perform much worse than their value function estimates would suggest. To address this issue, wide variety of methods have been proposed recently [1, 3, 4, 5J. Typically, these methods employ some mechanism stay "close" to behavior policies, such as policy constraints value conservatism.
While these methods in principle address overestimation and distributional shift, practice im- plementing these approaches requires various approximations (depending on the method) that can lead to hyperparameter sensitivity or performance that is worse than their theoretical formulation might suggest. This issue particularly pronounced on "narrow" datasets that consist of relatively deterministic behavior, such the demonstration data that is often used in robotic learning. Our key observation is that these issues can be mitigated by properly discretizing continuous action spaces, and then employing discrete action versions of these methods, where many of these approxima- tions are unnecessary. For example, policy constraints or any conservatism regularizer that requires expectations over actions can be computed exactly with discrete actions.
Unfortunately, naïvely discretizing the action space result in exponential blowup in the number of actions, while coarse adaptive discretization methods can lead imprecise actions. In this paper, we propose an adaptive scheme, state-conditioned action quantization (SAQ), for discretizaing
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

continuous action spaces. We perform cantiioond action discretization by utilizing a VQ-VAE model. SAQ is based on simple insight: given a particular dataset, there are often only a few in-distribution options available in each state, corresponding roughly to the "primitives" that are supported under the data. This allows us to use comparatively very small discrete action spaces, without suffering from the curse of dimensionality, while still enjoying the benefits of discrete action spaces and simpler offline RL algorithm implementations.
The main contribution of this work is a practical approach, SAQ, for learning quantized action representations for improving continuous-action offline RI. methods on variety of robotic learning tasks. We present a general method for learning condicondii action discretizations and then apply this method with three offline RL methods: conservative Q-learning (CQL) [3], implicit Q-learning (IQL) [4], and behavior regularized actor-critic (BRAC) [1]. All of these methods require some sort of approximation with continuous actions, while the discrete version provides for convenient implementation that avoids such approximations, We find that the discrete version of each method implemented with SAQ generally yields improved performance on commonly used benchmark tasks over each method's continuous-action counterpart, particularly on "narrow" datasets (e.g.. expert data). We also evaluate these methods on a set of challenging robotic manipulation tasks from the Robomimic environment [6], where continuous offline RL methods struggled to get good performance, and find that our approach outperforms both prior offline RL methods in this setting by large margin.
2 Related Work
Offline RL. Some offline RL methods use a policy constraint to mitigate overestimation from distributional shift, constraining the learned policy to stay close the data via density modeling or some other divergence measure [1, 2, 7, 8, 9, 10, 11, 12, 13, 4, 5, 14). Another line of work directly refer as conservative value function methods [3, 15, 16, 17, 18, 19, 20]. While both types of methods can be formulated in ways that in principle address distributional shift, in practice they require some kind of approximation to be applied with continuous actions, either in estimating the behavior policy, or in computing integrals or expectations over the action space to formulate pessimism penalties. This introduces approximation errors and complexity. We show that our approach can avoid the need for such approximations for three representative methods from both categories, leading improved performance.
regularizes Q values of unseen actions, which
Discretizing continuous action space for control. Recent work has shown discretizing continuous actions can yield good performance, and several discretization strategies have been introduced. To address the issue of exponential growth of actions in the naïve discretization scheme, one line of methods assumes independence of action dimensions [21, 22, 23, 24, 25, 26). or perform discretization in autoregressive way [27, 28, 29]. Our work is different in that we perform state-dependant discretization, which adaptively controls the precision of the discretization scheme. Perhaps the closest to our work is Dadashi et al. [30]. which also learns state-dependant discretization. It assumes access to human demonstration dataset. Given the current state, it uses neural network to index a set of plausible continuous actions and trains the network by comparing its output with the demonstration data. Our work differs in that we focus on the offline RL setting, where discretization could enforce conservatism and policy constraints exactly. To our knowledge, our work the first to propose adaptive action discretization with VQ-VAEs for offline RL.
3 Preliminaries
The RL problem is formally defined by a Markov decision processes (MDPs) M = (S, Po. 7). where S and denote the state and action spaces, and 7 [ss'|s, a), r(s, a) represent the dynamics and reward function, respectively. no (s) denotes the initial state distribution, and 7 E (0, 1) denotes the discount factor. The objective of RL is to learn a policy that maximizes the return (discounted sum of
rewards): max; J (TT) ;= (( Jova EE, (s,,, at)]. In offline RL, we provided with offline

---

dataset D = ((s, a, r,))} of transitions collected using a behavior policy and our goal is to find the best possible policy only using the given dataset, without any additional online data collection. In this paper, we focus on three offlin RL methods, conservative Q-learning (CQL), implicit Q-laarning (IQL), and behavior regularized actor-critic(BRAAC); though our approach could likely extended to other methods also.
Conservative Q-learning, Naïvely learning Q-value function from the offline dataset (e.g., via Q-learning or FQI) suffers from OOD actions [31, 2, 32], and the CQL algorithm [3] applies regularizer R(0) to prevent querying the target Q-function on unseen actions. R(0) minimizes the Q-values under the policy (als), and counterbalances this term by maximizing the values of the actions D. Formally, we have the following objective during Bellman backup:
min E a.a.a'D a/-xx
[[0o(s, (r- +
[Qo(s,
E s,a~D
(Q0(s,a))) (1)
where Q denotes the target Q-function.
The specific version BRAC-v explicitly subtracts the divergence (TT(-(s)) <-(sss)) from the target value while performing the Bellman update. Additionally, since the divergence between the learned policy and the behavior policy at the current state is not part of the Q-function, BRAC-v also explicitly adds the divergence value the current state the policy update. We instantiate the version of BRAC that uses the KL-divergence:
Behavior regularized actor-critic.
(T mT(-(s) (aaa)(-)s)) [log (als) log (als))
To estimate the second term log (a(s). BRAC trains an additional behavior policy, that we denote a-. Denoting the policy and the Q-function Ta and Qe. the BRAC-v perform following Bellman update:
min EADD (r(s, + (19) + Blog a (o(s.a))
(3)
then extracts policy by:
max
1Qo(s, Blog (a(ss)) - a log (a|s)]
Implicit Q-Learning. Instead of applying policy constraints or critic regularizations, IQL uses expectile regression learn value function approximate an expectile т over the distribution of actions by optimizing the value objective
(v) T (aa) (15a)) V=))))
where (s, a) is parameterized target critic and L5(u) = - I(u < O)lu². This value function then used update the Q-function with TD enror following the objective function
(Q = E a) + (s') (s a))
With the value function V& and f-fucctoon Q5 a), the policy is learned using advantage weighted regression following the loss below:
L= = Eisa)-nlexp(a(0/16 - Vs(s)) log T (a|s)).
Vector quantized variational autoencoders. Our method uses the VQ-VAE [33] as part of the discretization process. The VQ-VAE consists of an encoder and a decoder. The encoder network parameterizes a posterior distribution (2(2) over discrete latent variables & given the input data æ. The decoder that parameterizes the distribution po (a|z) then reconstructs the observations from these discrete variables. Specifically, the encoder first outputs continuous embedding eœ( RP Discretization is done with codebook defined on R xD composed of K vectors e, E R.,, where j 2, K. The embedding (s(z) is compared with all the vectors in the codebook, and the

---

VO-VE
Breoder
Dccoder
a (1...K)
VQ-VAE
Policy
Decoder
a (I...
Figure 1: Left: Training SAQ. Right: Discrete policy training/evaluation with SAQ. SAQ learns scalar discrete representation of continuous actions using condiccondi VQ- VAE. During policy training and evaluation time, we use the decoder of SAQ to reconstruct the continuous actions from discrete policy actions.
discrete state chosen to correspond to the nearest codebook entry. The discrete posterior is then defined as:
(1,
if = arg min, min))le(()) - =eil2
(= = k(x)
(5)
0,
otherwise
Let's denote Zgo (z) the final latent code from the encoder, i.e., Zqo (x) ex. The decoder will reconstruct the original input x based on 2g,, (z). Additionally, a codebook loss and commitment loss are added to move codebook vectors and encoder embeddings close to each other. The overall training objective is as follows:
T.(0, 0))
E
pog| (x)) + [sg[((((r)) ell2 + (|e((x) - sg(e]11
(6)
cE
where sg stands for the stop gradient operator. We can then optimize the parameters of the encoder and decoder with this objective.
4 State Conditioned Action Quantization for Offline RL
The central premise of this paper is that carefully and efficiently discretizing the action space of continuous-action RL problem can make it significantly simpler to implement offline RL methods based on critic or actor regularization, and that such discretized methods can attain better performance on especially difficult data distributions, such as narrow datasets. How should we discretize continuous action space? Discretizing too coarsely can make it difficult to perform delicate tasks that require fine control, and can collapse in-distribution and -uttr--distributtinn actions into the same bin, making offline RL more difficult. On the other hand, naïvely using very fine discretization can be intractable due the curse of dimensionality. Our key observation is that can construct discretization with relatively few discrete actions but with minimal loss of resolution if we employ learned Hate-conditionel discretization, which we can accomplish by means of a VQ-VAE model. In this section, we will describe this approach, and discuss how it can be combined with simple and effective discrete-action offline RL methods.
4.1 State-Conditioned Action Discretization via Scalar-Quantize Auto-Encodes
Abstractly, we aim to learn a state-conditionel action discretization (SAQ) scheme that can map a continuous action given state to discrete variable à for training, and then given choice of the discrete variable à also map it back to the original action space A for evaluation.
Given this requirement, a natural choice is to utilize auto-encoders that can produce state-conditioned discrete codes for a given action. Therefore, our approach adapts the rector-quantizer variational auto-encoder (VQ-VAEs) [33] framework for performing action discretization, with the difference that unlike conventional VQ-VAEs that typically learn multi-dimennional discrete latent code for an input, only aim to learn scalar discrete latent code for given input action as this is sufficient for performing downstream RL.

---

More formally, denoting the parameters of the VQ-VAE encoder with and the decoder with 6, we wish to learn an encoder ( (-1,, a) that produces a one-dimensional discrete latent action, and a decoder to map the discrete action code back to the original action space, po(a|s s, a). Following the training procedure
TQ-V,Es, we first obtaín latent embedding zq. (s, a) from the approximate posterior q6 (-[ss, a) and then compare it within the codebook, and obtain the nearest vector eg. Next, this latent action code through the decoder to obtain approximate reconstructed action, à p(-|s, ek), which is further trained to minimize the reconstruction emror against the original action a passed as input to the encoder. Following the training procedure proposed by Van den Oord et al. [33], the overall training objective for SAQ is shown in Equation 7, where sg is the operator of stopping gradient, D is the offline dataset:
|- loggoe po Ege a)) + (| a)] -e -ell ||ea(s. - sg[elll
min 0,
E 8,aD eE 11,,,3...
(7)
4.2
Offline RL with Discretized Actions
After training VQ-VAE to obtain discrete action representation, can transform the original problem with continuous action into a discrete problem by learning policies and value functions in the discrete action space.
SAQ-CQL. For CQL, we follow the discrete variant of CQL [3], where we learn the Q-function with a conservatism penalty and parameterize the policy as π (als) xx Q(s, a). Specifically, we employ the maximum entropy variant of CQL and replace the sampled log-integral-exp with log- sum-exp by summing over all discrete actions. This allows us compute the CQL conservatism loss exactly, without having to rely on the samples from policy to estimate the integral of Q-function. Our overall objective of SAQ-CQL is therefore
E [(Qo(s,a) ( + E s.a.s' WD a
log Σ exp(Qo(s, Qa a)
min 2
s,aDD
We note that in this specific variant of discrete CQL, the conservative penalty term Ex [log exp(Qo(s (0))s,)))) Qo(s, (o(s.a) ā)] is exactly equivalent to the discrete negative log- likelihood behavioral cloning loss:
log exp(Qu(s,@)
log Σ exp((Qo(s, @s)
E s,
Σ, exp(Qo as))
8,0DD
E 8,a~D
[log (ā|s)]
For IQL, we follow the original problem formulation in Kostrikov et al. [4]. then derive its closed-form solution in the discrete case. We denote the advantage as A (s, a) Q" V*) for a given policy т, which can be obtained by the same procedure detailed in Kostrikov et al. [4]. The policy extraction step IQL. objective find a policy n' so that A (s, a) can be maximized while staying close to behavior policy Tg.
SAQ-IQL.
n arg max Ra (s) (A* a)] s.t.
DEL mp <.
We can solve this constrained optimization in Eq. 9 by using the Lagrangian method, and the solution given by:
T" (als) exp - As. + log ng (a|s))
(10)
where À is the Lagrangian multiplier, which controls the amount of constraint deviation. We refer readers Appendix A for detailed derivation. In practice, obtain πg by training additional BC policy on the behavior dataset.
5

---

SAQ-BRAC For SAQ-BRAC, we first learn categorical cescrete-action behavior policy x (a|s) on the discretized actions by maximizing the log-likelihood of the offline dataset. We then implement discrete BRAC by exactly following Eq. 3 and Eq. 4, where we compute the KI-divergence between the learned policy and behavior policy by simply summing over all discrete actions.
4.3 Diagnosing offiine RL performance with constraint enforcement
As conjectured in Sec. 1, we associate inexact constraint enforcement with the degradation of performance of offline RL methods. To verify this hypothesis, in this section, we conduct an experiment in a pointmass navigation environment (maze2d-large from Fu et al. [34]), where the RI. agent controls a point mass to navigate through the maze from a fixed starting point to the goal point, as shown on the left side of Figure We collect three optimal demonstration trajectories as the offline dataset and both continuous CQL and SAQ-CQL this domain. We visualize the average return in the middle as well the sample estimated (shown in blue) and the exact CQL conservatism penalty (shown in green) on the right of Figure 2. We observe that the performance of continuous CQL rapidly degrades with more training steps, with the estimated conservative penalty R(Ø) diverging from the exact value in the period where the performance degrades. The error in estimation causes the policy training to overly penalize the wrong action samples, resulting in degraded performance. On the contrary, SAQ-CQL computes the penalty exactly, which enables the optimizer to smoothly minimize the penalty, leading to robust policy performance. While it may be that with more hy hyperparameter tuning the baseline continuous-actio performance in this case corresponds to the inexact estimate of the CQL regularizer, which is largely mitigated by our discretization approach.
version would perform better, the results suggest that poor
1.0
concenantie perelty 2103
20.0
WAl Lastiriaal Eq-eeinane m1
7.5
5.0
2.5-
25
0.4-
0.2-
Corkrruuu COL
3.0
Disrrte COL
4050
5000
Grodient
shess
the average return of SAQ-CQL and continuous CQL over the course of offline training. Right: estimated and exact CQL conservatism penalty values. The estimated conservative penalty for continuous-action CQL diverges significantly from the true value during training, resulting in rapid performance degradation, while the SAQ-CQL enjoys stable training by optimizing the exact penalty value. As a result, the training process (middle) significantly more stable with SAQ, which we expect should make the algorithm easier to use, simplifying checkpoint selection and tuning.
Figure 2: Left: visualization of the maze2d environment. Middle:
5 Experiments
Our experiments compare three continuous action offline RL methods to their discretized variants instantiated with SAQ, both on standard offline RL benchmarks and the Robomimic robotic manipu- lation environment [6]. We combine SAQ with: CQL [3], IQL [5], and BRAC [1]. We first evaluate on the D4RL [35] suite of tasks, and then the Robomimic tasks [6), which prior work has found to present particular challenge for offline RL methods. Appendix B presents detailed analysis and ablation study of SAQ to understand how it affects offline RL training, which we encourage readers to examine for more detailed study of the method. We also present a comparison with Aquadem [30] in Appendix C.
5.1 D4RL Benchmark Evaluations
We present the results the D4RL benchmark suite [34] in Table 5, with illustrations of the evaluation domains shown in Figure 3. Some of the benchmark tasks consist of narrow data distributions, while

---

Figure 3: D4RI. benchmark tasks: locomotion, antmaze, adroit and kitchen.
Robomimic tasks: lift, can, square, tool-hang, transport. Among these tasks, square features multi-modal behavior, tool-hang requires high-precision manipulation, transport is particularly challenging because its long-horizon complex multi-modal nature.
Figure 4:
others contain high-coverage data. For example, the "kitchen-" domains consist of data from expert human teleoperators controlling a robotic arm to perform a variety of manipulation skills, which must then be sequenced and recombined by the algorithm to solve specific tasks. The "-expert" and "-mediun-expprr"" datasets for the locomotion tasks also contain narrow expert data, possibly in combination with broad but suboptimal data, with the "-meddum-eeper"" version presenting particular challenge in terms of picking out the narrow but nigh-performing expert mode from the noisier overall distribution. The "adroit" dexterous manipulation tasks also contain narrow expert demonstration data from humans controlling the simulated robotic hand with a data glove. The narrow datasets are particularly challenging for prior continuous action methods, as the narrow behavior policies exacerbate challenges due to imperfect approximations for policy constraints and conservative regularizers. This is particularly pronounced for nigh-dimensiona domains, such as the 24-DoF "adroit" robotic hand, where methods that require integrals expectations over the action space (such as the regularizer in CQL) must sample and average over high- -ith--iinnsionlll action space, We see that for each algorithm (BRAC, IQL, and CQL), the version of each algorithm discretized with SAQ improves with respect to the average score in each of the domain types (locomotion, antmaze, adroit, and kitchen). Although some cases the improvement is small, in other cases very significant, particularly for narrow-data adroit and kitchen tasks, and the "-medium imedium-expert" versions of the halfcheetah and hopper task. We hypothesize that SAQ performs well in these domains because the discretization capture the individual (narrow) modes, while the discrete-action RL algorithm then select from among these modes to attain the best performance.
While SAQ can lead to significant improvements over the continuous-action counterpart of each method, the main benefit SAQ is not in raw performance, but in terms of the simplification of the downstream RL problem. Our experiments provide an "apples to apples" comparison for each RL method, comparing its discrete (SAQ) version to its original continuous formulation, but of course, other offline RI. methods might perform better some tasks, and do not aim to show that SAQ achieves Kfate-of-the-att performance over all possible offline RL techniques. However, we expect that the simple discrete-action RL problem that is presented via our proposed discretization will in the long run make it casier to scale offline RL to harder and more complex problem domains, and provide a valuable tool to the RL practitioner. RL methods tend to be complex and difficult to tune, so any improvement that simplifies the RL part of the problem likely to improve practical utility.
5.2 Robomimic Evaluation
Robomimic [6] is set of environments and demonstration datasets that require controlling 7-DoF robot arm to perform variety of manipulation tasks. Prior work [6] reported that these environments particularly challenging for offline RL algorithms, with the best-reported results obtained by
7

---

Task
BRAC
SAQ-BRAC
IQL
SAQ-IQL
CQL
SAQ-CQL
BC
SAQ-BC
locomotion avg
64.16
66.98
75.14
76.67
75.6
77.35
38.86
60.77
antmaze avg
18.84
29.5
55.34
55.92
55.15
56.87
0
adroit avg 23.86
40.09
20.32
22.82
10.58
21.37
12.35
21
kitchen avg
15.78
36.11
50.83
58.61
48.67
65.89
17.22
57
Table 1: Averaged normalized scores across locomotion, Adroit, AntMaze, and kitchen domains from D4RL. The version of each algorithm discretized with SAQ generally improves over the average score of the original algorithm in each class of domains, with particularly pronounced improvements on narrow dataset domains such as adroit and kitchen. The full results are deferred to Appendix
Task
IQL
SAQ- IQL
CQL
Robomimic
SAQ- CQL
BC
Robomimic
SAQ-
CQL
BC
BC
lift
58
90
64.2
92.7
90.8
59.47
100
90.13
can
33.73
19.6
38
71.2
31.73
95.3
66.4
square
26.93
46.67
0
5.3
44.27
19.33
78.7
45.33
tool-hang
2.67
28
0
0
3.87
1.87
17.3
3.47
transport
0
2
0
0
3.47
0.27
29.3
3.2
average 24.27
46.93
16.76
27.2
42.72
22.53
64.12
41.71
Table 2: Average success rates on Robomimic tasks using the Proficient Human dataset for each task.
simpler imitation learning methods. Using the "proficient human" (PH) datasets, which each consist of 200 successful trajectories with a binary reward, we trained policies with the continuous version of policies using the discretization from SAQ. The results,
IQL, CQL, and BC, well as discrete-acto presented in Table 2, show that SAQ indeed improves continuous offline RI. by large margins all the tasks considered. It's worthwhile to mention that our continuous BC results don't 't exactly match the original paper [6], as the authors point out, they adopt a Gaussian Mixture Model (GMM) for the policy class and extensively optimize parameters then perform checkpoint selection, whereas we directly train unimodal BC policies.
While the D4RL results suggest that discretization with SAQ can consistently improve the perfor- mance of each offline RL algorithm, these results further suggest that domains that especially challenging for offline RL, such narrow demonstration datasets, are particularly amenable for SAQ, where it can enable offline RL methods that were previously outperformed by simple imitation learning to attain significantly better results. This suggests that SAQ can be a particularly effective tool robotic learning, where narrow demonstration datasets might be commonplace.
Discussion, Limitations, and Future Work
We presented a method for state-conditioned action quantization to improve continuous offline RL algorithms. Our approach allows offline RL methods enforce policy constraints or value conservatism more exactly as compared to their continuous counterparts. This is particularly relevant and important in the robotic learning setting where usually assume narrow datasets of expert demonstrations where function approximation errors get even more exaggerated. However our approach does have number of limitations. First, we require sufficient state-action coverage to be able to perform state-conditioned action quantization. This is a common assumption in offline RL and is true in many curated datasets; however, it might be challenging to obtain such datasets real-world robotic settings. That said, our approach is able to achieve impressive performance in the Robomimic environment which largely composed of real human telcoperation data. Second, is not clear the best way adopt method in the online finetuning setting, where new data might invalidate the learned discretization. Adaptively adjusting the discretization during online training could be a valuable topic to explore in future work.

---

Acknowledgments
This research was partly supported through the Office of Naval Research through N00014-21-1- 2838 and N00014-20-1-2383. We acknowledge computing support from the Berkeley Research Computing(BRC) program and the NSF Cloudbank program.
References
[1] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911. 1361, 2019.
[2] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11761-11771, 2019.
[3] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779, 2020.
[4] Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021.
[5] I. Kostrikov, J. Tompson, R. Fergus, and O. Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprini arXiv:2103.08050, 2021.
[6] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang. R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. 5th Annual Conference on Robot Learning, 2021. URL https:
[7] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In Proceedings the 36th International Conference Machine Learning, 2019.
[8] N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 1XXvv:107.00055: 2019.
[9] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Srrattage-weighte regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 2019.
[10] A. Nair, M. Dalal, A. Gupta, and S. Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 2020.
[11] Z. Wang, A. Novikov, K. Zoina, J. T. Springenberg, S. Reed, B. Shahriari, N. Siegel, J. Merel,
C. Gulcehre, N. Heess, et al. Critic regularized regression. arXiv preprint arXi::2006.15134 2020.
[12] S. Fujimoto and S. S. Gu. A minimalist approach offline reinforcement learning. arXiv preprint 20t.::200600666600 2021.
[13] J. Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. In International Conference on Machine Learning, 2007.
[14] N. Y. Siegel, T. Springenberg. F. Berkenkamp, A. Abdolmaleki, M. Neunert, Lampe, R. Hafner, and M. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396 2020.
[15] T. Xie, C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal. offline reinforeement learning, Advances in neural information processing systems, 34, 2021.
pessimism for

---

[16] O. Nachum, B. Dai, I. Kostrikov, Y. Chow, L. Li, and D. Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:/912.02007 2019.
[17] Yu, G. Thomas, L. Yu, S. Ermon, Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005. 2020.
[18] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint urXiv:2102.08366 2021.
[19] Y. Jin, Z. Yang, and Z. Wang. Is pessimism provably efficient for offline rI? arXiv preprint 2rXi:2202..50085 2020.
[20] S. Rezaeifar, R. Dadashi, N. Vieillard, L. Hussenot, O. Bachem, O. Pietquin, and M. Geist. Offline reinforcement learning as anti-exploration. arXiv preprint arXiv:2106.06431, 2021.
[21] A. Tavakoli, Pardo, and P. Kormushev. Action branching architectures for deep reinforcement learning. 2018.
[22] N. Vieillard, M. Andrychowicz, A. Raichuk, O. Pietquin, and M. Geist. Implicitly regularized rl with implicit q-values. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, 2022.
[23] Y. Tang and S. Agrawal. Discretizing continuous action space for on-policy optimization. 2019.
[24] Y. Chebotar, Q. Vuong, K. Hausman, F. Xia, Y. Lu, A. Irpan, A. Kumar, T. Yu, A. Herzog,
K. Pertsch, K. Gopalakrishnan, Ibarz, O. Nachum, S. A. Sontakke, Salazar, H. T. Tran, J. Peralta, C. Tan, D. Manjunath, J. Singh, B. Zitkovich, T. Jackson, K. Rao, C. Finn, and
S. Levine. Q-transformer: Scalable offline reinforcement learning via autoregressive q-fnnctions In 7th Annual Conference Robot Learning. 2023. URL https htttps://openrevieww/ nett/
[25] N. M. M. Shafiullah, Z. J. Cui, A. Altanzaya, and L. Pinto. Behavior transformers: Cloning k modes with one stone. In Thirty-Sixth Conference on Neural Information Processing Systems, 2022.
[26] Seyde, P. Werner, W. Schwarting, I. Gilitschenski, M. Riedmiller, D. Rus, and M. Wulfmeier. Solving continuous control via g-leanning arXiv, 2022.
[27] L. Metz, J. Ibarz, N. Jaitly, and J. Davidson. Discrete sequential prediction of continuous actions for deep rl. arXiv, 2017.
[28] C. Tessler, G. Tennenholtz, and S. Mannor. Distributional policy optimization: An alternative approach for continuous control. In Advances in Neural Information Processing Systems, 2019.
[29] A. Tavakoli, M. Fatemi, and P. Kormushev. Learning represent action values as hypergraph on the action vertices. In International Conference on Learning Representations, 2021.
[30] R. Dadashi, L. Hussenot, D. Vincent, S. Girgin, A. Raichuk, M. Geist, and O. Pietquin. Continuous control with action quantization from demonstrations, In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 4537-4557, 17-23 Jul 2022.
[31] Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. arXiv preprint
2018.
[32] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
10

---

[33] A. Van den Oord, O. Vinyals, and k. kavukcuoglu. Neural discrete representation learning, In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[34] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072 07219, 2020.
[35] Fu, A. Kumar, O. Nachum, G. Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning. In arXiv, 2020. URL https://arxi: org/pdf/2ee 07219.

---

Appendix
A Detailed SAQ-IQL derivation
In this section, deseribe in detail the derivation for the SAQ-IQL algorithm. We start with the original optimization objective from IQL that employs explicit policy constraint w.r.t to a behavior policy
π arg max ( (A a)] xell
s.t. (T (T) Se
(1I)
Σ <a|s)) =1
We can write down the Lagrangian of 11 and expand it out:
L(T,,A, = E Ba (s, + A(E D + aal1 - Σ (alss)
(T(a|s) log( A(ass)
L(T, a,)== = E (T(als)4 +
Σ
(a(a))) ))) a(1 Σ xaass))
anv)(-118)
Differentiating L(T,, a) with respect to r(als) results in
OL( (s, a) Xlog(=(als)) log(Ts((ass) +1)
We divide the differentiated Lagrangian by the constant À
@L(T, a.
A (s. + log(T((als)) - log(T(a(s)) - X
A
and set the resulting expression to arrive at nT (a|s),
(s, a)
n"" (als) expl
log(TT(((ss)) -1 7
A (s,
म (a|s)
2(s) exp
+ log((T(a(ss))
X
where =(8) a normalizing term.
12

---

B Ablation Studies for SAQ
Task
No
State
State
State
State
State
State
SAQ-
SAQ.
SAQ-
SMQ.
SAQ-
SAQ-
cQI.
CQI.
IQL.
1Q1.
BRAC
BRAC
halfchoetah-
1.56
47.07
1.56
36.2
40.25
medium-rppla--22
supper-mediing
21.56
replay-v2
walker2d-medium-- replay-v2
4.67
81.72
6.89
45.64
53.52
0.25
average
7.16
74.51
6.73
47.09
6.57
54.21
Table 3: Comparing the performance state-conditioned action discretization against unconditioned action discretization with CQL. The state-conditioned discretization scheme significantly outperforms the unconditioned one since unconditioned action discretization cannot compress the action space into few number of bins.
Comparing discretization methods. To understand the importance of the state-conditionel dis- cretization method, we compare it against naive discretization method where the VQ- discretizes the actions without conditioning on the states and present the results in Table We see that the state-conditioning allows indeed highly important in compressing the action space into small number of bins, resulting in much higher performance than a state-agnostic discretization scheme.
Codebook size robustness. One key design choice we make in this paper is the use of VQ-VAE, one natural question is then our method's s robustness against the codebook size in the VQ-VAE; since that can be crucial in determining the quality the performed discretization through it. Towards this end, empirically experiment with varying codebook sizes across all three algorithms. We present the results in Table 4, and we found that method's performance is consistent across codebook sizes; which further resonates with the practical utility of adopting method.
Codebook Size
16
32
64
128
SAQ-CQL
108.7
111.6
110.8
103.2
SAQ-IQL
106.9
104.8
104.2
94.42
SAQ-BRAC
106.5
108.3
105.7
107
Table 4: Comparing the performance of SAQ-IQL, SAQ-CQL, and SAQ-BRAC on hopper-expert--2 while varying the codebook size. It can be observed that the discretized algorithms are invariant to codebook size changes.
Controlling policy constraint levels. As stated Sec.4.1, one key premise of SAQ is that we can enforce policy constraint or value conservatism exactly; which associates with the practical performance of offline RL methods. To further verify this hypothesis empirically; task from the Gym locomotion suite and vary the weight coefficients for policy constraint or value conservatism to observe the resulting performance. We present our results in Fig. 5, we can see that small constraint enforcement leads to poor performance initially; then the performance ramps up when increase the coefficients; finally con- verges with sufficient large coefficients. This observation confirms conjecture in the paper.
200 COL
200.0IC
pick one
bcreainn
Figure 5: Increasing policy constraint levels on hopper-expert-v2 environment from Gym locomotion.
13

---

C
Additional Experiment Results
Task
BRAC SAQ- BRAC
IQL
Aquadem-
SAQ-
CQL
Aquadem-
SAQ-
BC
Aquadem- SAQ-
IQL.
IQ1.
CQL.
CQI.
BC
BC
halfcheetah-expett-22
67.99
64
94.78
92.73
90.3
43.9
92.66
92.65
91.53
92.69
108 4
73.07
90.31
88.06
80.75
89.05
50.32
91.69
74.17
57.65
halfcheetah-mmediee medium-repline v2
-3.3
35.28
44.24
35.12
36.2
47.07
40.5
40.25
35.16
35.62
3.76
sullcheetah-mediumee
47.08
43
47.3
42.39
42.52
48.56
44.5
43.89
42.32
41.93
47.02
Sopper-expert-v22
67.34
110
108.8
109
100.3
100.5
110.3
109.9
98.8
110.1
92.63
hogger-meddiu-cxxett-22
50.74
98.75
32.85
54.57
81.56
67.08
86.7
98.47
44.48
57.9
58.94
32.54
61.75
38.05
59.43
94.73
90.3
68.87
17.96
20.94
32.93
hopper-medduu--22
57.06
54.16
54.3
50.53
70.32
58.5
38.25
50.31
52.98
42.47
walker2d-expert-22
108.4
110.12
108.5
107.6
109.2
108.2
107.3
108.3
108.3
104.5
woilke/2d-meiiu--xxpeee
109
102.9
109.56
108
100.3
110.7
108.1
108.6
91.79
83.55
103.2
walker2d-nn mulker2d--edium--ee -epllaa----
73.72
0.4
68.71
32.66
45.64
81.72
80.8
53.52
12.98
27.86
15.27
walker2dd-eeiuu--22
81.94
64.8
81,25
68.06
83.11
82.1
74.77
70.18
27.86
62.52
locomotion average
64.16
66.98
75.14
71.01
76.67
75.6
82,58
77.35
59.39
61.16
60.77
antmaze- hetp:az--mmmiiu-iiiveeeeee
26
47.6
76.67
40
68.33
72.75
22.67
75.47
0
0
0
Medmmaz--eddum-aee play-v2
48.67
56.93
78.67
46
74.33
67.04
30.67
68.67
1.3
untmaz--larg-divver
0.66
9.73
31.67
33.67
35.62
22.67
antmaze-large-play-22
3.73
34.33
14.33
45.18
25.33
47.33
antmaze average
29.5
55.34
55.92
25.34
56.87
door- coor-human-v0 hammer-human---0
-1.01
35.42
1.79
2.15
9.26
0.84
6.09
2.12
3.29
3.24
9.28
-142
20.52
1.41
1.56
1.57
0.27
2.64
0.6
2.09
1,38
pen-human-vd relocate-human-v0
98.15
98.41
69.69
73.91
80.25
41.24
85.66
82.73
45.28
72.12
73.3
0.28
8,38
0.55
0,2
0.05
0.28
0.02
0.04
0.48
0.02
adroit uverage
23.86
40.09
20.32
19.54
22.82
10,58
23.67
21.37
12.35
19.48
21
kitchen-mixed-oo kitchen-complete-00
10.33
53.33
48.92
56.94
52.92
62
0
57.67
37.9
58.89
34
31.67
10
66
70.92
76.76
21.5
47.67
39.13
68
90.33
kitchen- kitchen-partial-vr partial-vO
5.33
45
37.58
44.83
46.25
92.33
37.33
43.56
46.67
kitchen average
15.78
36.11
50.83
57.56
58.61
48.67
23.83
65.89
38.12
56.82
57
Table 5: Comparison of our method and Aquadem on various D4RL tasks. SAQ in general improves its continuous counterpart, especially in the narrow dataset setting; also outperforms Aquadem in most tasks.
14


Reviews:
[{"review_id": "w5DAKZ_6h_", "reviewer": "Reviewer_Pinn", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This proposes that offline RL methods like CQL, IQL, and BRAC employ a sampling step to optimize the constraints when the action space is continuous, making their performance poor. Thus, this paper discretizes the action space through state-conditioned reconstruction of action using a VQ-VAE (thus ensuring a small action space). The discretized versions of the above algorithms perform better than their continuous counterparts.", "summary_of_recommendation": "Overall, I liked the paper as it is interesting, clearly written, and shows significant improvements over the baselines. But, it is missing some analyses to understand the core issue in the continuous action spaces and why the discretization is performing much better. The current hypothesis that continuous action space offline RL suffers due to approximations in the estimation of the constraints is not empirically justified well. In this state, I cannot recommend acceptance despite the strong results the paper reports. I would be more than happy to reconsider my position if provided with more information such as implementation details, training curves, hyperparameter tuning approach, the author's answers, and analysis experiments.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Several questions are raised in the review above. Additionally, I am curious why the authors mention that \"the main benefit of SAQ is not in raw performance\", despite the results showing large improvements over the traditional continuous action space way of solving the problem. Is there something that I missed about why the improvement in performance is not claimed more strongly? For example, I did not check the performance of these algorithms in other prior works. Do the authors suspect that the continuous action space results can improve with some implementation optimizations or better hyperparameter tuning, and the true difference in performance against the discrete action space results is smaller?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, I liked the paper as it is interesting, clearly written, and shows significant improvements over the baselines. But, it is missing some analyses to understand the core issue in the continuous action spaces and why the discretization is performing much better.\n\n# Strengths\n- Clearly written, well-explained and the transformation of continuous to discrete versions of the algorithms is clear.\n- The idea of using state-conditioned VQ-VAE to discretize the action space is quite interesting and it's nice to see it works well too in practice.\n- The results often show a significant gain over the compared continuous action space counterparts. The results seem very strong.\n\n\n# Weaknesses\nMy main concern is that the paper is **missing qualitative and quantitative investigation of the contributions of this paper**: 1. the use of VQ-VAE to discretize and 2. discrete v/s continuous action space in offline RL:\n\n## VQ-VAE for discretization\nSome implementation details and crucial analyses are missing:\n- I wonder how difficult or unstable the training of VQ-VAE was, and is there any plan to open-source the code? What was the size of the codebook? Could the authors provide more training details, especially those pertaining to this paper's contributions.\n- Is the discrete codebook sufficiently able to recover all the action reconstructions? What is the performance of the action reconstruction?\n- What exactly is the role of the codebook and the VQ-VAE decoder? One trivial solution is that the VQ-VAE decoder learns to disregard the codebook and simply learns a BC policy $f(a | s, e_k) \\sim f_{BC}(a | s)$. Since, the decoder essentially does BC and the other offline-RL methods now use this decoder implicitly, it is possible that their performance improves more on expert dataset, because of this inherent bias towards BC, by assuming that the data is good for imitation. I am not at all claiming that this happens, but understanding what is the learned discrete action space, codebook, and VQ-VAE encoder-decoder seems important to justifying where the performance gains come from.\n- **Other discretization approaches**: The paper mentions several action discretization works [21-28] which either assume independence of action dimensions or model the action space autoregressively, and the claim is that naive discretization can result in exponential blowup or imprecise actions. While this makes sense intuitively, since the task is offline RL, the challenge of exploration is not there. This means that there is a chance that even with such a discrete action space, learning is feasible and at least easier than the continuous action space counterpart — for instance, if the discretized action space is quite large, then the sampling versions of the offline RL algorithms can still be applied. Nevertheless, since VQ-VAE as the discretization scheme is one of the core contributions, it is important to compare against alternate action discretization schemes to disentangle the roles of discretization scheme and the question of discrete v/s continuous offline RL optimization. I do appreciate the ablation of unconditioned VQ-VAE, but that only explores one style of discretization: reconstruction with VQ-VAE. It would be nice to see the effect of some other methods such as [23, 24, 25].\n\n## Discrete v/s continuous action space for offline RL\nIt is great to see the improvement with the discretization, but the current experiments do not convincingly show that the benefit is actually because of replacing the approximate calculations with exact versions, or due to other artifacts of discrete action spaces, such as the ease of multi-modal modeling of data.\n- Especially, the result of BC v/s SAQ-BC hints at the fact that there are major improvements even in the absence of any approximation made in the continuous action space.\n\t+ It's not even clear that multimodality is the core reason of improvement, because the SAQ-BC v/s BC gains are more significant in expert datasets (especially, halfcheetah and hopper) than medium datasets. It is unclear to me why SAQ-BC (92.63) would be so much better than BC (28.49) when the dataset is expert.\n- In Fig 2, at around 1200 grad steps in continuous CQL achieves optimal performance despite the estimated R($\\theta$) already diverging from the exact value. What explains this success despite the divergence? It would hint that even continuous action space offline RL can reach optimal performance despite the approximations over the constraints.\n\t+ Also, if instability only arises post optimality, then wouldn't early stopping be a simple solution to continuous CQL's issues?\n\t+ Can the authors provide the learning curves for all the results computed to observe the learning dynamics? Since the key claim is the poor optimization due to approximations in continuous action spaces, it is important to observe the performance over the course of training.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "X697wFfMmGG", "reviewer": "Reviewer_L3mY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a new method for online reinforcement learning with continuous action spaces.  This method uses a VQ-VAE to learn a state-dependent discretization of the action space.  They then show that this discretization provides significant benefits to existing offline reinforcement learning techniques.", "summary_of_recommendation": "The paper demonstrates new practical tools for addressing important problems in the learning and robotics communities.  If I were starting a new project in this area in the near future, I would want to read this paper.\n\nPost Rebuttal: see notes in main review above.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Some analysis of the issues mentioned above about SAQ harming performance by small amounts in many tasks, while improving performance by large amounts in a few tasks would be very helpful.\n\nIf you have time to run the ablation with AQUADEM's discretization method on a few tasks that would also be interesting to see.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The paper presents a novel solution to a challenging and practically useful problem\n- The paper is clearly written and easy to understand.\n- The method appears to provide substantial improvements over existing approaches.\n\nWeaknesses:\n- No D4RL experiments in the main paper.  The authors devote space to showing images from the D4RL benchmark (Figure 3) but not their actual results, which are instead in the appendix. \n- The average improvement in each category in D4RL hides an interesting observation that deserves more attention: SAQ appears to slightly harm performance for many individual tasks, but drastically improves performance in  others.  The locomotion tasks especially exhibit this phenomenon, where for most tasks, performance is actually harmed by SAQ, even though the average scores are propped up by very large gains in a few specific tasks.  A more thorough analysis here would be extremely useful.  In these tasks where SAQ hurts performance, what is the failure mode?  Are the continuous actions in successful rollouts not covered by the discretization, or is something else happening?  This may require more space than could fit in the main paper, but would be really great supplemental analysis.\n- A comparision against AQUADEM's log-sum-exp discretization technique seems important here.  Even though, AQUADEM is designed for demonstrations+RL and SAQ is designed for offline RL, they both share the technique of pretraining a state-dependent discretization, and so it would be extremely useful to know which discretization technique works better here.\n\nTypos:\n- \"re-visit once we have full results\" at the end of the 1st paragraph of 5.2\n\nPost rebuttal: I am keeping weak-accept at this time.  The other reviewers found some inconsistencies in the experiments, which the authors attempted to correct.  It seems like the others disagree about whether or not these issues were properly addressed.  The issues of the other reviewers are somewhat concerning, but to me there are enough interesting components and ideas here  that I don't think it's worth moving my score down.  I generally buy Aebd's argument in response to g7YW: \"I think that your robomimic concerns are orthogonal to the main point of the paper. In my mind, the point of the paper is to compare base algorithms (e.g. CQL, BC, IQL) with exactly the same algorithms plus SAQ. While they could tune hyperparameters for each domain like robomimic, it seems like a cleaner comparison between standard CQL and CQL+SAQ to use the simpler hyperparameter tuning strategy used in this paper. Moreover, if the hyperparameter tuning were that extensive, it would be the same for CQL+SAQ.\"", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "TJ_ikoDCi4", "reviewer": "Reviewer_g7YW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method to learn a quantized action representation for use in continuous action offline reinforcement learning approaches. The method is to train a state-conditioned VQ-VAE to learn a set of discrete latent actions, which can be reconstructed by a state-conditioned decoder. Several offline RL methods are then adapted to use discrete actions instead of continuous actions. The action representation and discrete variants of the offline RL methods are evaluated against their continuous counterparts on two benchmarks - D4RL and robomimic, and shown to increase performance.", "summary_of_recommendation": "Currently, the issues with the experimental evaluation in the paper outweigh the positives. To summarize, the issues include not showing the value of the VQ-VAE compared to other discretization schemes and the poor results on the robomimic benchmark compared to the ones reported there.\n\nPost-rebuttal: After the author rebuttal and the discussion, I have decided to raise my score to weak accept.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address the comments in the weaknesses section above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths**\n\nThis paper presents an interesting approach to learn action discretization and presents useful empirical evidence showing that the discrete offline RL method variants outperform their continuous counterparts, sometimes significantly. Furthermore, this is shown across 3 different methods (CQL, IQL, and BRAC). As an added benefit, some of the modifications to the offline RL methods seem to simplify their implementation and allows for computing quantities exactly, due to the use of discrete actions. \n\n**Weaknesses**\n\nThere are some issues with the experimental evaluation and the writing and clarity. Comments related to the experiments follow:\n\n- The paper only compares the proposed method against the continuous action variants of the offline RL methods. However, there is limited analysis on whether the choice of VQ-VAE is important for the method to work. The VQ-VAE appears to be a crucial choice compared to other ways to discretize the actions, but that isn't really shown. The appendix does have some experiments showing that the state-conditioning for the VQ-VAE is important, and that the codebook size does not matter much. However, there are no comparisons to other discretization methods. Some other choices of discretization include the AQuaDem paper (https://arxiv.org/abs/2110.10149), which is highly related, and acknowledged in the related work section. It seems as though their discretization learning method could be applied to offline RL (even though they apply it to online RL). Other alternatives include using a categorical VAE with gumbel-softmax trick (https://arxiv.org/abs/1611.01144), and potentially learning a state-conditioned Gaussian Mixture Model (as in the robomimic paper, https://arxiv.org/pdf/2108.03298.pdf) and using the mode centers.\n\n- Some of the robomimic results are questionable. Looking at Table 1 in the robomimic paper (https://arxiv.org/pdf/2108.03298.pdf), CQL obtains 92.7 and 38.0 respectively on the Lift (PH) and Can (PH) datasets, while the implementation in this paper achieves 64.2 and 19.6. The BC results in this paper are also significantly lower than the ones reported in the robomimic paper (e.g. Lift 100 vs. 59.5, Can 95.3 vs. 31.7, Square 78.7 vs. 19.3, Transport 17.3 vs. 0.3, and Tool Hang 29.3 vs. 1.87). Given that the implementation is open-source, it is hard to understand why these results are so much lower. \n\nSome of the writing should be edited to improve clarity. Comments related to writing and clarify follow:\n\n- The preliminaries section could be shortened - there are many details for CQL, BRAC, and IQL presented, and they are not all relevant to understand the core contribution of the method (using VQ-VAE to learn action discretization). The changes to each method are also highlighted in section 4.2 in a somewhat self-contained manner. It could be useful to make more explicit connections to the preliminaries section or just present all of the relevant details directly in section 4.2 and omit the sections in the preliminaries on these methods.\n\n- Section 4.1 could be explained a little more clearly and precisely. For example, what is meant by single dimensional latent code? It most likely corresponds to K vectors in the codebook with D=1 (using the notation from the preliminaries section). The preliminaries section seems to have redundant information (for example equation 6 vs. equation 7 in this section). It would be better to make explicit connections to the preliminaries section to identify how the general VQ-VAE formulation was modified. This seems to be the core part of the method, so making it extremely clear is very important.\n\n- The writing in Section 4.2 could also highlight the advantages of the use of discrete actions instead of continuous actions a little more. For example, it wasn't immediately clear that the CQL conservatism loss becomes an exact sum instead of an approximate sample-based integral. This would also help motivate the use of discrete actions over continuous actions more.\n\n- Related to the point above, it might also help to provide some more intuition about the state-conditioned VQ-VAE. While the number of discrete latent actions is fixed, the interpretation of these actions depends on the state. In this sense, these latent codes could probably be thought of as modes of a conditional state-action distribution, similar to a Gaussian Mixture Model (GMM) or a categorical VAE. Given this choice, it seems a natural point of comparison could be to a GMM or categorical VAE (as mentioned in a comment above on adding more comparisons).\n\n- It would be nice to see more useful analysis in section 5.1 and section 5.2 of the results in Table 1 and 2, with references to particular comparisons in the tables. For example, which improvements were the most significant and which were the least? Was this to be expected?\n\n- More details on the specific implementation details should be provided. For example, what kinds of networks and hyperparameters were used? Did the BC baseline use a simple L2 loss, and how else did it differ from the one in the robomimic paper? How was the SAQ-BC model trained? Were the actions labeled into discrete codes and used to train a network to map states to discrete codes via a cross-entropy loss?\n\nSome typos and potential mistakes:\n- line 79 \"actin\"\n- line 259 \"re-visit once we have full results\"", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "6s08F7VX90W", "reviewer": "Reviewer_Aebd", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents state-conditioned action quantization (SAQ) a method for discretizing action spaces with a state-conditioned VQ-VAE. The method can be added on top of any standard offline RL algorithm to simplify the action space. Experiments are presented on a variety of base algorithms and tasks from D4RL and robomimic that show substantial improvements in performance by providing a simpler action space with a strong prior for actions in the dataset.", "summary_of_recommendation": "The paper presents a simple method that provides clear cut improvements over baseline methods and seems to be most effective on tasks that are most applicable to robotics. I think the paper is a good contribution and should be accepted.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See weaknesses above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n\n1. SAQ is a simple and general method that can be applied on top of any base algorithm for offline RL. This makes its applicability potentially large and impactful.  \n\n2. The empirical results are clear and show substantial improvements in performance. These improvements seem to hold across most environments, but are especially strong on high dimensional tasks with more realistic demonstration data like robomimic, underscoring the potential importance for the CoRL audience. \n\n## Weaknesses\n\n1. The idea is not entirely original since Dadashi et al (cited in the paper as [29]) propose a very similar method for online RL. However, the paper provides this reference and builds on the idea to extend it into offline RL with a VQ-VAE.\n\n2. Section 4.3 provides an explanation of why SAQ is helpful on top of CQL, but leaves a few lingering questions. (1) While it is clear that CQL makes an approximation with continuous actions, this explanation does not as cleanly extend to IQL or BC, but these methods also benefit from SAQ. Thus, it seems that this explanation may not be complete. (2) It is unclear how the exact CQL penalty is calculated. Is it just by taking many more samples so the estimate concentrates around the true value?", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes action-discretization based on VQ-VAE to improve offline RL algorithms. The proposed method was combined with offline RL methods such as IQL and CQL, and the experimental results show that the proposed action discretization improves the performance of offline RL methods.

While the reviewers pointed out several unclear points, the authors addressed them in the rebuttal. Based on the overall assessment, AE recommends the acceptance of the paper. 
However, Reviewer Pinn pointed out that the authors do not provide the learning curves of the proposed method. Please consider adding some learning curves and further analyzing what happens in the proposed method.
Decision: Accept (Poster)