Continuous Control with Action Quantization from Demonstrations
Authors: Robert Dadashi
Leonard Hussenot
Damien Vincent
Sertan Girgin
Anton Raichuk
Matthieu Geist
Olivier Pietquin
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Reinforcement Learning
Action Discretization
Learning from Demonstrations

Abstract: In Reinforcement Learning (RL), discrete actions, as opposed to continuous actions, result in less complex exploration problem and the immediate derivation of the maximum of the action-value function which is central to dynamic programming-based methods. In this paper, we propose a novel method: Action Quantization from Demonstrations (AQuaDem) to learn a discretization of continuous action spaces by leveraging the priors of  demonstrations. This dramatically reduces the exploration problem, since the actions faced by the agent not only are in a finite number but also are plausible in light of the demonstrator’s behavior. By discretizing the action space we can apply any discrete action deep RL algorithm to the continuous control problem. We evaluate the proposed method on three different setups: RL with demonstrations, RL with play data --demonstrations of a human playing in an environment but not solving any specific task-- and Imitation Learning. For all three setups, we only consider human data, thus most challenging than synthetic data. We found that AQuaDem consistently outperforms state-of-the-art continuous control methods, both in terms of performance and sample efficiency.

Under review as a conference paper at ICLR 2022
CONTINUOUS CONTROL WITH ACTION QUANTIZATION FROM DEMONSTRATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
In Reinforcement Learning (RL), discrete actions, opposed to continuous actions, result in less complex exploration problems and the immediate computation of the maximum of the action-value function which is central to dynamic programming- based methods. In this paper, we propose novel method: Action Quantization from Demonstrations (AQuaDem) to learn discretization of continuous action spaces by leveraging the priors of demonstrations. This dramatically reduces the exploration problem. since the actions faced by the agent not only are in a finite number but also are plausible in light of the demonstrator's behavior. By discretizing the action space can apply any discrete action deep RL algorithm to the continuous control problem. We evaluate the proposed method on three different setups: RL with demonstrations, RL with play data -demonstrations of human playing in an environment but not solving any specific task- and Imitation Learning. For all three setups, we only consider human data, which is more challenging than synthetic data. We found that AQuaDem consistently outperforms state-of-the-an -ftt continuous control methods, both in terms of performance and sample efficiency on a variety of hard manipulation tasks.
INTRODUCTION
With several successes on highly challenging tasks including strategy games such as Go (Silver et al., 2016), StarCraft (Vinyals et al., 2019) or Dota 2 (Berner et al., 2019) as well as robotic manipulation (Andrychowiez et al., 2020), Reinforcement Learning (RI) holds a tremendous potential for solving sequential decision making problems. RL relies on Markov Decision Processes (MDP) (Puterman, 2014) as its cornerstone, a general framework under which vastly different problems can be casted.
There is clear separation in the class of MDPs between the finite discrete action setup, where an agent faces a finite number of possible actions, and the continuous action setup, where an agent faces an infinite number of actions. When the number of actions is small, the former is arguably simpler, since exploration is more manageable with a finite, reasonable number of actions, and computing the maximum of the action-value function is straightforward (and implicitly defines greedily-improver policy). In the continuous action setup. the parametrized policy either directly optimizes the expected value function that estimated through Monte Carlo rollouts (Williams, 1992), which makes demanding in interactions with the environment, or optimizes a parametrized state-action value function (Konda and Tsitsiklis, 2000) hence introducing additional sources of approximations.
Therefore, workaround consists in turning a continuous control problem into discrete one. The simplest approach to naively (e.g. uniformly) discretize the action space, an idea which dates back to the "bang-bang" controller (Bushaw, 1952). However, such discretization scheme suffers from the curse of dimensionality. Various methods have addressed this limitation by making the strong assumption of independence (Tavakoli et al., 2018; Tang and Agrawal, 2020; Andrychowicz et al., 2020) or causal dependence (Metz et al., 2017; Vinyals et al., 2019; Sakryukin et al., 2020; Tavakoli et al., 2021) between the action dimensions which are typically complex and task-specific (e.g. autoregressive policies, pointer networks based architectures).
In this work, we introduce novel approach leveraging the prior of human demonstrations for reducing a continuous action spaces to discrete set of meaningful actions. The proposed method does not suffer from the curse of dimensionality and does not require any task specific assumption.

---

Under review as a conference paper at ICLR 2022
Demonstrations typically consist of transitions experienced by human in the targeted environment, performing the task at hand or not. They are of particular interest in cases where the reward function is hard to define (Russell, 1998; Ng et al., 1999), to facilitate exploration (Salimans and Chen, 2018; Nair et al., 2018) or to build behavioral priors (Singh et al., 2020).
We thus propose Action Quantization from Demonstrations, or AQuaDem, novel paradigm where we learn a state dependent discretization of a continuous action space using demonstrations, enabling the use of discrete-action deep RL methods by virtue of this learned discretization. We formalize this paradigm, provide a neural implementation and analyze it through visualizations in simple grid worlds. We empirically evaluate this discretization strategy on three downstream task setups: Reinforcement Learning with demonstrations, Reinforcement Learning with play data, and Imitation Learning. We test the resulting methods on robotics tasks and show that they outperform state-of state-of-the-art continuous control methods both in terms of sample-efficiency and performance on every setup.
2 PRELIMINARIES
Markov Decision Process. We model the sequential decision making problem as Markov Deci- sion Process (MDP) (Puterman, 2014; Sutton and Barto, 2018). An MDP is tuple (S, A. P, 7, Po)) where S is the state space, A is the action space, P is the transition kernel, r the expected reward function, y the discount factor and po the initial state distribution. Throughout the paper, we distin- guish discrete action spaces, which simply amount to a set (l...., (l....,K] K}. from continuous action spaces which consist in an interval of Id where d' is the dimensionality of the action space. A stationary policy π is a mapping from states to distributions over actions. The value function V of a policy π is defined as the expected discounted cumulative reward from starting in a particular state and acting according to T: V*(ss) = E Er= 7'((s,, ax) 80 = 8, O2 ~ (8)), 8t+1 ~ (S(e. at))]. An optimal policy x² maximizes the value function V* for all states. The action-value function Q is defined as the expected discounted cumulative reward from starting in a particular state, taking an action and a(s.
then acting according to x: (T a) r(ss a) yB [V* (s')
Value-based RL. The Bellman (1957) operator T connects an action- value function Q for the state-action pair (s, a) to the oetion-value function in the subsequent state s': (Q) a) == r(s, a) + + (s',, a') ~ T(s), s' ~ a(s.a)] Value Iteration (VI) (Bertsekas, 2000) is the basis for methods using the Bellman equation to derive algorithms estimating the optimal policy n² The prototypical example is the Q-learning algorithm (Watkins and Dayan, 1992), which is the basis of egg. DQN (Mnih et al., 2015), and consists in the repeated application a stochastic approximation of the Bellman operator Q(s, a) == r(s, a) + max,'' Q(s', a²), where (s, a, s') is transition sampled from the MDP. The Q-learnigg algorithm exemplifies two desirable traits of VI-inspired methods in discrete action spaces that 1) bootstrapping: the current Q-value estimate at the next state s' is used to compute a finer estimate of the Q-value at state 8, and 2) the exact derivation of the maximum Q-value at given state. For continuous action spaces, state-of-the-an methods (Haarnoja al., 2018; Fujimoto et al., 2018) are also fundamentally close to a VI scheme, as they rely on Bellman consistency, with the difference being that the argument maximizing the Q-value, other words the parametrized policy, is approximate.
Demonstration data. Additional data consisting of transitions from an agent may be available. These demonstrations may contain the reward information or not. In the context of Imitation Learning (Pomerleau, 1991; Ng et al., 1999; 2000; Ziebart al., 2008), the assumption is that the agent generating the demonstration data is near-optimal and that demonstration rewards are not provided. The objective is then to match the distribution of the agent with the one of the expert. In the context of Reinforcement I Learning with demonstrations (RLfD) (Hester et al., 2018; Vecerik et al., 2017), demonstration rewards are provided. They are typically used in the form of auxiliary objectives together with standard learning agent whose goal is to maximize the environment reward. In the context of Reinforcement Learning with play (Lynch et al., 2020), demonstration rewards are not provided as play data is typically not task-specific.
Demonstration data can come from various sources, although a common assumption is that it is generated by single, unimodal Markovian policy. However, most of available data comes from agents that do not fulfill this condition. In particular, for human data, and even more so when coming from several individuals, the behavior generating the episodes may not be unimodal nor Markovian.
2

---

Under review as a conference paper at ICLR 2022
3 METHOD
In this section, we introduce the AQuaDem framework and a practical neural network implementation together with an accompanying objective funcctio. We provide a series of visualizations study the candidate actions learned with AQuaDem in gridworld experiments.
Step (offline)
Learn state-conditioned quantization.
Step 2 (online) Run discrete RT, on quantized actions.
Qla
V
(()) aag mAs
Visualization of the AQuaDem framework (offline) with a downstream algorithm (online).
3.1 AQUADEM: ACTION QUANTIZATION FROM DEMONSTRATIONS
Our objective is to reduce a continuous control problem to a discrete action one, on which we can apply discrete-action RL methods. Using demonstrations, we wish assign to each state s E S a set of K candidate actions from A. The resulting action space is therefore a discrete finite set of K state-conditioned vectors. In given state s E S, picking action {1...., K} stands for picking the 1tth candidate action for that particular state. The AQuaDem framework refers to the discretization of the action space, and the resulting discrete action algorithms used with AQuaDem on continuous control tasks are detailed in Section 4. We propose to learn the discrete action space through a modified version of the Behavioral Cloning (BC) (Pomerleau, 1991) reconstruction loss that captures the multimodality of demonstrations. Indeed the typical BC implementation consists in building a deterministic mapping between states and actions & S + A. But in practice, and in particular when the demonstrator is human, the demonstrator can take multiple actions in given state (we say that its behavior is multimodal) which are all good candidates for AQuaDem. We thus learn a of K candidate actions and optimize a reconstruction loss
mapping v S - AK from states based on a soft minimum between the candidate actions and the demonstrated action.
Suppose have dataset of expert demonstrations D {( a:) 1mm- In the continuous action setting, the vanilla BC approach consists in finding parametrized function fa that minimizes the reconstruction error between predicted actions and actions in the dataset D. To ease notations, we will conflate the function fa with its parameters T and simply note it $ S + A. The objective is thus to minimize: ming EE, B. (b(s) all Instead, we propose learn set of K actions (s) for each state by minimizing the following loss:
[log log Σ exp [Vac(s) - -1 T
min E, Ea [-
where the temperature T is hyperparameter. Equation (1) corresponds to minimizing a soft- minimum between the candidates actions V (1(s)...., V (s) and the demonstrated action a. Note that with K 1, this is exactly the BC loss. The larger the temperature T is, the more the loss imposes all candidate actions to be close to the demonstrated action a thus reducing to the BC loss. The lower the temperature T is, the more the loss only imposes single candidate action to be close to the demonstrated action a. We provide empirical evidence of this phenomenon in Section 3.2. Equation (I) is also interpretable in the context of Gaussian mixture models (see Appendix A). The V function enables us to define new MDP where the continuous action space is replaced by a discrete action space of size K corresponding the K action candidates returned by Vi at each state.
3.2 VISUALIZATION
In this section, we analyze the actions learned through the AQuaDem framework, in toy grid world environment. We introduce a continuous action grid world with demonstrations in Figure 1.
We define neural network V and optimize its parameters by minimizing the objective function defined in Equation (1) (implementation details can be found in Appendix D.1). We display the resulting candidate actions across the state space in Figure 2. As each color of the arrows depicts
3

---

Under review as a conference paper at ICLR 2022
Goal
Figure I: Grid world environment where the start state is in the bottom left, and the goal state is in the top right. Actions are continuous (2- dimensional), and give the direction in which the agent take a step. These steps are normalized by the environment to have fixed L2 norm. The stochastic demonstrator moves either right or up in the bottom left of the environment then moves diagonally until reaching the edge of the grid, and goes either up right to reach the target. The demonstrations are represented in the different colors.
Start
single head of the network, we observe that the action candidates are smooth: action candidates slowly vary the state vary, which prevents to have inconsistent action candidates in nearby states. Note that BC actions tend to be diagonal even in the bottom left part of the action space, where the demonstrator only takes horizontal or vertical actions. On the contrary, the action candidates learned by AQuaDem include the actions taken by the demonstrator conditioned on the states. Remark that in the of K 2. the action right is learned independently of the state position (middle plot in Figure 2) although it is only executed in subspace of the action space. In the case of K = 3, actions completely state-independent. In non-trivial tasks, the state dependence induced by the AQuaDem framework is essential, as we show in the ablation study in Appendix C and in the analysis of the actions learned in more realistic setup Appendix B.
Belavioral Cloning
AQuaDemn K-2
AQuaDem K-a
Figure 2: Visualisation of the action map- ping learned by BC and the candidate actions learned with AQuaDem for K and K 3 and T = 0.01. Each color represents a head of the V network.
Influence of the temperature. The temperature controls the degree of smoothness the soft- minimum defined in Equation (1). We show that with larger temperatures, the soft-minimum converges the average which well represented in Figure rightmost plot where the profile of AQuaDem's 's action candidates conflate with actions learned by BC. With lower temperatures, the actions taken by the demonstrator are recovered, but the temperature is too low (T' 0.001), some actions that are not taken by the demonstrator might appear candidates (blue arrows in the leftmost figure). This occurs because the soft minimum converges to a hard minimum with lower temperatures meaning that as long can be arbitrarily far off. In this work, we treat the temperature as hyperparameter, although a natural direction for future work is to aggregate actions learned for different temperatures.
candidate is close enough to the demonstrated action, the other candidates
AQuaDom, K-3
Figure 3: Influence of the temperature on resulting candidate actions leareed with AQuaDem.
k
k
k
K
k
t
l K
T-0.011
T-0.1
3.3 DISCUSSION
On losing the optimal policy. In any form of discretization scheme, the resulting class of poli- cies might not include the optimal policy of the original MDP. In the case of AQuaDem, this mainly depends on the quality the demonstrations. For standard continuous control methods, the parametrization of the policy also constrains the space of possible policies, potentially not including the optimal one. This a lesser problem since policies tend to be represented with functions with universal approximation capabilities. Nevertheless, for most continuous control methods, the policy improvement step is approximate, while in the case of AQuaDem is exact, since it amounts to selecting the argmax of the Q-values.
4

---

Under review as a conference paper at ICLR 2022
On the multimodality of demonstrations. The multimodality of demonstrations enables to define multiple plausible actions for the agent to take in a given state, guided by the priors of the demonstrations. We argue that the assumption of multimodality of the demonstrator should actually be systematic (Mandlekar et al., ).. Indeed, the demonstrator can be e.g. non-Markovian, optimizing for something different than reward function like curiosity (Barto et al., 2013), or they be in the process of learning how to interact with the environment. When demonstrations are gathered from multiple demonstrators, this naturally leads to multiple modalities in the demonstrations. And even in the case where the demonstrator is optimal, multiple actions might be equally good (e.g. in navigation tasks). Finally, the demonstrator can interact with an environment without any task specific intent, which we refer to as play (Lynch et al., 2020) and also induces multimodal behavior.
4 EXPERIMENTS
In this section, we evaluate the AQuaDem framework on three different downstream tasks setups: RL with demonstrations, RL with play data and Imitation Learning. For all experiments, we detail the networks architectures, hyperparameters search, and training procedures in the Appendix.
4.1 REINFORCEMENT LEARNING WITH DEMONSTRATIONS
Setup. In the Reinforcement Learning with demonstrations setup (RLiD), the environment of interest comes with reward function and demonstrations (which include the reward), and the goal is to learn a policy that maximizes the expected return. This setup is particularly interesting for sparse reward tasks, where the reward function is easy define (say reaching goal state) and where RL methods typically fail because the exploration problem is too hard. We consider the Adroit tasks (Rajeswaran et al., 2017) represented in Figure 9, for which human demonstrations are available (25 episodes acquired using a virtual reality system). These environments come with a dense reward function that we replace with the following sparse reward: 1 if the goal is achieved, 0 otherwise.
Algorithm & baselines. The algorithm we propose is two-fold training procedure: 1) we learn discretization of the action space in a fully offline fashion using the AQuaDem framework from human demonstrations; 2) we train a discrete action deep RL algorithm on top of this this discretization. We refer to this algorithm as AQuaDQN. The RL algorithm considered is Munchausen DQN (Vieillard et al., 2020) as it is the state of the art on the Atari benchmark (Bellemare et al., 2013) (although use the non-distributional version of it which simply amounts to DQN (Mnih et al., 2015) with a regularization term). To make much buffers: one containing interactions with the environment, the other containing the demonstrations that we sample using a fixed ratio similarly DQID (Hester et al., 2018), although we do not use the additional recipes of DQfD (multiple n-step evaluation of the bootstrapped estimate of Q, BC regularization term) for the sake of simplicity. When sampling demonstrations, the actions are discretized by taking the closest AQuaDem action candidate (using the Euclidean norm). We consider SAC and SAC from demonstrations (SAC1D) -a modified version of SAC where demonstrations are added to the replay buffer (Vecerik et al., 2017)- as baselines against the proposed method. We do not include naive discretization basclines here, as the dimension of the action space at least 24, which would lead to a 22 ~ 16M actions with binary discretization scheme, which is prohibitive without additional assumptions on the structure of the action-value function.
of the demonstrations as possible, maintain two replay
Evaluation & results. We train the different methods on 1M environment in- teractions on 10 seeds for the chosen hyperparameters (a single set of hyperam- eters for all tasks) and evaluate the agents every 50k environment interactions (without exploration noise) on 30 episodes. An episode is considered a success if the goal is achieved during the episode. The AQuaDem discretization trained offline using 50k gradient steps on batches of size 256. The number of actions considered were 10, 15, 20 and we found 10 to be performing the best. Figure 4 shows the AQuaDem loss through the training procedure of the discretization step, and the Figure shows the returns of the trained agents as well as their success rate. On Door, Pen, and Hammer, the AQuaDQN agent reaches high success rate, largely outperforming SACíD in terms of success and sample efficiency.
130K 1508 38.00 4008 3000
Figure 4: AQua- Dem discretization loss.
5

---

Under review as a conference paper at ICLR 2022
Lioor
Pin
110
ADLEDON
40
0.5
M0
04
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
012
0.2
022
2-1
5/6
0.0
1.5
0.0
0.00
1.0
Steps
Starge
Figure 5: Performance AQuaDQN against SAC and SACID baselines. Agents are evaluated every 50k environment steps over 30 episodes. We represent the median performance in terms of success rate (bottom) and returns (top) as well the interquartile range over 10 seeds.
On Relocate, all methods reach poor results (although AQuaDQN slightly
outperforms the baselines). The task requires a larger degree of generalisation than the other three since the goal state and the initial ball position are changing at each episode. We show in Figure that when tuned uniquely on the Relocate environment and with more environment interactions, AQuaDQN manages to reach a 50% success rate where other methods still fail. Notice that on the Door environment, the SAC and SACID agents outperform the AQuaDQN agent in terms of final return (but not in term of success rate). The behavior of these agents are however different from the demonstrator since they consist in slapping the handle and abruptly pulling it back, We provide videos of all resulting agents (one episode for each seed which is not cherry picked) to demonstrate that AQuaDQN consistently learns a behavior that is qualitatively closer to the demonstrator.
Relocate
SACID
Figure 6: Performance of AQuaDQN against SAC and SACID baselines when all are tuned on the Relocate environment. We represent the median performance in terms of success rate well as the interquartile range over 10 seeds.
BAC
Succos
Humae
00 33
Euctronment Steps
4.2 IMITATION LEARNING
Setup. In Imitation Learning, the task is not specified by the reward function but by the demon- strations themselves. The goal is to mimic the demonstrated behavior. There is no reward function and the notion of success ill-defined (Hussenot al., 2021). number of existing works (Ho and Ermon, 2016; Ghasemipour et al., 2019; Dadashi et al., 2021) 1) cast the problem into matching the state distributions of the agent and of the expert. Imitation Learning of particular interest when designing satisfying reward function -one that would lead the desired behavior to be the only optimal policy- harder than directly demonstrating this behavior. In this setup, there no reward provided, not in the environment interactions nor in the demonstrations. We again consider the Adroit environments and the human demonstrations which consist 25 episodes acquired via a virtual reality system.
Algorithm & baselines. Again, the algorithm we propose has two stages. 1) We learn -fully offline- a discretization of the action space using AQuaDem. 2) We train discrete action version of the GAIL algorithm (Ho and Ermon, 2016) in the discretized environment. More precisely, we interleave the training of a discriminator between demonstrations and agent experiences, and the training of Munchausen DQN agent that maximizes the confusion of this discriminator. The Munchausen DQN takes one of the candidates actions given by AQuaDem. We call this algorithm AQuaGAIL. As baseline, we consider the GAIL algorithm with a SAC (Haarnoja et al., 2018) agent directly maximizing the confusion of the discriminator. This results in a very similar algorithm as the one proposed by Kostrikov et al. (2019). We also include the results of BC (Pomerleau, 1991).

---

Under review as a conference paper at ICLR 2022
Lioor
Hestere
Pra
6.6
0.6
AQU
0.4
0.4
0.2
0.0
0.0
5.0
0.0
16
16
165
20-
Ahil
AQUaGAL
109
14
1.
12
12
23-
Ensfrommen
106
Figure 7: Performance of AQuaGAIL against GAIL and BC baselines. Agents evaluated every 50k environment steps over 30 episodes. We represent the median success rate (top row) on the task as well as the Wasserstein distance (bottom row) of the agent's state distribution to the expert's 's state distribution as well as the interquartile range over 10 seeds.
Evaluation & results. We train AQuaGAIL and GAIL for 1M environment interactions on 10 seeds for the selected hyperparameters (a single set for all tasks). BC trained for 60k gradient steps with batch size 256. We evaluate the agents every 50k environment steps during training (without exploration noise) on 30 episodes. The AQuaDem discretization is trained offline using 50k gradient steps on batches of size 256. The results are provided in Figure 7. Evaluating imitation learning algorithms has to done carefully as the goal to "mimic a behavior" is ill-defined. Here, we provide the results according to two metrics. On top, the success rate is defined in Section 4.1. Notice that the human demonstrations do not have a success score of on every task. We see that, except for Relocate, which is a hard task to solve with only 25 human demonstrations due to the necessity to generalize to new positions of the hall and the target, AQuaGAII. solves the tasks as successfully as the humans, outperforming GAIL and BC. Notice that results corroborate previous work (Orsini et al., 2021) that showed poor performance of GAIL on human demonstrations after 1M steps. The second metric we provide, on the bottom, the Wasserstein distance between the state distribution of the demonstrations and the one of the agent. We compute it using the POT library (Flamary et al., 2021) and use the Sinkhorn distance, regularized version of the Wasserstein distance, as faster to compute. The "human" Wasserstein distance score is computed by randomly taking episodes out of the 25 human demonstrations and compute the Wasserstein distance to the remaining 20. We repeat this procedure 100 times and plot the median (and the interquartile range) of the obtained values. Remark that AQuaGAIL is able to get much closer behavior to the human than BC and GAIL on all four environments in terms of Wasserstein distance. This supports that AQuaDem leads to policies much closer to the demonstrator. We provide videos of the trained agents as an additional qualitative empirical evidence to support this claim.
4.3 REINFORCEMENT LEARNING WITH PLAY DATA
Setup. The Reinforcement Learning with play data is an under-explored yet natural setup (Gupta et al., 2019). In this setup. the environment of interest has multiple tasks, a shared observation and action space for each task, and a reward function specific to each of the tasks. We also assume that have access play data, introduced by Lynch et al. (2020), which consists in episodes from human demonstrator interacting with an environment with the sole intention to play with it. The goal to learn an optimal policy for each of the tasks. We consider the Robodesk tasks (Kannan et al., 2021) 1) shown in Figure 9, for which we acquired play data. We expand on the environment as well as the data collection procedure in the Appendix D.2.
Algorithm & baselines. Similarly to the RLID setting, we propose a two-fold training procedure: 1) we learn discretization of the action space in a fully offline fashion using the AQuaDem framework on the play data, 2) we train a discrete action deep RL algorithm using this discretization on each AQuaPlay. Unlike the RLfD setting, the demonstrations do not include any task specific reward nor goal labels meaning that we cannot incorporate the demonstration episodes in the replay buffer nor use some form of goal-conditioned BC. We use SAC as a baseline,
tasks. We refer to this algorithm
7

---

Under review as a conference paper at ICLR 2022
which is trained to optimize task specific rewards. Since the action space dimensionality is fairly low (5-dimensional), we can include naive uniform discretization baselines that we refer to as (Bushaw, 1952). The original "bang-bang" controller (BB-2) is based on the extrema of the action space, we also provide a uniform discretization scheme based on and 5 bins per action dimension, that we refer as BB-3 and BB-5 respectively.
Evaluation & results. We train the different methods on 1M environment interactions on 10 seeds for the chosen hyperparameters (a single set of hyperameters for all tasks) and evaluate the agents every 50k environment interactions (without exploration noise) on 30 episodes. The AQuaDem discretization is trained offline on play data using 50k gradient steps on batches of size 256. The number of actions considered were 10, 20, 30, 40 and we found 30 to be performing the best. It is interesting to notice that higher than for the previous setups. It aligns with the intuition that with play data, several behaviors needs to be modelled. The results provided in Figure 8. The AQuaPlay agent consistently outperforms SAC in this setup. Interestingly, the performance the BB agent decreases with the discretization granularity, well exemplifying the curse of dimensionality of the method. In fact, BB with a binary discretization (BB-2) is competitive with AQuaPlay, which validates that discrete action RL algorithms are well performing the discrete actions are sufficient to solve the task. Note however that the Robodesk environment is relatively low-dimensional action environment, making possible have BB as baseline, which is not the case of e.g. Adroit where the action space is high- high-dimensional.
Open Slide
Open Drawer
Push Green
ADuaPlay
1500
Retura
SAC
500
500
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0,5
0.8
1.0
0.0
0.2
0.4
0.6
0.5
1.0
Upright Off Table
1500
2500
Return
600
1000
500
400
0.0
0.0
0.0
0.6
0.8
0.6
Flat Block In Shelf
Lit Uptight Block
LiB Boll
le6
250
Robamn
200
150
1.00
100
50
1200
Environment Stepe
Ecvironment Stepe
lnd
Eavironment Steps
165
Figure 8: Performance of AQuaPlay against SAC and "bang- baselines. Agents evaluated every 50k environment steps over 30 episodes. We represent the median return as well as the interquartile range over 10 seeds.
RELATED WORK
Continuous action discretization. The discretization of continuous action spaces has been intro- duced in control problems by Bushaw (1952) with the "bang- controller (Bellman et al., 1956). This naive discretization problematic in nigh-dimensional action spaces, as the number of actions grows exponentially with the action dimensionality. To mitigate this phenomenon, a possible strategy to assume that action dimensions are independent (Tavakoli al., 2018; Andrychowicz al., 2020; Vieillard et al., 2021; Tang and Agrawal, 2020), to assume or learn causal dependence between them (Metz et al., 2017; Tessler et al., 2019; Sakryukin et al., 2020; Tavakoli et al., 2021) The AQuaDem framework circumvents the curse of dimensionality as the discretization is based on the demonstrations and hence is dependent on the multimodality of the actions picked by the demonstrator rather than the dimensionality of the action space. Close to setup is the case where the action space is both discrete and continuous (Neunert et al., 2020) or the action space is discrete and large (Dulac-Arnold et al., 2015). Those setups interesting directions for extending AQuaDem.
8

---

Under review as a conference paper at ICLR 2022
Q-learning in continuous action spaces. Policy- based methods consist in solving continuous or discrete MDPs based on maximizing the expected return over the parameters of a family of policies. If the return is estimated through Monte Carlo rollouts, this leads to algorithms that typically sample- inefficient and difficult to train in high-dimensional action spaces (Williams, 1992; Schulman et al., 2015; 2017). As result, a number of policy- shaiic methods inspired from the policy gradient theorem (Sutton et al., 2000), aim at maximizing the return using an approximate version of the Q-value thus making them more sample-efficient. One common architecture to parameterize Q-value, which is estimated by enforcing Bellman consistency, and define policy using an optimization procedure of the parametrized Q-value. Typical strategies solve the Q-value maximization include enforcing the Q-value to be concave (Gu et al., 2016; Amos et al., 2017) making it easy to optimize through e.g. gradient ascent, to use a black box optimization method (Kalashnikov et al., 2018; Simmons- Edler et al., 2019; Lim et al., 201 8), to solve mixed integer programming problem (Ryu et al., 2020), or to follow a biased estimate of the policy gradient based on the approximate Q-value (Konda and Tsitsiklis, 2000; Lillicrap et al., 2016; Haarnoja al., 2018; Fujimoto et al., 2018). Recently, Asadi et al. (2021) proposed to use a network that outputs actions together with their associated Q-values, tuned for each of the tasks at hand, approximate Q-value is a key problem that does not appear in discrete action space environments, thus justifyin the interest of the AQuaDem framework.
lowddimensional action spaces. Note that maximizing the
Hierarchical Imitation Learning. A number of approaches have explored the learning of primi- tives or options from demonstrations together with a high-level controller that is either learned from demonstrations (Kroemer et al., 2015; Krishnan et al., 2017; Le et al., 2018; Ding et al., 2019; Lynch et al., 2020), or learned from interactions with the environment (Manschitz et al., 2015; Kipf et al., 2019; Shankar et al., 2019), or hand specified (Pastor et al., 2009; Fox et al., 2019). AQuaDem can be loosely interpreted as a two-level procedure as well, where the primitives (action discretization step) are learned fully offline, however there is no concept of goal nor temporally extended actions.
Modeling multimodal demonstrations. number of works have modeled the demonstrator data using multimodal architectures. For example, Chernova and Veloso (2007); Calinon and Billard (2007) introduce Gaussian mixture models in their modeling of the demonstrator data. More recently, Rahmatizadeh et al. (2018) use Mixture density networks together with a recurrent neural network to model the temporal correlation of actions as well as their multimodality. (Yu al., 2018) also uses Mixture density networks to meta-learn a policy from demonstrations for one-shot adaptation. Another recent line of works has considered the problem of modeling demonstrations using an energy-based model, which is well adapted for multimodalities (Jarrett et al., 2020; Florence et al., 2021). Singh et al. (2020) also exploit the demonstrations prior for downstream tasks by learning a prior using a state- conditiondi action generative model coupled with continuous action algorithm. This is different from AQuaDem that exploits the demonstrations prior to learn a discrete action space in order to use discrete action RL algorithms.
PERSPECTIVES AND CONCLUSION
With the AQuaDem paradigm, we provide simple yet powerful method that enables to use discrete- action deep RL. methods on continuous control tasks using demonstrations, thus escaping the com- plexity or curse of dimensionality of existing discretization methods. We showed in three different setups that it provides substantial gains in sample efficiency and performance and that it leads to qualitatively better agents, as enlightened by the videos provided in the supplementary material.
There are a number of different research avenues opened by AQuaDem. Other discrete action specific methods could be leveraged in similar way in the context of continuous control: count-based exploration (Tang et al.. 2017). planning (Browne et al., 2012) or offline RL (Lagoudakis and Parr, 2003; Riedmiller, 2005). Similarly number of methods Imitation Learning (Brantley et al., 2019; Wang et al., 2019) or in offline RL (Fujimoto and Gu, 2021; Wu et al., 2019) are evaluated on continuous control tasks and are based on Behavioral Cloning regularization which could be refined using the same type of multioutput architecture used in this work. Another possible direction for the AQuaDem framework is to be analyzed in the light of risk-MDPs as the constraint of the action space arguably reduces notion of risk when acting in this environment. Finally, the gain of sample efficiency is clear in different experimental settings, we believe that the AQuaDem framework could be an interesting avenue for learning controllers on physical systems.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. In International Conference Machine Learning, 2017. 9
OpenAl: Marcin Andrychowicz. Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray. et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 2020. 1,8
Kavosh Asadi, Neev Parikh, Ronald E Parr, George D Konidaris, and Michael L Littman. Deep radial-basis value functions for continuous control. In AAAI Conference on Artificial Intelligence, 2021. 9
Andrew Barto, Marco Mirolli, and Gianluca Baldassarre, Novelty or surprise? Frontiers in psychology.
2013. 5
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2013.
R. Bellman. A markovian decision process. Indiana University Mathematics Journal, 1957. 2
Richard Bellman, Irving Glicksberg, and Oliver Gross. On the "bang-bang" control problem. Quarterly of Applied Mathematics, 1956.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dkbiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06660 2019.
Dimitri Bertsekas. Dynamic programming and optimal control. Athena scientific Belmont, 2000.
Christopher M Bishop. Mixture density networks. 1994. 15
Kianté Brantley, Wen Sun, and Mikael Henaff. setprreemment-rgulrrizeee imitation learning. In International Conference on Learning Representations, 2019. 9
Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in games, 2012. 9
Donald W. Bushaw. Differential equations with a discontinuous forcing term. Technical report, 1952. 8
Sylvain Calinon and Aude Billard. Incremental learning of gestures by imitation in humanoid robot.
In ACM/IEEE international conference on Human-robot interaction, 2007. 9
Sonia Chernova and Manuela Veloso. Confidence-baase policy learning from demonstration using gaussian mixture models. In International Conference Autonomous Agents and Multiagent Systems, 2007.
Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imitation learning. International Conference on Learning Representations, 2021. 6
Yiming Ding, Carlos Florensa, Mariano Phiclipp, and Pieter Abbeel. Goal-conditioned imitation learning, Advances Neural Information Processing Systems, 2019.
Gabriel Dulac-Amold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv: 1512.07679 2015. 8
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport, Journal of Machine Learning Research, 2021.
10

---

Under review as a conference paper at ICLR 2022
Pete Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. arXiv preprint arXiv:2109.0000 100.:0017900177 2021. 9
Roy Fox, Ron Berenstein, lon Stoica, and Ken Goldberg. Multi-task hierarchical imitation learning for home automation. In International Conference on Automation Science and Engineering (CASE). IEEE, 2019.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2106.0688 2021. 9
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference Machine Learning, 2018. 2, 9.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. Conference on Robot Learning. 2019.
Shixiang Gu, Timothy Lillicrap, lly Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International Conference on Machine Learning, 2016. 9
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long horizon tasks via imitation and reinforcement learning. Conference on Robol Learning (CoRL., 2019. 7
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, 2018. 2, 6,9
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, al. Deep q-learning from demonstrations. In AAAI Conference on Artificial Intelligence, 2018. 5, 17
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural
Information Processing Systems, 2016. 6
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, et al. Acme: research framework for distributed reinforcement learning, arXiv preprint arXiv:2006.00977 2020. 17
Léonard Hussenot, Marcin Andrychowicz, Damien Vincent, Robert Dadashi, Anton Raichuk, Sabela Ramos, Nikola Momchev, Sertan Girgin, Raphael Marinier, Lukasz Stafiniak, et Hyperparameter selection for imitation learning. In International Conference on Machine Learning, 2021. 6
Daniel Jarrett, Ioana Bica, and Mihaela van der Schaar. Strictly batch imitation learning by energy- based distribution matching. Advances Neural Information Processing Systems, 2020.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on Robot Learning. 2018. 9
Harini Kannan, Danijar Hafner, Chelsea Finn, and Dumitru Erhan. Robodesk: A multi-task rein- forcement learning benchmark. bttps://qithub com/ www.c//wo./cccooeeeeo 2021. 7, 16
Thomas Kipf, Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez-Gonzalez, Edward Grefen- stette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. In International Conference on Machine Learning, 2019. 9
Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information
Processing Systems, 2000. 1,
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-acton criticc Addressing sample inefficiency and reward bias in adversarial imitation learning. International Conference on Learning Representations, 2019. 6
11

---

Under review as a conference paper at ICLR 2022
Sanjay Krishnan, Roy Fox, Ion Stoica, and Ken Goldberg. Ddco: options for robot learning from demonstrations. Conference on Robot Learning. 2017.
Discovery of deep continuous
Oliver Kroemer, Christian Daniel, Gerhard Neumann, Herke Van Hoof, and Jan Peters. Towards learning hierarchical skills for multi phale manipulation tasks. In International Conference on Robotics and Automation. IEEE, 2015. 9
Michail G Lagoudakis and Ronald Parr. Lquat-squares policy iteration. The Journal of Machine Learning Research, 2003. 9
Hoang Le, Nan Jiang, Alekh Agarwal, Miroslav Dudík, Yisong Yue, and Hal Daumé. Hierarchical imitation and reinforcement learning. In International Conference on Machine Learning, 2018.
Timothy P LLillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. International Conference on Learning Representations, 2016. 9
Sungsu Lim, Ajin Joseph, Lei Le, Yangchen Pan, and Martha White. Actor-expert: A framework for using q-learning in continuous action spaces. arXiv preprint
2018. 9
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning, 2020. 2, 5, 7,
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martinn What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03298, 2021. 5
Simon Manschitz, Jens Kober, Michael Gienger, and Jan Peters. Learning movement primitive attractor goals and sequential skills from kinesthetic demonstrations, Robotics and Autonomous Systems, 2015.
Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete sequential prediction of continuous actions for deep rl. arXiv preprint arXiv: 055055 2017. 1,
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 2015. 2, 5, 17
Ashvin Nair, Bob McGrew, Marein Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Overcom- ing exploration in reinforcement learning with demonstrations. In IEEE International Conference on Robotics and Automation. IEEE, 2018.
Michael Neunert, Abbas Abdolmaleki, Markus Wulfmeier, Thomas Lampe, Tobias Springen- berg, Roland Hafner, Francesco Romano, Jonas Buchli, Nicolas Heess, and Martin Riedmiller. Continuoss-discrtte reinforcement learning for hybrid control in robotics. In Conference Robot Learning, 2020. 8
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. International Conference on Machine Learning, 1999. 2
Andrew Ng. Stuart Russell, et Algorithms for inverse reinforcement learning. International Conference Machine Learning, 2000. 2
Manu Orsini, Anton Raichuk, Léonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, and Marcin Andrychowicz. What matters for adversarial imitation lcarning? arXiv preprint arXiv:2106.00672. 2021. 7
Peter Pastor, Heiko Hoffmann, Tamim Asfour, and Stefan Schaal. Learning and generalization of motor skills by learning from demonstration. In International Conference on Robotics and Automation. IEFÉ, 2009.
12

---

Under review as a conference paper at ICLR 2022
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 1991. 3,6
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley Sons, 2014. 1,2
Rouhollah Rahmatizadeh, Pooya Abolghasemi, Aman Behal, and Ladislau Bölöni. From virtual demonstration real-world manipulation using Istm and mdn. In AAAI Conference on Artificial Intelligence, 2018. 9
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Robotics: Science and Systems, 2017. 5, 16
Martin Riedmiller. Neural fitted q iteration- first experiences with a data efficient neural reinforcement learning method. In European Conference Machine Learning. Springer, 2005. 9
Stuart Russell. Learning agents for uncertain environments. In Conference on Computational learning theory, 1998. 2
Moonkyung Ryu, Yinlam Chow, Ross Anderson, Christian Tjandraatmadja, and Craig Boutilier. Caql: Continuous action q-learning. International Conference on Learning Representations, 2020.
Andrey Sakryukin, Chedy Raissi, and Mohan Kankanhalli. Inferring DQN structure for high-
dimensional continuous control. 2020. 1, 8
Tim Salimans and Richard Chen. Learning montezuma's 's revenge from a single demonstration. arXiv preprint 1812.01811.33350 2018. 2
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. International Conference Learning Representations, 2016. 17
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, 2015. 9
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017. 9
Tanmay Shankar, Shubham Tulsiani, Lemrll Pinto, and Abhinav Gupta. Discovering motor programs by recomposing demonstrations. In International Conference on Learning Representations, 2019.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schriuwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. Nature, 2016.
Riley Eimlons-Edllr, Ben Eisner, Eric Mitchell, Sebastian Scung, and Daniel Lee. Q-learning for continuous actions with cross-cntropy guided policies. arXiv preprint 1rXiv:1903.10603 2019.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. International Conference on Learning Representations, 2020. 2,
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, 2000.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, 2017. 9
13

---

Under review as a conference paper at ICLR 2022
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization. In AAAI Conference on Artificial Intelligence, 2020. 1, 8
Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action branching architectures for deep rein- forcement learning. In AAAI Conference on Artificial Intelligence, 2018. 1,
Arash Tavakoli, Mehdi Fatemi, and Petar Kormushev. Learning to represent action values as hypergraph on the action vertices. Internation Conference in Learning Representations, 2021. 1, 8
Chen Tessler, Guy Tennenholtz. and Shie Mannor. Distributional policy optimization: An alternative approach for continuous control. Advances in Neural Information Processing Systems, 2019. 8
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In AAAI Conference Artificial Intelligence, 2016. 17
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang. Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv: (XXx::1077008111 2017. 2, 5, 17
Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Munchausen reinforcement learning. Advances in Neural Information Processing Systems, 2020. 5, 17
Nino Vieillard, Marcin Andrychowicz. Anton Raichuk, Olivier Pietquin, and Matthieu Geist. Implic- itly regularized rl with implicit q-values. arXiv preprint urXiv:2108.07041, 2021.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 2019.
Ruohan Wang, Carlo Ciliberto, Pierluigi Vito Amadori, and Yiannis Demiris. Random expert distillation: Imitation learning via expert policy support estimation. In International Conference on Machine Learning, 2019.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 1992. 2
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 1992. 1,9
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint arXiv: 1361, 2019. 9
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. Robotics: Science and Systems, 2018. 9
Brian Ziebart, Andrew Maas, Andrew Bagnell, and Anind Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference Artificial Intelligence, 2008. 2
14

---

Under review as a conference paper at ICLR 2022
A CONNECTION TO GAUSSIAN MIXTURE MODELS
The BC loss can be interpreted as a maximum likelihood objective under the assumption that the demonstrator data comes from Gaussian distribution. Similarly Mixture density networks (Bishop. 1994), we propose to replace the Gaussian distribution by a mixture of Gaussian distributions. Suppose we represent the probability density of action conditioned on state by mixture of K Gaussian kernels: p(als) Σx Org (s)dg (a|s), where (g{()) the mixing coefficient (that can be interpreted as a state conditioned prior probability), and (als) is the conditional density of the target a. Now assuming that the kernels are centered on (s) km
k=1:: I:K and have fixed covariance
221 1 where a is hyperparameter, we can write the log-likeldhood of the demonstrations data D as:
CL(D)
Σ
log(p(s)p(als))
Σ log + log
Σ k=1
On (s())de(alss)
6
8,0ED
K
(s)(s) ( | al2
Σ s,0DD
log + log (C'
Σ
Ø²
Therefore minimizing the negative log likelihood reduces to minimizing:
K
(s) (- [ - all -
Σ - log Σ
k=1
We propose to use a uniform prior (s) L when learning the locations of the centroids, which leads exactly to Equation (1) where the variance o² the temperature T. Note that we initially learned the state conditioned prior as (s), but we found no empirical evidence that it may be used to improve the performance of the downstream algorithms defined in Section 4.
B ACTION VISUALIZATION IN A HIGH-DIMENSIONAL EN ENVIRONMENT
For the Door environment (see Figure 9) represent the actions candidates learned using the AQuaDem framework with videos that can be found in the supplementary material in the folder visualization (for the best hyperparameters in the RL with demonstrations setting see Ap- pendix D.4.1), As the action space is of high dimensionality, we choose to represent each action dimension on the x-axis, and the value for each dimension on the y-axis. We connect the dots on the x-axis to facilitate the visualization through time. We replay a trajectory from the human demonstrations and show at each step the actions proposed by the AQuaDem network, and the action actually taken by the human demonstrator. Each action candidate has color consistent across time (meaning that the blue action always correspond to the same head of the Y network). Interestingly, the video shows that actions are very state dependent (except some default O-action) and evolve smoothly through time.
C ABLATION STUDY
In this section, we provide two ablations the AQuaDQN algorithm. The first ablation to learn fixed set of actions independently of the state (which reduces to K-means). The second ablation consists in using random actions rather than the actions learned by the AQuaDem framework (the actions given by the AQuaDem network, randomly initialized and not trained). We use the same hyperparameters as the one selected for AQuaDQN. In each case, for a number of actions in (5, 10, 25), the success rate of the agent is for all tasks throughout the training procedure.
D IMPLEMENTATION
D.1 GRID WORLD VISUALIZATIONS
We learn the discretization of the action space using the AQuaDem framework. The architecture of the network is common hidden layer of size 256 with relu activation, and subsequent hidden layer
15

---

Under review as a conference paper at ICLR 2022
of size 256 with relu activation for each action. We minimize the AQuaDem loss using the Adam optimizer with the learning rate 0.0003 and the dropout regularization rate 0.1 on 20000 gradient steps.
D.2 ENVIRONMENTS
We considered the Adroit environments and the Robodesk environments, for which we described the observation space and the action space in Table
Environment
Observation Space
Action Space
Door
39
28
Hammer
46
26
Pen
45
24
Relocate
39
30
Robodesk
Table 1: Environment description of the Adroit and Robodesk observation and action space.
Pun
Ralorate
Ralsolok
Figure Visualizations of the Adroit and Robodesk environments.
The Adroit environments (Rajeswaran et al., 2017) consists in shadow hand solving tasks (Figure 9). The environments come with demonstrations which are gathered using virtual reality by a human.
Adroit
Robodesk The Robodesk environment (Kannan et al., 2021) consists of a simulated Franka Emika Panda robot interacting with table where multiple tasks are possible. The version of the simulated robot the Robodesk environment only includes DoFs (vs the 7 DoFs available, were made not controllable). We evaluate AQuaPlay on the 9 base tasks described in Robodesk: open_s de, open.drawez, push-green, stack, wtw.:igghh...ccof............ flat.block.in.shelf. htt.:ashhhe..........ggigh..lloo wittckiiiiiff.. lift.ball.
We used the RLDS creator github. com/ woom//gooleeeeee weee-rreeeeerrrbbeee creator to generate play data, together with Nintendo Switch Pro Controller. The data is composed by 50 episodes of approximately minutes where the goal the demonstrator is to interact with the different elements of the environment.
D.3 HYPERPARAMETER SELECTION PROCEDURE
In the section we provide the hyperparameter selection procedure for the different setups. For the RLfD setting (Section 4.1) and the IL setting (Section 4.2) the number of hyperparameters is prohibitive to perform grid search. Therefore, we propose to sample hyperparameters uniformly within the set all possible hyperparameters. For each environment, we sample 1000 configurations of hyperparameters, and train each algorithm including the baselines. We compute the average success rate of each individual value on the top 50% of all corresponding configurations (since poorly performing configurations are less informative) and select the best performing hyperparameter value independently. This procedure enables to 1) limit combinatorial explosion with the number of hyperparameters 2) provide fair evaluation between the baselines and the proposed algorithms as they all rely on the same amount of compute. the supplementary material, we provide histograms detailing the influence of each hyperparameter. For the RLiP setting, fixed the parameters related to the DQN algorithm with the ones selected in the RLfD setting to limit the hyperparameter search, which enables to perform grid search for 3 seeds, and select the best set of hyperparameters.
16

---

Under review as a conference paper at ICLR 2022
D.4 REINFORCEMENT LEARNING WITH DEMONSTRATIONS
D.4.1 AQUADQN
We learn the discretization of the action space using the AQuaDem framework. The architecture of the network is a common hidden layer of size 256 with relu activation, and subsequent hidden layer of size 256 with relu activation for each action. We minimize the AQuaDem loss using the Adam optimizer and dropout regularization.
We train DQN agent on top of the discretization learned by the AQuaDem framework. The architecture of the Q-network we use the default LayerNorm architecture from the Q-network of the ACME library (Hoffman et al., 2020), which consists in hidden layer of size 512 with layer normalization and tanh activation, followed by two hidden layers of sizes 512 and 256 with elu activation. We explored multiple Q-value losses for which we used the Adam optimizer: regular DQN (Mnih et al., 2015), double DQN with experience replay (Van Hasselt et al., 2016; Schaul et al., 2016), and Munchausen DQN (Vieillard al., 2020); the latter led to the best performance. We maintain a fixed ratio of demonstration episodes and agent episodes in the replay buffer similarly Hester al. (2018). We also provide hyperparameter an optional minimum reward to the transitions of the expert to have a denser reward signal. The hyperparameter sweep for AQuaDQN can be found in Table 2. The complete breakdown of the influence of each hyperparameter provided in hps_1 _. aquadqn. html in the supplementary.
Hyperparameter
Possible values
aquadem learning rate aquadem input dropout rate aquadem hidden dropout rate aquadem temperature aquadem # actions
3e-5, 0.0001, 0.0003, 0.001, 0.003
0, 0.1, 0.3
0, 0.1, 0.3
0,0001, 0.001, 0.01
10, 15, 20
dqn learning rate
0.00003, 0.0001, 0.003 1, 5
dqn step
dgn epsilon
0,001, 0.01, 0.1 0, 0.1, 0.25, 0.5 None, 0.01
dqn ratio of demonstrations
dqn min reward of demonstrations
Table 2: Hyperparameter sweep for the AQuaDQN agent
When selecting hyperparameters specifically for Relocate, for Figure 6, the main difference in the chosen values a dropout rate set
D.4.2 SAC AND SACFD
We reproduced the authors' implementations (with adaptive temperature) and MLP networks for both the actor and the critic with two hidden layers of size 256 with relu activation. We use an Adam optimizer to train the SAC losses. We use a replay buffer of size 1M, and sample batches of size 256. We introduce a parameter of gradient updates frequency n which indicates a number of n gradient updates on the SAC losses every n environment steps. SACID version of SAC inspired by DDPGfD (Vecerik et al., 2017) where we add expert demonstrations to the replay buffer of the SAC agent with ratio between the agent episodes and the demonstration episodes which is a hyperparameter. We also provide hyperparameter an optional minimum reward to the transitions of the expert to have a denser reward signal. We found that the best hyperparameters for SAC are the same for SACíD. The HP sweep for SAC and SACfD can be found in Table 3 and Table 4. The complete breakdown of the influence each hyperparameter provided in hps 1 1fd sac .tmll and hps 1 fd sacfd. html in the supplementary.
17

---

Under review as a conference paper at ICLR 2022
Hyperparameter
Possible values
learning rate
3e-5, 1e-4, 0.0003
step tau reward scale
1, 5 0.005, 0.01, 0.05 0.1, 0.3, 0.5
Table 3: Hyperparameter sweep for SAC.
Hyperparameter
Possible values
learning rate
3e-5, 1e-4, 0.0003
step tau reward scale
1,
0.005, 0.01, 0.05 0.1, 0.3, 0.5
ratio of demonstrations mini reward of demonstrations
0.001, 0.1, 0.25
None, 0.01, 0.1
Table 4: Hyperparameter sweep for SACfD.
D.5 IMITATION LEARNING
D.5.1 AQUAGAIL
We learn the discretization of the action space using the AQuaDem framework. The architecture ofthe network is common hidden layer of size 256 with relu activation, and a subsequent hidden layer of size 256 with relu activation for each action. We minimize the AQuaDem loss using the Adam optimizer and dropout regularization. The discriminator a MLP whose number of layers, number of units per layers are hyperparameters. We use the Adam optimizer with two possible regularization scheme: dropout and weight decay. The discriminator outputs value from which compute three possible rewards log(p), 0.5 (p) + log(1 - p). log(1 - p) corresponding to the reward balance hyperparameter. The direct RL algorithm Munchausen DQN, with the same architecture and hyperparameters described Section 1.4.1. The hyperparameter sweep for AQuaGAIL. can be found in Table 5. The complete breakdown of the influence of each hyperparameter provided in hps-il.aquagail html in the supplementary.
Hyperparameter
Possible values
discriminator learning rate discriminator num layers discriminator num units discriminator regularization discriminator weight decay discriminator input dropout rate discriminator hidden dropout rate discriminator observation normalization
le-7, 1e-6, 3e-5, le-4 1,2
16, 64, 256
none, dropout, weight decay
5, 10, 20
0.5, 0.75
0.5, 0.75 True, False 0., 0.S. 1.
discriminator reward balance
dqp learning rate
3e-5, le-4, 3e-4
dqn step
1.3.5
dqn epsilon
0.001, 0.01, 0.1
aquadem learning rate
3e-5, 1e-4, 3e-4, 1e-3, 0.0001, 0.001, 0.01
aquadem temperature
aquadem num actions aquadem input dropout rate
10, 15, 20
0, 0.1, 0.3
aquadem hidden dropout rate
0. 0.1, 0.3
Table 5: Hyperparameter sweep for the AQuaGAIL agent.
18

---

Under review as a conference paper at ICLR 2022
D.5.2 GAIL
We used the same discriminator architecture and hyperparameters as the one described in Section D.5.1. The direct RL agent is the SAC algorithm whose architecture and hyperparameters are described in Section D.4.2. The hyperparameter sweep for GAIL can be found Table The complete breakdown of the influence of each hyperparameter is provided in hps il Al_gail. .hmll the supplementary.
Hyperparameter
Possible values
discriminator learning rate discriminator num layers discriminator num units discriminator regularization discriminator weight decay discriminator input dropout rate discriminator hidden dropout rate discriminator observation normalization
1e-7, 3e-7, le-6, 3e-5, le-4 1,2 16, 64, 256
none, dropout, weight decay
10, 20
0.5, 0.75
0.5, 0.75 True, False 0., 0.5,
discriminator reward balance
sac learning rate sac step
3e-5, le-4, 3e-4 3. 0.005, 0.01, 0.05
sac tau sac reward scale
0.1, 0.3, 0.5
Table 6: Hyperparameter sweep for the discriminator part of the GAIL agent.
D.5.3 BEHAVIORAL CLONING
The BC network is MLP whose number of layers, number of units per layers and activation functions are hyperparameters. We use the Adam optimizer with two possible regularization scheme: dropout and weight decay. The observation normalization hyperparameter is set True when each dimension of the observation are centered with the mean and standard deviation of the observations in the demonstration dataset. The complete breakdown of the influence of each hyperparameter provided in hps_il b_b.l.b...... html in the supplementary.
Hyperparameter
Possible values
learning rate
le-5, 3e-5, le-4, 3e-4, le-3 1. 3
num layers num units
16, 64, 256 relu, tanh True, False 0, .011 0.1
activation observation normalization
weight decay
input dropout rate
0. 0.15, 0.3
hidden dropout rate
0. 0.25, 0.5
Table 7: Hyperparameter sweep for the BC agent.
D.6 REINFORCEMENT LEARNING WITH PLAY DATA
D.6.1 SAC
We used the exact same implementation as the one described Section D.4.2. The HP sweep can be found in Table 8.
D.6.2 DQN WITH NAIVE DISCRETIZATION
We used the exact same implementation as the one described in Section D.4.1, and also use the best hyperparameters found in the RLiD setting. As the action space (-1,1)5, we three different
19

---

Under review as a conference paper at ICLR 2022
Hyperparameter
Possible values
learning rate
le-5, 3e-5, 3e-4, 1e-4 3, 0.1, 1, 10
step
reward scale
tau
0.005, 0.01, 0.05
Table 8: Hyperparameter sweep for SAC for the Robodesk environment. The best hyperparameter set was chosen as the one that maximizes the performance on average on all tasks.
discretization meshes: (-1.1). (-1.0.,,( 0.5, 1} which induce discrete action space of dimension 2", 3,,3,, respectively. We refer to the resulting algorithm as BB-2, BB-3, and BB-5 (where BB stands for "Bang-bang").
D.6.3 AQUAPLAY
We used the exact same implementation as the one described in Section D.4.1, and also use the best hyperparameters found in the RLfD setting for the Munchausen DQN agent. We performed a sweep on the discretization step that report in Table 9.
Hyperparameter
Possible values
learning rate
0.0001, 0.0003, 0.001 0, 0.1, 0.3
dropout rate
temperature
le-4, le-3, le-2
# actions
10, 20, 30, 40
Table 9: Hyperparameter sweep for the AQuaPlay agent. The best hyperparameter set was chosen as the one that maximizes the performance on average on all tasks.
E SANITY CHECK BASELINES
We provide in Figure the results of our SAC implementation on the classical OpenAl Gym environments, for 20M steps. The hyperparameters are the one of the original papers (original paper, followed by the introduction the adaptative temperature). The results correspond to what provided in the literature.
We provide in Figure 11 the results on OpenAl Gym environment sof the configurations of SAC and AQuaDQN that were selected for Adroit (and not Mujoco!). The action discretization is learned on the medium-expert dataset from D4RL (no dataset is provided for Humanoid and that why we do in Figure 10 that SAC perform much better on Mujoco
not include in this figure). One Gym after 1M steps when selecting hyperparameters on the environments themselves.
Halltheetah-v22
Art-42
Walker2d-22
Hopper-v2
Humenoid-22
opetaade retura
15000
6000
6000
10000
7500
10000
4000
4000
5000
5000
2000
2000
1000
2500
steps
1c7
steps
1c7
stops
1e7
steps
167
steps
167
Figure 10: SAC median and interquartile range 10 seeds the Open Gym environments.
F
PROOF
For lighter notations, we write 2, -| all and = (II,..,, For a single state-action pair (the empirical expectation being relevant for studying the effect of the temperature), the
20

---

Under review as a conference paper at ICLR 2022
Wi her 22-22
A18-22
Hopper-22
SAC
A0u3tOn
2000
2000
1033
1320
165
140
1ng
Figure 11: SAC and AQuaDQN medians and interquartile ranges on 10 seeds on the Open Gym environments. These configurations the ones that were selected to perform best Adroit and not for Mujoco. One can see Figure 10 that SAC can perform much better on Mujoco Gym after IM steps when selecting hyperparameters the environments themselves.
AQuaDem loss can be rewritten:
2k
J(V) = llg
exp(
T
k=
K
1
Ik
--T log
exp(
T
T
k=1
K
(-T log(
exp(
-/ )))
log K.
T
K
Let's define fr(x) -7 log( K Σ exp(- The function fr is the same the loss up to (I) constant term that does not change the solution the optimization problem, (2) a scaling term that can be absorbed into the learning rate. So, we can study this function for the behavior of the loss with the respect to the temperature.
Now, denoting 2. ming 2k, we'll we'll first study the behavior for low temperature.
fT(x) Toog K Tlog
exp(
Lk T
km
Tlog K -loo
Jmm
exp(
Fk Fm
T
K
23 x T
T -og K + am 7 log +
exp(
1=1..mm
Tmn-
T-00
Therefore, when the temperature goes zero, fr behaves the minimum.
21

---

Under review as a conference paper at ICLR 2022
For large temperatures, we have, using Taylor expansions:
k
fr(s) = -olo
expl
1 T
))
K
K
Ik o Lk T
= =oloo
(1
K
T
k=1
K
1
Ik + al +
=Tllog(1
=
K k=
T
T
Fk + T
K
K
Tk.
T-000
K
So, when the temperature goes to infinite, fo behaves as the average.
22


Reviews:
[{"review_id": "qY_NM9GI4yx", "reviewer": "Reviewer_zgiZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem setting is interesting, timely, and could have a good amount of impact should the technique be adopted. Please consider the following comments;\n\n- The authors “reproduced” the SAC implementation. I am not fully convinced that the implementation is competitive to the original. While running a HP sweep is commendable, it is not enough to ensure that the baselines work as intended, especially in exploration tasks that can have a tremendous impact. One would need to (a) ensure that the implementation can reproduce the results in the original paper in the original environments (mostly Mujoco/Dm control suite/openai gym) and (b) run a number of experiments of Aquagail in these environments to compare performance. While I agree that it might be harder to generate meaningful play data, I do think that this shouldn’t be a problem for the imitation learning experiments. \n\n- Related work on exploration in continuous action spaces is largely missing. While exploration is not easy in this domain, there exists a large body of literature (all the different types of curiosity mechanisms, UCB inspired methods etc.) [2-6] If the approach claims superiority of discrete action spaces over continuous ones, a baseline with principled exploration in continuous action spaces is necessary.\n\n- Why not compare the approach against a hierarchical Rl/Imitation learning approach or a hybrid continuous-discrete baseline? E.g. [1] The authors also claim that this is the most related approach.\n\n- The approach states: “Our objective is to reduce a continuous control problem to a discrete action one, on which we can apply discrete-action RL methods.” I believe this alone is already a great contribution and it is not necessary to frame it as an advantage over continuous action-space RL. The paper makes a number of somewhat handwavy claims why discrete action spaces are superior but while they may sound intuitive they are not substantiated by theory or experiments.\n\n- How does the approach compare to state-of-the-art in offline RL, which provides a similar problem setting?\n\n- 10 seeds is amazing and I would love to see more papers following this direction. Nonetheless, it is not clear what the shaded regions in the plots mean. Is this the 1 standard deviation interval or a confidence interval based on the t-test? What p-value of the confidence interval is chosen? I agree that this is a minor detail point but it should be mentioned.\n\n\n[1] Neunert, M., Abdolmaleki, A., Wulfmeier, M., Lampe, T., Springenberg, T., Hafner, R., ... & Riedmiller, M. (2020, May). Continuous-discrete reinforcement learning for hybrid control in robotics. In Conference on Robot Learning (pp. 735-751). PMLR.\n\n[2] Osband, I., Blundell, C., Pritzel, A., & Van Roy, B. (2016). Deep exploration via bootstrapped DQN. Advances in neural information processing systems, 29, 4026-4034.\n\n[3] Osband, I., Van Roy, B., Russo, D. J., & Wen, Z. (2019). Deep Exploration via Randomized Value Functions. J. Mach. Learn. Res., 20(124), 1-62.\n\n[4] Chen, R. Y., Sidor, S., Abbeel, P., & Schulman, J. (2017). UCB exploration via Q-ensembles. arXiv preprint arXiv:1706.01502.\n\n[5] Seyde, T., Schwarting, W., Karaman, S., & Rus, D. (2020). Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles. 5th Conference on Robot Learning (CoRL)\n\n[6] Bai, C., Wang, L., Han, L., Hao, J., Garg, A., Liu, P., & Wang, Z. (2021, July). Principled exploration via optimistic bootstrapping and backward induction. In International Conference on Machine Learning (pp. 577-587). PMLR.\n\n[7] Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017, July). Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning (pp. 2778-2787). PMLR.\n", "summary_of_the_paper": "The paper proposes learning a state-dependent discretization of a continuous action space using demonstrations. The motivation is to enable the use of discrete-action deep RL methods instead of relying on policies with continuous action space parametrization. The authors argue that this results in improved exploration (avoiding the curse of dimensionality). The approach is evaluated on 5 environments in the Adroit and Robodesk environments on Reinforcement Learning with demonstrations, Reinforcement Learning with play data, and Imitation Learning. The authors claim that they outperform state-of-the-art continuous control methods both in terms of sample efficiency and performance on every setup.", "summary_of_the_review": "The paper presents an interesting framework for learning the discretization of action spaces. Related work needs to be extended to include exploration, the claims of benefits of discrete over continuous action spaces have to be substantiated, the evaluation has to be done on fair baselines and commonly used environments to allow the comparability. I am happy to adjust my rating should these issues be addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Xmh0I35tKTG", "reviewer": "Reviewer_teiU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper explores the connection between the continuous actions space and discrete actions space, proposes an automatic discretization. This idea is interesting, the writing is clear and easy to follow. But I have a few concerns on this paper:\n\n(1) The benefit of proposed method relies on the preference of discrete action space over the continuous action space, as it mainly provides an architecture to output multiple continuous actions and evaluate them with discrete action based Q-learning methods. However, this preference is not discussed fully in the paper. The most relevant discussion in the paper is practical comparison between proposed approach and SAC, a continuous action space based baseline. Though the practical performance of the proposed discretization looks better, it is not clear if this advantage is task specific. More analysis on this part could better support the motivation of this paper.\n\n(2) The role of the temperature T of equation (1) is not very clear. The paper states that this hyperparameter controls whether all output actions are similar to the demonstrated action or only one of them is similar to the demonstrated action. But it is not convincing enough. The proposed loss function sums up losses from all output K actions, and the temperature T is the same for all actions and hence does not seem to control significance of any individual action. As a result, the temperature seems to only control the scale of the loss.\n\n(3) Another concern is also related to the proposed loss. The proposed loss does not seem to provide guidance on the diversity of output actions. As they are bound to the same demonstrated action, it is difficult to tell how it can output multiple different actions for the same input. If the model always output similar actions in each candidates for the same input, the significance of the proposed design can be strongly weakened. Because the proposed loss is exactly the ordinary BC when K = 1, as the paper mentions.\n\n(4) As the supplementary (A) section shows, the proposed architecture looks very similar to the Gaussian mixture models. However, the paper does not compare ordinary Gaussian mixture models with the proposed method. The usage of Guassian mixture models in reinforcement learning and imitation learning problem does not seem to be novel, as we can find it in some previous works:\n\nChernova, Sonia, and Manuela Veloso. \"Confidence-based policy learning from demonstration using gaussian mixture models.\" Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems. 2007.\n\nAgostini, Alejandro, and Enric Celaya. \"Reinforcement learning with a Gaussian mixture model.\" The 2010 International Joint Conference on Neural Networks (IJCNN). IEEE, 2010.\n\nChoi, Yunho, Kyungjae Lee, and Songhwai Oh. \"Distributional deep reinforcement learning with a mixture of gaussians.\" 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.\n\nThen the novelty of proposed method may also strongly depends on how it is better than related previous works. The novelty and significance of this paper can be shown clearer if convincing analysis and comparison to previous Gaussian mixture models works are provided.", "summary_of_the_paper": "This paper looks into improving data efficiency in continuous action space reinforcement learning problem by connecting it to its discrete action space correspondence and taking advantage of demonstration data. To be specific, the paper proposes learning a state dependent discretization of the original continuous action space by mimicking actions in demonstration trajectories. This discretization output a fixed number of actions for each input state. In a grid world example, it is visualized that the proposed method learns multimodal demonstrated action better as it can catch different actions in the same state. In the experiment section, the proposed method is compared to baselines in various demonstration and reinforcement learning setting and shows benefit over compared baselines.", "summary_of_the_review": "The paper provides an interesting idea on connecting discrete action space and continuous action space for reinforcement learning and imitation learning. The writing is clear. But some important discussion seems to be insufficient and hence its novelty and significance is not clear enough.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SnQnxbVpE4d", "reviewer": "Reviewer_hp36", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* This paper proposed an action space discretization method, it claims to not suffer from the curse of dimensionality and not needing any specific assumptions about the task. I agree with the author that this is an important topic that RL community should keep an eye on. However, I might not agree that discrete control problems are harder than continuous control problems. I think the argument in the introduction that since continuous control problem is much harder than discrete ones, so we should turn a continuous problem into a discrete problem is debatable. \n\n* The main idea is to capture the mode of continuous actiosn taken under different states through a Gaussian Mixture Model type of neural network. The key novelty is Equation 1. where Figure 2 explains the key idea very well. \n\n* On the empirical side, the author has considered three settings: RL with demonstration, RL with play data, and imitation learning. The significance of the results is strong.  ", "summary_of_the_paper": "This paper proposes a novel method: Action Quantization from Demonstrations (AQuaDem) to learn a discretization of continuous action spaces by leveraging the priors of demonstrations. As such, one can apply any discrete action deep RL algorithm to continuous problems. In three scenarios: RL with demonstration, RL with play data and imitation learning, the effectiveness is validated. ", "summary_of_the_review": " I think the technical side of the story is sound, the method taken is reasonable and the empirical results are valid. However, I think the novelty of this paper is limited. The idea is rather simple, and learning discretization through a gaussian mixture model type of representation is trivial. I am not quite convinced at the current stage that it reaches an ICLR level of work yet. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GPK8rnYYSMq", "reviewer": "Reviewer_MjVt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths: \n1) Repurposing BC and extending it to serve as a discrete action-space learner is interesting.\n2) Paper is generally well written, and easy to follow. \n3) Choices of methods to combine the proposed approach with and the environments to test in are reasonable. \n4) The breadth of discussing the related work is reasonable. \n\n--- \n## Weaknesses:\n1) Some important missing baselines (some of which are referred to briefly but somewhat dismissed in experiments). \n2) Some missing related works.  \n3) Presentation at some points is misleading or incomplete.\n4) Some discussions missing. \n\n---\n## Issues:\n- **Sec. 1, paragraph 2** \"*[in finite discrete action spaces] exploration is more manageable [and] computing the maximum of the action-value function is straightforward*\"**:** A finite discrete action space can have a much higher intrinsic complexity than a continuous action space. As such, I wouldn't say that in general exploration is more manageable. For instance, finite but large combinatorial action spaces can be much harder to explore with action-composite methods (such as standard DQN) as opposed to when a continuous embedding can be assumed and exploited using PG methods (see, e.g., Dulac-Arnold et al. (2015)). In addition, the exponential complexity of maximization in combinatorial discrete action spaces can fast become computationally intractable --- it isn't straightforward in general.  \n\n- **Sec. 1, paragraph 2:** The end of this paragraph seems to imply two algorithmic choices for continuous-action control: (1) MC/Reinforce-style, (2) Q-learning-style. This could be written much better in my view to encapsulate the full landscape of approaches better (e.g. actor-critic methods). Otherwise, clarify what is meant.\n\n- **Bang-bang controller:** BB controller implies a very specific type of discretization scheme, where only the extrema of the action space are used in the control problem. However, in this paper, you use this to refer to the generic uniform discretization scheme. Not sure if referring to this scheme as BB is really correct or helpful. \n\n- **Sec. 1, paragraph 3:** Andrychowicz et al. (2020) does not assume a causal dependence between action dimensions as far as I know. If they do, please explain in your response. Also, I believe this paragraph is a good place to also refer to approaches that assume independence between action dimensions: (Tavakoli et al., 2018; Tang and Agrawal, 2020; Andrychowicz et al., 2020).\n\n- **Sec. 1, paragraph 3** \"*[...] but they are typically complex and task-specific.*\" **:** Most of these approaches are trivial to implement. Regarding task-specificity, they are no more task-specific than standard model-free methods such as DQN and DDPG are. Regarding being complex, could you clarify in what respect it is meant? Regarding task-specificity, I personally understand that your method relies less on the task specification, but do state more clearly for the reader in what way your method is less task-specific. \n\n- **Sec. 2, paragraphs 1-2:** *r* is used to denote both a *random variable* (when discussing the Q-learning update rule with sampled transitions) and the *reward function* (e.g. when expressing the recursive form of $Q^\\pi$ and $V^\\pi$). Also, while often in papers it is omitted that by \"reward function\" in fact the *expected* reward function is meant, it is useful to state that explicitly.  \n\n- **Sec. 4.1, paragraph 2 (last line):** Action dimensionality of 24 is manageable by all following approaches: \n  - Factored DQN methods (Metz et al., 2017, Tavakoli et al., 2018) and Refs. [2, 3].\n  - Factored PG methods (Tang and Agrawal, 2020).\n\n- **Sec. 4.3, Evaluation & results** \"*Interestingly, the performance of the BB agent decreases with the discretization granularity, well exemplifying the curse of dimensionality of the method.*\" **:** This has been discussed in Ref. [2], where exploiting compositionality has also been shown to be key in solving this problem. At the end of the same paragraph, again it is mentioned that the reason why uniform discretization is not used is that it is not possible to run them in high-dim. action spaces. These statements seem to ignore the line of works discussed at the beginning of your paper's Related Work. Is there a reason for this?\n\n- **Sec. 5, paragraph 1:** \n  - Insert (Tang and Agrawal, 2020): \"*[...] action dimensions are independent (Tavakoli et al., 2018; **Tang and Agrawal, 2020**, Andrychowicz et al., 2020 [...])*\".\n  - Remove (Tang and Agrawal, 2020) from *autoregressive discretization* citations.\n  - After referencing methods that assume independence or those assuming some specific causal dependence, refer also to methods that learn the causal dependence (Ref. [2,4]).     \n\n---\n## Questions:  \n1. Please respond to the questions posed in the *Issues* section of my review.    \n\n2. Why not compare with action representation learning methods (see, e.g., Ref. [1])? This could be quite a valuable comparison, and several possibilities could be explored.\n\n3. Why only try the discrete-action baseline in the low-dimensional case of the Robodesk domain? As you have noted in the related work, there are now a few approaches for scaling discrete-action Q-based and PG methods to high dimensional problems; why not try them?\n\n4. **Figure 6:** How does the tuning process on Relocate work?\n\n5. What is the difference between *Success Rate* and *Sparse Reward Returns* in Fig. 5? Given that sparse rewards are 1 if the goal is achieved, I expected these two to be the same. \n\n6. **Table 8 (and experiments in general):** Choosing hyperparameters that perform best on average across all tasks (including test ones) is not fair. Am I misunderstanding something here, or is there a reason why this makes sense here?\n\n7. Could you position your work w.r.t. the line of work of action-embedding/action-representation/latent-action learners (e.g. in the context of Refs. [1,5])? \n\n--- \n### References:\n\n[1] Chandak, Y., Theocharous, G., Kostas, J., Jordan, S., Thomas, P. (2019). Learning Action Representations for Reinforcement Learning. In *Proceedings of the 36th International Conference on Machine Learning*.\n\n[2] Tavakoli, A., Fatemi, M., Kormushev, P. (2021). Learning to Represent Action Values as a Hypergraph on the Action Vertices. In *Proceedings of the Ninth International Conference on Learning Representations*. \n\n[3] Van de Wiele, T., et al. (2020). Q-learning in enormous action spaces via amortized approximate maximization. *arXiv preprint arXiv:2001.08116*.\n\n[4] Sakryukin, A., Raissi, C., Kankanhalli, M. (2020). Inferring DQN structure for high-dimensional continuous control. In *Proceedings of the 37th International Conference on Machine Learning*.\n\n[5] Merel, J., et al. (2019). Neural probabilistic motor primitives for humanoid control. In *Proceedings of the Seventh International Conference on Learning Representations*.\n\n\n", "summary_of_the_paper": "This paper proposes a method for discrete action-space learning from originally continuous action spaces using demonstrations. In other words, the aim is to utilize demonstration data so as to find a set of K discrete actions in a given state. If expert demonstrations represent an individual greedy optimal policy, the demonstrations sufficiently cover the state space, and the demo-generating optimal policy and the learning agent's policy are fully observing of the state (no partial observability), then K=1 can be sufficient to yield the greedy demo-generating optimal policy. However, in reality, these assumptions are hardly ever fulfilled. In addition, the demonstration data may not even come from an *optimal policy* for a task (e.g. human demonstration is generally suboptimal), or be w.r.t. to any specific downstream task in mind at all (*free play*). In such general cases, learning a discrete subspace of the continuous action space is quite useful. First, it allows us to apply discrete-action methods instead of continuous-action ones (which have been recently shown to be better even without incorporating demonstration data). Second, the reduced action space makes exploration significantly easier. Regarding the latter, this method can be seen as learning a hard bias on exploration (removing a subset of actions from the original downstream action space) and only using those used more frequently in demonstrations. The method used to achieve this is similar in spirit to Behavioral Cloning (BC) and, in fact, becomes equivalent to BC in the extreme case of K=1.  ", "summary_of_the_review": "I generally like this paper: the proposed method is sound and useful, experiments are insightful and nicely designed, and good discussions around limitations (e.g. pruning optimal actions) are present.\n\nHowever, the paper makes assertions here and there that are not entirely true (e.g. while related work covers a group of methods for addressing the curse of dimensionality, on various occasions this line of work is dismissed and a limited set of baselines are used). Baselines that use uniform discretization and can scale to high dimensions (regarding computation and generalization capabilities) by exploiting compositionality in multi-dimensional spaces are simple to try and would be very informative. \n\nAlso, discussion and comparison with an important class of methods (*action representation learning* methods) are missing. I referred to one important work in this direction (Ref. [1]), but I'm sure there is more on this topic under different names (e.g. latent actions, action embedding learning).  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a reinforcement learning technique for problems with continuous actions. The proposed approach consists in learning  a discretization of continuous action spaces from human demonstrations. This discretization returns a fixed number of actions for each input state. By discretizing the action space, any discrete action deep RL technique can be readily applied to the continuous control problem. Experiments reported in the paper show that the proposed approach outperforms several RL baselines such as SAC.

The key criticism from the reviewers relates to the incremental nature of this paper's contribution. While the precise equation proposed by in this paper for learning discrete actions from demonstrations may be novel, there have been several very similar techniques in the literature. For example, Gaussian Mixture Models (GMMs), a closely related model, have been widely studied in the context of learning policies from demonstrations. 

In summary, the reviewers are not convinced that the paper contains sufficiently novel ideas for an ICLR publication.
Decision: Reject