Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
Authors: Yuping Luo
Huazhe Xu
Tengyu Ma
Venue:  (2020)
Keywords: imitation learning
model-based imitation learning
model-based RL
behavior cloning
covariate shift

Abstract: Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently. However, learning from demonstrations often suffers from the covariate shift problem, which results
in cascading errors of the learned policy. We introduce a notion of conservatively extrapolated value functions, which provably lead to policies with self-correction. We design an algorithm Value Iteration with Negative Sampling (VINS) that practically learns such value functions with conservative extrapolation. We show that VINS can correct mistakes of the behavioral cloning policy on simulated robotics benchmark tasks. We also propose the algorithm of using VINS to initialize a reinforcement learning algorithm, which is shown to outperform prior works in sample efficiency.

Published as a conference paper at ICLR 2020
LEARNING SELF-CORRECTABLE POLICIES AND
VALUE FUNCTIONS FROM DEMONSTRATIONS WITH
NEGATIVE SAMPLING
Yuping Luo
Huazhe Xu
Tengyu Ma Stanford University
Princeton University yupingl@cs.ppi www.cc..ppce.... cs.princeton.edu
University of California, Berkeley huazhe xu @ wecs. becc.berkkley.euu
tengyuma@e stanford.edu
ABSTRACT
Imitation learning, followed by reinforcement learning algorithms,, a promising paradigm to solve complex control tasks sumple-efficienlly However, learning from demonstrations often suffers from the covariate shift problem, which results in cascading errors of the learned policy. We introduce notion of conservatively- extrapolated value functions, which provably lead to policies with self-correction. We design an algorithm Value Iteration with Negative Sampling (VINS) that practically learns such value functions with conservative extrapolation. We show that VINS correct mistakes of the behavioral cloning policy simulated robotics benchmark tasks. We also propose the algorithm of using VINS to initialize reinforcement learning algorithm, which is shown to outperform prior works in sample efficiency.
INTRODUCTION
Reinforcement learning (RL) algorithms, especially with sparse rewards, often require large amount of moal-and-eroory Imitation learning from small number of demonstrations followed by RL fine- tuning a promising paradigm improve the sample efficiency (Rajeswaran al., 2017| Večerík
et al. 2017 Hester et al. 2018 Nair et al.. 2018 Gao et al., 2018).
The key technical challenge of learning from demonstrations the covariate shift: the distribution of the states visited by the demonstrations often has a low-dimension support; however, knowledge learned from this distribution may not necessarily transfer to other distributions of interests. This phenomenon applies to both learning the policy and the value function. The policy learned from behavioral cloning has compounding errors after we execute the policy for multiple steps and reach unseen states (Bagnell 2015 Ross & Bagnell 2010 The value function learned from the demonstrations can also extrapolate falsely to unseen states. See Figure Ta for an illustration of the false extrapolation in toy environment.
We develop an algorithm that learns a value function that extrapolates to unseen states more con- servatively, as an approach attack the optimistic extrapolation problem (Fujimoto et al. 2018a). Consider a state s in the demonstration and its nearby state ø that is not in the demonstration. The key intuition that 5 should have a lower value than 8, because otherwise likely should have been visited by the demonstrations in the first place. If a value function has this property for most of the pair (s, 3) this type, the corresponding policy will tend correct errors by driving back the demonstration states because the demonstration states have locally higher values. We formalize the intuition Section 4 by defining the so-called conservatively-extran extrapollle value function, which guaranteed to induce policy that stays close to the demonstrations states (Theorem 4.4)
In Section 5 we design a practical algorithm for learning the
value function by negative sampling technique inspired by work learning embeddings Mikolov et al. (0013) Gutmann & Hyvärinen (2012). We also learn a dynamical model by standard supervised learning that compute actions by maximizing the values of the predicted next states. This algorithm does not use any additional environment interactions, and we show that it empirically helps correct emrors of the behavioral cloning policy.

---

Published as a conference paper at ICLR 2020
(a) The value function leamned from the stan- dard Bellman equation (or supervised learning) on the demonstration states. The value function falsely extrapolates to the unseen states. For example, the top left corner has erroncously the largest value.
(b) The conservatively-extraiee tion (defined in equation (4.2) learned with negative sampling (VINS, Algorithm Zin Sec- tion5). The values unseen states tend to be lower than their nearby states in the demonstra- tions, and therefore the corresponding policy tend to correct itself towards the demonstration trajectories.
value func-
result, once the policy induced by the value function makes a mistake, the error will compound.
Figure 1: A toy environment where the agent aims to walk from starting state (the yellow entry) to a goal state (the green eatry). The reward is sparse: R(s, a) = -1 unless s is at the goal (which is also the terminal state.) The colors of the entries show the learned value functions, Entries in black edges are states in demonstrations. The eyan arrows show the best actions according to the value functions.
When additional environment interactions are available, we use the learned value function and the dynamical model to initialize an RL algorithm. This approach relieves the inefficiency in the prior work (Hester et al.) 2018 Nair et al. 2018 Rajeswaran al., 2017) that the functions require a significant amount of time and samples to be warmed up, even though the initial policy already has a non-trivial success rate. Empirically, the proposed algorithm outperforms the prior work in the number of environment interactions needed to achieve near-optimal success rate.
Q
In summary, our main contributions are: 1) we formalize the notion of values functions with conservative extrapolation which are proved to induce policies that stay close to demonstration states and achieve near-optimal performances, 2) we propose the algorithm Value Iteration with Negative Sampling (VINS) that outperforms behavioral cloning three simulated robotics benchmark tasks with sparse rewards, and 3) we show that initializing an RL algorithm from VINS outperforms prior work in sample efficiency on the same set of benchmark tasks.
2 RELATED WORK
Imitation learning.
Imitation learning is commonly adopted as a standard approach in
robotics (Pomerleau, 1989 Schaal 1997] Argall et al. 2009) Osa et al. 2017; Ye Alterovitz
2017
Alcotti & Caselli 2006 Lawitzky et al. 2012 Torabi et al. 2018 Le al. 2017:2011 and
many other areas such as playing games (Mnih et al. 2013). Behavioral cloning (Bain & Sommut
1999 is one of the underlying central approaches. See Osa al. (208)) for thorough survey and
more references therein. If we are allowed to access an expert policy (instead of trajectories) or an
approximate value function, in the training time or in the phase of collecting demonstrations, then, stronger algorithms can be designed, such as DAgger (Ross et al. (001I. Aggre VaTe (Ross & Bagnell 2014), AggreVaTeD VaTeD (Sun al. 2017), DART (Laskey et al. (017)) THOR Sun al. (201(218))) 8a). Our setting is that we have only clean demonstrations trajectories and a sparse reward (but we still hope learn the self-correctable policy.)
Ho & Ermon 2016);; Wang et al. (2017): Schroecker et al. (2018) successfully combine generative models in the setting where large amount of environment interaction without rewards are allowed.
The sample efficiency of (Ho & Ermon (016)) has been improved in various ways, including maximum mean discrepancy minimization (Kim & Park 2018)) a Bayesian formulation of GAIL (Jeon et al. 2018), using an off-policy RL algorithm and solving reward bias problem (Kostrikov al. 201 8) and bypassing the learning of reward function (Sasaki et al.) 2018). By contrast, we would like to minimize the amount of environment interactions needed, but are allowed to access sparse reward. The work (Schroecker & Isbell (017)) also aims to learn policies that can stay close to the demonstration sets, but through quite different approach of estimating the true MAP estimate of the
2

---

Published as a conference paper at ICLR 2020
policy. The algorithm also requires environment interactions, whereas of our main goals is to improve upon behavioral cloning without any environment interactions.
Inverse reinforcement learning (c.g., see (Abbeel & Ng 2004 Ng et al. 2000 Ziebart al. 2008
Finn et al. 2016a by Fu et al., 2017 ) is another important and successful line of ideas for imitation approach in the sense that it aims to learn reward function that the expert is optimizing. In contrast, we construct a model to learn the value function (of the trivial sparse reward
learning. It relates
R(s,a) 1), rather than the reward function. Some of these works (e.g., (Finn et al. 2016a b Fu et al. 2017) use techniques that are reminiscent of negative sampling or contrastive learning, although unlike our methods, they use "negative samples" that are sampled from the environments.
Leveraging demonstrations for sample-efficient reinforcement learning, Demonstrations have been widely used to improve the efficiency of RL (Kim et al. 2013 Chemali & Lazaric 2013 Piot et al. 2014 Sasaki et al. 2018), and common paradigm for continuous state and action space is to initialize with RL algorithms with a good policy or Q function (Rajeswaran et al., 2017 Nair et al.)
2018 Večerík et al. 2017 Hester et al. 2018 Gao et al. 2018). We experimentally compare with the previous state-of-the-an algorithm in Nair et al. (2018) on the same type of tasks. Gao et al. (2018) has introduced soft version of actor-critic to tackle the false extrapolation of Q in the argument of a when the action space is discrete. In contrast, we deal with the extrapolation of the states in a continuous state and action space.
Model-based reinforcement learning. Even though we will learn a dynamical model in our algo- rithms, we do not use it to generate fictitious samples for planning. Instead, the learned dynamics only used in combination with the value function to get a Q function. Therefore, we do not consider our algorithm model-based techniques. We refer (Kurutach et al. 20188 Clavera et 2018 Sun let al.
2018b Chua et al. 2018 Gonzlez et al.) 2018 Pascanu et al. 2017 Khansari-Zadel Bil Billard 201 Luo al. 2018) and the reference therein for recent work on model-based RL.
Off-policy reinforcement learning There is large body of prior works in the domain of off- off-po
RL, including extensions of policy gradient (Gu al. [0066] Degris et al. [2012] Wang al. 2016)
or Q-learning (Watkins & Dayan, 1992, Haarnoja et al.) 2018 Munos et al. 2016). Fujimoto
al. (2018a) propose to solve off off- policy reinforcement learning by constraining the action space, and Fujimoto et al. (2018c) use double Q-learning (Van Hasselt et al. (0166) to alleviate the optimistic extrapolation issue. In contrast, our method adjusts the erroneously extrapolated value function by explicitly penalizing the unseen states (which is customized to the particular demonstration off- policy data). For most of the off-policy methods, their convergence are based on the assumption of visiting each state- action pair sufficiently many times. In the learning from demonstration setting, the demonstrations states are highly biased or structured; thus off-policy method may not be able to learn much from the demonstrations.
3 PROBLEM SETUP AND CHALLENGES
We consider a setting with deterministic MDP with continuous state and action space, and sparse rewards. Let S = R2 be the state space and A = R be the action space, and let M :Rd x R -, R2 be the deterministic dynamics. At test time, a random initial state so generated from some distribution Dag. We assume Da, has a low-dimensional bounded support because typically initial states have special structures. We aim find a policy и such that executing æ from state so will lead to set of goal states G. All the goal states are terminal states, and we run the policy for at most T steps if none of the goal states is reached.
Let (So, 01, .1,...,) .) be the trajectory obtained by executing deterministic policy x from 80. where a, = T (8t), and 8,41 M (8(8,, at). The success rate of the policy и defined as
suco(m) = E (3 < T, E
(3.1)
where the expectation is taken over the randomness of sg. Note that the problem comes with a natural sparse reward: R(s, a) -1 for every and a. This will encourage reaching the goal with small number of steps as possible: the total payoff of a trajectory is equal to negative the number of steps the trajectory succeeds, or -T otherwise.
3

---

Published as a conference paper at ICLR 2020
Let Te be an expert policy from which set of n demonstrations sampled. Concretely, independent initial states {sh
) Ji=1 from Dx are generated, and the expert executes Te to collect set of n trajectories (-()))=- We only have the access to the trajectories but not the expert policy itself.
We will design algorithms for two different settings:
Imitation learning without environment interactions: The goal is to learn policy a from the demonstration trajectories (7(0) without having any additional interactions with the environment.
Leveraging demonstrations in reinforcement learning: Here, in addition to the demonstrations, we can also interact with the environment (by sampling so ~ Da, and executing policy) and observe if the trajectory reaches the goal. We aim is to minimize the amount of environment interactions by efficiently leveraging the demonstrations.
Let L be the set of states that can be visited by the demonstration policy from random state with positive probability. Throughout this paper, we consider the situation where the set U is only a small subset or low-dimensional munifold of the entire state space. This is typical for continuous state space control problems in robotics, because the expert policy may only visit a very special kind of states that are the most efficient for reaching the goal. For example, in the toy example in Figure I the set U only contains those entries with black edges
To put our thcoretical motivation in Section 4 into context, next summarize a few challenges of imitation learning that are particularly caused by that 2/ is only a small subset of the state space.
Cascading errors for behavioral cloning. As pointed out by Bagnell 2015); Ross & Bagnell (2010), the errors of the policy can compound into a long sequence of mistakes and in the worst case cascade quadratically in the number of time steps T'. From statistical point of view, the fundamental issue is that the distribution of the states that a learned policy may encounter is different from the demonstration state distribution. Concretely, the behavioral cloning TDe performs well on the states in 2/ but not on those states far away from U. However, small errors of the learned policy can drive the state to leave Ut and then the errors compound as we move further and further away from U. As shown in Section 4 our key idea is to design policies that correct themselves to stay close to the set u4
Degeneracy in learning value or Q functions from only demonstrations. When Ui is small subset or a low-dimensional manifold of the state space, off-policy evaluation of V/h and QTe is fundamentally problematic in the following sense. The expert policy Te is not uniquely defined outside U/ because any arbitrary extension of nee outside would not affect the performance of the expert policy (because those states outside U will never be visited by mee from sg Dag). As result, the value function VTa and QT is not uniquely defined outside U. In Section + we will propose a conservative extrapolation of the value function that encourages the policy to stay close to U. Fitting QTe is in fact even more problematic. We refer to Section A for detailed discussions and why our approach can alleviate the problem.
Success and challenges of initializing RL with imitation learning. A successful paradigm for sample-efficient RL is initialize the RL policy by some coarse imitation learning algorithm such as BC Rajeswaran et al. 2017 Večerík et al.] 2017 Hester et al.) 2018 Nair et al.) 2018]] Gao et al. 2018). However, the authors suspect that the method can still be improved, because the value function or the Q function are only randomly initialized so that many samples are burned to warm them up. As alluded before and shown in Section 4 we will propose a way learn a value function from the demonstrations so that the following RL algorithm can be initialized by a policy, value function, and Q function (which is a composition of value and dynamical model) and thus converge faster.
THEORETICAL MOTIVATIONS
In this section, we formalize our key intuition that the ideal extrapolation of the value function V should be that the values should decrease as we get further and further from the demonstrations. Recall that we use 2/ to denote the set of states reachable by the expert policy from any initial state so
In this work, we only consider deterministic expert policies. Onne may imagine that 2/ can he more diverse set if the demonstrations are more diverse, hut an expert will visit entries the top hottom few rows, because they are not on any optimal routes the goal state.
4

---

Published as a conference paper at ICLR 2020
Illustration of the correction effect. conservatively www.aiiieeleeeeppltttteee value function V, as shown in the figure, has lower values further away from U, and therefore the gradients of V point to- wards U. With such value function, suppose we are at state which is E-close U. The locally- correctable assumption of the dynamics assumes the existence of de that will drive us to state Sex that is closer to 24 than Since 8ox has relatively higher value compared to other possible future states that are further away from 2/ (c.g., s' shown in the figure), Sex will be preferred by the optimiza- tion (43).. In other words, if an action leads to state with large distance U, the action won't t picked by because it cannot beat dex-
Figure 2:
d'
#BC
Sex
Coal
lower value
ants' de
drawn with positive probability from D... We use ||- to denote norm in Euclidean space IRd. Let (s) be the projection of E Rd to a set uf c IR (according to the norm || ID
We introduce the notion of value functions with conservative extrapolation which matches V on the demonstration states U and has smaller values outside U. As formally defined in equation 4.I) and (4.2) in Alg. we extrapolate 4%8 in a way that the value at 8 é U is decided by the value of its nearest neighbor in u (that is (Tl()))))) and its distance to the nearest neighbor (that is, ||s 1s- (s) (). We allow a dv > 0 error because exact fitting inside outside U/ would be impossible.
Algorithm Self Self-correctable policy induced from a value function with conservative extrapolation
Require: conservatively-extrapobee values V satisfying
V(s) = V (s) ,v,
if EU
(4.1)
(s) TM (1))(s) A|Is ll + öv
if & u
(4.2)
and locally approximately correct dynamics M and BC policy TвC satisfying Assumption
(4.
Self-correctable policy
(s)
argmaa
V(M(s,))
(4.3)
la (S)
Besides conservativell-exxrapleee value function V, our Alg. Argleeslsss on learned dynamical model M and behavioral cloning policy TBC- With these, the policy returns the action with the maximum value of the predicted next state in around the action of the BC policy. other words, the policy π attempts to re-adjust the BC policy locally by maximizing the value of the next state.
Towards analyzing Alg. [ we will make a few assumptions. We first assume that the BC policy is correct in the set U, and the dynamical model M is locally correct around the set U and the BC actions. Note that these are significantly weaker than assuming that the BC policy globally correct (which is impossible to ensure) and that the model M is globally correct.
Assumption 4.1 (Local errors in learned dynamics and BC policy). We assume the BC policy mae makes at most & error in W: for all U, have [)))()) [s()]] óz. We also assume that the learned dynamics M has dm error locally around Z/ and the BC actions in the sense that for all that is s-close U, and any action that (-close (s). we have [JM a) A a)ll < da
We make another crucial assumption on the stability/corretaaiitty of the true dynamics. The following assumption essentially says that if we are at state that near the demonstration set, then there exists an action that can drive us closer to the demonstration set. This assumption rules out certain dynamics that does not allow corrections even after the policy making a small error. For example,
3 Recall that we assume that Dzg has a low-dimensional support and thus typically will also he a low- dimensional subset of the ambient space.
Any ticbreaker be used if there multiple closest points.
5

---

Published as a conference paper at ICLR 2020
robot, unfortunately, falls off a cliff, then fundamentally it cannot recover itself our algorithm cannot deal with such pathological situations.
that the dynamics M is (y, Le: dynamics), For some y E (0, 1) and E > 0, Le > O, we assume wirt to the set U in the sense that for all Eu e (0, e) and any tuple (8, a, 8') satisfying 8, 8 E and &' = M* (8, a), and any so-perturbation & of s (that is, E Neo (5)), there exists action Oex that is Le close a, such that it makes correction in the sense that the resulting state s' is yEo-close to the set U: s' = (s dex) E Nys0 Here No (K) denotes the set of points that are 8-close to K.
Assumption 4.2
Finally, we will assume the BC policy, the value function, and the dynamics are all Lipschitz in their arguments We also assume the projection operator to the set 2/ is locally Lipschitz. These are regularity conditions that provide loose local extrapolation of these functions, and they are satisfied by parameterized neural networks that are used to model these functions.
Assumption 4.3 (Lipschitz-ness of policy, value function, and dynamics). We assume that the policy TBC -ip-ishhhtt.. That is. || ( < L a|| for all s, We assume the value function V and the learned value function V are Lv-Lipschitz, the model M is M, Lipschitz w.r.t to the action and L.M.--Lisschhi w.r.t to the state s. We also assume that the set 24 has Ln- Ln--Lppshitz projection locally: for all s, that is e-close to 24, [[TT((()) - (s)] I Lu |s $|I.
Under these assumptions, now we are ready to state our main theorem. It claims that 1) the induced policy π in Alg. stays close to the demonstration set and performs similarly to the expert policy Te. and 2) following the induced policy т, we will arrive at a state with near-optimal value.
Theorem 4.4. Suppose Assumption 4.1
hold with sufficiently small & > O and errors LoB + d + Lm. Let X be sufficiently large so that X 2 2Ly Ln Ln (226 +2Lvôx, Then, the policy ar from equation (4.3) satisfies the following:
5M, 8n > so that they satisfy C 2
(1-2)5
1. Starting from so E U and executing policy for To S ' steps, the resulting states 81, 8oo are all E-close to the demonstrute states set U.
2. In addition, suppose the expert policy makes at least p improvement every step in the sense that for every s U, either V* (M*(s. Te Te())) 2 (s) + - or M (s, (s())) reaches the goal Assume E and dsg. 8v, 5n are small enough so that they satisfy p 2 5 8n.
Then, the policy will achieve a state ST with T < 2 2 steps which is s-close to a state ST with value at least V (sT) 2 (E + 8m)
The proof is deferred to Section B The first bullet follows inductively invoking the following lemma which states that if the current state is e-close to 21 then so is the next state. The proof of the Lemma is the most mathematically involved part of the paper and is deferred to the Section B We demonstrate the key idea of the proof in Figure 2 and its caption.
Lemma 4.5. In the setting of Theorem 4.4 suppose s is E-close to the demonstration states set U. Suppose U. and let a = (()) and s' = M (s,a). a). Then, s' is also E-close to the set U.
We effecttiely represent the Q function by ((M a)) in Alg. We argue in Section A.1 that this helps address the degeneracy issue when there are random goal states (which is the case in our experiments.)
Discussion: can we learn conservatively--xtaapoaaee Q-function? We remark that we do not expect a anneevvatvv-eetrpllaatt Q 2-functions would be helpful. The fundamental idea here to penalize the value of unseen states so that the policy can self-correct. However, to learn a Q function that induces self-correctable policies, we should encourage unseen actions that can correct the trajectory, instead of penalize them just because they are not seen before. Therefore, it is crucial that the penalization is done on the unseen states (or V') but not the unseen actions (or Q).
'ee note that technically when the reward function is R(s, a) = 11. the value function is not Lipschitz. Ba|a 12 which does not require additional
This can be alleviated by considering similar reward R(s, a) information.
Spis is when the reward is always -1 before achieving the goal.
Here 2 hides multiplicative constant factors depending the ippsciitz parameters TaMay L.Mss Tem; Tav

---

Published as a conference paper at ICLR 2020
Algorithm 2 Value Iteration on Demonstrations with Negative Sampling (VINS)
& No environment interaction will be used 2: Initialize value parameters Φ - Φ and model parameters randomly
1: R t- demonstration trajectories
3: for ==
4: sample mini- batch of N transitions (s, a, s') from R
5: update é to minimize Lia(d; [3) Lne (d;
6: update 0 to minimize loss Lmgdel B)
update target network: Φ 8:
7:
T
9:
function POLICY(S)
10:
Option 1: a = TBC (s): Option 2: a a==0 sample noises E .1... from Uniform[-1e 1)m argmax; V (M, a + a(;))
11:
m is the dimension of action space > a > is a hyper-parameter
12:
return + of,--
MAIN APPROACH
Learning value functions with negative sampling from demonstration trajectories. As moti- vated in Section Algorithm and Theorem 4.4] we first develop a practical method that can learn value function with conservative extrapolation, without environment interaction. Let Vo be a value function parameterized by Φ. Using the standard TD learning loss, we can ensure the value function be accurate on the demonstration states U (i.e., to satisfy equation (4.T). Let ő be the target value (r(s, a) + Va((s)
function the TD learning loss is defined as (o) )"spp
Vo())
where r(s, a) is the (sparse) reward, Ø is the parameter of the target network, p is the distribution of the ilaees-action-states tuples of the demonstrations. The crux of the ideas in this paper is to use negative sampling technique enforce the value function to satisfy conservative extrapolation requirement (4.2). It would be infeasible to enforce condition (4.2) for every & U. Instead, we draw random "negative samples" from the neighborhood of Wd, and enforce the condition (4.2). This in- spired by the negative sampling approach widely used in NLP for training word embeddings Mikollyy al, (2013); Gutmann Hyvärinen (2012). Concretely, we draw a sample ~ DT create random perturbation of to get point E UI and construct the following loss function 37
(o) Benp"e, p-pertur(()) (V/ A||s - 311 Vo(3))²
The rationale of the loss function can be best seen in the situation when 24 is assumed to be low- dimensional manifold high-dimensional state space. In this case, 5 will be outside the manifold UI with probability 1. Moreover, the random direction ş - is likely to be almost orthogonal to the tangent space of the manifold U, and thus s a reasonable approximation of the projection of š back to the U, and ||s a|| is an approximation of || ø||. If 2/ is not manifold but a small subset of state space, these properties may still likely to hold for a good fraction of s.
We only attempt to enforce condition (4.2 for states near 44 This likely suffices because the induced policy shown to always stay close to U. Empirically, we perturb s by adding a Gaussian noise. The loss function to learn Vo is defined as L(Φ) Lra(Ø) + JLCns (Φ) for some constant > 0. For a mini-batch
of data, we define the corresponding empirical loss by C(o; B3 (similarly we
define Lialo; B) and Lne(á: B)). The concrete iterative learning algorithm is described in line 7 of Algorithm (except line is for learning the dynamical model, described below.)
Learning the dynamical model. We standard supervised learning to train the model. We use l norm as the loss for model parameters 0 instead of the more commonly used MSE loss, following the success of (Luo al. 2018): Lomotee "=(,,,,='~>=== Mo a) s/|[2]-
Optimization for policy. We don't 't maintain an explicit policy but use an induced policy from V and Mo by optimizing equation (4.3) A natural choice would be using projected gradient ascent to
A target value function is widely used in RT. improve the stability of the training (Lillicrap et al. 2015 Mnih al. (015)..
"ith slight abuse of notation, we use ne denote both the distribution of (s, 8') tuple and the distribution of of the expert trajectories.
7

---

Published as a conference paper at ICLR 2020
optimize equation 4.3 It's also possible use cross-entropy methods in (Kalashnikov al. 2018
to optimize it. However, we found the random shooting suffices because the action space is relatively low-dimensional in our experiments. Moreover, the randomness introduced appears to reduce the overfitting of the model and value function slightly. As shown in line TO 13 Alg. 2 we sample k
actions in the feasible set and choose the one with maximum V₂ (M))
Value iteration with environment interaction. As alluded before, when more environment interac- tions are allowed, we initialize an RL algorithm by the value function, dynamics learned from VINS. Given that we have V and M in hand, we alternate between fitted value iterations for updating the value function and supervised learning for updating the models. (See Algorithm BJ Section ( We do not use negative sampling here since the RL algorithms already collect bad trajectories automatically. We also do not hallucinate any goals as in HER (Andrychowicz et al. 2017).
EXPERIMENTS
Environments. We evaluate our algorithms in three simulated robotics environments 10 designed by Plappert et al. 2018 based on OpenAl Gym (Brockman et al., (016)) and MuJoCo (ooorov ct al. 2012): Reach, Pick-And-Place and Push. A detailed description can be found in Section D.I
Demonstrations. For each task, we use Hindsight Experience Replay (HER) (Andrychowicz et al. 2017) to train a policy until convergence. The policy rolls out to collect 100/200 successful trajectories as demonstrations except for Reach environment where 100 successful trajectories are sufficient for most of the algorithms to achieve optimal policy. We filtered out unsuccessful trajectories during data collection.
We consider settings: imitation learning from only demonstrations data, and leveraging demon- stration in RL with a limited amount of interactions. We compare our algorithm with Behavioral Cloning and multiple variants of our algorithms in the first setting. We compare with the previous state-of-the thetthterttttt art by Nair et al. (0018), and GAIL Ho & Ermon (016) in the second setting. We do not compare with (Gao et al. 2018 because it cannot be applied to the case with continuous actions.
Behavioral Cloning (Bain & Sommut 1999). Behavioral Cloning (BC) learns a mapping from a state an action on demonstration data using supervised learning. We use MSE loss for predicting the actions.
Nair et al. '18 (Nair et al. 2018). The previous state-of-the-an algorithm from Nair et al. (0018) combines HER (Andrychowicz et al., (007)) with BC and a few techniques: 1) an additional replay buffer filled with demonstrations, 2) an additional behavioral cloning loss for the policy, 3) a Q-fiter for non-optimal demonstrations, 4) resets to states in the demonstrations to deal with long horizon tasks. We note that reseting to an arbitrary state may not be realistic for real-world applications in robotics. In contrast, our algorithm does not require resetting to a demonstration state.
GAIL (Ho & Ermon, 2016) Generative Adversarial Imitation Learning (GAIL) imitates the expert by matching the state action distribution with GAN-like framework.
HER (Andrychowicz et al., 2017) Hindsight Experience Replay (HER) is the one of the best techniques that deal with sparse-reward environments with multiple goals and can be combined with any off-policy RI. algorithm. The key idea is that HER extends the replay buffer by changing the goals. With reasonable chosen goals, the underlying off-policy RL. algorihtm can receive more signals from the generated experience, making policy optimization more complete.
DAC (Kostrikov et al., 2018) Discriminator-Acton Critic (DAC) is sample-efficient imitation learning algorihtm built on the top of GAIL. It addresses the reward bias problem by adapting AIRL reward function and introducing an absorbing state. Furthermore, it replaces the underlying RL algorithm in GAIL by TD3 (Fujimoto et al.) (0188) to make it more sample efficient.
VINS. As described in Section 5 in the setting without environment interaction, we use Algorithm 2 otherwise we use it to initialize an RL algorithm (see Algorithm 3). We use neural networks to param- eterize the value function and the dynamics model. The granularity of the HER demonstration policy is very coarse, and we argument the data with additional linear interpolation between consecutive states. We also use only a subset of the states as inputs to the value function and the dynamics model,
Available https: thub. com http://wwwta..//c////////////tt/
8

---

Published as a conference paper at ICLR 2020
which apparently helps improve the training and generalization of them. Implementation details can be found in Section [.2]
Our main results are reported in Table for the setting with environment interaction and Figure for the setting with environment interactions. Table[]show that the Reach environment is too simple so that we do not need to run the RL algorithm. On the harder environments Pick-And-Plan and Push, our algorithm VINS outperforms BC. We believe this is because our conservatively eeteraaaieelyeetappllaeaee value function helps correct the mistakes in the policy, Here we use 2k trials estimate the success rate (so that the errors in the estimation is negligible), and we run the algorithms with 10 different seeds. The error bars are for standard error.
VINS (ours)
BC
Pick-Aidd-Place
Push
Reach 10
99.3 + 0.1%
98.6 0.1%
0.8
Pick 100
75.7 + 1.0%
66.8 = 1.1%
VIS (cuss
Muus)
0.6
Pick 200
84.0 1 0.5%
82.0 = 0.8%
Push 100
44.0 1.5%
37.3 1.1%
0.2
0.2
Push 200
55.2 0.7%
51.3 0.6%
00
0.2
0.4
0.6
0.8
3.0
0.0
0.2
0.4
0.6
0.8
3.0
complles EstepoO
142
comm (trep))
142
Table 1: The success rates of achieving the goals for VINS and BC in the setting with- out any environment interactions. A random policy has about 5% success rate at Pick and Push.
Figure 3: The learning curves of VINS+RL (Algo-
rithm 3 vs the prior state-of-the-art Nair et al.'8 on Pick-And-Place and Push. Shaded areas indicates one standard error estimated from 10 random seeds 12
Figure 3 shows that VINS initialized RL algorithm outperforms prior state-of-the-ar in sample efficiency. We believe the main reason is that due to the initialization of value and model, we pay less samples for warming up the value function. We note that our initial success rate in RL is slightly lower than the final result of VINS in Table This because in RL we implemented a slightly worse variant of the policy induced by VINS: in the policy of Algorithm 2 we use option 2 to search the action uniformly. This suffices because the additional interactions quickly allows us to learn good model and the BC constraint is no longer needed.
Allaioon studies. Towards understanding the effect of each component of VINS, we perform three ablative experiments to show the importance of negative sampling, searching in the neighborhood of Behavioral Cloned actions (option in line 1o or Algorithm 2). and a good dynamics model. The results are shown in Table 2 We study three settings: (1) VINS without negative sampling (VINS w/o NS), where the loss Lons is removed; (2) VINS without BC (VINS w/o BC), where option in line 10 Algorithm 2 is used; (3) VINS with oracle model without BC (VINS w/ oracle w/o BC), where we use the true dynamics model to replace line 12 of Algorithm 2
Note that the last setting is only synthetic for ablation study because in the real-world we don't have access the true dynamics model. Please see the caption of Table for more interpretations. We use the same set of hyperparameters for the same environment, which may not be optimal: for example, with more expert trajectories, the negative sampling loss C... which can be seen as a regularziation, should be assigned a smaller coefficient μ.
7 CONCLUSION
We devise new algorithm, VINS, that can learn self-correctable by learning value function and dynamical model from demonstrations. The key idea is theoretical formulation of conservatively- extrapolated value functions that provably leads to self-correction. The empirical results show a promising performance of VINS and an algorithm that initializes RL with VINS. It's fascinating direction to study other algorithms that may learn conservatively-extrpolaa
value functions in
TThe standard error the paper means the standard error of average success rate over 10 (100 for Reach 10) different random seeds by the same algorithm, that is, the standard deviation of 10 numbers over /10 (or 10. respectively).
The curve for Nair et al's only starts after a few thousands steps because the code we use tpt / / chub. com/ jangi rri shabh/ BaabbhhOvvvrroo /vveroomroooooooooooo ng explexeplaaeeee on- inn-fram from- only evaluates after the first epoch.

---

Published as a conference paper at ICLR 2020
Pick 100
Pick 200
Push 100
Push 200
BC
66.8 + 1.1%
82.0 + 0.8%
37.3 + 1.1%
51.3 = 0.6%
VINS
75.7 = 1.0%
84.0 - 0.5%
44.0 + 0.8%
55.2 = 0.7%
VINS w/o BC
28.5 + 1.1%
43.6 1.2%
14.3 i 0.5%
24.9 = 1.3%
VINS w/ oracle
51.4 + 1.4%
62.3 + 1.1%
40.7 + 1.4%
42.9 = 1.3%
w/o BC
VINS w/ oracle
76.3 + 1.4%
87.0 + 0.7%
48.7 1.2%
63.8 = 1.3%
VINS w/o NS
48.5 + 2.1%
71.6 + 0.9%
29.3 + 1.2%
38.7 = 1.5%
Table 2: Ablation study of components of VINS in the setting without environment interactions. We reported the average performance of 10 runs (with different random seeds) and the empirical standard error of the estimator of the average performance. The success rate of VINS w/o NS is consistently worse than VINS, which suggests that NS crucial for tackling the false extrapolation. From comparisons between VINS w/o BC and VINS w/ oracle w/o BC, and between VINS and VINS w/ oracle, we observe that the learning of the dynamics can be improved (potentially by e.g.. by collecting data with random actions), then VINS or VINS w/o BC can be improved significantly. We also suspect that the reason why we need to search over the neighborhood of BC actions that the dynamics not accurate state-action pairs far away from the demonstration set (because the dynamics is only learned on the demonstration set.)
other real-world applications beyond the proof-of of-c-cc---cccs experiments in this paper. For example, the negative sampling by Gaussian perturbation technique in this paper may not make sense for high-dimensional pixel observation. The negative sampling can perhaps done in the representation space (which might be learned by unsupervised learning algorithms) so that we can capture the geometry of the state space better.
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-fir international conference Machine learning, pp. 1. ACM, 2004.
Jacopo Aleotti and Stefano Caselli. Grasp recognition in virtual reality for robot pregrasp planning by demon- stration. In Proceedings 2006 IEEE International Conference Robotics and Automation, 2006. ICRA 2006., pp. 2801-2806. IEEE, 2006.
Marcin Andrychowicz, Filip Wolski, Alex Ray. Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048 5058, 2017.
Brenna D Argall, Sonia Chernova, Manucla Veloso, and Brett Browning. A survey of robot learning from demonstration. Roboties and autonomous systems, 57(5) 443, 2009.
J Andrew Bagnell, An invitation to imitation. Technical report, CARNEGIF-MELLON UNIV PITTSBURGH PA ROBOTICS INST, 2015.
Michael Bain and Claude Sommut. A framework for behavioural claning. Machine intelligence, 15(15):103, 1999.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Jessica Chemali and Alessandro Lazaric. Direct policy iteration with demonstrations. In Twenty-Fourd International Joint Conference Artificial Intelligence, 2015.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754 4765, 2018.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbecl. Model-based reinforcement learning via meta-policy optimization. arXiv preprint 1009:v800.002//1 2018.
Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-eritic. arXiv preprint arXiv: arXiv:/205.4880 4205.4839 2012.
10

---

Published as a conference paper at ICLR 2020
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines, htttp::/gitthb:
oopenai baanaii/baslll lines 2017.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03833 2016a.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pp. 49-58, 2016b.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint 17X1v:1110.1124:: 2017.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv: 1rXiv:18/2.000 1812.0290, 2018a.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error actor-critic methods. arXiv preprint arXiv:1802.00447 2018b.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.0994: 2018c.
Yang Gao, Ji Lin, Fisher Yu, Sergey Levine, Trevor Darrell, et al. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv:1802.05311 2018
Shixiang Gu, Timothy L Lillicrap, Zoubin Ghahramani, Richard E Turner, and Sergey Levine. Q-prop: Sample- efficient policy gradient with an off-policy critic. arXiv preprini 1XXiv:661100224: 2016.
Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13(Feb):307-361, 2012.
Tuomas Haaroja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with a stochastic actor arXiv preprint arXiv: 1801.0190 2018.
Todd Hester, Matej Vecerik, Olivier Pietquin. Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565 4573, 2016.
Wonseok Jeon, Seokin Sco, and Bung Kim. bayesian approach to generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 7429-7439, 2018
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Iharz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 10293, 2018.
S Mohammad Khansari- Zadch and Aude Billard. Learning stable nonlinear dynamical systems with gaussian mixture models. IEEE Transactions on Robotics, 2795)9933- 2011.
Beomjoon Kim, massoud Farahmand, Joelle Pineau, and Doina Precup. Learning from limited demonstra- tions. In Advances in Neural Information Processing Systems, pp. 2859-2867, 2013.
Kee-Eung Kim and Hyun Soo Park. Imitation learning via kernel mean embedding. In Thirty- AAAI Conference on Artificial Intelligence 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-ccto-rrrtt::: Addressing sample inefficiency and reward bias in adversarial imitation learning. 2018.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802. 10592, 2018
Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust. imitation learning. arXiv preprint arXiv: 1703.09327, 2017.

---

Published as a conference paper at ICLR 2020
Martin Lawitzky. Jose Ramon Medina, Dongheui Lee, and Sandra Hirche. Feedback motion planning and learning from demonstration physical robotic assistance: differences and synergies. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3646-3652. IEEE, 2012.
Hoang M Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-agent imitation learning. In Proceedings of the 34th International Conference on Machine Learning-Volee Volume 70, pp. 1995 2905-2003. JMLR. org. 2017.
Hoang M Le. Nan Jiang, Alekh Agarwal, Miroslav Dudik, Yisong Yue, and Hal Daumé III. Hierarchical imitation and reinforcement learning, arXiv preprint (rXiv:180330050 2018
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton, Layer normalization. arXiv preprint arXiv: 167.006450 2016.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 1509.02971, 2015.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality In Advances in neural information processing systems, pp. 31 11-119, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529, 2015.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforce- ment learning. In Advances Neural Information Processing Systems, pp. 1054-1062, 2016
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Picter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEFE International Conference on Robotics and Automation (ICRA). pp. 6292-6299. IEEE, 2018.
Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement lcarning. In Icml, volume 1, pp. 2. 2000.
Takayuki Osa. Amir M Ghalamzan Esfahani, Rustam Stolkin, Rudolf Lioutikov. Jan Peters, and Gerhard Neumann. Guiding trajectory optimization by demonstrated distributions. IEEE Robotics and Automation Letters, 2(2):119-224 826, 2017.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 7(1-2)11 179, 2018.
Razvan Pascanu, Yujia Li, Oriol Vinyals, Nicolas Heess, Lars Buesing, Sebastien Racanière, David Reichert, Théophane Weber, Daan Wierstra, and Peter Battaglia. Learning model-based planning from scratch. arXiv preprint arXiv:1707.06170 2017.
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted bellman residual minimization handling expert demonstrations. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 549-564. Springer, 2014.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej. Peter Welinder, Vikash Kumar, and Wojeiech Zaremba. Multi-goal reinforcement. learning: Challenging robotics environments and request for research, 2018
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Advances in neural information processing systems, pp. 305-313, 1989.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. arXiv preprint arXiv: 170. 2017.
12

---

Published as a conference paper at ICLR 2020
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661 (61-668, 2010.
Stephane Ross and Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv: 1rXiv:1406..30 /406.5979, 2014.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 627-635, 2011.
Alvaro
Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. arXiv preprint 01X.v:280600122,,, 2018.
Fumihiro Sasaki, Tetsuya Yohira, and Atsuo Kawaguchi. Sample efficient imitation learning for continuous control. 2018.
Stefan Schaal. Learning from demonstration. In Advances in neural information processing systems, pp. 1040-1046, 1997.
Yannick Schroccker and Charles L Isbell. State aware imitation learning. In Advances in Neural Information Processing Systems, pp. 2911-2920, 2017.
Yannick Schroecker, Mel Vecerik, and Jon Scholz. Generative predecessor models for samplc-efficient imitation learning. 2018.
Wen Sun, Arun Venkatraman, Geoffrey J Gordon, Byron Boots, and J Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In Proceedings the 34th International Conference on Machine Learning- Volume 70, pp. 3309-3318. JMLR. org, 2017.
Wen Sun, Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining reinforcement learning & imitation lcarning. arXiv preprint 1805. 12400, 2018a.
Wen Sun, Geoffrey J Gordon, Byron Boots, and Bagnell. Dual policy iteration. In Advances in Neural Information Processing Systems, pp. 7059-7069, 2018b.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for modeel-based control. 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv: 1 885.00154, 2018.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learinng In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Matej Vežerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint 1rXiv:1707.088111 2017.
Ziyu Wang, Victor Bapst. Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint arXiv:161 1,0/224, 2016.
Ziyu Wang, Josh S Merel, Scott F. Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess, Rohust imitation of diverse hehaviors. In Advances in Neural Information Processing Systems, pp. 5320-5329, 2017.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-292, 1992.
Gu Ron Alterovitz. guided motion planning. In Robotics research, pp. 297-307. Springer, 2017,
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp.
Chicago, IL. USA, 2008.
A DEGENERACY OF LEARNING Q FUNCTIONS FROM DEMONSTRATIONS AND OUR SOLUTIONS
Fitting QT from only demonstration is problematic: there exists a function Q(s, a) that does not depend on all, which still match QT= all possible demonstration data. Consider Q(s,a) a) & ( Te()). We can verify that for any (s,a) pair in the demonstrations satisfying a = Te (s). it holds that Q(s, = (s. a). However, Q( a) cannot be accurate for other choices of a 's because by its definition, does not use any information from the action a.
13

---

Published as a conference paper at ICLR 2020
A.1 COPING WITH THE DEGENERACY WITH LEARNED DYNAMICAL MODEL
Cautious reader may realize that the degeneracy problem about learning Q function from demon- strations with deterministic policies may also occur with learning the model M. However, will show that when the problem has the particular structure of reaching random but given goal state learning Q still suffers the degeneracy but learning the dynamics does not.
The typical way to deal with random goal state is to consider goal-conditioned value function (ss. 9). policy ((s g). and Q function Q(s, a, g) [3] However, the dynamical model does not have to condition on the goal. Learning Q function still suffers from the degeneracy problem because (s, a.g) g) A (** (s. (s. g) matches Q on the demonstrations but does not use the information from a at all. However, learning M does not suffer from such degeneracy because given single state in the demonstration, there is still variety of action that can be applied to state s because there are multiple possible goals (In other words, we cannot construct a pathological M (s, a) (s, (s)))) because the policy also takes in an input). As a result, parameterizing Q by Q(s a. g) = (s, g), do not suffer from the degeneracy either.
B MISSING PROOFS IN SECTION
Proof of Lemma 4.5 Let x Thys and ā (8)) Because s is e-close to the set U, we have assumption of the dynamics, we have that there exists an action aes such that a) ||acx - all S L and b) s'xx & M* (s, axx) is ye-close to the set U.
S 5|
By the (7, Lc. (ooe)))ooally-coreetttt
Next we show that dex belongs to the constraint set fa: ||a Tc((s) I () in equation (4.3) Note [B (5) + (B) - (s)) I| S L8S + 6r + L because of triangle inequality, the closeness of dex and a. the assumption that TBC has ö error in the demonstration
that ||acx 1in--pa <T)( S ||aex - all + ||- -
state set Ut and the Lipschitzness of тBC. Since ( is chosen to be bigger than L + 8, + LgE, we conclude that aęx belongs to the constraint set of the optimization in equation (4.3)
This suggests that the maximum value of the optimization 443) is bigger than the corresponding value of axx
V(M(s, a)) 2 V(M dax))
(B.1)
Note that belongs to the constraint set by definition and therefore Ia- 0xx|1 S 2C. By Lipschitzness of the dynamical model, and the value function V=: we have that ||M (8, a) M (s, axx) S
Larlla dex < 2LMC. Let s' = MM (s, a) and 6'xx = MM* dex). We have || s/xx < 2LMS.
By the Lipschitz projection assumption, we have that | 155--11p -Iubaall S Lulls' - =a S 2Labac which in turns implies that (Ilesl) - V (Ile S 2Lv Ln by Lipschitzness of V It follows that
v(') S V* (Ils') - A|55 - 11rs'll + ōv
(by assumption 4..
S (I + (T V (Ilys')) - A11s/ -Iys' II + by
(by triangle inequality)
< V* + 2Ly LULMS- - A|Is' - Hus'll + or (by equations in paragraph above)
< V + AlIs's. - Iss=2l + 2Ly Ln - A||s' - -us'll + 25v (by assumption (.2
Note that by the Lipschitzness of the value function and the assumption on the error of the dynamical model.
[V(') (s.(ss)))) (s,)) (s) v(M(s,
$ ILoll - M(s, S Lvôn
(B.2)
Simlarly
[V((<) - V(M(s, axx)) = (VM (xx)) (M dcx))l
< Lo MTT (s, dex) - Ms, acx))l S Lv
(B.3)
1hts is equivalent to viewing the random goal part of an extended state 3 = (s.g). g). Here the second part of the extended state is randomly chosen during sampling the initial state, but never changed by the dynamics. Thus all of previous work does apply this situation via this reduction.
14

---

Published as a conference paper at ICLR 2020
Combining the three equations above, we obtain that
*| - 1s + 22v A||s' -Ilss'1 || + 28v
V(s') V(s'x)
2 v(M(s,a) - ((M(ss 2LvdM
(by equation (B.2 and
B.3
-LLvdM
(by equation [.I
Let K - 2LyLubMs + 20v + 2Ly and use the assumption that s is 76-close to the set U
(which implies that || IIs S yE), we obtain that
A|Rs - Illss'I - Alla - 1o + X7e +
(B.4)
Note that À 2 (1-g)e have that ||s 1l S.
inductively for T steps. To prove bullet 2, we will prove that long as 8; is e-close U, then we can improve the value function by at least p-? in one step. Consider 8, = (s;)) We triangle inequality, we have that [|(((;) (T()))l I [s.)) (s,)) + [[c()s ||TT||++[[((())/// can be bounded respectively by Lx s.ll S Lr and 8n, using the definition of the Lipschitzness of TBC, and the error assumption TBC on the demonstration state set respectively. It follows that IMM((s, (s;)) (s, (s.)))) I LM,=lls; 3,1 + LM (s()) =e(5)) S LM + LM Ç + L + 0.). It follows by the Lipschitzness of the projection that
Proof of Theorem 4.4
To prove bullet 1, we apply Lemma 4.5
- (Bc [5 + [[C(), - (=.))l These three terms
|s,41
Il, (5:))l [TuM (s:)) Iu, (s, (e())))
(B.5)
= (s,, - M (3,, m3()))
(B.6)
< Lu(LM, + Ls LaE + 84))
(B.7)
This implies that
V - // (Il ne (3:)) s Lv Ly + LMaal + L + 8z))
(B.8)
Note that assumed that V(M* (5,, Te (8:) > V(s.) + por M* (o(:)) reaches the goal. If the former, it follows that /* (3+41) 1 2 (8:) + Ly Ln(L,, + LM + L + 8x)) 2 (5,) + p/2. Otherwise, or 8:41 is E-close 5 whose value at most Lu (5 + LGE + 8n)) -0(E + 8n)
C ALGORITHMS VINS+RL
pseudo-code our algorithm VINS+RL can be found in Algorithm 3
Algorithm 3 Value Iteration with Environment Interactions Initialized by VINS (VINS+RI)
Require: Initialize parameters d, from the result of VINS (Algorithm) 2
1: R demonstration trajectories;
2: for stage f 1,... do
3: collect n samples using the induced policy æ in Algorithm 2 (with Option 2 in Line 10 and
add them to R
4: for 1,..., Tinner do
5: sample mini-batch B of N transitions (s,a,r, s') from R
6: update e to minimize Lua(s); B) update target value network: Φ o T( -5)
7:
8:
update minimize loss fuooeel (0; B))
15

---

Published as a conference paper at ICLR 2020
D IMPLEMENTATION DETAILS
D.1 SETUP
In the three environments, a 7-DoF robotics arm is manipulated for different goals. In Reach, the task to reach a randomly sampled target location; Pick-And-Place, the goal is to grasp a box in table and reach target location, and in Push, the goal is to push box to a target location.
The reward function is 0 if the goal is reached; otherwise -1. Intuitively, an optimal agent should complete the task in shortest possible path. The environment will stop once the agent achieves the goal or max step number have been reached. Reaching max step will be regarded as failure.
For more details, we refer the readers to (Plappert et al. 2018).
D.2 HYPERPARAMETERS
Behavioral Cloning We use a ford-forward neural network with 3 hidden layers, each containinig 256 hidden units, and ReLU activation functions. We train the network until the test success rate plateaus.
Nair et al.'18 Nair et al. 2018 We use the implementation from jangirri www.srrssshhhh/see Over
https://github: com/
frrratiooooo-o---- fennt-frrmmdemmo hyperparameterrs, except that we're using 17 CPUs.
We don change the default
GAIL (Ho & Ermon 2016) We use the implementation from OpenAl Baselines DDariwal et al.
2017)
We don't change the default hyperparameters.
HER (Andrychowiezet al. 2017) We also use the code from OpenAl Baselines and keep the default
hyperparameters.
Dicerininator--Atto--CCttttt (Kosrrikoy et al. 2018 We use the implementation from the of-
ficial implementation https: /tttpp://githubte com www.///oooooooooooooooooo//.oooo googl e- researchehhheeeee google research/ Eree/master/dac
VINS
Architecture: We use feed-forward neural networks as function approximators for values and dynamical models. For the Va, the network has hidden layer which has 256 hidden units and a layer normalization layer (Lei Ba et al. 2016).. The dynamics model is feed-forward neural network with hidden layers and ReLU activation function. Each hidden layer has 500 units. The model uses the reduced states and actions to predict the next reduced states.
Value augmentation: We augment the dataset by a linear interpolation between two con- secutive states, i.e., for a transition (s, a, r, s') in the demonstration, it's augmented to (s + A(s' - 8), a, Xr; for a random real ~ Uniform(0,11 1]. To minimize the losses, we use the Adam optimizer Kingma & Ba 2014) with learning rate 10 10-4. 44 We remove some less relevant coordinates from the state space to make the dimension smaller. (But we maintain the full state space for BCC BC will perform worse with reduce state space.) Specifically, the states of our algorithm for different environment are Reach: the position of the arm, b) Pick: the position of the block, the gripper, and the arm, and c) Push: the position of the block and the arm.
States representation and perturbation: Both our model M and value function V are trained on the space of reduced states. The perturb function is also designed separately for each task. For Reach and Push, we perturb the arm only; For Pick-And-Plan or the gripper. In the implementation, we perturb the state s by adding Gaussian noise from distribution A00,, pE), where Σ is diagonal matrix that contains the variances of each coordinate the states from the demonstration trajectories. Here > 0 is a tune.
perturb the arm
16

---

Published as a conference paper at ICLR 2020
Pick 100
Pick 200
Push 100
Push 200
BC
66.8 + 1.1%
82.0 + 0.8%
37.3 + 1.1%
51.3 = 0.6%
VINS
75.7 = 1.0%
84.0 - 0.5%
44.0 + 0.8%
55.2 = 0.7%
BC w/ data aug
58.8 = 1.3%
77.4 0.9%
30.1 + 0.8%
41.4 1.6%
VINS w/ oracle
47.2 = 1.9%
74.3 + 1.0%
30.7 + 0.8%
41.3 = 0.8%
w/o NS
VINS w/ oracle
76.3 + 1.4%
87.0 0.7%
63.8 = 1.3%
VINS w/o NS
48.5 + 2.1%
71.6 + 0.9%
29.3 + 1.2%
38.7 = 1.5%
Table 3: Ablation study of components of VINS in the setting without environment interactions. We reported the average performance of 10 runs (with different random seeds) and the empirical standard error of the estimator of the average performance. The success rate of BC w/ data augmentation is consistently worse than BC w/o data augmentation.
E ADDITIONAL EXPERIMENTS
We have some additional experiments to evaluate VINS, which are (a) BC with data augmentation (b) VINS with oracle without negative sampling, which are explained below. The results are summarized in Table
BC w/ data augmentation. As VINS augments the data to train the value function, it might be interesting to see the performance of BC w/ data augmentation to make the comparison more fair. The way we augment the data is similar to what we have done for the value function: Given two consecutive state-action pairs (s. a.) and (s' a'), where s' (s, a), sample t Uniform(0, 1] and construct a new pair (Is + (1 l)s', + (1 - l)a'). The new pair is used for behavior cloning. However, we found out that this kind of data augmentation hurts. We hypothesize that the augmented data provide incorrect information, as the actions between two consecutive states might be very non-linear, while value is more linear. We suspect that the biases/errors introduced by the data augmentation is particularly harmful for BC, because it lacks mechanism of self-correction that VINS has.
VINSw/ oracle w/o negative sampling. Another way to isolatethe effect of negative sampling is comparing VINS w/ oracle w/o NS to VINS w/ oracle. The results are shown at Table We can see that VINS w/ oracle achieves significantly higher success rate than VINS w/ oracle w/o negative sampling, which shows that negative sampling is essential to the success of VINS.
VINS with fewer demonstrations. To evaluate whether VINS works with few demonstrations. we also conduct our experiments wih only 40 expert trajectories. The result is shown in Table 44] The performance of BC is worse than that with 100 expert trajectories, VINS can still correct some actions via learned value function and model and achieves higher success rate.
VINS with stochatic dynamics model. As this paper mainly studies deterministic dynamics model, we'd 'dd like note that theory can be generalized to stochastic dynamics model easily. We also conduct proof-of-concep- experiments to show the performance of VINS in this case. The stochastic environment used in the experiments is based Push, and the modification is that the dynamics model adds noises to the received actions, i.e., s' ~ M a () where Ç (N,,, a² 2²T), f), M here is the original deterministic dynamics model. One intuitive interpretation is that the actuator on the robot might not be good enough to produce the exact forces. For simplicity we don't 't train stochastic dynamics model but learn a deterministic one approximate it. The result is summarized Table 4 in which we can find even with a determinisic dynamics model, VINS performs better than BC.
17

---

Published as a conference paper at ICLR 2020
VINS (ours)
BC
Pick 40
40.0 + 0.9%
36.3 + 1.7%
Push 40
26.5 0.6%
23.5 1 0.7%
StochasticPush 100
38.4 + 0.9%
29.6 1.4%
StochasticPush 200
51.5 = 0.7%
42.5 - 0.9%
Table 4: The success rates of achieving the goals for VINS and BC in the setting without any environment interactions. We reported the average performance of 10 runs (with different seeds) and the empirical standard error of the esitmator of the average performance. VINS outperforms BC in all tested environments.
18


Reviews:
[{"review_id": "S1liW_icKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work tried to address the covariate shift problem in imitation learning, which is due to the mismatch between training and test state distribution and may cause compounding errors. \n\nThe authors proposed the algorithm called value iteration with negative sampling (VINS) of which the main ideas can be summarized as follows. First, value iteration is used on expert trajectories with negative sampling. Specifically, the states that are randomly perturbed from expert’s states were used to enforce (4.1) and (4.2) in the submission (called *conservative extrapolation* requirements in the submission). By doing so, the state values outside the support of expert state visitation distribution become less than those inside the support. In the meantime, temporal-difference (TD) error was minimized to satisfy Bellman consistency among state values. The second main idea of this work is to use *self-correctable policy*, where the approximate dynamics and behavioral-cloning (BC) policy were used to select the action which is expected to give higher value at the successor states. \n\nTo consolidate their ideas, the authors proved Theorem 4.4 showing that state visitation distribution of resultant policy is approximately close to that of an expert under a few assumptions. Empirically, they considered two experiments. In the first experiment, assuming that the environment simulation is not allowed, the performance of VINS was compared with that of BC over 5 tasks, and VINS achieved higher success rates. In the second experiment, assuming the simulation is allowed, VINS was compared with existing methods such as HER+BC, GAIL and Nair et al ‘18 and shown to be much more sample-efficient compared to the selected baselines. \n\nAlthough the theoretical and empirical contributions of this work are clear to me when the environment simulation is not allowed (as shown in the first experiment in Table 1), I think the second experiment, which allows the environment simulation (as shown in Figure 3), is misleading, and this is why I vote weak reject for this work. For instance, we can simply think about GAIL with BC initialization, but it seems to me that GAIL with random initialization was used in the second experiment (since authors mentioned GAIL in OpenAI baselines was used without hyperparameter tuning (https://github.com/openai/baselines/blob/master/baselines/gail/run_mujoco.py#L53)). In addition to it, there have been some recent works on the sample efficiency of imitation learning with environment simulation which are not included as baselines in this work:\n\n[1] GMMIL (Kim and Park, 2018, “Imitation Learning via Kernel Mean Embedding”) - cost learning with maximum mean discrepancy minimization leads to sample-efficient training\n\n[2] BGAIL (Jeon et al, 2019, “Bayesian Approach to Generative Adversarial Imitation Learning”) - Bayesian cost is helpful for sample-efficient imitation learning\n\n[3] DAC (Kostrikov et al, 2019, “Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning”) - Solving reward bias in imitation learning with simulation and using off-policy RL algorithm to enhance the sample efficiency\n\n[4] Sasaki et al, 2019, “Sample Efficient Imitation Learning for Continuous Control” - Bypassing cost learning and introducing off-policy RL to enhance the sample efficiency. \n\nEspecially, off-policy imitation learning methods [3], [4] are shown to be extremely sample-efficient compared to GAIL. I think the authors should have compared the performance of VINS + RL with those baselines in the second experiment if they tried to emphasize the sample efficiency of VINS + RL. Otherwise, they should have focused more on the initialization effect of VINS and BC. For example, one can consider the convergence speed of GAIL to the expert performance when policies were initialized with either VINS or BC.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gCYBFkcB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper tackles an issue imitation learning approaches face. More specifically, policies learned in this manner can often fail when they encounter new states not seen in demonstrations. The paper proposes a method for learning value functions that are more conservative on unseen states, which encourages the learned policies to stay within the distribution of training states. Theoretical results are derived to provide some support for the approach. A practical algorithm is also presented and experiments on continuous control tasks display the effectiveness of the method, with particularly good results on imitation learning followed by reinforcement learning.\n\nThe proposed algorithm makes use of a natural intuition, that states visited by the expert probably have higher values, and the paper generally does a good job of supporting the approach through theory and experiments. Although the experiments seem sound, certain experimental details are not completely clear. The theory may also have some restrictive assumptions, limiting its significance.\nOverall, I am divided about this paper. While this submission has the elements of a good paper and the presentation is great, certain concerns make me hesitant to recommend acceptance. I would be willing to increase my score if those points are addressed. \n\nTheory:\n1) My main concern is the applicability of the theorem, due to Assumption 4.2. While the intuition is that there is an action that corrects the next state towards the demonstration states, the theoretical condition is more restrictive. In particular, the following part (paraphrasing): \"there exists an action a_cx that is close to a^bar and that makes a correction towards U\". This condition implies that there are correcting actions near any action a^bar, which sems unrealistic in most cases. For example, in a driving task, say s^bar is a state such that moving back to U require the vehicle to move to the left. Then, consider the action a^bar of steering towards the right (with some angle). There could be no action near a^bar that makes the vehicle turn left towards U. Note that this is not necessarily a pathological situation as described in the text.\n\n2) Also concerning assumption 4.2, I do not see why s^bar' is included in the quantifier of the statement since it is not used afterwards; after \"there exists an action...\" no mention of s^bar' is made.\n\n3) The projection function may not be well-defined if there are multiple states that are closest to the one being projected.\n\n4) It could be said explicitly that the expert policy is assumed to be deterministic. Currently, this is not said outright.\n\nExperiments:\n1) It seems like VINS relies heavily on the assumption that the environment is deterministic to learn an effective model. Was VINS tried in stochastic environments? \n\n2) Data augmentation is used for VINS. This seems like an unfair advantage compared to the baseline competitors since sample efficiency is a key concern to reinforcement learning algorithms.  To make the comparisons fair, either it should be removed or the other algorithms should also receive additional data. How is the performance of VINS without this addition?\n\n3) A description of how the hyperparameters were chosen and their final values would be needed for reproducibility. Also, a discussion of the importance of the hyperparameters and their sensitivity would be informative. For example, I was curious to know the value of \\alpha in Algorithm 2 compared to the ranges the actions could take. \n\n4) I am not convinced that using Q functions would necessarily fail. On p.6, the paragraph \"can we learn conservatively-extrapolated Q-function\" gives some reasoning why this could fail, that we may not want to penalize unseen actions. This is in opposition to the BCQ algorithm [1], which explicitly tries to avoid unseen actions and still has good performance. Trying a variant with Q(s,a) could be worthwhile. \nI am not exactly sure if I understood Appendix A properly but, from my understanding, I do not think the argument made there necessarily invalidates using Q functions. It seems to apply mostly to deterministic expert policies and also Q(s,a) could still have reasonable values due to function approximation (even if the particular action 'a' is not seen in demonstrations). \n\n5) Which RL algorithms were used for the imitation learning + RL set of experiments?\n\n6) For table 1, are the results also averaged over different sets of demonstrations? \n\n7) Are error bars one standard deviation or one standard error (divided by sqrt(n))?\n\n8) For figure 3, using RL without imitation learning would serve as a good additional baseline \n\n9) Ablation study: Trying no negative sampling with a perfect model could isolate the effect of negative sampling. \n\n10) Ablation study: What is the no behavior cloning and perfect model experiment trying to show?\n\n11) I think the name of the algorithm should be modified as \"value iteration\" refers to a specific dynamic programming algorithm for learning value functions, while the proposed algorithm does not resemble this at all. \n\nMinor comments and typos (no impact on score):\n- Using the cross-entropy method as in QTOpt [2] could be used to pick actions in a more refined manner.\n- There is a large amount of blank space on p.8\n- p.3 \"At the test time\" -> \"At test time\"\n- p.4 \"entire states space\" -> \"entire state space\", \"burned to warm up them\" -> \"burned to warm them up\"\n- p.9  \"option 2 by search the action uniformly.\"  -> \"option 2 to search the actions uniformly\"\n\n[1] \"Off-Policy Deep Reinforcement Learning without Exploration\" by Fujimoto et al.\n[2] \"QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation\" by Kalashnikov et al.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bylgr7yRFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents the value iteration with negative sampling (VINS) algorithm, a method for accelerating reinforcement learning using expert demonstrations.  In addition to learning an expert policy through behavioral cloning, VINS learns an initial value function which is biased to assign smaller expected values to states not encountered during demonstrations.  This is done by augmenting the demonstration data with states that have been randomly perturbed, and penalizing the value targets for these states by a factor proportional to their Euclidean distance to the original state.  In addition to the policy and value function, VINS also learns a one-step dynamics model used to select actions against the learned value function.  As the value function learned in VINS is only defined with respect to the current state, action values are estimated by sampling future states using the learned model, and computing the value of these sampled states.\n\nEmpirical results on a set of robotic control tasks demonstrate that VINS requires significantly less interaction with the environment to learn a good policy than existing, state of the art approaches given the same set of demonstrations.  While the paper presents a novel and highly effective approach, there are some apparent limitations to the algorithms which should be highlighted, and there is room for improvement in the empirical evaluations.\n\nIt is unclear that VINS would generalize well beyond robotic control domains.  For one, its theoretical guarantees depend on the local reversibility of the dynamics, that is, for small deviations from the desired state, it is possible to return to that state in a single step.  This isn't too significant a restriction, as the ability to recover quickly from small mistakes would seem to be a necessary for any method to be able to provide similar guarantees about its behavior.  The bigger issue is the use of the Euclidean metric (or any fixed metric) in the definition of conservative extrapolation.  Basically, a state is said to be similar to the states observed during the demonstrations if it is close, under the Euclidean metric, to at least one demonstrated state.  This is a reasonable approach in robotic control tasks, where Euclidean distance is a good measure of how similar two configurations are to one another, but it would seem to be unsuitable for domains where the observation space consists of images or other high-dimensional representations.  In those cases, a useful notion of similarity would likely have to be learned from the data.  In such domains, one might imagine that the conservative value function would simply learn to distinguish between real observations, and those that have been perturbed by random noise, which would never be observed in the actual task.\n\nWhile experimental results demonstrate a very significant advantage for VINS both in terms of sample complexity and final performance, results are presented for only two tasks, 'pick-and-place' and 'push', while VINS outperforms the alternatives on these tasks, it is worth noting that its initial performance (without additional environment interact) is not dramatically superior to pure behavioral cloning.  It would be helpful to see how well VINS compares against the alternatives for a much smaller number of demonstrations, say 5-20, a regime where we would expect initial performance to be poor.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces Value Iteration with Negative Sampling (VINS) algorithm as a method to accelerate RL using expert demonstrations. VINS learns an initial value function that has a smaller value at states not encounter during the demonstrations.

The reviewers raised several issues regarding the assumptions, theoretical results, and experiments. The method seems to be most natural for robotic control problems. Nonetheless, it seems that the rebuttal addressed most of the concerns, and two of the reviewers increased their scores accordingly. Since we have three Weak Accepts, I believe this paper can be accepted at the conference.
Decision: Accept (Poster)