Data augmentation for efficient learning from parametric experts
Authors: Alexandre Galashov
Josh Merel
Nicolas Heess
Venue: NeurIPS 2021 Submitted (2021)
Keywords: behavior cloning
expert-driven learning

Abstract: We present a simple, yet powerful data-augmentation technique to enable data-efficient learning from parametric experts. Whereas behavioral cloning refers to learning from samples of an expert, we focus here on what we refer to as the policy cloning setting which allows for offline queries of an expert or expert policy. This setting arises naturally in a number of problems, especially as a component of other algorithms. We achieve a very high level of data efficiency in transferring behavior from an expert to a student policy for high Degrees of Freedom (DoF) control problems using our augmented policy cloning (APC) approach, which combines conventional image-based data augmentation to build invariance to image perturbations with an expert-aware offline data augmentation approach that induces appropriate feedback-sensitivity in a region around expert trajectories. We show that our method increases data-efficiency of policy cloning, enabling transfer of complex high-DoF behaviours from just a few trajectories, and we also show benefits of our approach in the context of algorithms in which policy cloning is a constituent part.

Data augmentation for efficient learning from parametric experts
Anonymous Author(s)
Affiliation Address email
Abstract
We present simple, yet powerful data-augmentation technique to enable data- efficient learning from parametric experts. Whereas behavioral cloning refers to learning from samples of an expert, focus here what refer to the policy cloning setting which allows for offline queries of an expert or expert policy. This setting arises naturally in number of problems, especially other algorithms. We achieve very high level of data efficiency in transferring behavior from expert to student policy for high Degrees of Freedom (DoF) control problems using our augmented policy cloning (APC) approach, which combines conventional image-based data augmentation to build invariance image perturbations with an expert-aware offline data augmentation approach that induces appropriate feedback-sesstivityy in a region around expert trajectories. We show that our method increases data-effin data-efficiency of policy cloning, enabling transfer of complex high-DoF behaviors from just few trajectories, and we also show benefits of our approach in the context of algorithms in which policy cloning is constituent part.
2
4
component of
6
8
10
11
12
13
14
15
Introduction
16
17 In various control and reinforcement learning settings, there need to transfer behavior from an expert policy to student policy. Broadly, when only samples from the expert policy are available, the standard approach is to employ a version of regression from states to actions. This class of approaches for producing policy is known as behavioral cloning [Pomerleau] 1989 Michie and Sammut [996]. Behavioral cloning is quite flexible and supports the setting where expert trajectories come from human teleoperating the relevant system directly, well as various settings where the trajectories are sampled from other controllers, which themselves may have been trained or scripted. However, for
18
19
20
21
23
24 any of the settings where the expert policy is actually available, rather than just samples from the expert, it is reasonable to suspect that sampling random rollouts from the expert policy followed by performing behavioral cloning is not the optimally efficient approach for transferring behavior from the expert to the student. Once a trajectory has been sampled via an expert rollout, there is actually additional information available that can be ascertained in the neighborhood of the trajectory, without having to perform an additional rollout, via the local feedback properties of the expert.
25
27
29
30 In this work, we refer to this setting, where we want to transfer from an expert policy to student
31 policy, while assuming the expert policy can be queried, as policy cloning. Naturally, there still often an incentive to reduce the total number of rollouts, which may require actually collecting data
33 an unsafe or costly fashion, especially for real-world control problems. As such, there an aim to characterize any efficiency that be gained in learning from small numbers of rollouts without as 35 much concern for how many offline queries are required the expert policy. one has primarily encountered behavioral cloning in the context of learning from human demonstrations, policy cloning,
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

37
with an available expert policy may seem contrived. However, policy cloning naturally arises in many settings, For example, we may have multiple experts that we wish consolidate into single neural network policy or there may be memory considerations that motivate compressing a large expert network into a smaller model with similar behavior. Perhaps most natural are settings in which an expert policy is costly or slow to execute, for example due to running a compute intensive procedure such as model predictive control (MPC) on specialized hardware (e.g. GPU); in such settings, the aim is to transfer expert behavior to parametric student policy that amortizes the cost. DAGGER is one well known approach for efficiently transferring behavior from an expert to student under these kinds of constraints [Ross et al.) 2011]. In separate setting, the expert may be suboptimal and the student needs to learn from expert while also being able to exceed the expert performance, perhaps by continuing to learn from a task via RL. This problem has been described as kickstarting in one incarnation [Schmitt et al. 2018 but also can arise when learning from behavioral priors (Trumala
38
39
40
41
42
43
44
45
46
48
49
al. 2020 Galashov al. 2019 as also happens, for example, in Distral [Teh et al. 2017].
50
To improve data-efficiency in supervised settings generally, including in behavioral cloning settings, it is reasonable to consider data augmentation. Data augmentation refers to applying perturbations to a finite training dataset to effectively amplify its diversity, usually in the hopes of producing a model that is invariant to the class of perturbations performed. For example, in the well studied problem
51
52
53
54 of object classification from single images, it is known that applying many kinds of perturbation should not affect the object label, model can be trained with many input perturbations all yielding the same output [Shorten and Khoshgoftaar 2019). This setting is fairly representative, with data augmentation usually intended to make the model "robust" to nuisance perturbations of the input. has also been recently demonstrated to be effective in the context of
56
58
This class of control problems in the offline RL setting Yarats et al. 2021 Laskin et al. 2020
59
Critically, for control problems it is not the case that the action should be invariant to the input state. Or rather, while it does make sense for a control policy to be invariant to certain classes of sensor noise, an important class of robustness is that the policy is appropriately reesoscc-re This is to say that for small perturbations of the state of the control system, the optimal action is different in precisely the way that the expert implicitly knows. This has been recognized and exploited in previous research that has distilled seedback-contro) plans into controllers [Mordatch and Todorov 2014 Mordatch et al., 2015 Merel et al. 2019). A similar intuition also underlies schemes which 66
60
61
62
63
64
67
inject noise into the expert during rollouts to sample more comprehensively the space of how the expert recovers from perturbations [askey et al. 2017 Merel et al. 2019].
68
69 In this work, we leverage this insight to develop a highly efficient policy cloning approach that
70 makes use of both classes of data augmentation. For a high-DoF control problem that operates only from state (humanoid run task from DeepMind control suite (Tunyasuvunakool et al.) 2020)) we
71
72 demonstrate the feasibility of policy cloning that employs state-based data augmentation with expert
73 querying to transfer the feedback- senssiiiv behavior of the expert in region around small number
74 of rollouts. Then on a more difficult high-DoF control problem that involves both state-derived and egocentric image observations (humanoid running through corrdiors task from DeepMind control 76 suite Tunyasuvunakool et al. 2020 D) we combine the state expert-aware data augmentation with separate image augmentation intended to induce invariance to image perturbations. Essentially
78 our expert-aware data augmentation involves applying random perturbations to the state-derived observations, and training the student to match the expert-queried optimal action at each perturbed 80 state, thereby gaining considerable knowledge from the expert without performing excessive rollouts
79
81 simply cover the state space around existing trajectories. Our approach compares favorably to
82 sensible baselines, including the naive approach of attempting to perform behavioral cloning with
83 state perturbations, which seeks induce invariance (as proposed in [Laskin et al. 2020 rather than feedback to state-derived observations.
84
In the presentation that follows, we will describe the problem setting (Section as well our 86 approach (Section 3). describe the domains we employ and present our initial experiments (Section 4 show that our augmented policy cloning approach works well when used as component of other
88 algorithms like DAGGER and kickstarting (Section 5). and finally close with a discussion (Section ())
2

---

89 2
Problem description
2.1
Expert-driven learning
81 We start by introducing a notion of expert-driven learning that will be used throughout the paper. At first, we present general form of the expert-driven objective and then introduce few concrete examples. We consider standard Reinforcement Learning (RL) problem. We present the domain as an MDP with continuous states for simplicity, however the problem definition is similar for a POMDP with observations derived from the state. Formally, we describe the MDP in terms of a continuous state space S R" (for some > 0), an action space A, transition dynamies p(s'|s, ss a) S x A p(S), and a reward function r S x A + R. Let II be a set of parametric policies, i.e. of mappings 96 T : + p(AA) from the state space S to the probability distributions over actions A, where E R"
98
94
95
96
97
99
for some m > 0. For simplicity of the notation, we omit the parameter in front of the policy, i.c.
100 = T and optimizing over the set of policies would be equivalent to the optimizing over a set of parameters. A reinforcement learning problem consists in finding such a policy that it maximizes the expected discounted future reward:
102
J(T) = Eper) Er'r(an
103
where p(r) p(00) II, P(a, (s))p((((1())))))))))))))))))))( (e) is trajectory distribution. We assume the existence of an expert policy T (a)ss)) This policy could used to simplify the learning of a new policy on the same problem. Formally, construct new learning objective which aims maximize the expected reward the problem in hand as well as to clone the expert policy:
104
106
[ Æg) = aJ(()) - AD(T, TB);
107
where D is some measure of distance of π from TE and 2 0, A 2 0 are parameters measuring importance of both objectives. In most of the applications, a E {0, 1] and X 2 0 represents relative importance of cloning an expert policy with respect to the RL objective.
108
109
110
2.2
Behavioral cloning (BC)
One important instance of the objective 2 with 0, À 1 is behavioral cloning. In this case, the measure of distance is defined as:
112
DBC(T, ( TE) = (o,))) (log
[log ग (a|s)]
(3)
Here, Bg = {(s,, a.), i= 11 I...., N). N > O is fixed dataset containing expert data. Minimizing the objective 33) is be equivalent to maximizing the likelihood of the expert data under the policy т.
114
The action in eqn. (3) be replaced by (s))) for deterministic policies or by the mean the mode for stochastic policies (e.g., by the mean (s()) for Gaussian policies <<(--s) = ((uE()), (+))))
115
116
117
2.3
DAGGER
118
Performance of Behavioral Cloning (BC) can be limited due to the fixed dataset, since the resulting policy may fail to generalize to states outside the training distribution. different approach, known in the literature as DAGGER [Ross et al.) 2011] was proposed to overcome this limitation. In this setting, the expert queried in states Visited by the student, thus reducing distribution shift, In our notation, this corresponds to 0, X 1 in eqn. (2) and the measure of distance is defined as: DDAGGER (T,, πE) (r) [log (a, [s)]],
119
121
122
(4)
123 where (T). 8 E (0, 1] is trajectory distribution where actions are sampled according to the mixture policy between student and an expert:
p(als) (als) + (1 ()ss())s),
(5)
125
(t). As in Section 2.2 for stochastic experts this action can be replaced by the mean or mode of the distribution in eqn. (4) The policy 7 (a]s) corresponds to a frozen version of student policy x so
The action a; in eqn. is resampled from the expert policy for the state st, i.e., of (
126
127
that the gradient V DDAGGERRTT TE) is not taken with respect to p(als). Note that even though, in
129
eqn. 4 we collect data from the environment, the setting nevertheless corresponds to pure imitation
learning since expected reward is not directly maximized.
3

---

131
2.4
Kickstarting
132 In eqn. (2). we combine both maximization of expected task reward and minimization of distance to
133
the expert. In literature, it is known as Kickstarting (Schmitt et al. 2018). In this case, in the objective
134
from eqn. 2 a 1, and À 2 0. As the measure of distance, we use the erosspyntropy from expert
135 to student, similarly to [Schmitt et al. 2018):
= J(m) p() [=Rrg( log i [a]s)]
where p(T) is a trajectory distribution, where actions are sampled according to the student policy (-Is). Usually, in the Kickstarting setting, the expert is sub-optimal and the goal is to train a policy that eventually outperforms the expert. Thus, is customary to reduce À over the course of training.
136
137
138
139
Yet, for simplicity, in our experiments we keep this coefficient fixed.
140
3
Augmented policy cloning
141
The previous section has demonstrated that the objective corresponding to the cloning behavior from the parametric expert policy could arise in multiple scenarios. In this section we propose a new and simple method which can significantly improve the data efficiency of the approaches described in Section 2
142
143
144
We explain the basic idea for BC, but its generalization to other expert-driven learning approaches described in Section
2 is straightforward. In Section 5 we show results for these problems.
145
146
When optimizing the objective 33. for every state & E DD from the expert trajectories dataset, we consider a small Gaussian state perturbation:
147
us ~ ~00,00
148
which produces a new virtual state:
= = +58 5s
(8)
149
Then, for this state we query the expert and obtain a new action
a' ~ T + 8s)
(9)
150 We then augment the dataset D g with these new pairs of virtual states and actions. More explicitly the idea can be expressed in terms of the following objective:
D(T, APC = [[..)) (log [[ass) + Wbs.
log <(alls + 6s)]
(10)
We call this approach Augmented Policy Cloning (APC) as it queries the expert policy to augment the training data. This approach is different from a naive augtentgmmnttiionn technique, where a new state would be generated, but associated with the original action (and not a one). It therefore allows to build policies which are feedback-responsive with respect to the expert. We formulate APC algorithm for BC in Algorithm 1
153
155
156
4
Core Results: Evaluation of Augmented Policy Cloning
157
158
4.1
Domains
159
To study how our method performs on complex control domains, consider two complex, high-DoF continuous control tasks involving control of a physically simulated humanoid body. Both domains
160
161
implemented using the MuloCo physics engine [Todorov al.] 2012) and are available in the dm_ control repository (Tunyasuvunakool et al., 2020). The first task is the standard control suite
162
163
Run task, where the Humanoid body needs to run a target speed and observations are based on proprioception. The second task is the Walls task which requires the same Humanoid body to run along corridor and avoid walls, using both proprioception and egocentric vision as observations. Both of these problems are rather challenging insofar as they require stabilization and locomotion control of a relatively complex humanoid body with 21 actuated DoFs, in one case using vision to guide the movement. Note these environments are related to the domains that have been proposed
184
165
166
167
168
169
for use offline RL benchmarks [Gulcehre et al. 2020]; however, the experiments we perform in this work require availability of the expert policy, so we do not use offline data, but instead train
170
171
new experts and perform experiments in the very low data regime. For more details, please refer to
Section 1.1 in Supplementary Material.

---

Algorithm 1 Augmented Policy Cloning (APC)
Parametric student policy: TTa
Initial parameters: 0g
expert policy: T
Dataset Bg { a;),
....,.. NN,,,,,J N
of expert state-action pairs
State perturbation noise
Learning rate a
Number of augmented samples: M
Number of gradient updates: K
Size of a batch: L for kK do
Sample a batch of pairs {(ai, si) BE For each state 8;, sample perturbations 58j (1,00,,)))))) I,..., M Construct M virtual states s/, 8 + ō8,, 1,..., [, I....,M
Resample new actions from expert de, TE(-1=1.T)
For Gaussian experts, the action a; = (s(s;) and the new actions are al. (L(()))) Compute the empirical negative log-likelihod:
L log me (a,1ss) + "o E==1 M
log To, (d6)1(
Update the parameters 0x+1 = 02 Vo end for
173
For each task, we train expert policies to convergence using the MPO algorithm Abdolmaleki et al. (2018)- Since the expert policy essentially saturates task performance, for each task, we keep three partially trained experts such that we can assess the ability of the kickstarting approach to outperform sub-optimal experts. We refer to the different experts Low, achieving approximatively 25 % of the optimal policy reward, Medium, achieving around 50 % of the performance and High, corresponding to the converged policy. Each expert is represented by a Gaussian policy. For more details, please
175
178
177
178
179
refer to Section 1.2 in Supplementary Material.
180
4.2
Applying Augmented Policy Cloning
181
First, we evaluate the performance of APC in fitting a fixed dataset of expert trajectories. In order to study the data efficiency of the method, we construct datasets containing different numbers of expert trajectories. The expert policies are represented by conditionally Gaussian distributions,
182
183
184
iec (=(+s)) = (s)(s), (s)). Thus, to assess the robustness of our method to expert noise we produce trajectories using the experts' mean but adding different levels of (homoscedastic) cero-mean Gaussian noise dg.
185
186
a ~ (=NN)(S))
187
Note that in addition to policy noise og which is introduced when sampling trajectories, initial pose and environment layout (for the Walls task) are also sampled randomly for each episode. We consider 4 levels of expert policy noise: Deterministic, which uses the Gaussian mean for the action, Low, with og = 0.2, Medium og 0.5 and High = 1.0.
188
189
190
191
For the APC method, we rely on Algorithm I For baselines, we consider BC algorithm from eqn. (3) as well as a simple modification of BC, where we apply, similar to APC, state perturbations as in eqn. (7) and eqn. (8), but we do not produce new action from the expert. We call this approach Naive Augmented Behavior Cloning (Naive ABC) which essentially corresponds to robustification
192
183
194
195
of the student policies with respect state perturbation and is similar in spirit to standard data- augmentation approaches. For vision-based tasks, we consider random crop augmentations of size 48x48 (downsampled from the input image of 64x64), similar to Laskin et al. [2020). When the image augmentations are used we add "with image" to the method name. On top of that, we consider a variant, where only image augmentation used, which we call Naive ABC (image only). For all methods, as an action in the objective from eqn. (3), we use an expert mean (e(s). We train all approaches to convergence (300K learning iterations on Walls and 13M learning iterations on Run). Each learning iteration corresponds applying gradients to 64 trajectories, each containing
196
197
198
199
200
201
203
10 time steps. After each learning iteration, we evaluate the policy on both validation set (50 random instances of the environment) and a test set (150 random instances of the environments).
5

---

Full trajectories
Short trajectories
600
400
Run
Episode reward
1000
Walls
500
50
100
200
500
1000
50
100
200
500
1000
Number of trajectories
Method
APC BC
Naive ABC
Figure 1: Behavioral cloning results on Run and Walls tasks (represented by rows). The X-axis represents the number of trajectories, whereas the Y-axis corresponds to the episodic reward averaged among 150 independent evaluations. The highest point of the bar corresponds the mean, whereas the dashed lines indicate the standard deviation. The pink dashed line indicate average expert performance. The legend describes a method which used. On the plot on the left depicts a standard BC experiment, where dataset contains specified number of full trajectories from the expert. The plot on the right illustrates the experiment, where dataset contains full trajectories and the rest are the short ones, containing only 200 timesteps each.
We apply early-stopping based the validation performance select the best model and report corresponding performance on the test set. For more details, please refer to Section 1.3 in Supplementary Material, As an additional evaluation, we test robustness of the obtained policies to fixed amount of noise during execution. For a learned student policy TT-ss) = N((((s) we evaluate by executing an action:
208
207
208
209
~ (~((p((),
210
where the fixed amount of student noise. We consider similar noise magnitudes as for the expert. For APC and Naive ABC, we sweep over state perturbation noise levels and choose the ones performing the best on the validation set. For APC, we use a, = 0.1 for Run and a, 1.0 for Walls. 0.01 for Walls. The ablation experiments
212
For Naive ABC, we use over noise levels for APC and Naive ABC are presented in Section of the Supplementary Material (Figure and Figure 2).
0.001 for Run and
214
The first of results, in Figure (left) demonstrates the increased data efficiency of APC over BC and Naive ABC in terms of number of trajectories. The noise level of the expert and student are fixed to Low for the ease of comparison. We also see that Naive ABC performs similarly BC. To further push the limits of data efficiency, we conducted a variant where a dataset contains only I full trajectory (1000 timesteps for Run and around 2k timesteps for Walls) along with multiple short trajectories
217
218
219
220
200 time-step only). This dramatically reduces the amount of expert data available to learn from. However, we hypothesise that in the environments considered, much of the diversity of the trajectories
222
223
arises due to initial state variation. This setting might arise in domains where execution is costly, such as robotics applications. In such setting we might have few longer trajectories along with patchwork of shorter trajectories covering more diverse parts of the state space. The results for this experiment are given figure (right). Again, we see that APC significantly more efficient than [
224
226
BC and Naive ABC. Interesting note that APC picks up quite a high performance after observing

---

228 10 (1 full and 9 short) trajectories for Run task and 100 (1 full and 99 short) trajectories for the Walls task. In Section in Supplementary material, we provide additional results for Walls task when use image-based perturbations.
229
In the experiment, in order understand how robust method to noise, study the impact of different levels of student and expert noises on performance. For each run, we use a dataset of
232
233
100 trajectories. The results are given in figure where each column corresponds
different level of expert noise, and the X-axis represents different levels of student noise. At first, we observe that APC consistently more robust than BC and Naive ABC for any level of expert and student noise. On top of that, we can notice that for any fixed level of expert noise, the performance degrades when a student noise increases. Finally, we see that for higher noise levels of expert. the learned student performs better in the high noise regime. It is consistent with the intuition training on noisy trajectories leads to a more robust policy. Overall, APC leads the most robust policy.
234
235
237
239
Det.
Low
Medium
High
Epácde
Student noise
APC
BC
Naive ABC
Noise sensitivity results. We consider levels of noise for student and expert: Determin- istic, which uses the Gaussian mean for the action, Low, is the noise 0.2, Medium 0.5 and High o = 1.0. Each column corresponds to a different level of expert noise. X-axis corresponds to different level of student noise. Y-axis corresponds the episodic reward averaged among 150 independent evaluations. The highest point of the bar corresponds to the mean, whereas the dashed lines indicate the standard deviation. The legend denotes method and corresponds task. The pink dashed line indicate average expert performance.
Figure 2:
Additional Results: Augmented Policy Cloning as a subroutine
240
241
5.1
DAGGER with data augmentation
Run
Walls
Method
APC
730
with Image
Episode
Naive ABC (image only)
Naive ABC with image
208
Expert #
0.0
Zonk
200k
T50k
L00K
2006
200
- 0.3
Environment steps
Figure DAGGER results. On the X-axis we report the number of environment steps. On the Y-axis we report averaged across 3 seeds episodic reward achieved by the student. We report confidence intervals in the shaded areas. For Run task, the confidence intervals are very small and are not visible. In solid line report the performance without using expert policy during the acting. In dashed line, we report the performance of the policy which mixes 30% with the expert. All the methods use mean action during evaluation.

---

242 As described in Section 2.3
DAGGER Ross et al. 2011] is more sophisticated approach where data is collected from the real environment by executing a policy from eqn. (3). which is a mixture between a student and an expert. In this section we study how data augmentation approaches affect the data efficiency of the DAGGER algorithm.
243
245
248
We consider similar baselines for both tasks as in the previous section. For an expert policy that has been pre-trained via MPO [Abdolmaleki et al. 2018 we perform online rollouts for two values of the expert-student mixing coefficient, 3 = 0 and 3 0.3 (see eqn. 5 Since both student and expert are Gaussian distributions, instead of using log π in equ. 4. we could use cate--onniiional cross entropy from an expert to a student, H[ 8) | (-|s)]. Empirically, we found that it worked better Supplementary Material. We run the experiments in data-restricted setup such that for every collected trajectory (10 time-steps), we apply 10 gradient steps, using replay-buffer to store the past experience. Additional experimental details are given in Section 1.4 in Supplementary Material. Results are shown in Figure 3 We see that APC and its vision variant outperform BC and Naive ABC similarly to the behavior cloning experiments. While we observe that image augmentation can help, we see that the primary advantage comes from the state-based augmentation for APC. For the Run task, we observe that all DAGGER methods achieve slightly lower performance than an expert policy. We speculate that this is due to insufficient coverage of the state space during training.
247
248
249
250
251
than using log т.. We demonstrate comparison in Section
252
253
255
257
259
260
5.2
Kiekstarting with data augmentation
Run, High
Run, Medium
Run, Low
750
Episode reward
250
1.0gg
2.00
0.0
4.00
1.0g0
Drg
oeg
9.0g
Loeg
0.0
3.0e9
4.0e9
Walls, High
Walls, Medium
Walls, Low
750
500
250
0.0c9
0.5w9
1.ag
0.0e9
0.569
1.099
0.0e9
0.5a9
3.0a9
Environment steps
Method
APC
Naive ABC
Naive ABC (image only) MPO
APC with image
Naive ABC with image
Figure 4: Kickstarting results. On the X-axis show the number of environment steps. On the Y-axis we report averaged across 3 seeds episodic reward achieved by the student. We report confidence intervals in the shaded areas. For Run task, the confidence intervals very small and are not visible. Each row indicates a task, whereas a column corresponds to the expert type. Dashed black line shows the expert performance.
A similar in spirit approach is kickstarting Schmitt et al. 2018 where we solve an RL task as well as cloning the expert policy. Similarly to previous section, we apply APC in kickstarting on the cross entropy term in eqn. (6) We use 3 types of expert policy as described in Section 4.1 We run experiments using a distributed setup with 64 acting policies and learner, querying the batches of trajectories (of size 10) from a replay buffer. On top of running BC methods, we also report the performance MPO Abdolmaleki et al. [2018) learning from scratch on the task of interest. All details are given in Section 1.5 in Supplementary Material. The results are given in figure 4
262
263
254
286
8

---

268
We observe that APC performs better than Naive ABC on Run task and similarly on Walls task. Both approaches perform better than BC and learning from scratch. We hypothesise that the reason of not seeing consistent advantage could be due to two factors. As we are in a high-data regime, since there is no limit on relative acting learning ratio, and acting policies are not restricted to collect trajectories, it is unclear whether aatteatugenttiion should help. In addition, we use reward signal which makes the impact of expert cloning less important. Note that the resulting agent is less data efficient in these experiments; this is because we do not control the relative ratio between acting and learning (i.e., no rate-limiting on the learner, due to instability of kickstarting experiments when rate-limi was explored). Furthermore, unlike in kickstarting |Schmitt et al. 2018), we do not
269
271
272
273
275
277
use an annealing schedule of to make the experiments simpler, but we still observe that a fixed coefficient helps to kickstart an experiment and outperform expert policy. On top of that, see that image-basee augmentation have less of impact in this setting.
279
Discussion
280
281
Many expert-driven learning approaches actually have access to an expert that can be queried; however, this opportunity is rarely exploited fully. In this work we demonstrated a general scheme for more efficient transfer of expert behavior by augmenting expert trajectory data with virtual, perturbed states as well as the expert actions in these virtual states. This data augmentation technique is widely applicable and we demonstrated that it improves data efficiency when used in place of behavioral cloning both in the offline setting or when behavioral cloning used as step within DAGGER or kickstarting.
285
285
287
288
Critically, data efficiency is generally very important in realistic applications, where new data acquisition cost could be high. In particular, settings involving deployment of policies in the real world, such as robotics applications, may benefit from an ability to efficiently transfer expert policy behavior from one neural network to another (for compression or execution speed reasons), or to combine behavior from multiple experts into single neural network. While overall, we consider the present work to be fairly basic research with limited ethical impact, insofar as our approach decreases the amount of data which needs to be collected through processes which could potentially be unsafe or costly, there is a potential positive social value.
209
290
291
293
295
296
The limitations of our approach consist in the reliance on the ability to query expert policy for the perturbed states which reduces the amount of applications where the method could be used. Another limitation is the reliance on the continuous state spaces. In discrete state spaces, it is unclear whether
297
298
299
a small perturbation in state would result in a valid action from an expert.
300
In future work, we plan to explore how our proposed augmentation technique can be leveraged in the context of Kl-regularized RL with behavior priors.
301
References
303
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation, 2018.
304
305
Alexandre Galashov, Siddhant M. Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojciech M. Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in kl-regularized rl, 2019.
306
308 Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine. Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RI unplugged: A collection of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
310
312 Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on robot learning, pages 143-156. PMLR, 2017.
314 Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data, 2020.

---

316
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid con- trol, 2019.
317
319
Donald Michie and Claude Sammut. Behavioural Clones and Cognitive Skill Models, page 387-395. Oxford University Press, Inc., USA, 1996. ISBN 019853860X.
321
ggor Mordatch and Emo Todorov. Combining the benefits of function approximation and trajectory optimization. In Robotics: Science and Systems, volume 4, 2014.
323
Igor Mordatch, Kendall Lowrey, Galen Andrew, Zoran Popovic, and Emanuel V Todorov. Interactive control of diverse complex characters with neural networks. Advances in Neural Information Processing Systems, 28:3132-3144 2015.
324
325
328
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Technical report, Carnegie-Melloo, 1989.
327
328
Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning, 2011.
329
330
Simon Schmitt, Jonathan J. Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojeiech M. Czarnecki. Joel Z. Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, and S. M. Ali Eslami. Kickstarting deep reinforcement learning, 2018.
331
333
Connor Shorten and Taghi Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6, 07 2019. doi: 10.1 I 8666400577-099-01977..
335
Yee Whye Teh, Victor Bapst, Wojeiech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. arXiv preprint arXiv: /00:::007001114 2017.
338
337
338
Dhruva Tirumala, Alexandre Galashov, Hyeonwoo Noh, Leonard Hasenclever, Razvan Pascanu, Jonathan Schwarz, Guillaume Desjardins, Wojeiech Marian Czarnecki, Arun Ahuja, Yee Whye Teh, et al. Behavior priors for efficient reinforcement learning. arXiv preprint arXiv:2010. 14274, 2020.
339
341
342
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
343
344
345
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_con con_cotrol:: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.
346
347
348
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In 9th International Conference on Learning Representations, ICLR, volume 2021, 2021.
349
7 NeuRIPS 2021 checklist
352
Do the main claims made in the abstract and introduction accurately reflect the pa- per's contributions and scope?
354
Yes, we believe that the claims in the abstract and introduction are accurately reflected by the paper's contributions and scope.
Have you read the ethics review guidelines and ensured that your paper conforms to them?
357
358
Yes.
359
Did you discuss any potential negative societal impacts of your work?
Yes. We included this in the Discussion section.
10

---

Did you describe the limitations of your work?
Yes, we discuss these in the "Discussion" section.
383
Did you state the full set of assumptions of all theoretical results Not applicable.
385
Did you include complete proofs of all theoretical results
Not applicable.
387
Did you include the code, data, and instructions needed to reproduce the main exper- imental results? ?
388
369
We included complete details in the supplementary material which could be used to reproduce the experimental results.
Did you specify all the training details?
Yes, in the main paper and supplementary material.
373
Did you report error bars?
We did. For every plot we reported the mean and standard deviations averaged across number of independent random evaluations.
375
378
Did you include the amount of compute and the type of resources used?
377
Yes, we did, the supplementary material.
378
If your work uses existing assests, did you cite the creators?
379
Yes. We use DeepMind control suite tasks and we cite the appropriate publications.
380
Did you mention the license of the assets?
381
We did not explicitly mention the licence as it is clear from the reference.
382
Did you include any new assets either in the supplementary material os URL? No.
383
Did you discuss whether and how consent was obtained from people whose data you're 're using / curating?
385
386
Since the assets and references are in the public access, no consent was required.
387
Did you discuss whether the data you are using / curating contains personally identifi- able information or offensive content?
388
389
The assets which we used did not include any personally identifiable information nor offensive content.
Did you include the full text of instructions given to participants and screenshots, if applicable?
392
393
Not applicable.
Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable Not applicable.
385
387
Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
398
399
Not applicable.


Reviews:
[{"review_id": "po7H-BKCAf8", "reviewer": "Reviewer_u3Dw", "summary": "This paper proposes to study the non-interactive imitation learning setting where access to the parametric expert policy is available. The paper proposes to augment the logged expert data’s states with noise, and then determine then sample corresponding actions from the expert’s policy. These perturbed states and corresponding action are then use to train the agent. They show this method outperforms Behavior cloning and an approach where the logged states are perturbed but the corresponding actions are kept fixed on imitation learning tasks and RL tasks where IL is used for warmstarting. ", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and clear, but the setting seems fairly contrived. The examples they give where this problem may arise are not reflected in their experimental setup. For instance, they say the setting may come up when there are multiple experts which must be combined into one (this is not totally clear either, what do you do when the experts disagree? Do you randomly sample?), or if the expert is an expensive MPC procedure. However, in their experiments the experts have the same network architecture as the agent. In the examples they describe, there may be other issues, for example non-realizability which could complicate things. \n\nIn any case, given the setting, the proposed method seems more like an obvious first baseline to try rather than an original contribution. It’s completely expected that it would perform better than BC and the other naive method, so there isn’t much insight or takeaway from the experiments. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The setting is contrived and the proposed method seems obvious given the setting.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "oJRbzGF-gXz", "reviewer": "Reviewer_rKs4", "summary": "The paper explores a novel approach to using expert policies in RL context. The main idea is to enrich expert demonstrations by perturbing a state by Gaussian noise and querying expert policy to produce an action from the perturbed state. These additional state-action pairs are used in the policy learning algorithm. The authors show experimentally that the proposed Augmented Policy Cloning (APC) can significantly improve the quality of cloned policy, compared to Behaviour Cloning.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes using expert policies to perform policy cloning in RL context. The method, termed Augmented Policy Cloning, can be applied in situations where expert policy is available. The states from expert demonstrations are perturbed with Gaussian noise and expert policy is used to produce a target action. The student policy is adjusted to maximize probability of these additional state-action pairs.\nThe method allows to expand expert demonstrations without doing environment rollouts. Authors compare the performance of APC on two tasks(Humanoid Run and Walls) against two baselines: Behaviour Cloning and Naive Augmented Behaviour Cloning. The later(NABC) is a modification of BC  via state perturbations but without changing corresponding actions.In the regime of low number of expert demonstrations APC significantly outperforms BC and NABC.\n\nThis idea of using expert policies for policy cloning is new to me. It looks like a useful tool in practice for efficiently using expert policies without the need to perform rollouts. \n\nIn terms of presentation quality I found the paper hard to read. Especially experimental sections can be improved by highlighting high level results and potentially moving some details into the Appendix. \nThe detailed comparison of APC and BC/NABC in Fig. 1 seems not that informative to me: it is clear that using expert queries around demonstration state will give much more information to the policy then in BC case. I think to make experimental part more informative it would be very valuable to consider a set of standard continuous-control tasks (like form DM control suite) and describe appropriate levels of state perturbation and advantage in reward.\n\nI also found the Naive ABC not a very meaningful baseline to compare to: simply perturbing the state with Gaussian noise will unlikely lead to a state where the same action is optimal. From Fig 1, 2, 3 it does look that NABC performs slightly worse then BC. However, in the case of kickstarting interestingly Naive ABC performs on par with proposed APC. I would be interested in authors thoughts on why this is the case.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "izgIzEruWJ", "reviewer": "Reviewer_sn3m", "summary": "This paper proposed an data-augmentation technique -- \"augmented policy cloning (APC)\" to enable efficient policy learning from parametric experts. It achieve a high level efficiency in transferring knowledge from an expert to a student policy for high Degrees of Freedom environment. The augmentation is introduce not only for the states but also to the actions of underlying policy.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "General:\nThe paper is well-written and easy to follow even for one who is not quite familiar with the area.\nThe idea is simple, yet very interesting. I am not quit into related literature, so I would not comment over its novelty.\nMy greatest concern is that whether this setting- \"cloning a parameterised experts\" is a large restriction.\nExperiments are decent. However, there is space to improve.\n \nMajor:\n- More discussion on the real world use case of the proposed method. To me, a prameterized expert is a strong restriction that would hinder the use case of proposed method.\n\n- The main results Fig 1-4 are presented over two environments (Runs & Walls). I would like to see how this method perform over other environments.\n\n- A followup to last one would be to analysis the noise scale used in the method when running on other enviroments. In other words, I would like the author to further analysis what aspect of the environment or expert policy would affect the noise scale.\n\nMinor:\n\n- line103: No \\pi in J(\\pi)'s expression. p(a|s) --> \\pi(a|s)\n- the presentation of Figure 2 might be improved", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5-8"}, "extra_text": {}}, {"review_id": "dRSCaFltAO4", "reviewer": "Reviewer_7FbC", "summary": "This paper proposes Augmented Policy Cloning (APC) approach to improve the data efficiency of expert behavior by augmenting expert trajectory data with virtual, perturbed states and the expert actions in these virtual states. This method is simple by applying the existing image-based data augmentation method, but it can increase the data efficiency of policy cloning and transfer high-DoF behaviors. The proposed method outperforms BC and Naive ABC.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper describes the flow of APC well and makes it easy to follow. This paper proposes APC which can generate a new state associated with the original action, and authors intuitively explain well the necessity and motivation for this method by comparing to the image-based methods. I would say the idea is not entirely new, but it can be useful in this field and the authors are well aware of its social impact.\n\nIn the significance aspect, I want to see authors to apply this method on other environment tasks. They try to propose a general scheme for more efficient transfer of expert behavior by augmenting expert's trajectory data, but their experiments seem limited in proving it. I have some questions about the experimental results and analysis (see below limitations and societal impact section). ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Here are a few concerns and questions I have regarding the submission, and addressing these would help me in giving a final rating:\n\n1. I wonder why this method is limited to the high-DoF control problem. As far as I understand, this method seems to be easily applicable to other environments, but it seems that it has not been tested in diverse environments. If the experiment had been performed on other problems for which the effect was easy to show intuitively, it would have been possible to effectively show the effect and potential for general application of this method. \n\n2. Looking at the results of Figure 2, it can be seen that the standard deviation of the results of Walls is significantly larger than that of Run. I would like to know why there is such a difference and whether there is no problem with reliability in these results.\n\n3. Looking at Figure 4 in Section 5.2, the results of Run and Walls are slightly different. In the case of Run, there is a difference on the results of High and Medium compared to Naive ABC and BC, but it is almost the same in Low. Conversely, in the case of Walls, in the case of High and Medium, there is little difference from Naive ABC, but it is relatively more different in Low. I wonder what is the basis for this difference.\n\n4.  I know it's due to the limitations of the amount, but putting too many details and comparisons and analyses of experiments into the supplementary material make it less legible.\n\n\n*** After reading all the feedback, it seems that this paper needs more detailed explanation of all the insights and environment settings. \nYou should conduct more convincing experiments on the proposed method and supplement your logic. \nThis was reflected in my score.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "Three days"}, "extra_text": {}}, {"review_id": "ZEB5d0Twaaf", "reviewer": "Reviewer_HWtk", "summary": "The paper introduces an approach to learning from an expert where a learning policy can query the expert. In particular, the authors propose to perturb the original states with Gaussian noise and then query expert actions for these states.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Novelty\nThe paper has limited novelty. The idea of augmenting the states with Gaussian noise is known to improve the robustness of reinforcement learning on mujoco tasks (see [1, 2]). While learning from an oracle is studied in DAgger.\n\n### Strengths\nThe paper demonstrates improvements over several baselines including behavioral cloning and image-based behavioral cloning with standard image augmentations. \n\n### Weaknesses \nI believe that the major weakness of this work is that it makes a strong assumption that adding Gaussian noise will produce a correct state. This assumption might not hold on to a variety of tasks. Moreover, the approach is evaluated on a limited set of tasks that raises further concerns regarding the generality of the approach. \n\nI cannot recommend the paper for acceptance in its current form. In particular, I have concerns regarding the generality of the approach. The experimental evaluation is limited.\n\n[1] Reinforcement learning with augmented data, Laskin etal., 2020\n\n[2] S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning, Sinha, etal., 2021", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "8k92klS9Imb", "reviewer": "Reviewer_KLas", "summary": "The paper proposes a policy cloning method that transfers an expert to a student. In the considered setting, the expert policy is accessible at all times, while there is no access to the environment to perform RL. The proposed method leverages data augmentation to successfully train the student policy.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method appears essentially an offline variant of the Dagger algorithm. The key difference seems that In Dagger, queries to the expert follows the state distribution of the student policy. In contrast, the proposed method augments a spherical (e.g. standard Gaussian) region for all expert states, and query the expert for the augmented states. In some scenarios, not needing environment interaction is an advantage.\n\nHowever, the proposed method has very limited algorithmic novelty and the empirical results are not surprising. In most tasks, covariate shift begins from states similar to the demonstrations. By knowing how the expert recovers from these states, behavioral cloning has sufficient info for the policy transfer to the student.\n\nWhen the proposed method is used for kick-starting, environment interaction is needed again to outperform sub-optimal experts. In this case, Dagger becomes more attractive than the proposed method, with its theoretical guarantee while being similarly easy to implement.\n\nThe empirical evaluation is also very limited, with experiments from only two environments. It is thus unclear if the method applies to other environments, In addition, the number of demonstrations may not be a fair metric for measuring the sample complexity, since the expert is queried many more times after the initial demonstration. It would be interesting to compare Dagger to the proposed method on the sample efficiency (including the additional queries to the expert).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the main review. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The ideas presented in the paper have been deemed interesting by most reviewers. However a few issues have been identified: 

- The experimental setting is limited and authors should motivate better their choice of environments. 

- The motivation for the proposed strategy needs to be discussed further. At the moment the use case setting for the proposed method appears quite limited. 
Decision: Reject