Data augmentation for efficient learning from parametric experts
Authors: Alexandre Galashov
Josh Merel
Nicolas Heess
Venue: NeurIPS 2022 Accept (2022)
Keywords: behavior cloning
expert-driven learning

Abstract: We present a simple, yet powerful data-augmentation technique to enable data-efficient learning from parametric experts for reinforcement and imitation learning. We focus on what we call the policy cloning setting, in which we use online or offline queries of an expert or expert policy to inform the behavior of a student policy. This setting arises naturally in a number of problems, for instance as variants of behavior cloning, or as a component of other algorithms such as DAGGER, policy distillation or KL-regularized RL. Our approach, augmented policy cloning (APC), uses synthetic states to induce feedback-sensitivity in a region around sampled trajectories, thus dramatically reducing the environment interactions required for successful cloning of the expert. We achieve highly data-efficient transfer of behavior from an expert to a student policy for high-degrees-of-freedom control problems. We demonstrate the benefit of our method in the context of several existing and widely used algorithms that include policy cloning as a constituent part. Moreover, we highlight the benefits of our approach in two practically relevant settings (a) expert compression, i.e. transfer to a student with fewer parameters; and (b) transfer from privileged experts, i.e. where the expert has a different observation space than the student, usually including access to privileged information.

Data augmentation for efficient learning from parametric experts
Alexandre Galashov DeepMind agalashov@deepmindd.c com
Josh Merel"
Nicolas Heess
DeepMind jsmerel@gmail.com
DeepMind
heess@deepmind.com com
Abstract
We present a simple, yet effective aata-augmentation technique to enable data- efficient learning from parametric experts for reinforcement and imitation learning. We focus on what we call the policy cloning setting, in which we use online or of- fline queries of an expert expert policy to inform the behavior of student policy. This setting arises naturally in a number of problems, for instance as variants behavior cloning, as component of other algorithms such DAGGER, policy distillation or Kl.-regularized RI.. Our approach, augmented policy cloning (APC), uses synthetic states to induce scabbback-cssstiiityy trajectories, thus dramatically reducing the environment interactions required for successful cloning of the expert. We achieve highly data-efficient transfer of behav- ior from expert to a student policy for high-degrees-offfreed of-eeedoo control problems. We demonstrate the benefit of our method the context of several existing and widely used algorithms that include policy cloning as a constituent part. Moreover, we highlight the benefits of our approach two practically relevant settings (a) expert compression, i.e. transfer to a student with fewer parameters; and (b) transfer from privileged experts, i.e. where the expert has a different observation space than the student, usually including access to privileged information.
region around sampled
Introduction
In various control and reinforcement learning settings, there is need to transfer behavior from expert policy to a student policy. Broadly, when only samples from the expert policy are available, the standard approach is to employ a version of regression from states to actions. This class of approaches for producing a policy is known as behavioral cloning [Pomerleau 1989 Michie and Sammut 1996). Behavioral cloning is quite flexible and supports the setting where the expert trajectories come from human teleoperating the relevant system directly, as well as various settings where the trajectories are sampled from other controllers, which themselves may have been trained or scripted. However, for any of the settings where the expert policy is actually available, rather than just samples from the expert, is reasonable to suspect that sampling random rollouts from the expert policy followed by performing behavioral cloning is not the most efficient approach for transferring behavior from the expert to the student. Once a trajectory has been sampled via an expert rollout, there is actually additional information available that can be ascertained in the neighborhood of the trajectory, without having to perform an additional rollout, via the local feedback properties of the expert.
We refer to this setting, where we want to transfer from an expert policy to a student policy, while assuming the expert policy can be queried, as policy cloning. Naturally, there is still often incentive to reduce the total number of rollouts, which may require actually collecting data in an unsafe or costly fashion, especially for real-world control problems. As such, there is motivation to characterize any
Work was done while at DeepMind
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

efficiency that can be gained in learning from small numbers of rollouts without as much concern for how many offline queries are required of the expert policy.
If one has primarily encountered behavioral cloning in the context of learning from human demonstra- tions, policy cloning, with available expert policy may seem contrived. However, policy cloning naturally arises in many settings, For example, an expert policy may be too large to execute due to memory considerations as in Parisotto and Salakhutdinov [202I], where authors propose to distill a large transformer network into small MLPs to be able to execute the policy on data collection workers. In a different setting, we could aim to distill an expert which has access to additional privileged information into student without access to it, for example by distilling a full state policy into the one which has vision observations. In applications such as robotics control tasks where observation are provided via vision, it is typically much easier to train a policy if some additional features are provided to the observations. However, these features might not be available when the policy should be used on the real robot. And in the DAGGER setting [Ross et al.) 2011]. a student policy collects data and is trained by regressing on the expert policy where state distribution comes from the student. In yet another setting, the expert may be suboptimal and the student needs to learn from expert while also being able to exceed the expert performance, perhaps by continuing to learn from a task via RL. This problem has been described as kickstarting in one incarnation (Schmitt et al. 2018 but also can arise when learning from behavioral priors Tirumala et al. 2020), [Galashov et al. 2019]. as also happens, for example, in Distral [Teh et al. 2017]. In all of the aforementioned situations, we implicitly rely on the assumption that it is cheap to query the expert policy. This, however, is not typically restriction and the main motivation is to minimize the amount of data student policy needs to collect.
To improve data-efficiency in supervised settings generally, including in behavioral cloning settings, it is reasonable to consider data augmentation. Data augmentation refers to applying perturbations to a finite training dataset to effectively amplify its diversity, usually in the hopes of producing a model that is invariant to the class of perturbations performed. For example, in the well studied problem of object classification from single images, it is known that applying many kinds of perturbation should not affect the object label, so a model can be trained with many input perturbations all yielding the same output (Shorten and Khoshgoftaar 2019). This setting is fairly representative, with data augmentation usually intended to make the model "robust" to nuisance perturbations of the input. This class of maage-perturbation has also been recently demonstrated to be effective in the context of control problems in the offline RL setting (Yarats et al. 2021 Laskin et al.) 2020].
Critically, for control problems it is not the case that the action should be invariant to the input state. Or rather, while it does make sense for a control policy to be invariant to certain classes of sensor noise, an important class of robustness is that the policy is appropriately feedback-responsive This is to say that for small perturbations of the state of the control system, the optimal action is different in precisely the way that the expert implicitly knows. This has been recognized and exploited in previous research that has distilled feedback-contie
plans into controllers [Mordatch and Todorov Mordatch et al. 2015 Merel et al., 2019]. A similar intuition also underlies schemes which inject noise into the expert during rollouts to sample more comprehensively the space of how the expert recovers from perturbations [Laskey et al. 2017 Merel et al., 2019] Moreover, the idea of recovery from perturbations was explored in scenarios where strong domain knowledge is avaialable. In [Bojarski et al.] 2016) for imitation learning for self-driging cars, the authros propose shift the camera images and associated shifted labels with these, which requires to know the model of the shift. In [George al. 2018). the authors used the label augmentation approach by assigning fixed label to syntetically created images. In [Buhet et al.) 2019]. the authors proposed to perturb the inputs and associate the labels which corresponded to the recovery of the perturbed inputs. Even if these approaches are related to our work, they require a strong domain knowledge and in some cases the knowledge of the perturbation model. In our approach presented below, we do not require any domain knowledge, but only the ability to query the expert policy.
In this work, we leverage this insight to develop highly efficient policy cloning approach that makes use of both classes of data augmentation. For a high-DoF control problem that operates only from state (humanoid run and insert peg tasks from DeepMind control suite (Tunyasuvunakool al. 2020),, we demonstrate the feasibility of policy cloning that employs state-based data augmentation with expert querying to transfer the feedback-sensitee small number of rollouts. Then on more difficult high- DoF control problem that involves both state- derived and egocentric image observations (humanoid running through corridors task from DeepMind
behavior of the expert in region around
2

---

control (Tunyasuvunakool et al. 2020]), ), we combine the state- based expert- expert-aware data augmentation with separate image augmentation intended induce invariance image perturbations. Essentially our expert-aware data augmentation involves applying random perturbations to the state-derived observations, and training the student to match the expert-queried optimal action at each perturbed state, thereby gaining considerable knowledge from the expert without performing excessive rollouts simply to cover the state space around existing trajectories. Our approach compares favorably to sensible baselines, including the naive approach of attempting to perform behavioral cloning with state perturbations, which seeks to induce invariance (as proposed Laskin et al. 2020) rather than feedback- seestick-sensitivitty to state-derived observations. We demonstrate that approach significantly improves data efficiency on all the settings mentioned above, ie., behavioral cloning, expert compression, cloning privileged experts, DAgger and kickstarting.
2 Problem description
2.1 Expert-driven learning
We start by introducing a notion of expert-driven learning that will be used throughout the paper. At first, we present general form of the expert-driven objective and then introduce few concrete examples. We consider standard Reinforcement Learning (RL) problem. We present the domain as an MDP with continuous states for simplicity, however the problem definition is similar for POMDP with observations derived from the state. Formally, we describe the MDP in terms of continuous state space S E R", > O, an action space A, transition dynamics p(s'|s, a) S A P(S). and a reward function r S x A + R. Let II be a set of parametric policies, i.c. of mappings Tee : + p(AA) from the state space S to the probability distributions over actions A, where for some m > 0. For simplicity of the notation, we omit the parameter in front of the policy, i.c. = Tg and optimizing over the set of policies would be equivalent the optimizing over set of parameters. A reinforcement learning problem consists in finding such a policy π that it maximizes the expected discounted future reward:
R"
Σ (rrr(al
J(T) = Ep(r)
where p(T) = p(so) II, p(a; (s (s))p((++1ss ar) is trajectory distribution. We assume the existence of expert policy (g(a)s) This policy could be used simplify the learning of new policy the same problem. Formally, we construct a new learning objective which aims to maximize the expected reward of the problem at hand well clone the expert policy:
(T, π (J(()) - AD(T,TE).
(2)
where D is function which measures the closeness of to ag and 2 0, A 2 0 are parameters describing importance of both objectives. In most of the applications, a E {0, 1} and À 2 represents a relative importance of cloning expert policy with respect to the RL objective.
2.2 Behavioral cloning (BC)
Behavioral cloning (BC) corresponds to optimizing the objective (2) with a 0, where is:
DBO(T, TE) (logs)
(3)
Here, Bg = [ss, a.), = I,..., N}. N > a fixed dataset containing expert data. Minimizing the objective 3 is equivalent to maximizing the likelihood of the expert data under the policy æ. The action a in eqn. ( can be replaced by (s) for deterministic policies or by the mean (g(s) for Gaussian policies Th (+(s) ((px(s).
2.3 Expert compression
In case of expert compression, we are interested optimizing a similar objective as eqn. where the student has smaller number of parameters compared to Ng.
3 but
3

---

2.4 Learning from privileged experts
In case of learning from privileged experts, we optimize similar objective to eqn. (3) where student receives different observations from the expert gg. We assume that the expert has access to the privileged information, but the student does not. In particular, we consider the case where dataset with expert data contains observations (rather than full state), i.e. BE {o,,,;;, = =1, 1,...,, V....,N,, a;),
N > 0, but the expert has access to the full state 8;. More precisely, we consider an expert that observes the state (se, Spriv), where Se is set of observations common to both the student and expert, whereas Spriv is privileged information (containing some task-specific information), Then, the observations for the student are obtained
(5es Ouis) where Ovis is the vision-based input.
2.5 DAGGER
Performance of Behavioral Cloning (BC) can be limited due to the fixed dataset, since the resulting policy may fail to generalize to states outside the training distribution. A different approach, known in the literature as DAGGER [Ross et al. 2011] was proposed to overcome this limitation. In this setting, the expert is queried in states visited by the student, thus reducing distribution shift. In our notation, this corresponds to a = 0, À = in eqn. (2) and D is defined as:
DAAGER (T, Æg) - (r) (a, [s)]],
(4)
where (T)) 8 E [0, 1] is trajectory distribution where actions are sampled according to the mixture policy between student and an expert:
pa(a|s) (a + (1- B)T
(5)
The action af in eqn. (4) is obtained from the expert policy as at Th (-1s:), as n TE(s) for determin-
[s] = (H) (E(s)) (see Section 2.2 The policy # (als) corresponds to frozen version of student policy
istic or as (s) for Gaussian policies T
that the gradient V DDAGEE Fg)
ignores the acting distribution ps)a)s). Note that even though, in eqn. 44) we collect data from the environment, the setting nevertheless corresponds pure imitation learning.
2.6 Kickstarting
In eqn. (2). we combine both maximization of expected task reward and minimization of distance to the expert. In literature, is known as Kickstarting (Schmitt et al. 2018). In this case, the objective from eqn. 2 becomes:
[ TR) = (TT) - ABp(r) [- [-ERg(o)a) log log [a]s)]
(6)
where p(r) is a trajectory distribution, where actions are sampled according to the student policy (-Is). corresponds having = and À 0 and D be state-conditional (across trajectory) cross-entropy from expert to a student. Usually, in the Kickstarting setting, the expert is sub-optimal and the goal train a policy that eventually outperforms the expert. Thus, is customary to reduce A over the course of training. Yet, for simplicity, in our experiments we keep this coefficient fixed.
3 Augmented policy cloning
The previous section has demonstrated how the goal of cloning expert behavior can arise in different scenarios. In this section propose new and simple method which can significantly improve the data efficiency in the settings described in Section We explain the basic idea for BC, but its generalization to other expert-driven learning approaches described in Section 2 is straightforward. In Section 6 we show results for these problems.
When optimizing the objective 3 for every state E Dg from the expert trajectories dataset, we consider a small Gaussian state perturbation:
8s NN(, )
which produces a new virtual state:
s' = + 8s
(8)

---

Naive data augmentation
augment
use original
states
actions
APC
resample appropriate
augment
states
actions
Figure 1: Schematic of APC (ours) method compared with naive data augmentation approach. The original states (magenta circles) and actions (magenta arrows) pairs are then augmented by new virtual states (green circles). In Naive data augmentation, the same actions (magenta arrows) are used for all new virtual states. In APC, however, for each new virtual state, we resample new action (green arrow) from the expert policy.
Then, for this state we query the expert and obtain new action
a' TE(- + 6s)
(9)
We then augment the dataset BE with these new pairs of virtual states and actions. More explicitly the idea can be expressed in terms of the following objective:
D(T, APP = B( lleeBBBloogoee [(alss + (ff= log (a'ls + 8s)] (10)
We call this approach Augmented Policy Cloning (APC) as it queries the expert policy to augment the training data. This approach is different from naive aata-aaugmnnttiion technique, where new state would be generated, but associated with the original action (and not a new one). It therefore allows to build policies which feedback-responsive with respect the expert. We illustrate it in Figure T and we formulate APC algorithm for BC in Algorithm
Algorithm 1 Augmented Policy Cloning (APC)
TTy parametric student policy, 6 initial parameters, TE expert policy, a learning rate state perturbation noise, M # augmented samples K # gradient updates, L, batch size Dataset 33 {(8;, az), 1,...,Ny N). N of expert state-action pairs for k=1 do
Sample a batch of pairs {aa, 1= ~ BE
For each state 5; sample M perturbations òs; (0000,)),j M Construct M virtual states s/,,, = 8; + 88,, 1,......,............ I,..,,
Resample new actions from expert at, ~ {
For Gaussian experts, the action a; = ME(s;) and the new actions are ar = (E(())))
Compute the empirical negative log-likelihood:
L= log ne, (agls:) + n E log To, (dt,)>%b,))
Update the parameters 0x+1 = 02 voL end for
4 Experimental details
In this section we provide details common to all experiments. We provide additional details for each of results the beginning of Section and Section
4.1 Domains
To study how our method performs on complex control domains, we consider three complex, high- DoF continuous control tasks: Humanoid Run, Humanoid Walls and Insert Peg. All these domains
5

---

are implemented using the MuJoCo physics engine Todorov et al. 2012] and are available in the dm_control repository (Tunyasuvunakool et al., 2020]. These problems are rather challenging. requiring stabilization of complex body (for humanoid tasks), vision to guide the movement (Walls task), and solving a complex control problem with a weak reward signal (Insert Peg). These environments are related to the domains that have been proposed for use in offline RI. benchmarks [Gulcehre et al.) 2020]; however, the experiments we perform in this work require availability of the expert policy, so we do not use offline data, but instead train new experts and perform experiments in the very low data regime. We compare all methods on Humanoid Run and Humanoid Walls tasks and report report subset of results on Insert Peg, due to complex nature of the experiments.
4.2 Baselines
As baselines we consider simple BC as described in eqn. (3) as well as a simple modification of BC. where, similarly to APC, we apply state perturbations to expert trajectories in eqn. 7 and eqn. (8) but we do not produce a new action from the expert (i.e.. we augment the states but keep the same action). We call this approach Naive Augmented Behavioral Cloning (Naive ABC). Essentially, this method trains a student policy to produce the same action in response to small state perturbations.. This approach is motivated by analogy to how one might build robustness in classifier. However, this is naive when applied to continuous control problems where even small changes in input should lead to a change in action. Moreover, for Humanoid Walls task, considered additional random crops augmentations applied to visual input of the student (not the expert) which is similar in spirit to Laskin et al. [2020]. Note that in this case, would also robustify the student to these vision augmentations as it will not produce a new action even in the APC (since the expert was not trained with data augmentations). When vision augmentations are used together with either APC or naive ABC, we add "with image" to the method name. When only image augmentations are used (without any state- based augmentations), we call it "image only". The purpose of combining vision and state augmentations is to study the interplay between APC and more traditional data augmentation methods. We only report results with vision augmentations for Aggeer and kickstarting in the main paper and we provide additional offline policy cloning results in Appendix 8.
5 Core results: offline policy cloning
Training and evaluation protocols. We train expert policies till convergence using MPO algo-
rithm Abdolmaleki et al.] [2018) for Humanoid tasks and VMPO algorithm Song et al. [0019] for Insert Peg task, as we found MPO was unable to learn on this task. The policies are represented
by the Gaussian distribution TE (-1s) = (uu (s), (s)). We create datasets as in eqn. 3 using
pre-trained experts with a different number of expert trajectories. To asses the sensitivity of different methods to the expert noise, when constructing dataset, the expert action is drawn according to Gaussian distribution with a fixed variance, i.e. ~ (spg(s), og). where dg is the fixed amount of expert noise. In the subsequent BC experiments, we use F = 0.2. Moreover, in order to analyze the noise robustness of the student policy is trained via BC, 7 (-1ss) N (N(((s), (s))), we evaluate it by executing the action drawn from a Gaussian with a fixed variance, i.e. a ~ N (j((s), o), where a is the fixed amount of student noise. In all the experiments we use = 0.2. We apply early stopping and select hyperparameters based on the evaluation performance on a validation set. We always report performance based on 150 random environment instantiations. For more details, see Appendix 4.
5.1 Applying Augmented Policy Cloning
We evaluate the performance of APC when fitting the fixed dataset of expert trajectories. For the APC method, we rely on Algorithm [ [ We use baselines described in Section 4.2 In Figure 2 we show the performance of different methods on different tasks as a function of number of trajectories available in the dataset. We see that APC needs significantly fewer expert trajectories to achieve a high level of performance. Moreover, we see that Naive ABC performs very similarly to BC. The results suggest that when cloning an expert using small fixed dataset APC can provide significant advantages.
In Appendix 6, we report additional results for a scenario, where instead of full long trajectories for each task, we consider only short trajectories (i.e., the early portion of episodes). The motivation for this experiment is to see whether we can further improve the data efficiency of the methods. Incidentally, this supplemental comparison shows that for Insert Peg all methods performed better

---

Run 1250
Walls
Insert Peg
Episode reward
6:0
730
Method
APC
320
BC
200
250
Naive ABC
Number of trajectories
Figure 2: Offline expert cloning results. The X-axis represents the number of trajectories, the Y-axis corresponds to the episodic reward across 150 independent evaluations. The red color corresponds to APC, the blue to Naive ABC and the green to BC. The pink line depicts the average performance of the teacher policy. Each subplot represents a different task.
with short trajectories, because initial snippets episodes actually include the full solution to the task (i.e., the expert policy rapidly inserts the peg and the episode doesn't It immediately terminate). We report only long trajectories in Figure 2 since APC performs relatively well in both cases and full length trajectories correspond to the most straightforward setting.
5.2 Expert compression
APC, Run
APC, Walls
APC, Insert Peg
1253
1003
730
400
Episode reward
Student
Torso
1000
500
1000
Architecture
Walls
[256), r=65536
600
2000
[256, 64), r-1024
[256, 256), r-256
[256, 256, 256], r-1
200
1000
50
100
201
5000
1000
Number of trajectories
Figure Teacher compression results. We plot performances different methods (rows) on different tasks (columns) as function of number of trajectories available in the expert dataset. The legend corresponds to different student architectures used as well as the relative difference in parameter count in baseline torso architecture compared to the student torso architecture. We see that the performance degrades much more drastically for BC compared APC. The pink dashed line corresponds to the performance of the expert policy. We report more complete results in Appendix 8.1
To study APC in practically motivated setting we consider expert compression as discussed in Section 2.3 where student policy has fewer parameters than the expert. This setting occurs, for instance, when the system is subject to computational contstraints (time, memory, etc.), as in Parisotto and Salakhutdinov 2021]. To study APC's data efficiency in this setting, we consider different sizes of the student network torso, where [256, 256, 256] corresponds to the original network size (see Appendix 4.1 for more details). We define the relative difference in parameter count in baseline torso with respect the student architecture torso as the product of baseline torso dimensions, i.e. 2563 divided by the student torso dimensions (i.e., for a student torso [256, 256), is equal 256?).
The results are given F observe that the performance of all methods degrades when the student network is smaller than the original one, but the degradation is much less severe for APC, while maintaining high level of data efficiency. See Appendix 8.1 for more complete results (with naive ABC) as well as additional ablation over student network torso sizes.
7

---

5.3
Learning from privileged experts.
Insert Peg
Insert Peg From vision
1000
Episode reward
750
Method
APC
500
BC
Naive ABC
500
10000
100
600
10000
Number of trajectories
Figure 4: Learning from privileged experts results. On the left plot we present the performance of different methods on Insert Peg task where additional privileged information (target position) available. On the right, we present the results where instead of privileged information (target position), we use visual input. We observe that learning from vision is less data efficient and more complicated, but APC manages to obtain comparable performance to the scenario with privileged information, whereas BC and Naive ABC fail to learn. The dashed pink line corresponds to the performance of the expert policy.
Next, we consider scenario where the expert has access to privileged information that is not available to the student, as discussed Section 2.4 study the impact APC in this scenario, we train the expert on Insert Peg task where the full state contains common information (proprioception, sword position and orientation) and privileged information the target position. The student is given access to the common observations well information about the target position. The latter setup is similar in spirit to [Laskin et al., 2020) For more details, see Appendix 4.2. From Figure 4 observe that APC achieves similar performance in both settings, provided sufficient amount of trajectories are available, whereas BC and Naive ABC fail to transfer the expert's 's behavior to the student.
third person (camera) view of the scene providing
6 Additional Results: Augmented Policy Cloning as subroutine
6.1 DAGGER with data augmentation
DAgger variant
OOiginal DAgger
Episode reward
AAC
Naive ABC
400
APC with
Naive ABC (image only)
Naive ABC with image
Expert 0.0
Aeer
0.3
Environment steps
Figure 5: DAGGER results. On the X-axis we report the number of environment steps. On the Y-axis we report averaged across 3 seeds episodic reward achieved by the student. The legend corresponds the DAGGER variant with additional data augmentation applied. We report confidence intervals in the shaded areas. For Run task, the confidence intervals are very small and are not visible. In solid line we report the performance without using expert policy during the acting. In dashed line, we report the performance of the policy which mixes 30% with the expert. The dashed pink line corresponds to the performance of the expert policy trained via MPO.
As described in Section 2.5 DAGGER Ross et al. 2011 is more sophisticated approach where data is collected from the real environment by executing a policy from eqn. (5). which is a mixture between student and expert. In this section the data efficiency of the DAGGER algorithm. For each task, we train expert policies convergence using the MPO algorithm Abdolmaleki et al. (2088) We consider similar baselines for both tasks
study how data augmentation approaches affect
8

---

as in the previous section. For an expert policy that has been pre-trained via MPO [Abdolmaleki al. 2018], we perform online rollouts for two values of the expert-student mixing coefficiett 3= 0 and B = 0.3 (see eqn. 5). Since both student and expert are Gaussian distributions, instead of using a log x in eqn. (4) we could use state-conditional cross entropy from an expert to a student, (-[=((1))) (-)(s)() [-|s)] Empirically, we found that it worked better than using log π (see Appendix 8.6). We run experiments in a data-restricted setup. For more details, see Appendix 5.1.
Results are shown in Figure 5 We report the performance of both the original DAGGER as well as DAGGER with data augmentation. We see that APC and its vision variant outperform BC and Naive ABC similarly to the behavior cloning experiments. While we observe that image augmentation can help, we see that the primary advantage comes from the state-based augmentation for APC. For the Run task, we observe that all DAGGER methods achieve slightly lower performance than an expert policy. We speculate that this is due to insufficient coverage of the state space during training.
6.2 Kickstarting with data augmentation.
Run
Walls
Kickstarting variant
Episode reward
-Orggial kickstarting
600
APC
400
Naive ABC
APC with image
Naive ABC (image only)
Naive ABC with image
MPO
1.000
2.0ed
dedt
ded
0.5e0
0.8
Environment steps
Figure 6: Kickstarting results. On the X-axis show the number of environment steps and on the Y-axis we report averaged across 3 seeds episodic reward achieved by the student. The legend corresponds the kickstarting variant where additional data augmentation is applied. We report confidence intervals the shaded areas. For Run task, these intervals are small and are not visible. Dashed pink line shows the expert performance.
A similar in spirit approach is kickstarting Schmitt et al. 20188. where we solve an RL task as well as cloning the expert policy. Similarly to previous section, we apply APC in kickstarting on the cross entropy term in eqn. (6). For each task, we train expert policies to convergence using the MPO algorithm Abdolmaleki et al.] [2018). Since in the kickstarting we are interested in outperforming sub-optimal expert, for each task, we select experts such that they achieve around 50 % of the optimal performance. On top of kicktarting, we report the performance of MPO Abdolmaleki et al. [2018) learning from scratch on the task of interest. We run experiment in a distributed, high data regime. All the details are given in Appendix 5.2.
The results are given in Figure 6 We report the performance of both the original kickstarting as well as kickstarting with additional data augmentation. We observe that APC performs better than Naive ABC on Humanoid Run task and similarly on Humanoid Walls task. Both approaches perform better than BC and learning from scratch. We hypothesise that the reason of not seeing a consistent advantage could be due several factors. Firstly, as we are in high-data and distributed regime, since there is no limit on relative acting learning ratio, and acting policies are not restricted collect trajectories, it is unclear whether data-augmeetatton should help. We tried to explore the rate-limiting regime, but we experienced instability of kickstarting experiments. Secondly, we use reward signal which makes the impact of expert cloning less important. Thirdly, on top of learning the policy, we also need to learn an state-action value ((,, a) function. Unfortunately, we cannot use APC-style data augmentation for learning Q function, therefore it might be the bottleneck. Finally, unlike in kickstarting (Schmitt et al. 22188), we do not use an annealing schedule of to make the experiments simpler, but we still observe that fixed coefficient helps to kickstart an experiment and outperform an expert policy. On top of that, we see that image- based augmentation have less of impact this setting and generally leads to poor performance.

---

7 Discussion
Many expert-driven learning approaches actually have access expert that can be queried; however, this opportunity is rarely exploited fully. In this work we demonstrated a general scheme for more efficient transfer of expert behavior by augmenting expert trajectory data with virtual, perturbed states as well as the expert actions in these virtual states. This data augmentation technique is widely applicable and demonstrated that it improves data efficiency when used in place of behavioral cloning in various settings including offline cloning, expert compression, transfer from privileged experts, or when behavioral cloning is used subroutine within online algorithms such as DAGGER or kickstarting. One could argue that other way to clone the expert would be to query it on densely covered state space. Such an approach, however, would require exponentially more number of queries as the dimensionality of the state space grows. Our approach, however, would scale similarly to baseline methods (such as DAGGER or Behavioral Cloning) as it would require only M additional queries for each state during learning (see Algorithm[] D. In case when the queries to the expert are expensive, our approach may become problematic.
Critically, data efficiency is generally very important in realistic applications, where new data acquisition cost could be high. In particular, settings involving deployment of policies in the real world, such as robotics applications, may benefit from an ability to efficiently transfer expert policy behavior from one neural network to another (for compression or execution speed reasons). While overall, we consider the present work to be fairly basic research with limited ethical impact, insofar as our approach decreases the amount of data which needs to be collected through processes which could potentially be unsafe or costly, there is potential positive social value.
Our approach neither intended for nor suitable for all control settings. Fundamentally, our approach relies upon the ability to query expert policy for the perturbed states. This arises frequently enough to be worth our investigation, but is limiting assumption. Our approach was also developed with continuous control problems, essentially with continuous observation spaces as well as continuous action spaces mind. Related approaches may be worth pursuing in discrete control problems, but that has not been focus of the present work.
In future work, it may be interesting develop an extension of our approach that is compatible with offline RI. While offline RL operates with different assumptions, i.e., we generally do not have access to the policy which produced the data, may wish to consider adjacent scenarios where the queryable expert can be combined with offlline learning RL or where augmentations facilitate learning the Q-function.
References
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation, 2018.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars, 2016.
Thibault Buhet, Emilie Wirbel, and Xavier Perrotton. Conditional vehicle trajectories prediction in carla urban environment, 2019.
Alexandre Galashov, Siddhant M. Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojciech M. Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in kl-regularized rl, 2019.
Laurent George, Thibault Buhet, Emilie Wirbel, Gaetan Le-Gall and Xavier Perrotton. Imitation learning for end to end vehicle longitudinal control with forward camera, 2018.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RI unplugged: A collection of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on robot learning, pages 143-156. PMLR, 2017.
10

---

Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data, 2020.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilisticc motor primitives for humanoid con- trol, 2019.
Donald Michie and Claude Sammut. Behavioural Clones and Cognitive Skill Models, page 387-395. Oxford University Press, Inc., USA, 1996. ISBN 01 019853860X.
Igor Mordatch and Emo Todorov. Combining the benefits of function approximation and trajectory optimization. In Robotics: Science and Systems, volume 4, 2014.
Igor Mordatch, Kendall Lowrey, Galen Andrew, Zoran Popovic, and Emanuel V Todorov. Interactive control of diverse complex characters with neural networks. Advances in Neural Information Processing Systems, 28:3 112-3142000 2015.
Emilio Parisotto and Ruslan Salakhutdinov. Efficient transformers in reinforcement learning using actor-learner distillation, 2021.
Dean Pomerleau. Alvinn: An autonomous land vehicle neural network. Technical report, Carnegie-Mellon, 1989.
Stephane Ross, Geoffrey Gordon, and Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning, 2011.
Simon Schmitt, Jonathan J. Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojeiech M. Czarnecki, Joel Z. Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, and S. M. Ali Eslami. Kickstarting deep reinforcement learning. 2018.
Connor Shorten and Taghi Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6, 07 2019. doi: 10. (0.11866/40377-019-009744
H. Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W. Rae, Seb Noury. Arun Ahuja. Siqi Liu, Dhruva Tirumala, Nicolas Heess, Dan Belov, Martin Ried- miller, and Matthew M. Botvinick. V-mpo: On-policy maximum posteriori policy optimization for discrete and continuous control, 2019.
Yee Whye Teh, Victor Bapst, Wojeiech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. arXiv preprint arXiv: 170..0477......0 2017.
Dhruva Tinumala, Alexandre Galashov, Hyeonwoo Noh, Leonard Hasenclever, Razvan Pascanu, Jonathan Schwarz, Guillaume Desjardins, Wojciech Marian Czarnecki, Arun Ahuja, Yee Whye Teh, et Behavior priors for efficient reinforcement learning. arXiv preprint arXiv:2010. 14274, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5000 IEEE, 2012.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Sigi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. 9th International Conference on Learning Representations, ICLR, volume 2021, 2021.

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [No], we won't include the code and the datasets in
Did you include the license the code and datasets? [No] proprietary.
The code and the data
Did you include the license to the code and datasets? [N/]]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the pa- per's contributions and scope? [Yes], we believe that the claims in the abstract and introduction are accurately reflected by the paper's contributions and scope.
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] we included details in the supplementary material, which could be used reproduce the experimental results
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]. we provided the details in the supplementary material
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes]. for every plot we reported the mean and standard deviations averaged across number of independent random evaluations.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [TODO]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]. we used DeepMind control suite tasks and we cited the appropriate publications.
(b) Did you mention the license of the assets? [No] we did not explicitly mention the licenese as it is clear from the corresponding references.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No], since the assets and references in the public access, consent was required.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No], the assets which we used did not include any personally identifiable information nor offensive content.
5. If you used crowdsourcing conducted research with human subjects...
12

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "uH-ABUcfOfW", "reviewer": "Reviewer_e72c", "summary": "This paper proposes a simple yet effective way APC, which queries the parametric experts to augment the demonstrations for imitation learning. APC is shown to outperform naive behavior cloning in both expert compression and transferring from privileged expert settings.", "questions": "I think the key problem here is that the authors don't explore the methods of querying experts enough. From my point of view, the advantage of APC is totally brought by a new setting where the expert could be queried, which is not an innovation of the methods. I suggest the authors compare APC with more baselines that can query the experts. \n\nAlthough this method is reasonable, I'm afraid the current technical contribution is far from enough. Adding the Gaussian noise is the simplest way to make a data augmentation, and previous works (e.g., RAD, https://arxiv.org/pdf/2004.14990.pdf) also have systematically studied the data augmentations for general reinforcement learning problems. It could be better if the authors come up with novel methods for data augmentations in your settings.", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\nThis paper focuses on a novel problem where a parametric expert can be queried. The paper writing is clear and easy to follow. The empirical results do demonstrate the advantage of APC over BC and naive ABC. \n\nCons:\nHowever, I don't feel this paper proposes any novel idea. One serious issue is that when compared with baselines, all these baselines don't query the expert as APC does. This means that the baselines have fewer demonstrations to imitate from. It is clear that APC will beat naive ABC since APC has more high-quality/ground-truth demonstrations. Therefore, I think this doesn't make enough technical contributions to the problem it studies.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qmsubTaF3mK", "reviewer": "Reviewer_pnsx", "summary": "This paper proposes an approach for imitation learning from parametric experts, by augmenting the data and then querying the experts for action labels on the new states. The authors emphasize two practical settings where learning from parametric experts is useful - expert compression and learning from privileged experts. Includes evaluation on continuous control mujoco tasks (humanoid and peg insertion).\n", "questions": "-> How does the proposed approach compare to regular dagger.  Specifically, how much worse is the policy performance (if at all), if the policy isn’t rolled out to collect new states, but instead the recorded states were just perturbed to get new states, and the expert was then queried for new actions ? What are the sample efficiency gains ? \n \n-> What is the performance of the expert MPO policies on the domains reported? \n \n-> For expert compression results, please plot ratio of performance of a low capacity model to that of the high capacity model for each method for easier comparison of compression ability. \n", "limitations": "The limitation are adequately discussed, primarily being requiring a parametric expert. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality - \n\n-> The proposed idea is very simple and intuitive. Data augmentation is commonly used for machine learning approaches and recently also in RL for improved performance [1,2]. Given parametric experts, querying them for action labels on the new augmented states seems like it would give rise to better performance, since more information is being extracted from the expert. This is similar to how in dagger when new states are encountered, the expert is queried for labels, except here the new states don’t require a policy rollout but rather a direct perturbation of the recorded states. While the paper does include experiments on dagger with data augmentation, I’d be curious to see how regular dagger compares to the proposed approach, in terms of policy performance and overall sample efficiency.\n \n-> The paper is missing a related works section that goes into depth about prior work in the area, but the structure of the paper does include some discussion on prominent imitation learning methods (dagger, kickstarting, privileged experts) but doesn’t mention specific works adequately. \n \n[1] : Reinforcement Learning with Augmented Data (Lashkin et al.)\n[2] : Regularizing Deep Reinforcement Learning from Pixels (Kostrikov et al.)\n \nSignificance - \n\n-> The proposed method can be of interest to a large section of the community interested in imitation learning, and is quite simple to adopt and implement. Sample efficiency is critical especially for robot learning, and the proposed approach shows the ability to learn control policies in the very low data regime. The only caveat is that the method needs parametric experts, but the authors are very clear about this and point out practical areas where the proposed approach is relevant - i.e expert compression and learning from privileged experts. The latter is specifically relevant for robotics, since sim to real pipelines [3,4] can adopt this idea and become more efficient. \n \n[3] : Rapid Motor Adaptation (Kumar et al.)\n[4]: Solving Rubik’s Cube with a Robot Hand (OpenAI)\n \nQuality -\n\n-> The experimental evaluation seems quite thorough (performance is reported on 150 random environment instantiations). The authors show that the proposed approach fits the expert data much better than the baselines given an offline dataset, and the results on learning from privileged experts do seem to show that the proposed approach is effective\n \n-> However, on the expert compression task, the only environment where there’s a clear advantage for the proposed method is humanoid run. On peg insertion, the baseline’s high capacity model has quite poor performance so it’s difficult to judge compression ability, and on humanoid walls the relative compression of the proposed approach and the baseline appear similar. I would recommend defining a new metric for compression (i.e ratio of performance of a low capacity model to that of the high capacity model) so that performance can be more accurately judged. \n \nClarity - \n\nThe paper is well written and motivated, and easy to follow. It includes clear algorithm boxes and equations that are sufficient to understand the details of the method. \n \n \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jV11WGGDy1", "reviewer": "Reviewer_kiE1", "summary": "The paper introduces a data augmentation technique for behavioral cloning algorithms. It consists of adding Gaussian noise to the states used for training and storing the perturbed states and the action the expert policy would produce in such states. Using this augmented dataset, the proposed algorithm achieves better performance than the baselines over three complex continuous tasks.", "questions": "While in continuous control environments adding Gaussian noise to a state often produces a perturbed state that could be reached by an agent (i.e. valid states), it is not the case when operating with visual observations. Is there an intuition why data augmentation would be useful in the visual domain?\n\nAssume that, when the expert was trained, it observes in the environment states with mean $\\mu$ and standard deviation $\\sigma$ (this could be estimated from the last few trajectories during the training of the expert). What is the performance of APC when using zero trajectories and only synthetic data coming from a normal distribution with mean $\\mu$ and standard deviation $\\sigma$? \n\n\nFigure 1 and Algorithm 1 appear many pages before the first time they are mentioned in the paper. I suggest putting them when they are mentioned.", "limitations": "The authors discuss the possible limitations of this work. The method queries the expert many times, so if expert queries are expensive, their approach might suffer. I suggest providing specific examples for this setting.\n\n\n\n\n---------------------------------------------------------------------------------------------------------------------------------------------------------------------\nEdit: updating my score after author-reviewer discussion", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and contains many implementation details and ablation studies.\n\nThe idea proposed is very simple and effective. Unfortunately, apart from the experimental results in complex environments, the novelty in this work is limited. It is well known that augmenting the dataset in Behavioral Cloning is useful and there are prior works applying observation perturbation and label corrections via experts queries [1,2,3].\n\nThe main motivation for using the proposed approach is the case when collecting more data in the environment is expensive, but querying the expert many times is not. The authors motivate this by saying that \"an expert policy may be too large to execute due to memory considerations\" and that \"Many expert-driven learning approaches actually have access to an expert that can be queried\", but no specific example is provided to the reader. \n\nThe comparison with baselines BC and ABC does not seem very fair. If the policy has enough capacity and the distribution from which we sample the states covers the state-space enough, one could clone the expert by repeatedly querying it on such random states without even seeing a single trajectory. Therefore it is quite straightforward that the proposed method works better than the baselines.  \n\n[1] M. Bojarski et al., “End to end learning for self-driving cars,” arXiv preprint, 2016\n\n[2] L. George, T. Buhet, E. Wirbel, G. Le-Gall, and X. Perrotton, “Imitation learning for end to end vehicle longitudinal control with forward camera,” in NeurIPS, 2018.\n\n[3] T. Buhet, E. Wirbel, and X. Perrotton, “Conditional Vehicle Trajectories Prediction in CARLA Urban Environment,” in ICCV, 2019\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hexaJy2NLFW", "reviewer": "Reviewer_immT", "summary": "They use augmentatio to improve data efficiency with learning from parametric experts. In naive data augmentation approach, parametric experts act same on augmented states and original states. However, APC get different action on augmented states using parametric experts.\nThey add a samll Gaussian state perturbation to state, and parametric experts predict action on perturbed state. \nThey experiment on continual control tasks, and show improved data efficiency. ", "questions": "1. I want to check the change of APC performance according to experts level. \n2.  Ablation study of sigma for Gaussian perturbation. (If the perturbatsion is small, a new action sampling from experts can be not needed.) \n3. Do you try other perturbation methods on states? ", "limitations": "In my opinion, their contribution is not enough, because APC can only used with parametric experts. Thus, we have to need well trained experts. Therefore, the scope of utilization is bound to be limited.\nFurthermore, baseline is not enough to assure the contribution of APC. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\nThis paper is well written with proper experiments. \nAPC significantly improve data efficiency, while naive ABC have almost similar data efficiency with BC. \nTheir experiement section is well structured for application of their method APC such as Expert compression and Privileged experts. \n\nWeakness \nBaselines are not enough. They have to add other offline RL algorithms to compare with APC. \nThe performance of experts is not revealed in the Figures. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept