Inferring Versatile Behavior from Demonstrations by Matching Geometric Descriptors
Authors: Niklas Freymuth
Nicolas Schreiber
Aleksandar Taranovic
Philipp Becker
Gerhard Neumann
Venue: CoRL 2022 Poster (2022)
Keywords: Imitation Learning
Versatile Skill Learning
Distribution Matching

Abstract: Humans intuitively solve tasks in versatile ways, varying their behavior in terms of trajectory-based planning and for individual steps. Thus, they can easily generalize and adapt to new and changing environments. Current Imitation Learning algorithms often only consider unimodal expert demonstrations and act in a state-action-based setting, making it difficult for them to imitate human behavior in case of versatile demonstrations. Instead, we combine a mixture of movement primitives with a distribution matching objective to learn versatile behaviors that match the expert’s behavior and versatility. To facilitate generalization to novel task configurations, we do not directly match the agent’s and expert’s trajectory distributions but rather work with concise geometric descriptors which generalize well to unseen task configurations. We empirically validate our method on various robot tasks using versatile human demonstrations and compare to imitation learning algorithms in a state-action setting as well as a trajectory-based setting. We find that the geometric descriptors greatly help in generalizing to new task configurations and that combining them with our distribution-matching objective is crucial for representing and reproducing versatile behavior.

Inferring Versatile Behavior from Demonstrations by Matching Geometric Descriptors
Niklas Freymuth 1*
Nicolas Schreiber
Philipp Becker
Aleksandar Taranovic
Gerhard Neumann'
"Autonomous Learning Robots Karlsruhe Institute of Technology Karlsruhe, Germany
Bosch Center for Artificial Intelligence (BCAT)
Renningen, Germany
A Asstrcc:: Humans intuitively solve tasks in versatile ways, varying their behavior in terms of trajectory-bann planning and for individual steps. Thus, they can easily generalize and adapt to new and changing environments. Current Imitation Learning algorithms often only consider unimodal expert demonstrations and act state-action- based setting, making it difficult for them to imitate human behavior in case of versatile demonstrations. Instead, we combine a mixture of movement primitives with distribution matching objective learn versatile behaviors that match the expert's 's behavior and versatility. To facilitate generalization to novel task configurations, distributions but rather work with concise geometric descriptors which generalize well unseen task configurations. We empirically validate method on various robot tasks using versatile human demonstrations and compare to imitation learning algorithms in a state-action setting as well trajectory-badt based setting. We find that the geometric descriptors greatly help in generalizing to new task configurations and that combining them with our distribution-matee matching objective is crucial for representing and reproducing versatile behavior.
do not directly match the agent's 's and expert's trajectory
Keywords: Imitation Learning, Versatile Skill Learning, Distribution Matching
Introduction
Imitation Learning (IL) [1, 2, 3] from human demonstrations is challenging as humans often solve tasks in versatile ways. Even the same person might solve a task differently when confronted with multiple times. Behavior can be versatile in terms of task-level decisions, such planning route to a target, and individual actions, such as randomly pausing during a movement to think about the next steps. Most recent IL approaches [4. 5, 6, 7] model the behavior in state-action space using Gaussian policies, which assumes the behavior is unimodal and cannot capture this planning versatility. Yet, this assumption is violated by most human expert datasets, often causing poor generalization to human demonstrations [8]. Additionally, II. approaches often need immense amounts of data to achieve generalization [9, 8]. We tackle these challenges by feature matching approach to IL that able to generalize to novel contexts from a small amount of expert demonstrations. Such contexts are (typically (ow-dimensional) vectors that describe specific task configuration within a family of related tasks, such as e.g., the coordinates of goal locations to reach. The resulting approach, Versatile Imitation from Geometrically Observed Representations (VIGOR), models distributions that match expert trajectories in terms of concise geometric behavioral descriptors. VIGOR represents distributions using Gaussian Mixture Models (GMMs)) over Probabilistic Motion Primitives (ProMPs) [10], and utilizes descriptors that are designed to abstract away from concrete contexts and thus facilitate generalization novel contexts sample-efficient way. An example of such descriptors would be geometric features in the form of (only) the distance between target
correspondence to niklas www.yrrtteiit.eee...
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

I. Train Conseat
2. Trann Conseat
Generalization Novel Test Consent
Solution
End-Elieatr Time
Robot Joints
Tagget
Target
Goomecrrc Representttion
Goometric Repecsentaionn
Govmetric Regresentttion
Solutio
Distacee to Theaett
Sohatioon
Dietonce Target
Distance to Target
2-
2-
0-
10
Eima
Gimac
Gima
Figure 1: End-effector traces (Top) and distances to target centers (Bottom) for a planar point-reaching task on 3 different contexts. (Left, Middle) Human experts solve the task by reaching the blue target and ending their trajectory in the grey one. (Right) VIGOR matches distributions of behavioral descriptors, such the depicted target distances, of these demonstrations. This produces versatile behavior in unseen contexts, such as new target positions, Depicted are 2 different component means from a trained GMM Policy. For each context, we show the joint configurations for the first trajectory in blue and the second trajectory in orange.
and the robot's end-effector, plus features that denote the smoothness of the robot. While similar behavioral descriptors have found use in Inverse Optimal Control [11, 12] to recover cost functions from demonstrations, common IL approaches instead use state representations that include both relative and absolute information [5, 13]. The exact form of this set of behavioral descriptors is task-dependent and allows the user include domain knowledge, Figure shows how VIGOR works on an exemplary reaching task. For training the GMMs, we rely on recent approaches [14, 15, 16] for distribution matching and Variational Inference. For inference, we train individual GMMs for each training context and an arbitrary number of test contexts.
We evaluate VIGOR human experts through teleoperation. We compare our method to standard Behavioral Cloning (BC) [17], BC with a GMM policy [13, 18), Generative Adversarial Imitation Learning (GAIL) [5] and Inverse Reinforcement Learning (IRL) techniques [7]. We find that these methods either fail to learn useful behavior because they average over multiple solutions, or that they are unable to capture the full versatility of the demonstrations. Contrary to this, VIGOR accurately models highly versatile behavior from already a small number of trajectories demonstrated by human experts. We perform extensive ablation studies to showcase the importance of different parameters and design choices. Datasets and code can be found at https: / //wwww github. com com/Nikkaat MiklasFreymmn wwwareymtt//Vooe/ /VIGO.
suite of versatile robotic tasks, where demonstrations collected from
To summarize, our list contributions is as follows:
We infer multi-modal distributions of desired trajectories from highly versatile human expert trajectories by matching distributions over behavioral descriptors between learner and expert.
Our approach is, to the best of our knowledge, the first Adversarial Imitation Learning method to utilize concise behavioral descriptors to facilitate generalization to novel contexts from already a small number demonstrations.
We conduct experiments with humun demonstrations in simulation and on a real robot and find that VIGOR accurately models highly versatile human behavior, outperforming various Imitation Learning baselines.

---

2 Related Work
Imitation Learning for Skills. A common way to represent skills over trajectories is via Movement Primitives (MPs) [19, 10J. While learning individual MPs from human demonstrations is a simple regression problem, modeling versatile behavior requires a more sophisticated model that can represent such multi-modality, e.g., a mixture model. Using Expectation Maximization (EM) [20], both [21] and [22] fit a single GMM over MP parameters of multiple demonstrations for different contexts and use them to generalize to novel contexts by conditioning. Yet, as they only fit single GMM for all contexts, they do not explicitly focus on representing versatility. Recent work [23] instead learns mixture over contextualized MPs using Gaussian Mixture Regression. This has been extended to non-linear relations between MP parameters and context [18] by using Mixture Density Networks (MDNs) [24]. While the above approaches work for tasks with small number of modes, optimizing MDNs can be challenging in the case of many modes in the demonstrations, Osa et al. [25] use planning algorithms reproduce human demonstrations in a trajectory-basse setting. They tune the parameters of a cost function such that the planned trajectories match the demonstrations and use the extracted cost function guide the trajectory optimization process. Yet, this approach is limited to uni- uni-modal demonstrations and requires engineered cost functions.
Imitation Learning by Distribution Matching, Classical IL approaches [26, 27] employed feature expectation matching to learn a policy from behavioral descriptors of expert demonstrations, Similarly, recent body of work [11, 12] in Inverse Optimal Control utilizes geometric behavioral descriptors that are similar to ours to learn cost functions for manipulation tasks from a few demonstrations. Whereas these approaches linearly match the moments/expectations of their features, our method instead matches a versatile distribution over non-linear features using the reverse Kullback-Leibh Divergence (KL) [28]. Directly matching the distribution rather than its moments means that our method can represent complex and multi-modal distributions. More recently, a new class of distribution matching approaches has gained popularity with the advent of Generative Adversarial Nets [29]. Starting from GAIL [5]. a whole class of distribution matching based IL approaches was developed [4, 30, 31, 32. 33, 34J. All of these approaches commonly work in a step-based setting and minimize some divergence between the state-action occupancy or state marginals. They generally do not consider versatile behavior, and additionally need to interact with the environment during training in order to generate states to match. We instead work in a trajectory- based setting which allows us to abstract away the environment's dynamics and learn entirely offline, i.e., without any environment interactions. Common methods in this setting usually employ maximum likelihood approach, which, as previously discussed, performs poorly on versatile data. As a solution to this problem, Becker et al. [16] propose using the Information-Propye Projection [35] instead, which is able to focus on individual modes of data rather than averaging over all modes of data. This approach has been extended to IRL [36]. Here, we build on these methods, generalizing them to sequential data and geometric behavioral descriptors.
Comparison-Basse Based Approaches. Brown et al. [6] perform IRL by utilizing provided rankings of demonstrations to train a discriminator on a comparison-ber based loss. As the discriminator learns to favor samples with a high rank, it can be used as a reward function after training. This has been extended by Disturbance-based Reward Extrapolation (D-REX) [7], which automatically generates rankings by fitting a BC policy on the demonstrations and subsequently draws samples with different levels of noise from this policy. We adapt D-REX to our trajectory- -base setting as a baseline. Similar to VIGOR, this adapted method trains model expert demonstrations on training contexts to optimize GMM policies on novel test contexts. However, while D-REX learns a reward function to do so, VIGOR iteratively (re- (re-)trains discriminator on the expert demonstrations and policy samples.
3 Foundations
In this section, we briefly cover the foundations and previous work that our method builds on.
(T.....,7y)) ProMPs represent the elements as T. (t) w. where 7; represents a vector of the desired joint angles at time-step & [18] Note that ur does not depends on and that t can thus be computed at an arbitrary resolution T. Here (() are time-dependent features, usually radial basis functions centered around different time points, and w denotes the parameter vector. For a single demonstration, the parameters ur are fitted using simple linear regression and compactly represent the entire trajectory.
Probabilistic Movement Primitives. For a trajectory
3

---

Training Contexts
Human Demonstrations
Preprocesss
Expert Trajectories
Transform
Geometric Behavioral Descriptors
Train
Discriminator
Training
Gaussian Mixture Policy
Sample
Learner
Transform
Test Contexts
Trajectories
Update
Figure 2: Schematic of VIGOR. We generate set of expert trajectories from human demonstrations and then transform them into geometric behavioral descriptors. These descriptors are then fed into a discriminator. This process is repeated for samples of a separate GMM policy for each training and test context. We then train the discriminator to distinguish between human and policy trajectories. The geometric descriptors are designed to abstract away from concrete task configurations, causing the discriminator to distinguish how well a trajectory performs rather than which context acts on. As result, the discriminator can be used to improve the policies.
Variational Inference for Gaussian Mixture Models. We build on a recent class of efficient Variational Inference approaches that allow modeling versatile behavior with GMMs [16, 14]. These [35] to minimize the reverse Kullback-Leibler Divergence
approaches use the (L((( (w) ())((w)) between a model q(w) and a target distribution p(w). In our case, p(w) is the distribution of expert trajectories. Arenz et al. [14] introduced an upper-bound objective based on variational decomposition. The resulting approach, Variational Inference by Policy Search (VIPS) [14, 15]. repeatedly solves the objective
(w) = arg min (q(w,=)) log q (w) (w))
+ KL(((=))l((=)) + Ex(=)) KL (w)l)) ll (w|l)) (I)
((w)
where ; denotes the model from the previous iteration. This objective decomposes into individual optimizations for the components and categorical distributions, which Arenz et al. [14] solve using
trust-region methods from policy search [37, 38].
Distribution Matching for Gaussian Mixture Models. We cannot directly work with VIPS, as we do not know p(w) but only have the expert demonstrations. To address this issue, Becker et al. [16] propose using density ratio estimation [39] approaches to approximate the log( ((w)(()))))) ()) tt)pp(p)) w)) term in
Equation 1, removing the dependency p(w). Specifically, they logistic regression to estimate the density ratio. The resulting approach, Expected Information Maximization (EIM), resembles Generative Adversarial Nets [29] where the discriminator effectively also estimates a density ratio.
4 Inferring Versatile Behaviors by Matching Geometric Descriptors
Our approach estimates versatile behavior in form of GMM distributions over ProMP weights for all training and test contexts of the given task. We note that while this setup requires the test contexts in advance, it does not need interaction with the real environment since the behavioral descriptors directly computed from the proposed trajectory and its context. To capture correlations in the demonstrations, we utilize full-covariance GMM components. For each planned trajectory, we compute concise descriptors such distances to key-points that isolate the performance of the trajectory from its context to allow for sample-efficient generalization to novel contexts. We then build HIM [16] and
discriminator to infer GMMs from demonstrations by matching the learner's 's and expert's 's distributions in the resulting geometric feature space. Towards this end, we generalize EIM to sequential data and modify the policy parameterization accommodate for setting. Figure 2 illustrates an overview of our approach. We provide pseudocode in Appendix A.
Sontext-Specific Mixture Policies We want to model the expert's behavior for given small set of training configurations of the task, denoted as training contexts Cuain- and a set of unseen test contexts Crain U Cust- As context-dependent GMMs² are hard
Cysst We consider the set of all contexts
Such GMMs could in principle be represented by MDNs. However, attempts with MDN and coll-covariance Gaussian components highly unsuccessful in experiments.

---

to learn and we assume a relatively small amount of training contexts, 6 in all our experiments, with multiple demonstrations per training context, we resort maintaining single GMM- q.(w) for each context c. This non-amortized formulation simplifies the policy updates as we can optimize the policy for each context individually, given the density ratio estimator, using highly efficient, tailored methods for full-rank GMM approximations [14, 15].
Distribution Matching of Behavioral Descriptors We assume access behavioral descriptors O = fo( that represent features encoded in the parameter vector w with respect to the context c of w. For simplicity, we will assume that the mapping fe between w and O deterministic, such that we can easily evaluate O for any desired plan w. We want to match the distribution of behavioral descriptors of the demonstrator while optimizing for q(w). i.e.
V. p( w)) log
(0) do
qº (w) arg min (O)) <(w)
IIp()) = arg min En(un) g(w))
((0)w)
p(0)
p( w)q ddw. where p( O (w) is Dirac delta distribution defined by the mapping fo and (O)) p
As the mapping between w and O is deterministic, the EIM algorithm can be directly applied to this setup with the difference that the discriminator is trained using O instead of Moreover, while we a learn different distributions ge( (w) for each context c, we use the same discriminator for all contexts. This way, we can infer distributions {go(w)le E Crea). i.e. infer versatile trajectories for unseen scenarios.
Density Ratio Estimation for Sequential Behavioral Descriptors As in our case, encodes the desired trajectory т, the behavioral descriptors are typically computed per time-step. i.e., O where each o, is a feature vector. To enable the classifier to deal with sequential data,
(o1..... :or) or)", we consider a soneence-to-squenne
neural network (0O) = ((((........ )r) = (y1....,, yrr" The network receives sequence of inputs O (01,--- OT ) and outputs sequence of values (u....,,rr)! where each y/ may depend on multiple oy. In each iteration of our optimization, we (re-)train this network to discriminate between the provided demonstrations equal number of policy samples (44) drawn uniformly from c. We first consider the case where we discriminate full trajectories as in EIM. Denoting the sigmoid function a, the discriminator can straightforwardly be trained on a binary cross-entropy loss of the sum of sequence values
O(p) of Crain. and an
ÿ Eizo Vt. i.e., by minimizing
(2) w.r.t. ((O). Equation 2 recovers the log density ratio ((O) log O(p) log (4)) at conver- gence [39]. However, this cost function only provides single training sample per trajectory and is therefore hard to use for small number of demonstrations. By utilizing the sequential structure of the trajectories, the classification error can be computed for each time-step, i.c. we minimize
BCE(O(O), (PP), (()) Eoo [log (a - [log (1 a (0))
T
BCEssep ( (O), (P)) ,(()) -oow
Σ
log (a (ut) - EoG
Σ log (1 (vi) 4==
(3)
w.r.t. (0) Finally, we train an ensemble of discriminators instead of a single one, using their average logit as the log density ratio estimate for the policy updates. For the network architecture, we find that simple 4 layer 1d convolutonaal neural networks (Id-CNNs) work best in our experiments. We compare this choice to other sequential network architectures the ablations in Appendix D.
Geometric Behavioral Descriptors VIGOR utilizes geometric descriptors that abstract away from a concrete task configuration to generalize to new configurations. To this end, we encode the current state along the desired trajectory with respect relative geometric features rather than absolute values. The resulting geometric descriptors cause trajectories of similar performance to appear similar in feature space regardless of their context, allowing for sample-efficient generalization to novel contexts. We note that such a descriptor space can often be straightforwardly constructed from the geometry of the task by composing distances of the end-effector For example, in object manipulation, this space can be composed of distances of the end-effector to the corners of box to push.
scy-points of the objects in a scene.
5 Experiments
All experiments use human demonstrations. The tasks were selected due to their highly multi-modal nature and because they allow for an easy collection of human demonstrations. We instructed the
5

---

demonstrators to solve the task in varying ways to create versatile solutions. As preprocessing step. we fit ProMP for each human demonstration and filter the resulting ProMPs such that only successful demonstrations remain. This step ensures that the human demonstrations can be imitated by the learner and conveniently allows to compress all demonstrations to a fixed length in a principled fashion. We use the resulting preprocessed expert demonstrations for all experiments unless otherwise stated. Similarly, unless noted otherwise, all experiments use training and 6 test contexts, and 5 GMM components for each configuration and method. Experimentally, this number of components is sufficient for matching the distribution of expert descriptors for the considered tasks, whereas more components generally only lead spurious improvements. We experiment with less components in the ablations in Appendix D. Unless noted otherwise, we do not train the categorical distribution of the GMM components, using uniform distribution instead. We evaluate the GMM policies using a number of samples for each component of each test context, and then rank these components according to the average performance of their samples. We then report statistics of the best component of the resulting value over the random seeds. Note that reporting the best component leads to fair comparison unimodal approaches. For baselines that do not use GMM policies, samples of the trained policies for each test context and average the performance of all of them. For each experiment, we report measure of how well the task performed in the main paper, and additional success rates in Appendix C. We repeat each experiment for 10 random seeds. Appendix E shows all hyperparameters and training details.
instead draw
Baselines. We compare VIGOR to a number of IL basclines, using three state-action baselines, namely GAIL [5], behavioral cloning (BC(S)) and behavioral cloning with a GMM policy (BC- GMM(S)) [13]. For all experiments, the state-action baselines receive geometric descriptors of the current state and time-step state information, and output the velocities of the robot joints as actions. Further, we also consider three trajectory-based methods. These are standard behavioral cloning (BC(T)), behavioral cloning with a GMM policy (BC-GMM(T)) (see e.g., Zhou et al. [18],, and a modified version of D-REX (EM+D-REX). The latter is chosen due to its similarities to VIGOR, which are detailed in Appendix I. We refer to Appendix B for details on the baselines and to Appendix C for further results on all experiments described below.
Planar Reacher. First, we consider the introductory task shown in Figure 1. In this task, the goal is to reach an intermediate target area with the end-effector of 5 Degrees of Freedom (DoF) robot with joints of length 1 before ending the trajectory in final target area. The geometric descriptors for this task are given by the euclidean distance of the end-effector to both targets, paired with the average velocity and acceleration of the joints to encode the smoothness of the motion. This yields total of 4 features per time-step. The left and middle panels of Figure depict example demonstrations for two training contexts (top) and the corresponding distances between the end-effector and targets over time (bottom). We explore other choices for the geometric descriptors in the ablations in Appendix D. For evaluation, we use the Distance to Boundaries of the target areas. For the first target, we consider the minimum distance of the end-effector to the boundary, for the second target, we consider the distance at the last time-step. We train all methods on 5 demonstrations per training context. The left of Figure 3 shows results on test contexts. We find that only VIGOR and EM+D-REX can consistently get close to both targets. Yet, EM+D-REX can only learn a single mode and fails to reproduce the versatility of the demonstrations. We investigate this in more detail in Appendix C.I.
7-D Panda Reacher. We repeat the above point-reaching task using 7 DoF Franka Emika Panda robot and 3-D intermediate and goal positions. Since we do not care about the end-effector's rotation, the last DoF can be ignored, leading to an effective action dimension of 6. We use 8 basis functions per action dimension for the MPs, leading to a total of MP dimension of 48 and thus a high-dimensional problem space. The expert demonstrations are collected using teleoperation setup with a virtual twin that records joint values over time. Appendix C.2 explains the setup in more detail. We use the same geometric descriptors as for the planar reacher task. The right of Figure 3 shows target distances on test contexts. Similar to the previous task, only VIGOR and EM+D-REX can consistently get close to both targets while the other methods regularly fail to pursue at least one of the two targets.
Box Pusher. We also evaluate our approach on contact-rich robot manipulation task. In this task, a simulated Franka Emika Panda robot has a rod attached to its end-effector and needs use push a rectangular box to a given goal position and orientation. The box always starts at the same position, and the task context is given by the desired (Ir, y) translation of the box, plus its desired rotation in degrees. For simplicity, the task is learned in task space with a fixed height. The demonstrations contain the starting position of the robot as well as its trajectory. Here, the solutions are versatile

---

Distance Boundarics
0.8
Demonstrations
Distance Boundaries
0.02
0.005
0.6
3
0.00
0.000
2
0.2
VIGOR
EM+
D-REX
BC
GMM(T)
BC(T)
GAII
BC
GMM(S)
BC
VIGOR
EM+
D-REX
BC-
GMM(T)
BC(T)
GAIL
BC-
GMM(S)
BCIS)
Figure 3: Mean target distance of samples from the best policy component the Planar (left) and
Panda (right) Reaching tasks for test contexts. We find that a trajectory- setting and the use of mixture policies help performance, Similarly, using discriminator to iteratively optimize the policies is advantageous on the reaching tasks, as seen from VIGOR and GAIL. Finally, the clear distribution matching objective of VIGOR and EM+D-REX seems crucial for imitation from a low number of versatile demonstrations. VIGOR uniquely combines the above properties and is able reliably reach both targets for both tasks.
be seen from VIGOR, EM+D-REX, and BC-GMM (T).
Distance Target [m]
0.3
Demonstrations
Distance to Target [m]]
VIGOR EIM+D-REX
Initial Box Position
BC(T)
GAIL
10-1
0.2
0.1
10-2
VIGOR
EM+
D-REX
BC-
GMM(T)
BC(T)
GAIL
BC-
GMM(S)
BC(S)
10
15
20
25
Number Training Contexts
Figure 4: Mean target distance of samples from the best policy component on the Box Pushing task for test contexts. The dotted line denotes the average corner distance between the initial and the desired box position. The dashed line shows the performance of the human demonstrations. (Left) VIGOR able consistently push the box the right configuration. Opposed to this, the other methods only marginally improve over the initial box positions, and in some cases perform worse than it. (Right) We find that VIGOR benefits most from additional training contexts, almost matching the performance of the demonstrator on unseen contexts for 24 training contexts.
because the box can be pushed both from the inside and outside. The geometric descriptors consist of the (ar, y) distances of the end-effector the initial and desired final position of the corners of the box. To facilitate generalization between contexts, we translate the basis of these distances such that the desired final box aligns with the origin of the coordinate system. We additionally add the end-effector velocity and acceleration as well as the time-step. resulting in 19 features per time-step. This geometric description of the task allows for IL without simulating explicitly modeling the box. significantly speeding the training process and facilitating generalization to both new contexts and real-world robots. We use 3 demonstrations per training context and evaluate the average distance of the corners of the final box to that of the desired final position. The contact with the box causes mismatch between planned trajectory and its execution on the robot, which we illustrate in Appendix C.3. Results are shown in the left of Figure 4. Additionally, the right of Figure 4 shows how the performance varies when changing the number of training contexts. Finally, we illustrate how planned trajectories of VIGOR perform on a real robot. To this end, we create real-world replica of the simulated box that the robot manipulates. Rollouts for different policy components of VIGOR trained on 24 training contexts exemplary test context are shown in Figure 5.

---

Figure 5: Trajectory executions of three component means of a VIGOR policy trained on 24 training contexts for the same test context. The first two rows show a real Franka Panda executing the planned end-effector trajectories. The last row shows the same setup in the simulation, where the target box is overlaid in green. All three executions push the box in different ways, capturing the versatility of the human demonstrations; while the robot in the first row pushes the box from the inside, the robot the second row pushes from outside. The third trajectory also pushes from the inside but much farther away from the corner of the box.
Ablations. We conduct extensive ablation studies to investigate how our design choices affect our approach. We find that a concise choice of behavioral descriptors, preprocessing of the expert demonstrations to ProMPs, and a 1d-CNN discriminator are crucial for performance. We also notice that the policy benefits from additional components and that we see modest improvements for an ensemble of discriminators and the stepwise loss proposed in Equation 3. We refer to Appendix for the full results and more thorough ablations of both VIGOR and EM+D-REX.
6 Limitations
Scale. We currently train and maintain a separate GMM over ProMP parameters for each context, leading to linear space and time complexity w.r.t. the total number of contexts. One way address this challenge would be to instead train joint Mixture Density Network q(w) c) over all contexts, which is left for future work. Additionally, representing trajectories with single ProMP limits them to single smooth movements. To alleviate this, MP chaining [40, 41, 42] can be straightforwardy integrated into our approach to allow for the representation more complex movements
Geometric Descriptors. VIGOR facilitates generalization to novel task configurations by using geometric descriptors. While this allows the user to include domain knowledge, it also requires clear idea about which aspects of the task are important. Instead, geometric descriptors could be automatically extracted from the task for example by using NNs processing point clouds [43, 44].
Re-training, Finally, our method assumes that the test configurations Cest are known in advance. Since the test configurations are optimized jointly with the training configurations, the only way to integrate new test configurations
re-train the algorithm. the future, we want to alleviate this issue by recovering reward representation from ranked intermediate policies of the given training configurations, essentially combining our approach with some the ideas of D-REX.
Conclusion
We proposed VIGOR, novel feature-matchin approach to Imitation Learning in environments with versatile solutions. Utilizing a combination movement primitives, mixture policies and matching geometric behavioral descriptors, our method closely imitate the behavior distribution of human experts from few demonstrations. We show the effectiveness of our approach on a suite of challenging robot coordination tasks. The results show that VIGOR is able closely match the distribution of the demonstrator, outperforming the chosen basclines in all considered settings.

---

Acknowledgments
NS and GN were supported by the Carl Zeiss Foundation under the project JuBot (Jung Bleiben mit Robotern). The authors acknowledge support by the state of Baden- Weiee--Wrttebberrg through bwHPC.
References
[1] S. Schaal. Learning from demonstration. Advances in Neural Information Processing Systems (NeurIPS), 9, 1996.
[2] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469 483, 2009.
[3] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2)11-35 2017.
[4] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations (ICLR), 2018.
[5] J. Ho and Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems (NeurIPS), 29, 2016.
[6] D. Brown, W. Goo, P. Nagarajan, and S. Niekum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. In International Conference on Machine Learning (ICML), pages 783-792. PMLR, 2019.
[7] D. S. Brown, W. Goo, and Niekum. Better-than demonsttrntt imitation learning via automatically-rankedd demonstrations. In Conference on Robot Learning (CoRI.), pages 330-359. PMIR, 2020.
[8] M. Orsini, A. Raichuk, L. Hussenot, D. Vincent, R. Dadashi, S. Girgin, M. Geist, O. Bachem, O. Pietquin, and M. Andrychowicz. What matters for adversarial imitation learning? Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[9] T. Osa, J. Pajarinen, G. Neumann, J. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 7(1-2):1-179, 2018.
[10] A. Paraschos, C. Daniel, J. R. Peters, and G. Neumann. Probabilistic movement primitives. Advances in Neural Information Processing Systems (NeurIPS), 26, 2013.
[11] P. Englert, N. A. Vien, and M. Toussaint. Inverse kkt: Learning cost functions of manipulation tasks from demonstrations. The International Journal of Robotics Research, 36(13-14):1474- 1488, 2017.
[12) P. Englert and M. Toussaint, Learning manipulation skills from single demonstration. The International Journal of Robotics Research, 37(1):337-154, 2018.
[13] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin What matters in learning from offline human demonstrations for robot manipulation. In 5th Annual Conference on Robot Learning, 2021.
[14] O. Arenz, G. Neumann, M. Zhong, et al. Efficient gradient-fres variational inference using policy search. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80, pages 234-243. Proceedings of Machine Learning Research, 2018.
[15] O. Arenz, M. Zhong, and G. Neumann. Trust-region variational inference with gaussian mixture models. J. Mach. Learn. Res., 21:163-1, 2020.
[16] Becker, O. Arenz, and G. Neumann. Expected information maximization: Using the i- projection for mixture density estimation. In International Conference on Learning Representa- tions (ICLR), 2020.
[17] M. Bain and C. Sammut. framework for behavioural cloning. In Machine Intelligence 15, pages 103-129, 1995.

---

[18] Y. Zhou, J. Gao, and T. Asfour. Movement primitive learning and generalization: Using mixture density networks. IEEE Robotics Automaatio Magazine, 27(2):2-33, 2020.
[19] S. Schaal. Dynamic movement primitives-a framework for motor control in humans and humanoid robotics. In Adaptive Motion of Animals and Machines, pages 261-2800 Springer, 2006.
[20] A. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1): 1-22, 1977.
[21] K. Mülling, J. Kober, O. Kroemer, and J. Peters. Learning select and generalize striking movements in robot table tennis. The International Journal of Robotics Research, 32(3):
263-279, 2013.
[22] M. Ewerton, G. Neumann, R. Lioutikov, H. B. Amor, J. Peters, and G. Maeda. Learning multiple collaborative tasks with a mixture of interaction primitives. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 1535-1542. IEFF, 2015.
[23] A. Pervez and D. Lee. Learning task pas-pparmeterrized dynamic movement primitives using mixture of gmms. Intelligent Service Robotics, 1 11(1):61-78, 2018.
[24] C. M. Bishop. Mixture density networks. Aston University, 1994.
[25] T. Osa, A. M. G. Esfahani, R. Stolkin, R. Lioutikov. J. Peters, and G. Neumann. Guiding trajectory optimization by demonstrated distributions. IEEE Robotics and Automation Letters, 2(2):819-826, 2017.
[26] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first International Conference on Muchine Learning (ICML), page 1, 2004.
[27] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008
[28] S. Kullback and R. Leibler. On information and sufficiency. The Annals Of Mathematical Statistics, 22(1 797-666 1951.
[29] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Bengio. Generative adversarial nets. Advances Neural Information Processing Systems (NeurIPS), 27, 2014.
[30] I. Kostrikov, K. K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson. Discriminator-actor- critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference Learning Representutions (ICLR), 2018.
[31] F. Torabi, G. Warnell, and P. Stone. Adversarial imitation learning from state-only demon- strations. Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 2229-2233 2019.
[32] I. Kostrikov, O. Nachum, and Tompson. Imitation learning via off-policy distribution matching. arXiv preprint arXivv 2019.
[33] K. S. Ghasemipour, Zemel, and Gu. divergence minimization perspective on imitation learning methods. In Conference on Robot Learning (CoRL.,, pages 1259-1277. PMIR, 2020.
[34] X. Zhang, Y. Li, Z. Zhang, and Z.-L. Zhang. f-gail: Learning f-divergence for generative adversarial imitation learning. Advances in Neural Information Processing Systems (NeurIPS), 33:12805-12815. 2020.
[35] K. P. Murphy. Machine Learning: A Probabilistic Perspective. MIT press, 2012.
[36] N. Freymuth, P. Becker, and G. Neumann. Versatile inverse reinforcement learning via cu- mulative rewards. 4th Robot Learning Workshop at Neural Information Processing Systems (NeurIPS), 2021.
10

---

[37] J. Peters, K. Mulling, and Y. Altun. Relative entropy policy search. In Twenty-Fond Fourth AAAI Conference on Artificial Intelligence, 2010.
[38] A. Abdolmaleki, R. Lioutikov, J. R. Peters, N. Lau, P. Reis, and G. Neumann. Model-based relative entropy stochastic search. In Advances in Neural Information Processing Systems (NeurIPS), pages 3537-3545, 2015,
[39] M. Sugiyama, T. Suzuki, and T. Kanamori. Density Ratio Estimation in Machine Learning. Cambridge University Press, 2012.
[40] C. Daniel, G. Neumann, O. Kroemer, J. Peters, al. Hierarchical relative entropy policy search. Journal of Machine Learning Research, 17:1-50, 2016.
[41] G. Neumann, W. Maass, and J. Peters, Learning complex motions by sequencing simpler motion templates. In Proceedings of the 26th Annual International Conference Machine Learning, pages 753-760, 2009.
[42] Manschitz, J. Kober, M. Gienger, and J. Peters. Learning to sequence movement primitives from demonstrations. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4414-4421 IEEE, 2014.
[43] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pages 652 660, 2017.
[44] C. R. Qi, I.. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets a metric space. Advances Neural Information Processing Systems (NeurIPS), 30, 2017.
[45] S. Wang, S. Toyer, A. Gleave, and S. Emmons. The imitation library for imitation learning and inverse reinforcement learning. https://github. https////gghhhab.ccoo//aaaannnn com/HumanCompat bltt:miitaimmiiiiiioiiioo 2020.
[46] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 17XXiv::7000000 1707.06347 2017.
[47] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Emestus, and N. Dormann. Stable-baselines: Reliable reinforcement learning implementations. Journal of Machine Learning Research 22(26))))1))11111111111) 1-8, 2021. URL http:
htte//appr///2///// 1364. html.
[48] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1997.
[49] A. Paszke, S. Gross, F. Massa, A. Lever, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Detooo, M. Rai- son, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Py- torch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems (NeurIPS) 32, pages 8024- 8035. Curran Associates, Inc., 2019. URL http://papers. neurips. neurips.cc/paper 9015-pytorch an- immeratiiv- hewpriveestttll--ig-----r--ooooe hhigh-performance deep- learning- 1ibrary.r.ryrr pdf.
[50] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations (ICLR) 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[51] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Research (JMLR), 15(1):1929-1999 1958, Jan. 2014. ISSN 1532-4435.
[52] loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning (ICML)- Volume 37, ICML. 15, page 448 4484456. JMLR.org. 2015.


Reviews:
[{"review_id": "bdOhD6ofhq2", "reviewer": "Reviewer_24LK", "summary": "This paper proposes a method for learning skills from multiple diverse demonstrations using an approach that infers multi-modal distributions in a geometric feature space.  It is motivated by skills that can inherently be performed/taught in different ways and a robot may be (and perhaps should be) learning from a dataset that leverages multiple (human) demonstrators.  The method is evaluated against several imitation learning baselines on a reaching task and a pushing task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper addresses an important, open problem in the Learning from Demonstration (LfD) literature.  It is very relevant in the age of big data, as we can much more feasibly scale up the number of humans to collect data from.  The method proposed is principled and builds from existing literature.  My primary questions are around the evaluation of the method.  \n\nOverall strengths and suggestions for improvement are summarised below.\n\nMore detailed strengths of paper:\n+  The method proposed is mathematically principled and allows for adaptability to multiple learning contexts.  It models a set of demonstrations as multi-modal distributions (Gaussian Mixture Models) in order to capture diversity/versatility in the way humans provide successful demonstrations of the task. Furthermore, it learns multi-modal distributions conditioned upon learning context and trains a discriminator to infer the correct context. This allows the method to account for more diverse contexts. \n+ The paper evaluates its method on real robot platforms.\n+  Figure 4 (right) is a nice result.  It empirically confirms that the proposed method can better leverage (and hence take advantage of) variation in training contexts, as compared to baseline imitation learning methods tested.\n\nI would suggest the following to improve the paper submission:\n-  RE the evaluation: How and why were these particular tasks (reaching and pushing) selected?  What interesting properties does each exhibit with respect to what it highlights about the difficulty of the problem or insights about this method as compared to other imitation learning methods, on inherently multimodal learning tasks?  What hypotheses were being tested with each of the experiments? \n-  RE the experimental results shown in Figures 3 & 4: Any insights as to why there was significantly more variance in how the methods performed in the reaching tasks as compared to the pushing task?  For the reaching tasks, I also was a bit unclear. Is the entire demonstration trajectory used for learning or just the final goal position? If the latter, I’m not entirely clear on where the multi-modality in the data is expected to derive from and would request more clarity on this.  In particular, if only the goal position is used as learning signal for the task, what geometric features are expected to vary sufficiently as to induce a multi-modal distribution?  \n-  Regarding the step of the method that performs distribution matching of behavioural descriptors (subsection 4.2), this seems related to the idea of optimising weights for feature expectation matching in Inverse Reinforcement Learning. I would be interested to understand how these are similar and different?  Or perhaps, does the proposed method build upon feature expectation matching imitation learning approaches?  And if so, in what ways? Why is generalising to geometric representations a difficult challenge?\n-  The first sentence under Section 4 (page 4) is causing confusion for me.  Does saying that the approach estimates weights for all training and test configurations imply that the method has access to test configurations at training time?  That would seem like “cheating”, from the perspective of generalising the learned model.  I would recommend clarifying what is intended here.\n\nAdditional minor comments:\n-  On page 1 (line 30), what does context-insensitive mean in this context?  Is this synonymous with saying context-independent?\n-  In the Experiments section (page 5, line 175), why n=5 components? How was this selected?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I would recommend accepting the paper, based upon its contribution of a method for addressing inherent multi-modality in skill learning from demonstration.  This is an important and (to my knowledge) currently open problem in the Imitation Learning literature.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See limitations above.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "SIMozKJFo0yt", "reviewer": "Reviewer_TvrN", "summary": "The paper is concerned with the problem of using imitation learning (IL) to find behavior policies that allow machines to accomplish tasks when the set of demonstrations that should be imitated are \"versatile,\" meaning that it contains multiple ways to accomplish a particular task. The paper proposes a new IL method called VIGOR which is based on probabilistic motion primitives and distribution matching over \"geometric features,\" and presents an evaluation of this new method for both simulated and real robot arm tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+) The paper does an excellent job of illustrating the problem of \"versatile\" demonstrations and performs experiments using real human demonstrations, which is compelling. \n\n(+) Experiments are performed on both simulated and real robots. \n\n \n\n(-) It's not clear to me how strong the contribution of the paper is here. The introduction of the paper makes it seems as though the authors are claiming that the use of geometric features constitutes the novelty of the proposed method, but merely representing the state features of a problem in terms of, eg, distance to goal, seems commonplace in the community. If the authors disagree, could they provide citations to important recent papers that do _not_ do this? Additionally, from the ablation studies, it also seems important for VIGOR that the demonstrations be \"filtered\" by first fitting a ProMP; are the authors claiming that doing this is also a novel part of their approach, or has this been done in other work? An explicit list of claimed contributions—along with justifications for why these are indeed contributions—would go a long way to improving the paper. \n\n(-) The technical discussion needs to be improved for clarity. For example, Section 3 starts off by defining a trajectory $\\tau = {\\tau_1, …}$, but never actually defines for the reader what type of mathematical object $\\tau_t$ is (eg, is it a scalar? a vector? what specifically does it encode? a state?). The same goes for the precise meaning of $\\phi$ and $O$. This missing information makes it hard for me to evaluate the proposed algorithm. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The problem of performing imitation learning in the presence of \"versatile demonstrations is interesting and the paper's illustration of this phenomenon, along with interesting experiments is good. However, the specific contributions to the literature are unclear to me and the technical discussion also needs to be improved for clarity.\n\nPOST-DISCUSSION UPDATE: During the discussion phase, the authors addressed several of my questions, and made corresponding revisions to the paper. I now think I have a better understanding of the novelty of the paper here, and am raising my score to \"Weak Accept.\" However, I still think that the authors can do a better job in the presentation here by more explicitly focusing on the truly novel portion of their algorithm as opposed to the features that it relies on having access to. I understand this is an important point, but the authors have provided very little here to the community regarding how to obtain these features.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "(see weaknesses identified above)", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "LBMgHdu5itq", "reviewer": "Reviewer_9xkZ", "summary": "This paper proposes an imitation learning approach to capture the multi-modality (versatility) of expert demonstrations. This paper focuses on the trajectory generation aspect of imitation learning while conditioning on relative geometric features such as relative distance to the goal positions.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- I can’t tell the strengths of the approach given that it’s missing  a very important baseline, as mentioned in the “weakness” and “issues”\n\nWeakness:\n- The work is about learning behaviors / skills / motion primitives, but not a plan technically. The title itself is very misleading.\n- It’s missing a very important baseline, behavior cloning + gaussian mixture model (BC-GMM) [1]. This previous approach can outputs GMM actions, and the model is trained with a very simple supervision loss. Solely comparing against vanilla BC is insufficient.\n- It’s very vague about the term “context”. What is exactly the context here? Different goals? Then why not just say “goal-conditioned”? There is no clear semantic of “context” in any of the tasks shown. Please be more specific about the definition of “context” (possibly with rigorous definition in the text) or use other terms without ambiguity.\n- The choice of comparison baselines is also ambiguous. What is the purpose of comparing against IRL? The authors are not learning rewards for their approach, then why compare against IRL? \n- all the tasks involvely only free motion generation (reaching, and pushing but not really involve contact-rich behaviors), and not in the closed-loop manner. How does the approach generalize to pick-place behaviors, and more complicated pushing behaviors?\n\n[1] Mandlekar et al. What Matters in Learning from Offline Human Demonstrations for Robot Manipulation", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Given that one of the basic baselines is missing to showcase the proposed formulation is indeed necessary to learn versatile behaviors, I vote for weak reject but will change the review if the authors address all my comments and issues properly.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- revise the title to be precise about the choice of words.\n- missing related works, especially related to the reference that has BC-GMM. And lack comparison to this baseline\n- Revise the experiment session to explain the reason of the specific choices of baselines. This also helps to illustrate what the contribution is exactly.  \n- The last paragraph of the related work is not well-written. The paragraph name needs a more precise summary other than “further relevant approaches”. \n- Some grammar issues: “Few modes” -> “a few modes”. \n- Explicitly state whether this approach can be applied/extended to pick/place behaviors. If it can, what are the major changes to the approach the authors need? Otherwise, this approach would only be incremental to solve pushing / reaching tasks that are not really complicated not all."}}]

Author rebuttal:

Decision: Accept (poster)