Dynamics-aware Skill Generation from Behaviourally Diverse Demonstrations
Authors: Shibei Zhu
Rituraj Kaushik
Samuel Kaski
Ville Kyrki
Venue: Submitted to ICLR 2023 (2023)
Keywords: Learning from Demonstration
Reinforcement Learning

Abstract: Learning from demonstrations (LfD) provides a data-efficient way for a robot to learn a task by observing humans performing the task, without the need for an explicit reward function. However, in many real-world scenarios (e.g., driving a car) humans often perform the same task in different ways, motivated not only by the primary objective of the task (e.g., reaching the destination safely) but also by their individual preferences (e.g., different driving behaviours), leading to a multi-modal distribution of demonstrations. In this work, we consider an LfD problem, where the reward function for the main objective of the task is known to the learning agent; however, the individual preferences leading to the variations in the demonstrations are unknown. We show that current LfD approaches learn policies that either track a single mode or the mean of the demonstration distribution. In contrast, we propose an algorithm to learn a diverse set of policies to perform the task, capturing the different modes in the demonstrations due to the diverse preferences of the individuals. We show that we can build a parameterised solution space that captures different behaviour patterns from the demonstrations. Then, a set of policies can be generated in solution space that generate a diverse range of behaviours that go beyond the provided demonstrations.

Under review as a conference paper at ICLR 2023
DYNAMICS- SKILL GENERATION FROM BEHAVIOURALLY DIVERSE DEMONSTRATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Learning from demonstrations (LfD) provides data-efficient way for a robot to learn task by observing humans performing the task, without the need for an explicit reward function. However, in many real-world scenarios (e.g., driving a car) humans often perform the same task in different ways, motivated not only by the primary objective of the task (e.g., reaching the destination safely) but also by their individual preferences (e.g., different driving styles), leading to a multi- modal distribution of demonstrations. In this work, consider Learning from state-only Demonstration setup, where the reward function for the common ob- jective of the task is known to the learning agent; however, the individual prefer- ences leading to the variations in the demonstrations are unknown. We introduce imitation-guided Reinforcement Learning (RL) framework that formulates the policy optimisation as a constrained RL problem to learn a diverse set of policies to perform the task with different constraints imposed by the preferences. Then we propose an algorithm called L.BD and show that we can build parameterised solution space that captures different behaviour patterns from the demonstrations. In this solution space, a set of policies can be learned to produce behaviours that not only capture the modes but also go beyond the provided demonstrations.
INTRODUCTION
Learning from demonstrations (LiD) (Schaal 1996) provides an alternative way to Reinforcement Learning (RI. for an agent to learn policy by observing how humans perform similar tasks. How- ever, in many real-world scenarios (e.g., driving a car), humans often perform the same task in different ways. Their behaviours are not only influenced by the primary objective of the task (e.g., reaching the destination safely) but also by their individual preferences or expertise (e.g., different driving styles) (Fürnkranz & Hüllermeier] 2010 Babes et al., 201T). In other words, all the under- lying policies maximize the same task reward, but under different constraints imposed by individual preferences. This leads to multi-moda. distribution of demonstrations, where each mode represents a unique behaviour.
With multi-modal demonstrations, typical LfD) methods, such Behaviour Cloning and Generative Adversarial Imitation Learning, either learn a policy that converges to one of the modes resulting in a mode-seeking behaviour or exhibit mean-seeking behaviour by trying to average across different modes (Ke et al. [2200] Ghasemipour et al.) 2020) Zhang et al. 2020). While the former will still recover a subset of solutions, the latter may cause unknown behaviour (see Fig T. Furthermore, none of these approaches is able to learn policies that correspond to the behaviours of wide range of individuals. The problem becomes even more challenging when there are only "state observa- tions" in the demonstrations without the actions. In such situations, supervised or unsupervised learning approaches cannot be applied directly to find a policy, and the agent must interact with the environment or with simulator (Torabi et al. 2019).
Being able to learn a diverse set of policies from demonstrations is often desirable to serve the requirements of a wide range of individual users. For instance, every self-driving car can have driving policy (selected from the diverse set of pre-trained policies) that matches the preferences of user. Many recent works show the advantages of having diverse set of policies. for instance, rapid damage adaptation in robotics (Kaushik et al, 2000) (hal//hahaartuuuccii et al. 2018 Cully et al. 2015) and safe sim-to-real policy transfer in roboties (Kaushik et al. 2022).

---

Under review as a conference paper at ICLR 2023
BANNI
demonstrations
B. Mean-seeking LfD
Mode-seeking LFD
D. LFBD (Ours)
Figure I: Given the demonstrations from several individuals in A, the mcan-sceking policy produces unseen behaviour that is unsafe as shown in B, the mode-sccking policy only recovers one mode as shown in C. We propose a new framework that recovers all the possible solution modes as shown in D. The example is inspired from (Ke et al. 2020)
In this work, we consider a specific setup of LID, known as Imitation Learning from Observations alone (ILFO) (Sun al., 2019). where the learning agent only has access the state observations without their corresponding actions. We propose a new framework that combines Reinforcement Learning with ILfO to solve the issues of learning from the multi-modal states-only demonstrations, especially with small set of unlabelled demonstrations. Unlike most of the LfD methods, our goal not just to learn how to perform a task or how to mimic humans, but rather how to perform task in all possible ways shown in Fig Thus, we focus applications where high-level task reward function can be defined easily, but the preference components that cause diverse behaviours cannot be explicitly defined. These include tasks such autonomous driving robotic manipulations, where the agent needs to mimic the human's behaviour pattern (i.e., preference) while reaching the intended goal (i.e., task reward).
In practice, defining a high-level task reward can be straightforward, i.e., for autonomous driving, this be function of the distance to target location and the penalty for collisions. However, defining the preference component is far from easy, as it may be impossible to find mathematical expression for each individual's preferences. Thus, we formulate the multimodal policy genera- tion guided through demonstratinn as constrained optimisation problem; where the generation of multimodal behaviours results from optimising policies for a given task reward function that satisfy different preference constraints.
As contributions, we first propose new ciitaation-uidden RL framework and an algorithm called Learning from Behaviourally diverse Demosntration (LfBD) to solve the problem of policy genera- tion from multimodal (state-only) demonstrations. We then propose a novel projection function that captures preferences state-region visitations. This projection function allows us to build pa- rameterised solution space allocate policies such that they satisfy different preference constraints. We show that approach is capable of generating multimodal solutions beyond the provided demonstrations, i.e. the resulting solutions also include interpolations between the provided demon- strations. Furthermore, in the solution space: 1) Given a (sate-only) demonstration, find the closest policy capable of gen- erating this demonstration. 2) Search policies in the solution space that have high/low likelihood according to the provided demonstrations (i.e., similar to the provided demonstrations) 3) Find solutions that satisfy different constraints.
method allows
perform different types of post-hoc policy searches
2 BACKGROUND
2.1 IMITATION LEARNING AS DIVERGENCE MINIMISATION
In this section, we discuss why current Imitation Learning (IL) methods are incapable of dealing with multi-modal demonstration distributions: especially, when only a small set demonstrations is available. Zhang et al. 2020 Ke et al. (2020); Ghasemipour et al. (2020) have shown that current the Imitation learning (IL) methods can be derived as family of of f-divergence minimisation methods, where the divergence of the state-action distributions of the expert Pra.p (s, a) and learning
2

---

Under review as a conference paper at ICLR 2023
Mode-seeking
Mean-seeking
Figure 2:
Fitting a Gaussian to a multimodal distribution (blue) by minimising the reverse-KL (A, mode- seeking) and the forward-KL (B, mean-seeking)
agent Pare (s, a) is minimised using either Kullback-Lebler (KL) divergences or Jensen-Shange (JS) divergence. As KL divergence is not symmetric, both forward KL and reverse KL are used. And depending on the choice of the divergence, the optimisation yields different results (Nowozin et al,
2016
Ke et al. 2020 Ghasemipour al. 2020
(see Fig 2). Additional details in Appx. A..
Behaviour cloning (Pomerleau 1988). a supervised off-line learning approach, is one of the ex- amples that use forward KI, to optimise its policy. In practice, it either follows mean-seeking behaviour under a stochastic policy and assumes data sampled during training are balanced or ex- hibits mode-seeking pattern. As it is supervised learning method that lacks interaction with the environment, it often requires large amount of demonstration data in order to accurately estimate the true state-action distribution (Neu & Szepesvári 2009).
One example of using reverse KL is Adversarial Inverse Reinforcement Learning (Fu et al. 2017) which recovers both the reward function and the policy by formulating IL as min max opti- misation using Generative Adversarial Networky (GAN) (Goodfellow et al. 2014). Here, the generator serves as the policy network and a discriminator as the reward function. The opti- mised objective is the reverse KI. divergence, which can be expressed in terms of entropy H as: Epe, (s.a) [log Teap(s, (s, a]] - H(Psa (s, a)). In practice, it faces additional challenges, such as the am- biguity of the reward function (Ng et al. 2000; Arora & Doshi 2021) (i.e., many reward functions could generate the same behaviours). Thus, its policy learning counterpart, Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016) is more frequently used. This latter optimises the
IS divergence (Goodfellow et al. al.] 20014 Ghasemipour al. 2020 Ke et al. 2020) with the causal entropy as policy regularisation term.
Optimising the JS-divergence result in the same mode-seeking behaviour the reverse KL (Theis jet al. 2015). It further faces one of the most notorious problems the GAN-based approach, commonly known mode collapse (Srivastava et al. 2017 Bau et al. 2019), where the genera- tor collapses to produce only a small set of data samples (partial collapse) or even a single sample (complete collapse). which disregards the modli nature of the distribution. Furthermore, the adversarial optimisation scheme in GAN-based methods further imposes other challenges, such as instability the optimisation. It is often difficult to guarantee that both the generator and the dis- criminator converge optimally at the same time (Radford et al. 2015). It is common to observe that the loss of the discriminator converges very quickly to zero (Arjovsky & Bottou| 2017]] Xu et al.) 2018) that impedes the gradient updates of the generator. It often needs a considerably large number of data and fine-tuning or additional regularization optimisation of IL methods is convex if the actions deterministic and the demonstrations cover the complete state space (Neu & Szepesvári] (009). In practice, it is always difficult to make sure that the demonstrations span the entire state space; instead, they just cover one of the global optima which ensures the convergence of the solution.
order to stabilise the training process. The
2.2 IMITATION LEARNING FROM OBSERVATION ONLY
Towards ILfO, Torabi et al. (2018) proposes a new approach called BCO, where an inverse dynamics model learnt through interaction with the environment and policy is learnt from state observa- tions by using the action inferred by the inverse dynamics. However, as with most LfD) methods, BCO does not handle multimodal demonstrations. Similarly, another approach called ILPO (Ed- wards et al. 2019) learns a forward dynamics model and latent policy to circumvent the need for the actions; however, is limited to discrete action space. Latent policy learning from demon- strations has also been proposed by Ren et al.] (2020); Fujiishi et al. (2021); Wang al. (2022a). GAIFO (Torabi et al. 2019) extends GAIL to state-only observations by using a discriminator that discriminates the state transitions instead of state-action pairs from the expert. State-alignment
3

---

Under review as a conference paper at ICLR 2023
based Imitation Learning (Liu et al. 2019) uses state alignment to recover state sequences close to the demonstrations using local state alignment and global state alignment. While it shares some sim- ilarities with method (i.e., state distribution matching). it needs state-action pairs to pre-training their model. A similar work is proposed in Lee et al. (2019). InfoGAIL (Li et al. (017) combines InfoGAN (Chen et al. 2016) with GAIL to capture the variations of diverse demonstrations using interpretable latent factors. However, it is not a state-only approach and suffers, in addition, from the mode-collapse problem of GAN. (Gavenski the sampling-basd -bsed method to imitate unknown policies but with state-action demonstrations. The closest work is perhaps by Ghasemipour et al. (020), where they recover state distributions that preserves 2 solution modes using a modified reverse KL. However, this mul- timodality is due to the use of the stochastic policy(i.e. outputs mean and variance of a Gaussian distribution). Unlike our approach, it does not allow the post-hoc policy search we propose, there- fore, it cannot obtain a specific policy deterministically. Shafiullah et al. (2022) uses a transformer to learn multimodal demonstrations; however, it does not deal with state-only data. Similar examples include the diffusion model (Wang et al.] 2022b) for path planning that focuses on in-distribution data generation with access to the action set.
al. 2020) shares similarities with by using
policy from
PROBLEM SETUP
Consider a finite-horizon Markov Decision Process (MDP) (S, A. R,,, T), where C Rd, and A C Rda are the continuous state and action spaces, p(s' (s, a) is the state-transition probability, 8 and s' are the current and the next state, a is the applied action, Po the initial state distribution, E [0, the discount factor, R : S x A + E the reward function, and T the time horizon. We assume that this MPD has differentiable dynamics with a continuous states transition function S141 fe(st, at), such that d(st,88+1)
< 6, with d as distance metric, and 5 a small constant.
Let = {To, Th..... .n--1 be a set of state- traeetraaectooris (demonstrations) such that т; = (sh. s si..... ... s) E Rd, To E Txxd, The demonstration set D results from set of n hu- man policies (als)li 0 :0:00 n 1}. where the subscript h indicates that is a human policy, which is inaccessible for us.
We hypothesise that while all human policies maximise the same task reward Reask (s, a), each policy is constrained by a preference component. We define the preference component as the state distribution, such that different individuals prefer to visit different states while solving the task. Therefore, human policies are the results of solving the following constrained optimisation problem:
J arg max E
>' Brask am)
subject to DKLL (s)] <E
(2)
where (s) = p(s(T))) is the state distribution preferred by the individual i, and px the state distribution of the learning agent. As different individuals optimise the task reward function with different preferred states, the demonstration distribution can have multiple solution modes.
Our goal is to learn large set of policies capable of solving the same task in different ways by satisfying different preference components of the demonstrators, rather than learning a single way of mean-seeking policy). Thus,
solving the task (i.e., modee-sekingg policy the learning agent has access to a simulator to receive the task reward from the environment and to observe the true dynamics of the system. At the same time, receives the demonstrations ID, where it needs to optimise its policies for the different unobserved (as it is not explicitly defined) preference components from these demonstrations. In other words, we propose to solve the objective above by combining the optimisation scheme of RL with IL that uses divergence minimisation scheme.
propose setup where
4 SKILL GENERATION FROM BEHAVIOURALLE DIVERSE DEMONSTRATIONS
Our method consists of two steps: (1) latent space modelling from the demonstration. and (2) diverse policy generation on the latent space. In addition, our method allows us to perform post-hos solution searching this space in order obtain solutions (policies) that satisfy different constraints.
4

---

Under review as a conference paper at ICLR 2023
To find out large set of policies, one for each g:(+) per the KL constraint in Eq. to construct a latent space 2, such that any 9e(-) can be mapped onto that space. Now, we use projection function that maps two similar 9:(-)s closer to each other in the latent space Z. the KI. constraint in Eq. can be approximated using a Huclidean constraint in this space 2. More concretely, for some constant d E IR+ 2 2, and projection function Ene(-) the problem can be reformulated on the latent space 2 as
first need
m. = arg max (Brask [=]]
(3)
subject to || Bnc(TT) -z:l22
Practically, we want to find one policy for each 2, E 2. optimising the objective in Eq. where 2, are uniformly and densely distributed in the space 2. In other words, we want optimize policies for each niche so that the optimized policies maximise the task reward while producing state trajectories specified by own niche only. This a quality-diversity optimization problem (Cully & Demiris 2017): finding high-quality solutions according to cost/fitness function in a space that specifies the behaviour of the solutions. We solve this optimisation problem using a quality-diversity algorithm called MAP-Elite (Mouret & Clune 2015 Vassiliades al. 2017), where we determine the behaviour of the solutions as the latent preferences extracted from our projection function.
4.1 LATENT SPACE MODELLING FROM DENSITY ESTIMATION
We first model the latent factors that explain the diversity in the behaviours from the demonstrations. As mentioned in Sec. 3 we hypothesise that the diversity is caused by individual preference over the state visitations; i.e., different individuals prefer visiting different regions in the state space. A similar hypothesis has also been made in previous works such as max-margin planning (Ratliff al., 2.06), where the reward function is recovered by matching the sate-visitation frequency for every single state and the state-action feature vector. While they use single state visitation counts, propose to estimate the state region visitation frequencies as latent preferences.
A. State observations (Demos)
Density distribution (GMM)
Latent space
Figure 3: Generation of latent space from demonstration. state observations (A) are modelled by a
GMM (B). The latent space (C) is huilt hased on different combinations of mixture components.
Given the state distributions from the demonstrations p(s) defined in Sec we use Gaussian Mixture Model (GMM) of mixture components to model the density as p(s)
Σ (sN(ss)p oz), where de is the mixture weight for each mixture component. This state density p(s) models the state observations of all the demonstrations. Under the assumption of a continuous state transition,i.e. states close in time will be in the same state region (see Sec 3 each mixture component will cover a state region in the state space. The state distribution of each individual, Ic. belongs to a subset of the entire state distribution. As mixture component can be seen as a state region, different trajectories would have different state region visitations or assigned mix- ture components. Given fitted GMM model, can estimate the state distribution of each 9a (s() given its observations by a subset of mixture components according to their corresponding posterior probability or responsibilities p(k)ss) That is, given the sequence of states, we can estimate the responsibilities p(k]s) for these states using the fitted GMM, and use the corresponding mixtures to approximate its state distribution. Given the assigned mixture components (ky) }. and their assign-
ment frequencies (i.e., number of assignment for each component w.r.t the total number of states T) for (s). we can approximately define the density of g((s) using a new GMM model composed of these mixture components as (s(s) = Σ 1 (sl(ss ok2)) where the assigned mixture compo- nents correspond state region visitations, and the new mixture weight is readjusted based on their assignment frequencies 4.
5

---

Under review as a conference paper at ICLR 2023
Now, define projection function, encoder function Enc RTxx, R d2 (T x d,), based on the assignment of the mixture components. Given a set of states from a policy, the GMM is used components. There are different ways to represent this information. For instance, we can have an encoding of size K and give the frequency values of each component [fi, f2.-.--- fi.l. However, to limit the size of the encoding, we choose to take the indices of the most frequently assigned components without explicit referring the actual frequencies (detailed algorithm in Appx. Algo I]. The precision of the mapping depends on our choice of m as we will have information loss with small value of m (further discussion in Appx. A.5.2 and A.4).
unsupervised clustering method to classify these states into different mixture
Implicitly, we are defining the feature vector of a trajectory as the mixture components that are responsible for its states. The visitation frequency is explicitly taken into account by taking the top d- most assigned mixture components in descending order. In addition, the use of GMM allows us evaluate the likelihood of given trajectory according to the density distribution modelled from demonstrations. For instance, given the example of Fig belong to in-distribution data or out-of-distribution based on their likelihood (of the state sequence).
evaluate whether these trajectories
A
T2
Average likelihood 0.25
Average likelihood 0.78
Figure 4: Mixture components/clusters assignment (represented by coloured confidence ellipsoids of GMM) for the state distributions of trajectories Tı E R.00 and T2 E R A. Out-of-distrrbutiio trajectory т, with low average likelihood according to fitted GMM. B. In-distribution trajectory 72 with high average likelihood.
Solution matching We can measure the similarity of 2 trajectories by measuring their associated mixture components. As have the analytical form of each component (e.g., Gaussian with its mean and variance), we can directly measure the distance between their distributions by using mea- surement our choice, such as Bhattacharyya distance or KL divergence.
4.2 MAP-ELITES (EA) AS DYNAMICS-AWARY CONDITIONAL GENERATIVE MODEL.
Given Z as the latent space defined by the projection function, aim to generate solution archive P= parameterised by the latent encoding 2, where each entry in this archive has a unique encoding The goal is generate policies and map them into this space according encoding values for their respective g(-). While the latent space contains all the possible state distributions defined by the mixture components of the GMM, the solution space constrained by the system's dynamics, as the possible solutions are subject to the system constraints that cannot generate certain state distributions. Thus, P. C 2. As the solution archive the result of filling the latent space with policies, we may use latent space interchangeably to refer to the empty solution space.
EA can be seen as an implicit state density model over states with high fitness/reward (Murphy) 2023). In our case, as EA optimises each instance of encoding, which is associated (deterministi- cally) with a given mixture model, the generation this space equivalent to an implicit density modelling conditioned on the latent encodings. Thus, refer our method as conditional gen- erative model that generates policies conditioned on different encodings. The workflow is shown in 5 and the detailed algorithm in Appx. 2
As mentioned Sec. we use MAP-Elites as the algorithm of our choice. starts with sampling N random policy parameters from uniform distribution: 01=0:N11 Ua,,). Then it evaluates the policies in simulation using the reward function Reast().)). :). and at the same time generates the state trajectories T1= and their corresponding encodings 2-= Then the policies are inserted into the closest cells in the solution archive P. If two policies compete for the same cell, the one with higher reward occupies the cell. Once this initialization is done, MAP-Elite randomly selects policy from the archive, adds small random noise (mutation) to the parameters, and evaluates the policy on the simulator for the reward, state trajectory, and the corresponding encoding. This new policy is inserted into the archive if either the cell is empty the new policy has higher reward; the

---

Under review as a conference paper at ICLR 2023
Policy
SIMULATOR
Projection function
Initial random sampling
Latent encoding
Policy insertion
Trajectory
generation
State space
Parameter space
Policy selection
random variation
Latent space
Figure 5: Workflow of policy generation and allocation in the latent space. To initialise, the parameters of the policy (here we use a neural network) are sampled uniformly, and then the policies are executed on the simulator get the corresponding trajectories (set of states), in order to allocate the policies into the latent space according to their encoding. Then, in loop, policies selected from the archive, small variations added, and new encodings are obtained to insert them
archive based on their reward.
policy is discarded otherwise. The selection, variation, and insertion continue until the maximum assigned policy evaluation count is reached.
As the highest rewarding policies occupy the nearest cells in the archive, those policies produce maximally rewarding behaviour staying close the state distribution specified the cell. other words, the policies in the archive essentially optimise the objective in Eq. for different 2,.
Post-hoc policy search Having set of solutions parameterised by latent encodings allows to find solutions post-hoc manner that satisfy different constraints. For instance, solution archive built for 2D navigation environment with one obstacle contains solutions for an equivalent environment J with two obstacles, as the latter space more restricted, where J G (see example find the closest policy in our archive by finding a policy with the closest distribution distance (see Sec. 4.1 and Alg. according to the encodings.
in Sec. 5). In addition, given demonstration,
EXPERIMENTS
We present 3 experiments with continuous state and action spaces to show our method is capable of generating solution archive that optimises implicitly the density distribution from demonstra- tions. We propose (motion) planning examples where the task is to generate a non-colliding path from two points in end-to-end manner. The first is 2D toy path planning environment and the second a robotic arm motion planning environment using the Franka Emika robot with Py- bullet (Coumans & Bai
as the physic simulator. In both cases, position coordinates as the state observation. For Franka, we fixed the Z axis, as the demonstrations do not contain Z coordinates. Finally, we show a driving experiment modified (Leurent) 2018) as close-loop control example, using neural networks for a step-wise decision-making policy. The experiments are run times. The average performance and further ablation study (time, number of valid solutions vs encoding choices) can be found in Appx. A.4 The latent encoding is vector of 6 dimensions defined by the top most frequent mixture components responsible for the state sequence. A discussion of model selection and reproducibility is included in Appx. A.5.2 and A.3
only the XY
2D path planning We use 28 non-colliding trajectories with fixed lengths that are generated arti- ficially using Bézier curves shown Fig/6A. We use B-spline as the policy function, which takes 5 control points in total to generate flexible curve as parameters, where the first and last control points are start and end coordinates, and the resulting points are the parameters of the policy, O e TRo We use GMM of 20 components for the multimodal distribution and 10 for the single-mode dis- tributions. Under the exact same setup, we show that the control points are optimised towards state distributions modelled by their respective GMM. As shown in Fig 6 the resulting control points from the single-mode setup are scattered mainly in the region within single- mode state distribution.
While for the multi-moda setup, the control points scattered in both regions of the state space. This comparison is even more noticeable after filtering out the solutions with a higher likelihood (according their respective GMM models). We show that even though the demonstrations only

---

Under review as a conference paper at ICLR 2023
cover solution mode, method is still capable of discovering multimodal solutions, where the unseen solutions from the demonstrations will have lower likelihood values.
005
Figure 6: Solution archives optimised for different GMM models fitted to single mode and multimodal demos. A. Multimodal demonstration. B. Trajectories with GMM from multimedal demonstration with colour bar as their corresponding average likelihood. C. Control points for policies from B shown with their colour and size re-adjusted according to trajectories's ' likelihood. D. Trajectories from solution space build for the single-mode demonstrations (i.e. lower regions) with their average likelihood. E. Control points for the policies of Fig. D.
As VAE is commonly used as feature encoding method as well as a (conditional) generative model, show its capability of feature encoding and generation ability in this setup. We the same coding size, and trained the model for 30000 iterations (details Sec[A.3.1] As shown in Fig for small dataset, the generator suffers from partial collapse that only generates samples from certain modes and fails to produce diverse samples. The same observation occurs for the encoder, it fails produce unique encoding values as shown Fig. B, D). While training the same model using the data generated from our solution, both its capability of reconstruction and encoding improves considerably. However, unlike our projection function, it does not provide an interpretable encoding value. And it cannot take the advantage of the simulator that is available to interact with. Another alternative method GAN, but it does not work when a small amount of data provided, and suffers from the mode collapse even though large amount of data is available (see See 2.1) We show this latter using trajectories from our archive in Appx. A.2)
Franka arm manipulation We the same setup here as in the previous case, except the mixture component to model the state density increases to 25 as more data is available. We collected 140 demonstrations in real life using motion tracker track the movement of pointer draw the trajectories without considering the real dynamics of the robot. The length of demos spans between 240 and 726 time-steps. This 3D environment contains 3D obstacle, where the resulting trajectories are plotted in 2D space for visualisation purpose in Fig
Highway driving We demonstrate here that our method is capable of optimising high-dimensional parameters of neural networks, where 0 20 We collected 40 demonstrations of 151 time- steps using a manual controller from the simulator. Here the observations contain both the position coordinates and the rotation angles of the vehicle, E R4, the action IR is the steering applied to the vehicle to overtake the obstacle without going off the road as shown in Fig 9 We show the resulting behaviours when multimodal and single-mode demonstrations are used in Fig
Post-hoc solution search In this case, we show that we can still find valid solutions when the constraints in the environment change. Here, we add another obstacle the 2D path planning
A
D
E
type means Variaticss
type
type archiive demos original denos
30a
means variamccs
10
30a
LO
volues
valuus
VAE
Figure 7: VAE failed on the original demonstrations, but can be fully trained using trajectories from our method. A. 1000 samples from VAE trained on the original demonstrations, B. Histogram of the predicted encodings for the samples (f.e, mean and variance for the isotropic Gaussian). C. 1000 samples from VAE trained trajecto- ries generated from our solution. Histogram of the predicted encodings for corresponding demonstrations. Histogram of the unique encodings from both VAEs.
8

---

Under review as a conference paper at ICLR 2023
0.50
0.20
02-
010
-04
02
0.4
022
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Figure 8: A. Franka environment. B. Demonstrations of various lengths. C. Trajectories from the solution archive. D. Trajectories with their corresponding likelihood visualised with colour map.
1.00
1.00
1,00
0..7
0.75
0.75
0.25
025
0.25
A
B
0.000
0.00
D
0.00
Figure 9; method finds feasible driving styles that overtake the obstacle in different ways. Demonstra- tion. B. Trajectories from the multimodal solution archive with the likelihood values. C/D. Trajectories from the single-mode solution archives with their respective likelihood values.
environment. By simulating the policies in this modified environment, obtained 1738 valid tra- jectories as shown Fig 10 Notice here that, we can observe more solution modes as results of interpolation between the two existing solution modes from the demonstrations. It is also possible to obtain a set of trajectories close to a given demonstration by matching the latent encodings (see Sec. 4.T in the solution archive (see Fig TOP. Here compare the result of encoding matching using VAE trained on demonstrations generated by our method (see Fig 7C) and the result from our proposed projection function (more details Appx. A.3.2). The results show that our projec- tion function enables better matches, as we can measure the distances between the underlying state distributions based on the encoding values (see Sec. 4.1.
A. New constraints
B. Our matching
VAE matching
D. Our matching
E. VAE matching
polcca demeescluutore
CoRnE polidoe
disset
Sooret pilicie
Figure 10: Post-hoc policy searching in the solution archive to find policies that satisfy different constraint and to find policies for given demonstration. A: 1738 valid solutions for new obstacle in the solution archive 6b. B/D: 5 closest policies to given demo using proposed projection function with GMM. C/E: closest policies using VAE encodings.
CONCLUSIONS
In this paper, we propose novel framework that allows build solution space where we gen- erate a diverse range of behaviours from a small set of state-only observations. We show empirically that our method optimises all the policies in the space towards different state density constraints while optimising the task reward. With our proposed projection function that encodes state transi- tions into meaningful statistical encoding, we can perform a post-hoc policy search the projected space searching for different policies. Our experimental results show that we are capable of generat- ing multimodal solutions deterministically beyond the provided demonstrations. Further discussion regarding its application use is included in Appex. A.8

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative ad adversarial networks. arXiv preprint arXiv: 04862, 2017.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, meth- ods and progress. Artificial Intelligence, 297:103500, 2021.
Monica Babes, Vukosi Marivate, Kaushik Subramanian, and Michael Littman. Apprenticeship learning about multiple intentions. In Proceedings of the 28th international conference on ma- chine learning (ICMI,-I)), pp. 897-904. Citeseer, 2011.
David Bau, Jun- Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Seeing what gan cannot generate. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4502-4511,2015
Konstantinos Chatzilygeroudis, Vassilis Vassiliades, and Jean- CannBBapiiste Mouret. Reset-free trial- and-error learning for robot damage recovery. Robotics and Autonomous Systems, 100:236-250, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, 2016-2021.
robotics and machine learning. http://pybullette
Antoine Cully and Yiannis Demiris. Quality and diversity optimization: A unifying modular frame- work. IEEE Transactions on Evolutionary Computation, 22(2):245-259, 2017.
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. Robots that can adapt like animals. Nature, 521(7553):503-507, 2015.
Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent policies from observation. In International conference on machine learning, pp. 1755-1763. PMLR, 2019.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint 1rXiv:1710.112444 2017.
Hidehito Fujiishi, Taisuke Kobayashi, and Kenji Sugimoto. Safe and efficient imitation learning by clarification of experienced latent space. Advanced Robotics, 35(16):1012-1022 2021.
Johannes Fürnkranz and Eyke Hüllermeier. Preference learning and ranking by pairwise compari- son. In Preference learning, pp. 65 02-822.55 Springer, 2010.
Nathan Gavenski, Juarez Monteiro, Roger Granada, Felipe Meneguzzi, and Rodrigo C Barros. Imi- tating unknown policies via exploration. arXiv preprint arXiv:2008. 056660 2020.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Conference Robot Learning, pp. 1259-1277.
PMLR, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Rituraj Kaushik, Pierre Desreumaux, and Jean-Baptiste Mouret. Adaptive prior selection for
repertoire-hased online adaptation in robotics. Frontiers in Robotics and AI, pp. 151, 2020.
Rituraj Kaushik, Karol Amdt, and Ville Kyrki. Safeapt: Safe simulation to-aaal robot learning using diverse policies learned in simulation. IEEE Robotics and Automation Letters, 2022.
10

---

Under review as a conference paper at ICLR 2023
Liyiming Ke, Sanjiban Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization. In International Workshop on the Algorithmic Foundations of Robotics, pp. 313-329. Springer. 2020.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdi- nov. Efficient exploration via state marginal matching. arXiv preprini (rXxv::906.05274, 2019.
Edouard Leurent. An environment for autonomous driving decision- maaiing. https://githa com/ www.eeeeetttgwwweeeeeeeeeeeeeeee
2018.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. Advances in Neural Information Processing Systems, 30, 2017.
Fangehen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State slignment-based imitation learning. arXiv preprint 1xXiv/991..0944 2019.
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint 1rXiv:1504.04900 2015.
Kevin P. Murphy. probml. ai
Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. URL
Gergely Neu and Csaba Szepesvári. Training parsers by inverse reinforcement learning, Machine learning, 7722)330--337, 2009.
Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, vol- ume 1, pp. 2, 2000.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06444 2015.
Nathan D Ratliff, Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In Proceedings of the 23rd international conference Machine learning, pp. 729-736, 2006.
Allen Ren, Sushant Veer, and Anirudha Majumdar. Generalization guarantees for imitation learn- ing. arXiv preprint arXiv:2008.019)) 2020.
Stefan Schaal. Learning from demonstration. Advances neural information processing systems, 9, 1996.
Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning SkS modes with one stone. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
URL httpppss//ppeeeeeiee...... net. forum? www.rooooommiinooooeaa dd
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. Advances in neural informa- tion processing systems, 30, 2017.
Wen Sun, Anirudh Vemula, Byron Boots, and Drew Bagnell. Provably efficient imitation learning from observation alone. In International conference on machine learning, pp. 6036-6045. PMLR, 2019.
Ajay Kumar Tanwani, Andy Yan, Jonathan Lee, Sylvain Calinon, and Ken Goldberg. Sequential robot imitation learning from observations. The International Journal of Robotics Research, 40 (10-1 (10-11): (00111)1306-1325, 2021.
11

---

Under review as a conference paper at ICLR 2023
Lucas Theis, Aäron den Oord, and Matthias Bethge. A note the evaluation of generative models. arXiv preprint arX1v:1511.0184,, 2015.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint 1rX5v:1805.011 00954, 2018.
Faraz Torabi, Garrett Warnell, and Peter Stone. Adversarial imitation learning from state-only demonstrations. In Proceedings of the 18th International Conference Autonomous Agents and MultiAgent Systems, pp. 2229-2222222222 2231, 2019.
Vassilis Vassiliades, Konstantinos Chatzily and Jean- Bann-Battttte Mouret. Using centroidal voronoi tessellations to scale up the multidimensional archive of phenotypic elites algorithm. IEEE Transactions on Evolutionary Computation, 22(4):623-630 2017.
Tianyu Wang, Nikhil Karnwal, and Nikolay Atanasov. Latent policies for adversarial imitation learning. arXiv preprint arXiv:2206. 11299, 2022a.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.061 0000800669.... 2022b.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 1316-1324, 2018.
Xin Zhang, Yanhua Li, Ziming Zhang, and Zhi-Li Zhang. f-gail:
Learning f-divergence for gen- erative adversarial imitation learning. Advances neural information processing systems, 33: 12805-12815, 2020.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 ALGORITHMS
Algorithm State region visitation frequency
function ENcODE(gmm model, state sequence, size descriptor)
2:
list components=gmm model.predicct(staee sequences) # Predict the list of mixture compo- nents responsible for each state
3:
nnmque_componett, n_counts unigue(liss.coopmnenns
4: comp.indices = www.ppoonetuuuuuuqqeu..coooooooa n.counts, size_descriptor)
5: return comp_indices
6: end function
Algorithm Solution archive generation
function ARCHIVE BRCHIVE.GENERATTE Www.GGEEEEEEEEnnn.... CEENER.ATEgmmmmmolee model, size .ezecripct)))
2: Initialise an instance of policy π, solution archive P. reward archive R, number of iteration
N. number of initial random samples M
3:
while < .M do
4:
unifurmmsaanaae (trtoomm.amppin((.... sample the parameter vectors from uniform distribution.
5:
Te policy.set policy.set.parameteoe parameter(z, ()
6:
Т, Ttask = =simulate(Tp))
7:
encode(gmm model, size descriptor)
8:
P(z) t- 0
9:
R(z) Ptask
10:
end while
while < do
12:
= ramdom solution select(P)
#' mutate(0) Gaussian noise injection to the parameters. Ptask =imulate(TTo))
14:
15:
= encode(gmm model, т, size descriptor)
16:
if Pzz) 0 or R(z) < Ttask then
17:
P(() 8'
18:
R(x) Trask
19: end
20:
end while
21:
return P, R
end
function
A.2 KL DIVERGENCE MINIMISATION FOR MULTIMODAL DISTRIBUTION
As KL divergence is not symmetric, the reverse KL and forward KL exhibit very different be- haviours. These are also known as Moment projection and Information projection respectively.
Given the state-action observations from the demonstrations from the expert's policy Teaps M- projection minimises the following forward KL:
JØ arg minDmiDer kI. (pr.e) (s, |l Pro (s,
DKL f Prusp (s, a) log
Preep (3, d(s,a) p (s.a) log
Prezp (r.aa(s,.))
)
(4)
Pro (8,
Pxp (x,(5,
While the I-projection minimises the following reverse KI.:
13

---

Under review as a conference paper at ICLR 2023
Th = arg minDRKE (P) Pao a))
(5)
DRKL = f Pro a) log
(rean(s,
Prtaa (3,
Pro (8, )
d( a) = Epng (s,a) (ro(s,a)
log
For M-projection, the difference between Preep (s, a) and Pno (s, a) is weighted by Prezp (s, a). Which means that when Pr..,y (8, a) = O, the discrepancy of Pro (8, a) > 0 from Pracy (8, a) will be ignored. Mathematically, this means that Preay (8,a) will typically over-estimate the support of Px, (s, a), due to Px..p (8, a) > whenever Pxo > to make sure KI. divergence stays finite. As result, the projecting exhibits mean-seeking, also known moment-matching, behaviour that averages over several modes given a multimodal distribution. While it avoids the low probability assignment of the region with data (e.g., the other mode of the distribution), it inevitably assigns probability mass to non-data region (Theis et al.) 2015) as shown in 2
For I-projection, difference between Prezp. (s, a) and Pro (s, a) is weighted by Pro (s, a). PHs. (s. a) will typically under-estimate the support of Pry and concentrate on one of its modes that results in mode-seeking behaviour. This is due to Proup (8,a) sure KL divergence stays finite. Using this projection is not straightforward as we might only have access to samples from the distribution rather than the data density itself (Murphy (023), where M-projection can be done easily by maximising the average log- ligelikei with respect to the given training dataset as an equivalent way of minimising the KL.
O whenever Pr, (s, a) 0 to make
A.2.1 EXPERIMENTAL ANALYSIS
Given the multi-mode and single-mode demonstrations of the 2D path planning environment, we proposed in Sec. [ We show the results of Gaussian Mixture Regression (GMR) model, build on the same GMMs we used for our method (see A.44). While it performs well in the single-mode demonstrations, the result from the multimodal distribution exhibits mean-seeking behaviour as shown in Fig
onridenca
contie
Figure 11: Results of GMR on multimodal demonstrations and single-mode demonstrations. Left: Prediction with 95% confidence interval from a GMR trained on multimodal demonstrations. Middle/L.eft: GMR predic- tions on single-mode demonstrations.
Extending the discussion in Sec 2.1 about GAN in dealing with multimodal distribution. We show experimentally the performance GAN in generating trajectories for the 2D path planning environ- ment. As GAN cannot deal with a small dataset, we trained it using the trajectories generated by our archive, which contains 3881 trajectories. The results are shown Fig 12 The discriminator takes the entire trajectory as the input and has 3 hidden layers with 128, 32, and 8 neurons respectively. The generator takes an input of a 7-dimensional vector and uses hidden layers with 128, 64, and 32 neurons respectively to generate the trajectory of the same size. The model is trained for 23000 iterations. The results are consistent with mode-seeking behaviour where converges to one of the solution modes discussed in See 2.1
A.3 EXPERIMENT SETUPS AND REPRODUCIBILITY
We train all the GMM models using sklearn library with fixed random state ensure reproducibil- ity. The experiments were run times per setup (e.g., multimode and single-mode demonstrations), with 2 different types of encodings. We show the average result for songle-mode demonstrations and
14

---

Under review as a conference paper at ICLR 2023
Figure 12: A: Samples from GAN after 3000 iterations. B. Samples from GAN after 15000 iterations. C. Samples from GAN after finished training. D. Training loss the discriminator and generator during training.
Experiments
iterations
Real time
CPU time
Planning (multimodal)
105
3m49s + 1m6s
13mls = 0m28s
Planning (upper)
10s
2m33s + 0ml 22
11m5ls + 1m18s
Planning (lower)
105
2m27s + 0m44s
12m3s + 0m21s
Franka
106
6h49m + 11m57s
6h49m + 11m57s
Highway (multimodal)
10a.
2h46m + 4m19s
13h29m = 36m
Highway (upper)
5 X 103.
1h54m + 0mm66
12h35m = 4mm4s
Highway (lower)
5 x 109.
1h26m + 11157s
12h36m + 44229s
Table I: Training time
multimodal demonstrations. most of the experiments allow us run parallel threads. We record both the average true clapsed time and the average CPU time, with their standard deviation. The sults are shown in Table ! For 2D path planning, we use Intel Core i7-6850K CPU @3.60GHz with 6 cores and trained our model using 12 parallel threads. For Franka, with I CPU core of Xeon Gold 6148 @2.40GHz, and 16GB RAM without multi molti-threading due the high memory demand from the physical simulator. For the highway environment, we cores of Intel Xeon Gold 6248 @ @250GGHz with 12 parallel threads and 8GB of RAM.
single thread process
In the highway environment, respectively as policy. The total number of parameters is 201 elements. The parameters of the networks are purely optimised using random sampling and perturbation from EA without any back- propagation. As we use the exact setup from 2D path planning in the Franka environment to show the applicability of our methods in "real" applications besides the 2D path planning experiment, we only test the result in the multi-modal setup. To ensure maximum reproducibility, the solution archive initialised with 104 solutions with uniformly sampled parameters from a given parameter space to minimise the advantage of good initialisation. The parameter space for 2D path planning defined within the range of [-5, 5]. while for Franka it is defined within [0, 1] and for highway within [-1, 1].
neural networks with 2 hidden layers of 8 and 16 neurons
As we tested 2 different types encoding, we report the performance results in the ablation study (see Appx. A.5), where we discuss the impact of the size of the encodings vs the size of the solution archive, among the others.
A.3.1 SETUP FOR VAE
We use VAE with 5 hidden layers with 128,128,64, 64,32 neurons correspondingly. Unlike projection function, the VAE cannot handle data of different sizes, instead, it receives the full tra- jectory with fixed length. Thus, it cannot deal with the demonstrations intended for Franka Arm environment, as they have various sizes.
A.3.2 POLICY SEARCH USING ENCODING MATCHING
Given demonstration, search for the closest policy by finding the with the minimal distance in the encoding space. The algorithm can be found in Alg. 3
To compare the result of VAE with method, the samples generated by VAE (see Fig. 7
15

---

Under review as a conference paper at ICLR 2023
Algorithm 3 Policy searching
1: function POLICY SEARCH(archive selllaarrcivepolliieeeee archive ieciivveencdings.. encoding matches)
2: Initialise an array distances
3:
for encoding in archive archive_encoding: do
4:
Dosstance_uunction(nnne
encoding.demo)
5:
D d
6: end for
7: closest_policies top_n www.eloess...llleceeeee potiesstpplicceeeaacheeppppict, D, n_matches)
8: return closest.policies
9:
end function
A.4 MODEL SECTION OF GMM
The model selection, i.e. the number of GMM components used to model the state density distri- bution from the demonstration, determined by Bayesian Information-thoonm choose the model with lower BIC the common practice. The GMM used for the 2D path planning environment is shown Fig 13 using confidence ellipsoid. The GMM used for the Franka envi- ronment is shown in Fig 14 We omit the visualisation for the highway environment, Gaussian which cannot be visualised with confidence ellipsoid.
criteria (BIC). We
4I)
Figure 13: GMM models used for 2D path planning environment. Left: GMM confidence ellipsoid for the multimodal demonstrations. Middle: GMM confideuce ellipsoid for the single-mode demonstrations (upper region). Right: GMM confidence ellipsoid the single demonstrations (lower region).
Figure 14: GMM model for the multimodal demonstrations recorded using motion tracker.
A.5 ABLATION STUDIES AND NUMERICAL. QUANTIFICATION SOLUTION ARCHIVE
In this section, we analyse the performance of our method by analysing the resulting solution archive generated by our method. We include as the performance indicator the total number of policies generated and the total valid solutions that reach the given task reward, i.e., reaching the goal while preventing the collisions. We report the results under different setups.
A.5.1 COMPARISON OF DIFFERENT ENCODINGS
We compare 2 different type of encoding: 1) Encoding with the top most frequently assigned
mixture component discussed in Sec 4.1
shown in Table 2) Same encoding but with an
16

---

Under review as a conference paper at ICLR 2023
additional likelihood value added. This latter is proposed to avoid the case of having two different iraeectories with the same encoding but with different likelihood values (i.e.. outto--distribution trajectories that happened to be mapped to the same encoding), shown in Table [ In the case of the point environment, the inclusion of likelihood does not increase the size of the final solution archive. We hypothesise that this is caused by the segmentation of the solution space, the average likelihood is low in general for all the trajectories (see the likelihood values for the trajectories in Fig ()) it worsens the uniform segmentation of the space. In reality, while different trajectories can have the assigned mixture components, their respective frequencies may rarely coincide. For instance, in Fig(4. two trajectories can have the same assigned component with low and high average likelihood respectively, but their resulting encodings are different due to the different frequency values. Thus, these trajectories will not compete for the same niche in the solution space. While in the case of highway environment, the inclusion of the likelihood does increase the size of the descriptor as the average likelihood values are higher, which span within [0, 1] (see the colour map for the trajectories in Fig D). which allows a uniform segmentation of the archive space.
Experiments
Demos's size
Encoding/GMM"
Total solutions
Valid
Planning (multimodal)
28 x 50
6/20
7703 + 20
6584 + 38
Planning (upper)
14 x 50
6/10
2329 + 47
1999 +
Planning (lower)
14x 50
6/10
3920 + 29
3314 + 34
Highway (multimodal)
40 x 151
6/20
3900 + 59
1009 + 42
Highway (upper)
20 X 151
6/10
1657 = 33
464 + 16
Highway (lower)
20 x 151
6/10
1187 + 14
261 -
Table 2: Number of demonstrations archive solutions with encoding without likelihood.
"Number compo-
nents for the encoding total number of mixture components.
Experiments
Demos's 's size
Encoding/ *MMM
Total solutions
Valid
Planning (multimodal)
28 50
7/20
3810 + 16
3554 + 6
Planning (upper)
14 x 50
7/10
2685 12
2415 + 15
Planning (lower)
14x 50
7/10
3761 +
3270 +
Franka
140 x (various)
7/25
8581 + 24
6948 + 48
Highway (multimodal)
40 151
7/20
4781 + 23
1116 25
Highway (upper)
20 x 151
7/10
1919 + 64
515 = 25
Highway (lower)
20 151
7/10
1237 40
260 = 11
Table 3: Number of demonstrations archive solutions with encoding that has likelihood as additional dimension. + Herr we have mixture components plus one value for the average likelihood.
Approaches as Tanwani et al. (202)) has used Hidden Mami-Markov Models extract the temporal sequence of the demonstration for imitation learning, which takes the transition probability between different mixture models into account. Our encoding does not consider the transition probability
between different mixture components; the use of Hidden Markov Models (HMM) could be done as a potential future work.
A.5.2 COMPARISON OF DIFFERENT CHOICES FOR GMM
Encoding space dimensionality. Given a GMM with dymn components and an encoding size of dy. The number of possible encodings is defined by different permutation of dymm its size has a theoretical upper bound of (dymm
dymm! ". In practice, its real upper bound depends the system's
dynamics, e.g., it may be impossible to have a state distribution with only different state regions that are far away from each other the state transition is continuous.
As discussed, the size of the latent (encoding) space depends on the number of mixture components that represent different state region visitation and the size of the deseriptor to choose. Here show the result of using 2 different GMMs with 10 and 5 mixture components respectively that have higher BIC scores compared to the GMM with 20 components (see model selection in A.4
17

---

Under review as a conference paper at ICLR 2023
For the GMM with 10 components, keep the size of descriptor unchanged, i.e., 7, while for the GMM of 5 components, we reduced it to 4 as the total number of components is reduced. For the former case, the number of total solutions is reduced to 1163 trajectories with 854 valid trajectories on average for the former case. While for the latter case, the sizes reduces to 120 total trajectories and 105 valid trajectories respectively. This is due the fact that a lower number of mixture components will reduce the size of the latent space. The result is shown in Fig 15
0.20
0.20
0.15
0.15
0.10
0.10
0.05
0.05
Figure 15: Comparison of solution archives optimised with different size of GMM mixture components in the projection function. Left: Mixture component with 10 components, and the resulting solution archive with valid trajectories shown with colour map that corresponds to their respective likelihood values. Right: Mixture component with 5 components and the associated solution archive with valid trajectories with their respective likelihoods.
A.6 POST-HOC SOLUTION SEARCH FOR CHANGING SIMULATOR
In this section, we show that the solution archive trained on the toy 2D path planning environment with 2D obstacles contains valid solutions for the Franka arm environment with 3D obstacles. Where the former does not have any physic engine or accurate collision detection mechanism; while the latter is high-fidelity simulator with a real physic simulator and sophisticated collision detection. And the state space for the 2D environment is defined to be the same as the Franka environment.
We configure the toy environment to have the approximate location of the obstacle. In the case of the Franka environment, the obstacle is a 3-dimensional cube instead of a 2D obstacle (see Fig A).. Thus, the collision detection of the toy environment is far from perfect compared to the Franka environment. We use the same parameters set up as detailed in Sec 5 and Appx. AA.3 The GMM is fitted the demonstrations recorded for the Franka arm (see Fig 14 Training
simplified simulator has its advantage such as faster convergence speed; reduces the training time as the simulation time is faster in the toy environment. As we see from Table [1 the training time for Franka (6/49m) is considerably longer than the 2D environment (3m49s) due to the slow physic simulation. By allowing access the simulator, we can filter out solutions by simulating the policies from the trained solution archive in the environment to validate them under different constraints. In this case, the constraints are different due to the change in the environment. The final solution archive trained in the 2D toy environment contains 9076 solutions, with 5978 valid collision- solutions for the Franka environment that reach the goal location.
A. Toy simulator
B. Pybullet simulator
0.5
-0.5
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.2
0.3
0.4
0.5
C.6
C.7
0.8
Figure 16: Solution archive trained using toy environment deployed in the 3D Franka Arm environment with Pybullet engine. The resulting trajectories are plotted in 2D space for better visualisation. A: Valid solutions for the toy environment with 2D obstacles without physic engine. B: Valid solutions Franka Arm environment with Pybullet as physic engine with 3D obstacle collision detection.
18

---

Under review as a conference paper at ICLR 2023
A.7 ADDITIONAL MULTIMODAL EXPERIMENT
In addition to bimodal demonstrations for the 2D path planning in Sec. 4.1 nario where deal with more solution modes. Here we use a total of 12 demonstrations generated using sine and cosine functions and used the same parameters' setup as the previous experiment. The GMM takes 20 mixture components, and we use a 6-dimensional encoding value. The resulting solution archive a
we include a new sce- solution modes are present in the demonstrations show that our method can
contains 6309 trajectories as shown Fig 17 While the solution archive built from bimodal demon- strations contains also multiple solution modes such as interpolation results, these demonstrations present with lower likelihood values as shown by its colour map (see Fig 6 I0AA. However, in this four-mode solution case, can observe clearly that the intersecting trajectories have higher likelihood values due to the presence of data points in the demonstrations.
004
-..2
Figure 17: A: GMM fitted to demonstration B. 12 multimodal demonstrations with 4 solution modes. C. Valid solution archive with trajectories colour with their respective trajectory likelihood.
A.8 POTENTIAL APPLICATIONS
We dedicate this section to discussing the applicability of our method. One of the potential applica- tions data augmentation. Given small set of state-only demonstrations, our method is capable generate diverse state-action demonstrations. These can be used, for instance, to provide more data input to other approaches. One example shown in the performance analysis of VAE in Fig 7 VAE suffers from mode collapse when it is trained using a small dataset. However, our method can gener- large set of demonstrations train the VAE to produce more diverse samples. However, notice here that in this case, VAE does not necessarily extrapolate beyond the provided demonstrations.
In practice, more useful use case is the recollection of visual demonstrations in high-fidelity environment. For instance, as shown from our Franka Arm environment, we are capable to generate a large set of demonstrations. While our approach currently does not support image-based input, can be used to generate training data for image-hased methods by recording the resulting trajectories in the simulator.
Another potential application is the search for policies that satisfy different constrains, such con- straints from different environments or policies that are capable of producing certain behaviours using encoding matching, shown in Appx. and Sec. 5
19


Reviews:
[{"review_id": "ov-FsPdh5Dn", "reviewer": "Reviewer_Vtfi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a learning from demonstration approach to generate diverse policies representing the different preferences of the expert. The authors seek to learn a GMM in the latent space, and match the state-visitation frequency from state observations only. Experiments on simulated reaching tasks with robot arm and car driving suggest the feasibility of the approach.", "summary_of_the_review": "This paper presents a learning from demonstration approach to encode and decode diverse expert demonstrations. The problem is of interest to a wider community. Consolidating the problem formulation further, quantifying the performance in the experiments and comparison with other approaches will improve the quality of the paper.", "strength_and_weaknesses": "- Learning from demonstrations is an active area of research, and of interest to the wider community. Encoding and generating diverse behaviors from demonstrations are important from both what to imitate and how to imitate perspective.\n\n- Problem formulation of the paper is similar to apprenticeship learning (or inverse reinforcement learning) under different preferences of the expert. In the problem formulation, the reward function needs to be parameterized in the preferences that the expert demonstrations are looking to encode. Eq.3 objective of matching state-visitation frequency under different preferences of the expert has also been well-studied in discrete spaces. Moreover, ,ean and mode seeking LfD is well-known in literature.\n\n- In continuous spaces, the hidden markov model/hidden semi-markov model is better suited to encode the temporal sequence in the state observations, as compared to the GMM (see [1]). Subsequently, Viterbi decoding can be used for generating diverse demonstrations.\n\n- There are no comparisons with other behavior cloning approaches to establish the merits of the approach. Moreover, there is no numerical quantification between expert demonstrations and generated demonstrations to quantify the performance.\n\n[1] Sequential robot imitation learning from observations, IJRR, 2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lZRKPbnDxUU", "reviewer": "Reviewer_KHmd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel algorithm to learn diverse behavior styles from demonstrations. It uses a projection function to encode demonstration trajectories into latent spaces and then uses MAP-ELITES to find a set of possible policies. Experiment results how the proposed model is able to learn a better coverage of different behaviors.", "summary_of_the_review": "The paper proposes a novel framework and the experiment results clearly show its capability to learn diverse behavior styles. ", "strength_and_weaknesses": "Strength:\nThe approach or the combination of approaches is novel and the experiments clearly show its capability to learn from diverse demonstrations.\n\nWeaknesses:\nWould be better to have more ablation studies on the algorithm design choices, e.g. compare some different projection methods, as well as different generative models.\n\nWould make the results more convincing by comparing with more than one baseline, especially other approaches that are designed for multimodal behaviors (e.g. VAE with multimodal latent space distributions).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ey8G2MYCWK", "reviewer": "Reviewer_NygL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This study built a parameterized solution space to help model driving behaviors for planning, which considering the effect of individual differences of driving demonstrations. ", "summary_of_the_review": "1. How about planning efficiency? Planning time consumption is vital to driving safety, and generally, an improper solution space could be a huge burden for computation speed. Some comparisons with some classic planning methods may be helpful.\n2. How to define the feasibility of solution space? Because there are always some detrimental demonstrations existing. Maybe a not suitable space can cause an undesirable outcome. The authors can make this part clearer. \n3. For a Post-hoc solution search (Fig. 10), a set of trajectories close to the demonstration are obtained, but the relative extent is not discussed. A comparison between similar methods can make your contributions outstanding.\n4. The authors proposed a method to find a potential solution from demonstrations. However, the authors should elaborate the application values. For example, learning from demonstrations in auto-driving technologies usually serves for human-like driving or driving safety. However, the treatment proposed in this study seemingly discusses a new approach without telling us what its meaning is. Following the descriptions of this study, the advantage is to find a solution through learning from demonstrations, but there have been many ways that can complete this. Maybe application potential and corresponding validations can help to make your contributions more valuable.\n5. Reproducibility is unknown because the codes are not open.", "strength_and_weaknesses": "Strength: writing and novelty\nWeakness: Reproducibility and application potential \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZK9tK05ul7", "reviewer": "Reviewer_9jjm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method that learns a set of policies that can cover all modalities of the demonstration dataset. The proposed method first models the state distribution of the demonstrations using a Gaussian Mixture Model (GMM). In this paper, a trajectory is represented with indices of the top $d_z$ mixture components that best describe state distributions of the trajectory. Then, this paper searches for policies that best represent each of the trajectory representations using an evolutionary algorithm, which randomly samples policies, rollouts the policies, encodes resulting trajectories into trajectory representations, stores the best policy for each trajectory representation, and iterates this process. Since each trajectory representation holds a policy that generates the corresponding trajectory, this method can achieve a set of policies that generate multi-modal trajectories. The experiments show that the proposed method can generate bimodal trajectories. ", "summary_of_the_review": "This paper tackles an important problem of learning multi-modal behaviors from demonstrations. However, the proposed approach does not seem to scale and the evaluation is done on too simple tasks to show its impact and practical use. Thus, I am recommending rejecting this paper.", "strength_and_weaknesses": "### Strengths\n\n* This paper tackles an important problem of learning policies that perform multi-modal behaviors.\n\n* The experiments demonstrate that the proposed method can acquire policies that correspond to diverse behaviors.\n\n\n### Weaknesses\n\n* The proposed method uses a Gaussian Mixture Model to approximate the state distribution. This may work on a small state space but would not scale to a high-dimensional state space, which is more practical. Moreover, this does not entail temporal information of the trajectory, which can limit its applicability to complex behaviors (e.g. loop).\n\n* There are some alternative approaches to achieve multi-modal behaviors without need for the simulator and task reward [a,b]. Discussion about comparisons to [a,b] would be great.\n\n* The tested environments and tasks are very simple -- 2D path planning with bimodal trajectories.\n\n* The experiments tested on datasets with only two modes. It would be great to verify the proposed method with more modes.\n\n* Instead of comparing with vanilla VAE, a comparison to VQ-VAE or VAE with a discrete latent space could be more appropriate since a quantized embedding in the proposed method can be the main source of generating multi-modal trajectories.\n\n* The paper claims that the proposed projection function provides the interpretable encoding value. However, the encodings shown in the paper are interpretable mainly because the experiments are done in 2D path planning. The proposed encoding will not be interpretable with the high-dimensional observations.\n\n* The proposed method assumes access to a simulator and a task reward, which limits its scalability to many applications.\n\n\n\n\n[a] Shafiullah et al. Behavior Transformer: Cloning k modes with one stone. NeurIPS 2022\n\n[b] Wang et al. Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning. arXiv 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject