Information Maximizing Curriculum: A Curriculum-Based Approach for Learning Versatile Skills
Authors: Denis Blessing
Onur Celik
Xiaogang Jia
Moritz Reuss
Maximilian Xiling Li
Rudolf Lioutikov
Gerhard Neumann
Venue: NeurIPS 2023 poster (2023)
Keywords: Imitation Learning
Verstile Skill Learning
Curriculum Learning
TL;DR: The paper proposes a novel algorithm for training mixture of experts models for versatile skill learning.

Abstract: Imitation learning uses data for training policies to solve complex tasks. However,
when the training data is collected from human demonstrators, it often leads
to multimodal distributions because of the variability in human actions. Most
imitation learning methods rely on a maximum likelihood (ML) objective to learn
a parameterized policy, but this can result in suboptimal or unsafe behavior due
to the mode-averaging property of the ML objective. In this work, we propose
Information Maximizing Curriculum, a curriculum-based approach that assigns
a weight to each data point and encourages the model to specialize in the data it
can represent, effectively mitigating the mode-averaging problem by allowing the
model to ignore data from modes it cannot represent. To cover all modes and thus,
enable versatile behavior, we extend our approach to a mixture of experts (MoE)
policy, where each mixture component selects its own subset of the training data
for learning. A novel, maximum entropy-based objective is proposed to achieve
full coverage of the dataset, thereby enabling the policy to encompass all modes
within the data distribution. We demonstrate the effectiveness of our approach on
complex simulated control tasks using versatile human demonstrations, achieving
superior performance compared to state-of-the-art methods.

Information Maximizing Curriculum: A Curriculum-Based Approach for Imitating Diverse Skills
Denis Blessing "t Onur Celik Xiaogang Jiat Moritz Reuss Maximilian Xiling Lif Rudolf Lioutikor* Gerhard Neumann 18 t Autonomous Learning Robots, Karlsruhe Institute of Technology Intuitive Robots Lab, Karlsruhe Institute of Technology FZI Research Center for Information Technology
Abstract
Imitation learning uses data for training policies solve complex tasks. However, when the training data is collected from human demonstrators, it often leads to multimodal distributions because of the variability in human actions. Most imitation learning methods rely on a maximum likelihood (ML) objective to learn a parameterized policy, but this can result in suboptimal or unsafe behavior due to the mode-averaging property of the MI objective. In this work, we propose Information Maximizing Curriculum, a curriculum-base a weight to each data point and encourages the model to specialize in the data it can represent, effectively mitigating the mode-averaging problem by allowing the model to ignore data from modes it cannot represent. To cover all modes and thus, enable diverse behavior, we extend our approach to mixture of experts (MoE) policy, where each mixture component selects its subset of the training data for learning. novel, maximum entropy- objective is proposed to achieve full coverage of the dataset, thereby enabling the policy to encompass all modes within the data distribution. We demonstrate the effectiveness of our approach on complex simulated control tasks using diverse human demonstrations, achieving superior performance compared to state-of-the-ar methods.
approach that assigns
Introduction
Equipping agents with well-perforning policies has long been prominent focus in machine learn- ing research. Imitation learning (IL) (1 offers a promising technique to mimic human behavior by leveraging expert data, without the need for intricate controller design, additional environment interactions, or complex reward shaping to encode the target behavior. The latter are substantial advantages over reinforcement learning techniques [2] 31 that rely reward feedback. However, a significant challenge in IL lies in handling the multimodal nature of data obtained from human demonstrators, which stem from differences in preferences, expertise, or problem-solving strate- gies. Conventionally. maximum likelihood estimation (MLE) is employed to train a policy on expert data. It is well-known that MLE corresponds to the moment projection (4, causing the policy to average over modes in the data distribution that it cannot represent. Such mode averaging can lead unexpected and potentially dangerous behavior. We address this critical issue by introducing Information Maximizing Curriculum (IMC), a novel curriculum-based approach.
In IMC, view imitation learning conditional density estimation problem and present mathematically sound weighted optimization scheme. Data samples are assigned curriculum weights, which are updated using information projection. The information projection minimizes the reverse
"Correspondence to denis. blessingokit.ed.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

KL divergence, forcing the policy to ignore modes it cannot represent 4 As a result, the optimized policy ensures safe behavior while successfully completing the task.
Yet, certain modes within the data distribution may remain uncovered with a single expert policy. To address this limitation and endow the policy with the diverse behavior present in the expert data, we extend our approach to mixture of expert (MoE) policy. Each mixture component within the MoE policy selects its own subset of training data, allowing for specialization in different modes. Our objective maximizes the entropy of the joint curriculum, ensuring that the policy covers all data samples.
We show that our method is able to outperform state-of-the of-the-art art policy learning algorithms and MoE policies trained using competing optimization algorithms complex multimodal simulated control tasks where data is collected by human demonstrators, In our experiments, we assess the ability of the models avoid mode averaging and cover all modes present in the data distribution.
Preliminaries
Our approach heavily builds minimizing Kullback-Leibler divergences well as mixtures of expert policies. Hence, we will briefly review both concepts.
2.1 Moment and Information Projection
The Kullback-Leibler (KI.) divergence (5] a similarity measure for probability distributions and is defined as DKL (pll'' = Ex p(x) log p(x)/p' pp (x) for discrete random variable x. Due to its asymmetry, the KL divergence offers two different optimization problems for fitting model distribution p to target distribution p* that is,
arg min DKL(p" (pl(p")lp)) (lp)
and arg min DKL. (plle")
Mcomen)-Prejoction
Pofoecntiion--Poocciion
The M-projection - or equivalently maximum likelihood estimation (MLE) 6 probability forcing, meaning that the model is optimized to match the moments of the target distribution, causing it to average over modes that it cannot represent. In contrast, the I-projection is zero forcing which leads the model to ignore modes of the target distribution that it is not able represent. The I-projection can be rewritten as maximum entropy problem, i.e.,
arg max Ep(x) [log p" (x)] + H(x). p
Using this formulation, it can be seen that the optimization balances between fitting the target distribution and keeping the entropy H(x) Σx (xx) p(x) high.
2.2 Mixtures of Expert Policies
Mixtures of expert policies are conditional discrete latent variable models. Given some observation o E O and action a E A, the marginal likelihood is decomposed into individual components, i.e.,
p(aoo) Σ p(zlo)p(alo,2)
Here, O and A denote the observation and action space respectively, and z stands for the discrete latent variable that indexes distinct components within the mixture.
The gating p(2 o) is responsible for soft- poft-partitioning the observation space into sub-regions where the corresponding experts p(alo, 2) approximate the target density. Typically the experts and gating are parameterized and learned by maximizing the marginal likelihood via expectation- maximization (EM) 7 or gradient ascent [8] In order to sample actions, that is, a' ~ "paajo)) for some observation o', we first sample a component index from the gating, i.e,, p(z]o'). The component index selects the respective expert to obtain a' ~ p(alo', z').

---

Information Maximizing Curriculum
In this section, we propose Information Maximizing Curriculum (IMC), a novel algorithm for training mixtures of expert polices. We motivate our optimization objective using a single policy. Next, we generalize the objective support learning mixture of experts policy. Thereafter, discuss the optimization scheme and provide algorithmic details.
3.1 Objective for a Single Expert Policy
We propose objective that jointly learns currieulum p(o, and parameterized policy (a)o) with parameters 0. The curriculum is categorical distribution over samples of a dataset {(o,, a,,)]_ |== assigning probability mass to samples according the performance of the policy. To allow the curriculum to ignore samples that the policy cannot represent, we build on the I-projection (see Equation Db. We, therefore, formulate the objective function
[p((, a), 0) Ep(o.a) [log (a(o)) + ()),
(2)
which is optimized for p(o, a) and 0 in an alternating fashion using coordinate ascent 19. We additionally introduce trade-off factor 7) that determines the pacing the curriculum. For r) + oo the curriculum becomes uniform, exposing all samples to the policy and hence reducing to maximum likelihood estimation for 0. In contrast, if r + the curriculum concentrates on samples where the policy log-likelihood log po (a ()) is highest. The objective can be solved in closed form for p(o, a) (see Appendix B resulting in
p (On: an) x (an on,)
Maximizing the objective w.r.t 0 reduces to a weighted maximum-likelihodd estimation, that is,
l* arg max Σ p(on an) log pe a., (o,).
The optimization is repeated until reaching a local maximum, indicating that the curriculum has selected fixed of samples where the policy attains high liglliies log po po(a) aalo). Proposition 3.1 establishes convergence guarantees, the details of which are elaborated upon in Appendix I
Proposition 3.1. Let J be defined as in Equation 2 and 0 < r < co. Under mild assumptions on and the optimization scheme for 0, it holds for all i, denoting the iteration index of the optimization process, that
[p(o, ())++)), g((+))) 2 J(p(o, a) a) (()),
where equality indicates that the algorithm has converged to a local optimum.
The capacity to disregard samples where the policy cannot achieve satisfactory performance mitigates the mode-averagie averaging problem. Nevertheless, drawback of employing a single expert policy is the potential for significantly suboptimal performance the ignored samples. This limitation is overcome by introducing multiple experts, each specializing in different subsets the data.
3.2 Objective for Mixture of Experts Policy
Assuming limited complexity, a single expert policy is likely to ignore large amount of samples due the zero-foreing property of the I-projection. Using multiple curricula and policies that specialize to different subsets of the data is hence a natural extension to the single policy model. To that end, we make two major modifications to Equation 2 Firstly, we use mixture model with multiple components 2 where each component has its own curriculum, i.e., p(o, a) Σ, p(2)p(o, a 22). Secondly, we employ an expert policy per component pe, pa, (alo, 2), that paced by the corresponding curriculum p(o, z). The resulting objective function is given by
J ((b)
(alo, 2)] + (7H(O,,)) a).
where vb summarizes the dependence on p(z), {p(o, a|²)}= and {0.).. However, Equation 3 is difficult to optimize as the entropy of the mixture model prevents us from updating the curriculum of each component independently. Similar to [10], introduce auxiliary distribution g a) to decompose the objective function into a lower bound L(bb, q) and an expected DKL term, that is,
((b) L(bb 9) + ppppio. a) ()) [l/ a)),
3

---

with a() - - (lo.aa=)p(z2/pp(0) a) and I( = Ep(s) Bp(o.ale)
1/H a|=]]] + nfH()),
J=(r(o.a))).0))
with R2 (o,, a,)) log po =) + 1) wogggogg2a0 [(=|0n, an), allowing for independent updates for p(o, alz) and 0g by maximizing the per-component objective objjecte function (p(o,, 0l)),0,)- A derivation can be found in Appendix Since a (p(z|o, o, a) {)]]){)))) 2 0, L is lower bound on J for n 2 Please note that the per-component objective function J/ is very similar Equation with J/z including an additional term, log g(= a). which serves the purpose of preventing different curricula from assigning probability mass to the same set of samples: Specifically, a component is considered to 'cover' a datapoint (On, an) when (= a,) = 1. Since Σ, q(> o,,, a,) 1, it follows that for other components z' + it holds that q(> On, an) = Consequently, log q(" oo, an) - -c, implying that R2 (On, an) -00. As result, the other curricula effectively disregard the datapoint, as p(O,; a², xx exp R2.(0n. an) Rs O.
We follow the optimization scheme of the dnpectation-maxinizatiin algorithm 77. that is, we iteratively maximize (M-step) and tighten the lower bound (E-step) L().g))
3.3 Maximizing the Lower Bound (M-Step)
We maximize the lower bound L( q) with respect to the mixture weights p(2), curricula p(o, al*) and expert policy parameters 0g. We find closed form solutions for both, p(2) and p(o, al2) given by
p" (z) exp (Ep(o.als)) (R. ao.a//) + H(oo al=)), and p(on, aml=)) = exp
an)/n), (5)
where plon, am|z) |z) are the optimal unnormalized curricula, such that holds p" p0 a,|=²
p(o,, a,,22) En p(on, an|²) However, due to the hierarchical structure of L( q) we implic- itly optimize for p(z) when updating the curricula. This result is frequently used throughout this work and formalized in Proposition (.2 A proof can be found in Appendix
Proposition 3.2. Let p" (z) and p(o, alz) be the optimal mixture weights and unnormalized curricula for maximizing L( 9). holds that
(2) Σ plon. ag anl2)/
>Σ p(o.. a2
The implicit updates of the mixture weights render the computation of p (z) obsolete, reducing the optimization to computing the optimal (unnormalized) curricula p(o, alz) and expert policy parameters - particular, this result allows for training the policy using mini-batches and thus greatly improves the scalability to large datasets explained in Section Maximizing the lower bound with respect to 8 results in a weighted maximum likelihood estimation, i.e.,
0 arg max 0,
Σ p(o.. an log po, (a,, On,
(6)
where the curricula p(o,, a, ( assign sample weights. For further details on the M-step, including derivations of the closed-form solutions and the expert parameter objective see Appendix B.2
3.4 Tightening the Lower Bound (E-Step)
Tightening of the lower bound (also referred to as E-step) done by minimizing the ex- pected Kullback-Leibler divergence in Equation 4 Using the properties of the KL divergence, it can easily be seen that the lower bound is tight if for all n E [1,...NN q(2(on) = p(2)o,, a,) holds. To obtain p(z)on, a,) leverage Bayes" rule, that is, p(=]On, an) p" (2)p" a, /2)/ /,)/E,, p" (2))p" (on, 22).). Using Proposition 3.2 we find that
p(z an) = p(on, an /n/2)/ Σ o( an|z).
Please note that the lower bound is tight after every E-step as the KL divergence is set to zero. Thus, increasing the lower bound L. maximizes the original objective assuming that updates 0, are not decreasing the expert policy log- likelihoddo log pe. (alo, 2).

---

3.5 Algorithmic Details
Convergence Guarantees. Proposition establishes convergence guarantees for the mixture of experts policy objective J. The proof mainly relies on the facts that IMC has the same convergence guarantees as the EM algorithm and that Proposition 3.1 can be transferred the per-component objective J.. The full proof is given in Appendix A.1
Proposition 3.3. Let J be defined as in Equation 4 and O < 11 < oc. Under mild assumptions pe. and the optimization scheme for g, it holds for all i, denoting the iteration index of the optimization process, that
Hab (
where equality indicates that the IMC algorithm has converged to local optimum.
Stopping Criterion. We terminate the algorithm if either the maximum number of training iterations reached or if the lower bound L( 9) converges, i.e.,
A = L ( (v) q) - L ((- (bb, S
with threshold and two subsequent iterations (i) and (i-1), The lower bound can evaluated efficiently using Corollary 1
Corollary 3.2 Consider the setup used in Proposition 3.2 For (2) E ab and {p" (o, al2)}² E +
holds that
9) 77 log Σ p(o.. an|z).
See Appendix
A.3 for proof.
Inference. In order to perform inference, i.c., sample actions from the policy, need to access the gating distribution for arbitrary observations o E which not possible as p(z oo a) is only defined for observations contained in the dataset o, We therefore leverage Corollary 2 to learn an inference network (=(=))) with parameters & by minimizing the KL divergence between p(2(o)
and ( (2 o) under p(o) (see Appendix A.4 (.4 for proof).
Corollary 2. Consider the setup used in Proposition 3.2
It holds that
min +
= max
Σ
plon, go
+
Once trained, the inference network can be used for computing the exact log- likelihodoo as p(alo) = Σ. 96 (z)o)ppp oa 2) or sampling component index, i.e., z' (z(z)o).
Mini-Batch Updates. Due to Proposition 3.2, the M- and E-step in the training procedure only rely on the unnormalized curricula p(o, a|2). Consequently, there is no need to compute the normalization constant Σ, (( a, /2). This allows utilize mini-batches instead of processing the entire dataset, resulting in an efficient scaling capability for large datasets. Please refer to Algorithm [ for detailed description of the complete training procedure.
Related Work
be grouped into two categories: Inverse reinforcement learning [12 [3]. which extracts a reward function from demonstrations and optimizes policy subsequently, and behavioral cloning, which directly extracts policy from demonstrations. Many works approach the problem of imitation learning by considering behavior cloning
Imitation Learning. variety of algorithms in imitation learning I [1
distribution-matchin problem, in which the state distribution induced by the policy is required to align with the state distribution of the expert data. Some methods [14 [5] are based on adversarial methods inspired by Generative Adversarial Networks (GANs) [66]
policy is trained to imitate the expert while discriminator learns to distinguish between fake and expert data. However, these methods are not suitable for our case as they involve interacting with the environment during training. Other approaches focus purely offline training and use various policy representations such as energy-based models [17. normalizing flows 19], conditional variational
autoencoders (CVAEs) 20 21, transformers
22 diffusion models 23 24 25
26
27 28). These
5

---

Figure I: Behavior learning environments: Visualization the obstacle avoidance task (left), the block pushing task (middle), and the table tennis task (right).
models can represent multi-modal expert distributions but are optimized based on the M-Projection. which leads to a performance decrease. Recent works (2 30 have proposed training Mixture of
Experts models with an objective similar to ours. However, the work by (29) requires environment-
specific geometric features, which is not applicable in our setting, whereas the work by [30] considers
linear experts and the learning of skills parameterized by motion primitives [3I]. For detailed
differentiation between our work and the research conducted by [30], please refer to Appendix D.1
Curriculum Learning. The au-
Algorithm IMC training procedure
thors of [32) introduced curriculum learning (CL) as a new paradigm
Require: Data {(0,, a., an) )
for training machine learning mod-
2: Require: Number of components N,
els by gradually increasing the dif-
Require: Curriculum pacing !7
ficulty of samples that are exposed
4: while | S c do
to the model. Several studies fol-
Perform E-step:
lowed this definition
[33 34 35
6: (2/on,, a.) + p(o,,. an |m12)/ Σ, p(o,. aml2) Vn
36
37
38 (39] Other studies
for z 1, ,,., N2 do
used the term curriculum learning
8: Perform M-step for curricula:
for gradually increasing the model
9:
R2 a.) < log Po, (an jo., 2) + 1y log g(=|on; an) Vn
42 or task 45 46). All of these approaches assume that the rifficulty-ranking of the samples known a-priori. In contrast, we consider dynamically adapting the curriculum according to the learn- ing progress of the model which is known as self-paced learning (SPL). Pioneering work in SPL was done in 47 which is related to our work in that the authors propose update the eurriculum well model parameters iteratively. However, their method is based on maximum likelihood which is different from our approach. More- over, their algorithm is restricted to latent structural support vector machines. For comprehensive survey on curriculum learning, the reader is referred to 334.
complexity [40
41
complexity 44
10:
p(on. a,, exp (R. an)/n) Vn
11:
Perform M-step for experts:
12:
0: + arg maxg, Σ. p(o.,, an logpo, (anlon,
13:
end for
14:
end while
5 Experiments
We briefly outline the key aspects of our experimental setup.
Experimental Setup. For all experiments, employ conditional Gaussian expert policies, i.e., Po, (alo, 2) = _N(aluo, (o), I2).. Please note that we parameterize expert means foo using neural networks. For complex high-dimensional tasks, we share features between different expert policies by introducing a deep neural network backbone, Moreover, we use a fixed variance o² = 1 and N2 50 components for all experiments and tune the curriculum pacing For more details Appendix
Baselines. We compare our method to state-of-the-ar generative models including denoising dif- fusion probabilistic models (DDPM) 24, normalizing flows (NF) [8] and conditional variational autoencoders (CVAE) [20]. Moreover, we consider energy- based models for behavior learning (IBC) 17 and the recently proposed behavior transformer (BeT) [22. Lastly, we compare against mixture
of experts trained using expectation maximization (EM) 48 and backpropagation (MDN) g and the MI.-Cur algorithm (30]. We extensively tune the hyperparameters of the baselines using Bayesian optimization [49] on all experiments.

---

DVAB
IBC
DET
Ml,Cu
BBC
Figure 2: Obstacle Avoidance: Visualization of 100 end-effector trajectories for all trained models. Every method is trained on diverse demonstration data (left). MDN, NF, and IBC mostly fail to successfully solve the task. Other methods (EM, DDPM, CVAE, BET, ML-Cur) either fail to have high success rates disregard modes in the data. Only IMC is able perform well on both metries.
Table 1: Performance Table: manipulation tasks. The best results use bold formatting. f indicates that higher and that lower values are better. For further details, please refer to the accompanying text.
We compare IMC against strong basclines three complex robot
OBSTACLE AVOIDANCE
BLOCK PUSHING
TABLE TENNIS
SUCCESS RATE (T)
EXTROPY (f)
SECCESS RATE (f)
ENTROPY (1)
DISTANCE FRBOR (}) SUCERSS RATE (T)
DISTANCE FRBOR (1)
MDN
0 2.20
0.000
2.0
0.000
0.50
0.051
0.5492a
EM
0.875.10.03
0.902
0.458.9.045
0.707 107011002
0.127
0.725
0.220.0.012
DDPM
0.719gmmt
0.63M
0.516
0.713an
0.1852an
NF
0:340at.ate
0.322
0. 1.379gn
0.068
758
25
0.910
BET
0 17
a..372
0.376
10
0.151 Lo
0. 0.758
0.285
ML-CUR
0. 0.455110223
0.005-4..24
0.000
018088 caw
0..88 ioun
IMC
0.855 nax
0.930 eon
0.52122000
0.654go
0. 120 on4
D. 8.07ggo
0. 0.158ganmm ner
Evaluation. For all experiments we perform multiple simulations using the trained policies to compute the performance metrics. Firstly, we report the success rate, which is the fraction of simulations that led successful task completion, reflecting the susceptibility to mode averaging. Secondly, we compute a task speciffc (categorical) distribution over pre- pre-defined behaviors that lead successful task completion. Thereafter, we compute the entropy over this distribution quantify the diversity in the learned behaviors and therefore the ability to cover multiple modes present in the data distribution. An entropy of implies that model executes the same behavior, while higher entropy values indicate that the model executes different behaviors. Further details are provided in the descriptions of individual tasks.
We report the mean and the standard deviation over ten random seeds for all experiments. For
detailed explanation of tasks, data, performance metries, and hyperparameters see Appendix C For
further experimental results see Appendix E The code is available online 2
5.1 Obstacle Avoidance
The obstacle avoidance environment is visualized in Figure (left) and consists of a seven DoF Franka Emika Panda robot equipped with cylindrical end effector simulated using the MuloCo physics engine (50). The goal of the task to reach the green finish line without colliding with one of the six obstacles. The dataset contains four human demonstrations for all ways of avoiding obstacles and completing the task. It is collected using a game- pad controller and inverse kinematics (IK) in the xy-plane amounting to 7.3k (o, a) pairs. The observations o E E24 contain the end-effector position and velocity of the robot. The actions a E R² represent the desired position of the robot. There are 24 different ways of avoiding obstacles and reaching the green finish line, each of which we define as different behavior 3. At test time, we perform 1000 simulations for computing the success rate and the entropy H(B) E. p(3) log24 p(B). Please note that we use log24 for the purpose of
https://githhb httpettttb..coa.lauubbbbaee
7

---

OFL
(a) N, =1
(b) N,
(c) N, N5
N, = 10
(e) Dataset
Figure 3: Curriculum Visualization: Visualization of the curricula p(o, a|=) for different number of components N₂ on the obstacle avoidance task. The color indicates different components z and the size of the dots is proportional to p(,,, an|2). For N₂ most samples in the dataset as a single expert is not able achieve high liglliiel values pe, on all samples (Figure 3a Adding more components to the model results in higher coverage of samples
we observe that the model ignores
as shown in Figure 3b 3d
Figure Be visualizes all samples contained in the dataset.
enhancing interpretabilityy, as it ensures H(B) [0, 1]. An entropy value of 0 signifies policy that consistently executes the same behavior, while an entropy value of represents a diverse policy that executes all behaviors with equal probability and hence matches the true behavior distribution by design of the data collection process. The results are shown in Table Additionally, we provide a visualization of the learned curriculum in Figure Further details provided in Appendix C.I.I
5.2
Block Pushing
The block pushing environment is visualized in Figure [ (middle) and uses the setup explained in Section 5.1] The robot manipulator is tasked to push blocks into target zones. Having two blocks and target zones amounts to four different push sequences (see c.g. Figure 8), each of which we define as a different behavior B. Using a gamepad, we recorded 500 demonstrations for each of the four push sequences with randomly sampled initial block configurations On (i.c., initial positions and orientations), amounting
a total of 463k (o, a) pairs. The observations E R contain information about the robot's state and the block configurations. The actions R2 represent the desired position of the robot. We evaluate the models using three different metrics: Firstly, the success rate which is the proportion of trajectories that manage to push both boxes to the target zones. Secondly, the entropy, which is computed over different push sequences 3. Since high entropy values can be achieved by following different behaviors for different initial block configurations Og in a deterministic fashion, the entropy of p(3) can be a poor metric for quantifying diversity. Hence, evaluate the expected entropy conditioned the initial state ou, i.c., Ep(oo) [H))) & No Eoo-poo E3 p((3)oo) log, p(3)oo). If, for the same Og, all behaviors can be achieved, the expected entropy is high. In contrast, the entropy is 0 if the same behavior is executed for the same Ou. Here, p(ou) and No denote the distribution over initial block configurations and the number of samples respectively. See Appendix C.I.2 for more details. Lastly, we quantify non-successful trajectories. via distance error, i.e., the distance from the blocks to the target zones at the end of a trajectory. The success rate and distance error indicate whether model is able avoid averaging over different behaviors. The entropy assesses the ability to represent multimodal data distributions by completing different push sequences. The results are reported in Table [ and generated by simulating 16 evaluation trajectories for 30 different initial block configurations Og per seed. The difficulty of the task is reflected by the low success rates of most models. Besides being challenging manipulation task, the high task complexity is caused by having various sources of multimodality in the data distribution: First, the inherent versatility in human behavior. Second, multiple human demonstrators, and lastly different push sequences for the same block configuration.
the performance
5.3 Franka Kitchen
The Franka kitchen environment was introduced in [5]] and uses seven DoF Franka Emika Panda robot with two DoF gripper to interact with simulated kitchen environment. The corresponding dataset contains 566 human-collected trajectories recorded using a virtual reality setup amounting to 128k (o, a) pairs. Each trajectory executes a sequence completing four out of seven different tasks. The observations o E 3R contain information about the position and orientation of the task-relevant objects in the environment. The actions E Rº represent the control signals for
8

---

EM DDPM
NF
CVAE
BET
IMC
0.5
Dumberr of tusk completions
mumber of task completionns
Figure 4: Franka Kitchen: Performance comparison between various policy learning algorithms, evaluating the success rate and entropy for a varying number of task completions. IMC is able to outperform the baselines in both metrics, achieving a high success rate while showing higher diversity in the task sequences. Performance comparison between various policy learning algorithms. IMC is more successful in completing tasks (success rate) while the same time having the highest diversity in the sequence of task completions (entropy).
the robot and gripper. To assess a model's ability to avoid mode averaging we again use the success rate over the number of tasks solved within one trajectory. For each number of solved tasks E {1, 2, 3, 4). we define behavior 5 as the order in which the task is completed and use the entropy (3) = eel p(3) log p(B) to quantify diversity. The results are shown Figure 5 and are generated using 100 evaluation trajectories for each seed. There are no results reported for IBC, MDN and ML-Cur as we did not manage to obtain reasonable results.
5.4 Table Tennis
The table tennis environment is visualized in Figure 1 (right) and consists of a seven DOF robot arm equipped with a table tennis racket and simulated using the MuJoCo physics engine. The goal is to return the ball to varying target positions after it is launched from randomized initial position. Although not collected by human experts, the 5000 demonstrations are generated using a reinforcement learning (RL) agent that is optimized for highly multimodal behavior such as backhand and forehand strokes (52). Each demonstration consists of an observation o E R2 defining the initial and target ball position. Movement primitives (MPs) (31 are used to describe the joint space trajectories of the robot manipulator using two basis functions per joint and thus a E R14, We evaluate the model performance using the success rate, that is, how frequently the ball is returned to the other side. Moreover, we employ the distance error, ie., the Euclidean distance from the landing position of the ball to the target position. Both metrics reflect if model is able to avoid averaging over different movements. For this experiment, we do not report the entropy as we do not know the various behaviors executed by the RL agent. The results generated using 500 different initial and target positions. Note that the reinforcement learning agent used generate the data achieves an average success rate of 0.91 and a distance error of 0.14 which is closely followed by IMC.
shown in Table and
5.5 Ablation Studies
Additionally, we compare the performance of IMC with EM for a varying number of components on the obstacle avoidance and table tennis task. The results are shown in Figure and highlight the properties of the moment and information projection: Using limited model complexity, e.g. 1 or 5 components, EM suffers from mode averaging, resulting in poor performances (Figure (andddd5 Figure 5b). This is further illustrated in Figure 5c] In contrast, the zero forcing property of the information projection allows IMC to avoid mode averaging (see Figure which is reflected the success rates and distance enror for a small number of components. The performance gap between EM and IMC for high model complexities suggests that EM still suffers from averaging problems. Moreover, the results show that IMC needs fewer components achieve the same performance EM.

---

ELS
S10000 stle 64
110000
0.4
12
(a)
Obstacle avoidance performance comparison.
(b) Table lennis performance comparison. IHE O4 3o
3Bd (N
IDV &C
Esd IS
DEC IN
(c) Obstacle avoidance end-effictor trajectories EM.
(d) Obstacle avoidance end-effector trajectories for IMC.
Figure Obstacle Avoidance: IMC and EM improve success rate and entropy with an increasing number of components (a). For small number of components, IMC archives a high success rate, as it allows the policy to focus on data that can represent. contrast, the policy trained with EM fails is forced to cover the whole data. This visualized in the end-effector trajectories (c + d). Similar observations can made for Table Tennis: Both performance metrics increase with a higher number of components. IMC manages achieve good performance with small number of components.
Conclusion
We presented Information Maximizing Curriculum (IMC), a novel approach for conditional density estimation, specifically designed address mode-averaging issues commonly encountered when using maximum likelihood-baate optmization in the context of multimodal density estimation. IMC's focus on mitigating mode-averaging particularly relevant in imitation learning from human demonstrations, where the data distribution is often highly multimodal due to the diverse and versatile nature of human behavior.
IMC uses a curriculum to assign weights to the training data allowing the policy to focus on samples it can represent, effectively mitigating the mode-averaging problem. We extended our approach a mixture of experts (MoE) policy, where each mixture component selects its own subset of the training data for learning, allowing the model imitate the rich and versatile behavior present in the demonstration data.
Our experimental results demonstrate the superior performance of our method compared to state-of- the-art policy learning algorithms and mixture experts (MoE) policies trained using competing optimization algorithms. Specifically, complex multimodal simulated control tasks with data collected from human demonstrators, our method exhibits the ability to effectively address two key challenges: i) avoiding mode averaging and ti) covering all modes present in the data distribution.
Limitations. While our current approach achieves state-of-the-ar performance, there are still areas for improvement in parameterizing our model. Presently, we employ simple multilayer perceptrons to parameterize the expert policies. However, incorporating image observations would require a convolutional neural network (CNN) (53) backbone. Additionally, current model relies on the Markov assumption, but relaxing this assumption and adopting history-basee models like transformers [44 could potentially yield significant performance improvements. Lastly, although this work primarily concentrates on continuous domains, an intriguing prospect for future research would to explore the application of IMC in discrete domains.
Broader Impact. Improving imitation learning algorithms holds the potential to enhance the acces- sibility of robotic systems in real-world applications, with both positive and negative implications. We acknowledge that identifying and addressing any potential adverse effects resulting from the deployment of these robotic systems is crucial responsibility that falls sovereign governments.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by funding from the pilot program Core Informatics of the Helmholtz Association (HGF). The authors acknowledge support by the state of bwHPC, well as the HoreKa supercomputer funded by the Ministry of Science, Research and the Arts Baden-W Württembrrg and by the German Federal Ministry of Education and Research.
through
References
[1] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, al. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 7(1-2):1-79, 2018
[2] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
[3] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[4] Kevin P Murphy, Machine learning: probabilistic perspective. MIT press, 2012.
[5] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(():79--86 1951.
[6] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
[7] Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1-22, 1977.
[8] Christopher M Bishop. Mixture density networks. 1994.
[9] Stephen Boyd, Stephen Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[10] Oleg Arenz, Gerhard Neumann, and Mingjun Zhong. Efficient gradient-fre variational inference using policy search. In International conference on machine learning, pages 234-243, PMLR, 2018.
[11] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5))469-445 483, 2009.
[12] Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. Xirl: Cross-embens embodimnbod inverse reinforcement learning. In Conference on Robor Learning, pages 537-546. PMLR, 2022.
[13] Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforeement learning. In Aaai, volume 8, pages 1433 1438. Chicago, IL. USA, 2008.
[14] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[15] Justin Fu, Katie I Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
[16] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
[17] Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pages 158-168, PMLR, 2022.

---

[18] George Papamakarios, Eric T Nalisnick, Danilo Jimenez Rezende. Shakir Mohamed, and Balaji Lakshminarayanan Normalizing flows for probabilistic modeling and inference. Mach. Learn. Res., 22(57):1 2021.
[19] Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. International Conference on Learning Representations, 2021.
[20] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
[21] Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4): 1205 2022.
[22] Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning modes with one stone. arXiv preprint arXiv:2206. 1251, 2022.
[23] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[24] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 1854, 2020.
[25] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[26] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song, Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303. 04737, 2023.
[27] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgesou, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representationss 2023.
[28) Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies. arXiv preprint wrXiv:2304.025:2 2023.
[29] Niklas Freymuth. Nicolas Schreiber, Aleksandar Taranovic, Philipp Becker, and Gerhard Neumann. Inferring versatile behavior from demonstrations by matching geometric descriptors.
In 6th Annual Conference on Robot Learning, 2022.
[30] Maximilian Xiling Li, Onur Celik, Philipp Becker, Denis Blessing, Rudolf Lioutikov, and Ger- hard Neumann. Curriculum-bass imitation of versatile skills. arXiv preprinl arXiv:2304.05171, 2023.
[31] Alexandros Paraschos, Christian Daniel, Jan R Peters, and Gerhard Neumann. Probabilistic movement primitives. Advances in neural information processing systems, 26, 2013.
[32] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48,
2009.
[33] Valentin Spitkovsky, Hiyan Alshawi, and Daniel Jurafsky. Baby steps: How "less is more" in unsupervised dependency parsing. 2009.
[34] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey, International Journal of Computer Vision, pages 1-40, 2022.
[35] Xinlei Chen and Abhinav Gupta. Webly supervised learning of convolutional networks. In Proceedings of the IEEE international conference computer vision, pages 1431-1439, 2015.
12

---

[36] Radu Tudor Ionescu, Bogdan Alexe, Marius Leordeanu, Marius Popescu, Dim P Papadopoulos, and Vittorio Ferrari. How hard can it be? estimating the difficulty of visual search in an image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2167-2166, 2016.
[37] Anastasia Pentina, Viktoriia Sharmanska, and Christoph H Lampeer. Curriculum learning of multiple tasks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5492-5500, 2015.
[38] Yangyang Shi, Martha Larson, and Catholijn M Jonker. Recurrent neural network language model adaptation with curriculum learning. Computer Speech & Language, 33(1):136-154, 2015.
[39] Wojeiech Zaremba and Ilya Sutskever. Learning execute. arXiv preprint arXiv:1410.461 2014.
[40] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for 2017.
improved quality, stability, and variation. arXiv preprint
[41] Pietro Morerio, Jacopo Cavazza, Riccardo Volpi. René Vidal, and Vittorio Murino. Curriculum dropout. In Proceedings of the IEEE International Conference on Computer Vision, pages 3544 3552, 2017.
[42] Samarth Sinha, Animesh Garg, and Hugo Larochelle. Curriculum by smoothing. Advances in Neural Information Processing Systems,
2020.
[43] Antoine Caubrière, Natalia Tomashenko, Antoine Laurent, Emmanuel Morin, Nathalie Camelin, and Yannick Estève. Curriculum-based transfer learning for effective end-to-end spoken 2019.
language understanding and domain portability. arXiv preprins
[44] Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on robot learning, pages 482-495. PMLR, 2017.
[45] William Lotter, Greg Sorensen, and David Cox. A multi-scale enn and curriculum learning strategy for mammogram classification. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pages 169-177. Springer, 2017.
[46] Nikolaos Sarafianos, Theodore Giannakopoulos, Christophoros Nikou, and Ioannis A Kakadiaris. Curriculum learning for multi-task classification of visual attributes. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 2608-2602322 2017.
[47] M Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. Advances neural information processing systems, 23, 2010.
[48] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation. 3(1):79-8 87, 1991
[49] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
[50] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEFF, 2012.
[51] Abhishek Gupta, Vikash Kumar, Corey Lynch. Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv: 1910. 1956, 2019.
[52] Onur Celik, Dongzhuoran Zhou, Ge Li, Philipp Becker, and Gerhard Neumann. Specializing versatile skill libraries using local mixture of experts. In Conference on Robot Learning, pages 1423-1433. PMLR, 2022.
13

---

[53] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence Jackel. Backpropagation applied handwritten zip code recognition. Neural compulation, 1(4):541-551, 1989.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[55] Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review, arXiv preprint arXiv: 00909,, 2018.
[56] Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3):334-334, 1997.
[57] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.
[58] You Zhou, Jianfeng Gao, and Tamim Asfour. Movement primitive learning and generalization: Using mixture density networks. IEEE Robotics & Automation Magazine, 27(2):22 32, 2020.
[59] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabílistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[60] George Papamakarios, Theo Pavlakou, and lain Murray, Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
[61] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International conference on learning representations, 2017.
[62] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
14

---

A Proofs
A.1 Proof of Proposition 3.1 and 3.3
We perform coordinate ascent on J which is guaranteed to converge stationary point if updating each coordinate results in monotonic improvement of [9] For fixed expert parameters we find the unique p(o, a) that maximizes
Convergence of the Single Expert Objective (Proposition 3.1
[55] (see Section 3.1 and hence we have ((p(o, a) a) ,00,0 0) 2 J(p(o, a) ()---) 0) where i denotes the iteration. Under suitable assumptions (log pe is differentiable, its gradient L-Lipschitz, 02 is updated using gradient ascent and the learning rate is chosen such that the descent lemma [56]
holds), it holds that J [p((,, a). ) 2 [[p(o, a), 0 (()-1)) Hence, we are guaranteed to converge to stationary point of J.
Convergence of the Mixture of Experts Objective (Proposition 3.3 As tighten the lower bound L in every E-step, it remains show that L( .) 2 L( ..) to prove that ((b) is guaranteed to converge to a stationary point, with 3 = {p(z), [p(o, al2) (02)e)- To that end, we again perform coordinate ascent on L to show monotonic improvement in every coordinate. Note that we find the unique p(z) and {p(o. a|²]}= that maximize L via Proposition 3.2 and Equation 5
Analogously the proof of Proposition 3.1 we can show monotonic improvement of L in (0.
under suitable assumptions on log
Remark. Although stochastic gradient ascent does not guarantee strictly monotonic improvements the objective function J, our empirical observations suggest that indeed tends to increase monotonically in practice shown by Figure
10.5
10
20
40
60
80
100
Figure 6: Convergence the Mixture of Experts Objective. Objective function value J for 100
training iterations on Franka Kitchen.
A.2 Proof of Proposition 3.2
Expanding the entropy in Equation obtain
p (2) x exp EEp ( (o, lo,
Using p" po a|z) p(o, a 2 Σn p( an (z) yields
p" (=) xexp (o. (o.a [ (0, - alg + log Σ p(on. an an|=)]).
Next, leveraging that log a|2) R. (o,, an we see that
pd (2) x exp F (o.te)) [log Σ p(on, an an|1)])
Σ p(on: an 12)
which concludes the proof.
A.3 Proof of Corollary
3.2
We start by rewriting the lower bound as L(bb
Epr(s)
- 17 log a|2)] 17 17log (x))
15

---

U al/ x p(o, a) and Proposition 3.2 we obtain
L( (=) (a (o, a) -7 log pl, a|z)
ong Σ p a, [2]] 77 log Σ p(o.. an 22 + n7 log ΣΣ pl an [2]]
With llg a R₂ (O,; an) all most terms cancel, giving
L( q) Ba (=) n log ΣΣ pl a []]=
1 log Σ
Σ p(on, an|=),
which concludes the proof.
A.4
Proof of Corollary 3.2
min P(o) DRL (2apll))))) = mmxx / Σ p(o, 2) log (2
max 1o Σ p(o. a, logg0))))) max / [ol p(o, a) pp(z a) (2)
@
min Ep(o,a) DRL (p(2lo (2)0))))))))
Expanding the expected KL divergence, get
p(z|0n, an)
min Epro,a) DKL. (p(= a)]lg> = min Σ p(Ons an) pp(z|on an) log O O0))
Noting that p(2 On an) is independent of Φb rewrite the objective as
max Σ p(on, an) Σ p(z an) log (2(o)),
Using that a() = p / Σ. p(o, al2) together with p(o, (=)"(") yields
p(0n, a,
max ΣΣ p (2)x a Σ
log 96 (2(o,))
p(o,, a. (2)
Using Proposition 3.2 can rewrite p" (z)p" (o, a|2) as p(o, 2)/ >, En p(on an an|z). Since the
constant factor 1/ Σx In p(on, a, 22) does not affect the optimal value of we obtain
p(O,. a,122)
max +
Σ>
on an (z) Σ
log (2(o)))) = max Σ> Σ
p(on. an am/²) log go (z/on),
Σ. p(o,, an 22
+
which concludes the proof. 11
B Derivations
B.1 Lower Bound Decomposition
To arrive at Equation 4 by marginalizing over the latent variable o for the entropy of the joint curriculum, i.c.,
H(o,a) = Σ p(on, an) logp a.)
Σ p(on, an) Σ p(2 a, a.)
Next, we use Bayes' theorem, that is, p(o,. a.) = p( ( a, l2)/p() on., an), giving
H(o,a) = Σ p(On, an) Σ p(2/on, an) (logp(=)) + logp an/=) logp(=//o,, an))
16

---

Moreover, we add and subtract the log auxiliary distribution log 9(= an)) which yields
H(o. a)
Σ p(on, an) Σ p(z)on, an (logp(() + p(o,, an
log p(= an) + log (= a.) log 0(2/on, an)). Rearranging the terms leads and writing the sums in terms of expectations arrive
H(o, a) =pp(=)) [Bp(() (ltg:looallogg///oo,oe. a)] + H( a|z)] H(22) + DKL (p(zlo, a)]lq(z|o, a)).
Lastly, multiplying H(o a) with 1) and adding Epp=) [log po. (y) 2)] we arrive at Equation
4 which concludes the derivation.
B.2 M-Step Objectives
Closed-Form Curriculum Updates. In order derive the closed-form solution to Equation 5 (RHS)
we solve
max al() = max ( p(o.a|z)
subject to
Σ p(o.a) = 1.
p(o.a(2)
Following the procedure of constrained optimization, we write down the Lagrangian function 9 as
L(P, A) = Σ p(on, an (2,, a.) -1 Σ plon, log am 4n|z)+X0 +X p(On, a. -));,
where À is the Lagrangian multiplier. As p is discrete, solve for the optimal entries of p(o.. a,, 22) that is, pl (on, an A12) = arg max, C(p, A). Setting the partial derivative of C(p. A) with respect to p zero, ie..
a
C(v, A) R2 a.)) lo an/=) + - 0.
Op(On, an 2)
yields (o,,, an, A/2) cxp (o,, a.) - + /17
Plugging y/ back in the Lagrangian gives the dual function g(A), that is,
g(n) L(p', 11 Σ exp (Rz an) + A)/n.
Solving for X arg min g(A) equates to
0x ( -1 + Σ exp (Rs an) -1 >)/n =
0
X = log (n Σ exp (Ra an) an) 1n)
Finally, substituting X into p' we have
cxp R (O,, am)
(on, an/z) = p'(o,, a., \112) Σ, exp (On, an)/n)
which concludes the derivation. The derivation of the optimal mixture weights p² (z) works analo- gously.
Expert Objective. In order to derive the expert objective of Equation we solve maxe, J. (p(o, al2), 02)
Σ p(on, (z) (oogpo (a, lo, 2) + v log g(= an) - r log p(o., an lz)). max 6,
Noting that q(2 an) and p(On, ax|=) are independent of O2 and p(On, an|z) p(O,, a, /2) Σn p(o,. 2n we find that
Σ p(on, an 12)
max J=(p(o, max e.
log Po, (a,, /on, 2).
8,
E,, p(o,, 2,
Noting that Σn p(O,, an/2) is constant scaling factor concludes the derivation.
17

---

C Experiment Setup
C.1 Environments and Datasets
C.1.1 Obstacle Avoidance
Figure 7: The left figure shows 6 out of 24 ways of completing the obstacle avoidance task. The middle figure shows 30 randomly sampled initial block configurations for the block pushing task. The right figure visualizes the Franka kitchen environment.
Dataset. The obstacle avoidance dataset contains 96 trajectories resulting in total of 7.3k (o, a) pairs. The observations o IR contain the end-effector position and velocity in Cartesian space. Please note that the height of the robot is fixed. The actions a E R² represent the desired position of the robot. The data is recorded such that there are an equal amount trajectories for all 24 ways of avoiding the obstacles and reaching the target line. For successful example trajectories Figure
Performance Metries. The success rate indicates the number of end-effector trajectories that successfully reach the target line (indicated by green color in Figure 2 The entropy
H(BB Σ p(3) log24 P(3),
is computed for successful trajectories, where each behavior & is one of the 24 ways of completing the task. To assess the model performance, we simulate 1000 end-effector trajectories. We count the number of successful trajectories for each way of completing the task. From that, we calculate categorical distribution over behaviors p(3) which used to compute entropy. By the use of log21 we make sure that H24(T) E [0, 1]. If model is able to discover all modes in the data distribution with equal probability, its entropy will be close to In contrast, H(3) = 0 if a model only learns one solution.
C.1.2 Block Pushing
Dataset. The block pushing dataset contains 500 trajectories for each the four push sequences (see Figure 8 resulting in total of 2000 trajectories 463k (o. a) pairs. The observations E R contain the desired position and velocity of the robot in addition to the position and orientation the green and red block. Please note that the orientation of the blocks is represented quaternion number system and that the height of the robot is fixed. The actions a E 8² represent the desired position of the robot. This task is similar to the one proposed in [17]. However, they use deterministic controller to record the data whereas we use human demonstrators which increases the difficulty of the task significantly due to the inherent versatility in human behavior.
Performance Metrics. The success rate indicates the number of end-effector trajectories that successfully push both blocks to different target zones. To assess the model performance on non- successful trajectories, we consider the distance error, that is, the Euclidean distance from the blocks to the target zones at the final block configuration of an end-effector trajectory. As there are total of four push sequences (see Figure
we use the expected entropy
Ep(oo) [3(3]),)]
Σ
((33)0 log, p(3)o)).
No
ogmp(oo) 3
to quantify model's ability to cover the modes the data distribution. Please note that we use log, for the purpose of enhancing interpretability, as it ensures H(3) [0, 1]. An entropy value
18

---

Figure 8: Block pushing: Top view of four different push sequences. Starting from the black cross, the gray line visualizes the end-effector trajectory of the robot manipulator. The small rectangles indicate different box configurations in the push sequence, the big rectangles mark the target zones.
of 0 signifies policy that consistently executes the same behavior, while entropy value of represents a diverse policy that executes all behaviors with equal probability and hence matches the true behavior distribution by design of the data collection process. Furthermore, as we sample 30 block configurations uniformly from configuration space (see Figure 77. For each on we simulate 16 end-effector trajectories. For given configuration, we count how often each of the four push- sequenee is executed successfully and use the result to calculate categorical distribution p(3 oo). Once repeated for all 30 configurations, we compute Ep(ao) [H(3)oo)].
set p(og) = 1/30
C.1.3 Table Tennis
Dataset. The table tennis dataset contains 5000 (o, a) pairs. The observations o E TR contain the coordinates of the initial and target ball position projection the table. Movement primitives (MPs) (11] are used to describe the joint space trajectories of the robot manipulator using two basis functions per joint and thus j2
Metrics. evaluate the different algorithms on the demonstrations recorded using the table tennis environment quantitatively, we employ two performance metries: The success rate and the distance error. The success rate the percentage of strikes where the ball is successfully returned to the opponent's is side. The distance error, is the distance between the target position and landing position of the ball for successful strikes.
C.1.4 Human Subjects for Data Collection
For the obstacle avoidance well the block pushing experiments used data collected by humans. We note that all human subjects included the data collection process are individuals who are collaborating on this work. The participants did, therefore, receive any financial compensation for their involvement in the study.
C.2 IMC Details and Hyperparameter
IMC employs parameterized inference network and conditional Gaussian distributions represent experts. For the latter, we use a fixed variance of and parameterize the means as neural networks. For both inference network and expert means we use residual MLPs (57). For all experiments, we use batch-size [B] (D), number of components N₂ = 50 and expert learning rate equal to 5 x 10 Furthermore, we initialized all curriculum weights as p(O,, a., [2) 1. For the table tennis and obstacle avoidance task, we found the best results using multi-head expert parameterization (see Section E.2 where we tested
4 layer neural networks. We found that using 1 layer with 32 neurons performs best on the table tennis task and layer with 64 neurons for the obstacle avoidance task. For the block pushing and Franka kitchen experiments, we obtained the best results using sigle-head parameterization of the experts. We used 6 layer MLPs with 128 neurons for both tasks. For the inference network, we used fixed set parameters that are listed in Table For the entropy scaling factor we performed hyperparameter sweep using Bayesian optimization. The respective values are 7 = 1/30 for obstacle avoidance, 7) = for block pushing and Franka kitchen and r = 1 for table tennis.
C.3 Baselines and Hyperparameter
We now briefly mention the baselines and their hyperparameters. We used Bayesian optimization to tune the most important hyperparameters.
19

---

Table IMC EM Hyperparameter.
PARAMETER
VALUE
EXPERT LEARNING RATE
10-4 44
EXPERT BATCHSIZE
1024
EXPERT VARIANCE (α²)
1
INFERENCE NET HIDDEN LAYER
INFERENCE NET HIDDEN UNITS
256
INFERENCE NET EPOCHS
800
INFERENCE NET LEARNING RATE
10-3
INFERENCE NET BATCHSIZE
1024
Mixture of Experts trained with AppectationMMaximizatinn (EM), The architecture of the mixture of experts model trained with EM [48] is identical to the one optimized with IMC: We employ parameterized inference network and conditional Gaussian distributions to represent experts with the same hyperparameters as shown in Table 2 Furthermore, we initialized all responsibilities as p(z|o) = 1/N,, where N₂ is the number of components.
shared backbone neural network with multiple heads for predicting component indices as well as the expert likelihood. For the experts, we employ conditional Gaussians with a fixed variance. The model likelihood is maximized in an end-to-end fashion using stochastic gradient ascent. We experimented with different backbones and expert architectures. However, found that the MDN is not able partition the input space in a meaningful way, often resulting in sub-optimal outcomes, presumably due to mode averaging. To find appropriate model complexity tested up 50 expert heads. We found that the number of experts heads did not significantly influence the results, further indicating that the MDN is not able to utilize multiple experts to solve sub-tasks. We additionally experimented with a version of the MDN that adds an entropy bonus to the objective (58) to encourage more diverse and multimodal solutions. However, we did not find significant improvements compared to the standard version of the MDN. For list of hyperparameter choices see 3
Mixture Density Network (MDN). The mixture density network [
Table 3: MDN Hyperparameter. The 'Value' column indicates sweep values for the obstacle avoidance task, the block pushing task, the Franka kitchen task and the table tennis task (in this order).
PARAMETER
SWEEP
VALUE
EXPERT HIDDEN LAYER
(1,2)
1,
EXPERT HIDDEN UNITS
(30,
50. 30.
BACKBONE HID. LAYER
4, 10)
3, 2,
BACKBONE HID. UNITS
(50, 100, 150, 200} [0.1,1]
200, 200, 200,
LEARNING X10
5.949. 7.748, 1.299, 2.577
EXPERT VARIANCE (o²)
MAX. EPOCHS
2000
BATCHSIZE
512
Denoising Diffusion Probabilistic Models (DDPM). We consider the denoising diffusion proba- bilistic model proposed by 24 Following common practice we parameterize the model as residual
MLP [27] with sinusoidal positional encoding (54] for the diffusion steps. Moreover, use the casine-based variance scheduler proposed by [59] For further details on hyperparameter choices see Table
Normalizing Flow (NF). For all experiments, we build the normalizing flow by stacking masked autoregressive flows [60] paired with permutation layers (18). As base distribution,
conditional isotropic Gaussian. Following common practice, we optimize the model parameters by maximizing its likelihood. See Table 5 for list of hyperparameters.
Conditional Variational Autoencoder (CVAE). We consider the conditional version of the autoen- coder proposed in (20). We parameterize the encoder and decoder with neural network with mirrored
20

---

Table 4: DDPM Hyperparameter: The 'Value' column indicates sweep values for the obstacle avoidance task, the block pushing task, the Franka kitchen task, and the table tennis task (in this order).
PARAMETER
SWEEP
VALUE
HIDDEN LAYER
(4,6, 10, 12)
6, 6, 8,6
HIDDEN UNITS
(50, 100, 150, 200}
200, 150, 200,
DIFFUSION STEPS
(5, 15, 25,50)
15, 15, 15,
VARIANCE SCHEDULER
COSINE
LEARNING RATE
10
MAX. EPOCHS BATCHSIZE
2000
512
Table 5: NF Hyperparameter. The 'Value' column indicates sweep values for the obstacle avoidance task, the block pushing task, the Franka kitchen task and the table tennis task (in this order).
PARAMETER
SWEEP
VALUE
NUM. FLOWS
(4, 6, 8, 10. 12) (50, 100, 150, 200) [0.01, 10]
6,6, 4,4
HIDDEN UNITS PER FLOW
100, 150, 200, 150
LEARNING RATE x10-4
7.43, 4.5, 4.62,
MAX. EPOCHS BATCHSIZE
2000
512
architecture. Moreover, consider additional scaling factor (3) for the KL regularization in the lower bound objective of the VAE as suggested in [6]].
Table 6: CVAE Hyperparameter. The 'Value' column indicates sweep values for the obstacle avoidance task, the block pushing task, the Franka kitchen task and the table tennis task (in this order).
PARAMETER
SWEEP
VALUE
HIDDEN LAYER
[4,6,8 8, 10, 12) 50, 100, 150, 200}
8, 10, 4,4 100, 150, 100, 100
HIDDEN UNITS
LATENT DIMENSION
(4, 16, 32, 64) [0 ,103]
32, 16, 16, 16
DEL SCALING (3)
1.641, 1.008, 0.452, 0.698 10-3
LEARNING RATE
MAX. EPOCHS
2000
BATCHSIZE
512
Implicit Behavior Cloning (IBC). IBC proposed in [1 and uses energy- based models to learn a joint distribution over inputs and targets. Following common practice we parameterize the model as neural network. Moreover,
use the version that adds gradient penalty to the InfoNCE loss [17]. For sampling, we use muddient-based Langevin MCMC [57]. Despite our effort, we could not achieve good results with IBC. A list of hyperparameters is shown in Table 7
Behavior Transformer (BET), Behavior transformers were recently proposed in 22 The model employs a minGPT transformer (22 to predict targets by decomposing them into cluster centers and residual offsets. obtain fair comparison, we compare our method to the version with no history. A comprehensive list of hyperparameters is shown in Table g
21

---

Table 7: IBC Hyperparameter. The 'Value' column indicates sweep values for the obstacle avoidance task and the table tennis task (in this order). We do not get any good results for the block push task and the Franka kitchen task.
PARAMETER
SWEEP
VALUE
HIDDEN DIM
(50, 100, 150, 200, 256}
200, 256
HIDDEN LAYERS
(4,6,8100 10}
4,6
NOISE SCALE
0.51
0.1662, 0.1
TRAIN SAMPLES
[8, 64]
44, 8
NOISE SHRINK
0.5
TRAIN ITERATIONS
20
INFERENCE ITERATIONS
40
LEARNING RATE
10
BATCH SIZE
512
EPOCHS
1000
Table 8: BET Hyperparameter. The 'Value' column indicates sweep values for the obstacle avoidance task, the block pushing task, the Franka kitchen task and the table tennis task (in this order).
PARAMETER
SWEEP
VALUE
TRANSFORMER BLOCKS
{2,3, 4, 6}
3,4, 6,
OFFSET LOSS SCALE
[1.0, 100.0, 1000.0}
1.0, 1.0, 1.0, 1.0
EMBEDDING WIDTH
(18, 72, 120}
96. 72, 120, 18
NUMBER BINS
(8, 10, 16, 64)
50, 10, 64,
ATTENTION HEADS
(4,6)
4,4,6,4
CONTEXT SIZE
1
TRAINING EPOCHS
500
BATCH SIZE
512
LEARNING RATE
10
D Extended Related Work
D.1 Connection to ML-Cur
This section elaborates on the differences between this work and the work by (30), both studies leverage curriculum learning paired with Mixture of Experts to facilitate the acquisition of diverse skills. However, it's essential to discern several noteworthy distinctions between our approach and theirs:
Firstly, Li et al. primarily focus on linear policies and gating distribution with limited flexibility. constraining the expressiveness of their learned policies. contrast, our approach allows for the use of arbitrarily non-linear neural network parameterizations for both, expert policies and gating.
Another divergence lies in the handling of mini-batches during training. While algorithm accommodates the use of mini-batches, Li et al.'s method does not support this feature. The ability to work with mini-batches can significantly enhance the efficiency and scalability of the learning process, especially when dealing with extensive datasets or intricate environments.
Additionally, Li et al.'s evaluation primarily centers around relatively simple tasks, that do not require complex manipulations, indicating potential limitations in terms of task complexity and applicability. In contrast, work is designed to address more intricate and challenging environments, expanding the range potential applications and domains.
I Lastly, it's worth noting that Li et al. specifically focus on the learning of skills parameterized by motion primitives [3T]. In contrast, our framework offers the versatility to encompass various skill types and parameterizations, broadening the scope of potential applications and use cases.
22

---

D.2 Connection to Expectation Maximization
In this section we want highlight the commonalities and differences between our algorithm and the expectation-maxiiizatoo (EM) algorithm for mixtures of experts. First, we look at the updates of the variational distribution q. Next, we compare the expert optimization. Lastly, we take a closer look at the optimization of the gating distribution.
The EM algorithm the variational distribution during the E-step
po_ (a, On 2 2))( O0)
c(2 = =p(=|0,, an)
Σ3 Po. (a,, n z) =)p(z(),,)
for all samples n and components 2. In the M-step, the gating distribution p(z|o) is updated such that the KL divergence between q(21o) and p(=[o) is minimized. Using the properties of the KL divergence, we obtain global optimum by setting p(zlon) = (==lon) for all and all 2. This allows us to rewrite Equation using the recursion in q, giving
pa. an On 2 2) )9)
(zzlo,((13)
Σ. Po_ (an On; =) (=(o0)))))
where (i) denotes the iteration of the EM algorithm. The update for the variational distribution of the IMC algorithm given by
p(o,, a. (1(2)))+1)
po. (an =))////200,, a,,
q(2 an (++1)
E. p(nn an An|=)(it1) Σ. Pa, (an g On a, )a)
Consequently, we that (220) (2) for 7 1. However, the two algorithms mainly differ in the M-step for the experts: The EM algorithm uses the variational distribution to assign weights to samples, i.e.
max Σ (=lo,,) log pe. (an on, 2),
whereas IMC uses the curricula as weights, that is,
NN
Σ p(o,, anl=)]og Po. (a, 1o z).
max O=
1=
This subtle difference shows the properties of moment and information projection: In the EM algorithm each sample o, contributes to the expert optimization as Σ, ((2/o,,) = However, all curricula ignore the nth sample, it will not have impact on the expert optimization. Assuming that the curricula ignore samples that the corresponding experts are not able to represent, IMC prevents experts from having average over 'too hard' samples. Furthermore, this results in reduced outlier sensitivity as they are likely to be ignored for expert optimization. Lastly, we highlight the difference between the gating optimization: Assuming that both algorithms train gating network (2(0)) have
max Σ Σ Φ
((=lon) log (2 (2(o)))))
for the EM algorithm and
max ΣΣ p(o.. a,/²) log
for IMC. Similar the expert optimization, EM includes all samples to fit the parameters of the gating network, whereas IMC ignores samples where the unnormalized curriculum weights p(0,, a,/22) are zero for all components.
E
Algorithm Details & Ablation Studies
E.1
Inference Details
provide pseudocode to further clarify the inference procedure of our proposed method (see Algorithm 2)
23

---

Algorithm 2 IMC Action Generation
1: Require: Curriculum weights {p(o,, a, nn (I. N7.2 [l,..,,N.].
2: Require: Expert parameter [0z] E {I,.. Nz}}
3: Require: New observation o"
4: if not parameter_aa pptptted then
5: Φ arg m En, E. p(o,, 2n log (z((o,))
6: parameter_updated f- True
7: end if
8: Sample z' ~ Is (zlo*)
9: Sample a' po. (alo", 2²)
10: Return a'
E.2 Expert Design Choices
Distribution. In our mixture of experts policy, we employ Gaussian distributions with a fixed variance to represent the individual experts. This choice offers several benefits in terms of likelihood calculation, optimization and ease of sampling:
To perform the M-Step for the currieula (Section 3.3 exact log-likelihood computation is necessary. This computation becomes straightforward when using Gaussian distributions, Additionally, when Gaussian distributions with fixed variances are employed to represent the experts, the M-Step for the experts simplifies to a weighted squared-error minimization. Specifically, maximizing the weighted likelihood reduces to minimizing the weighted squared error between the predicted actions and the actual actions.
The optimization problem for the expert update be formulated follows:
0: arg max 0,
Σ p(on. an 2n log po. (an/on: 2) = arg min
Σ p(on. ml (o.) a, l1..
6:
This optimization problem can be efficiently solved using gradient- based methods. Lastly, sampling from Gaussian distributions is well-known be straightforward and efficient.
Parameterization. We experimented with two different parameterizations the Gaussian expert means μ4a,, which we dub single-head and multi-head: For single-head, there is parameter sharing between the different experts. Each expert has its own set of parameters 0z. As a result, we learn N; different multi-layer perceptrons (MLPs) μe. R| + (lA, where N₂ is the number of mixture components. In contrast, the multi- parameterization experts and hence allows for feature sharing. We thus learn a single MLP μr R|O) - N.AA
global set of parameters 0 for all
To compare both parameterizations, we conducted ablation study where evaluate the MoE policy on obstacle avoidance, table tennis and Franka kitchen. In order to have a similar number of parameters, we used smaller MLPs for single-head, that is, 4 layers whereas for multi-head we used a 6 layer MLP. The results are shown in Table 9 and are generated using 30 components for the obstacle avoidance and table tennis task. The remaining hyperparameters are equal to the ones listed in the main manuscript. For Franka kitchen, we report the cumulative success rate and entropy for different number of completed tasks. We report the mean and standard deviation calculated across 10 different seeds. Our findings indicate that, in the majority of experiments, the single-head parameterization outperforms the mutli-head alternative. Notably, we observed substantial performance disparity, especially in the case of Franka kitchen.
Table 9: Expert Parameterization Ablation:
We compare IMC with single- and multi-head expert parameterization. For further details, please refer to the accompanying text.
OBSTACLE AVOIDANCE RATE (1)
TABLE TENNIS
FRANKA KITCHEN
ARCHITECTURE
(1)
SUCCESS RATE
ERR.
RATE (T)
(1)
SINGI E-HFAD
0. 0.800110 035
0.887190013 0.930+2.0SL
0
O. 1.008
3.644 200
6.180
MULTI-HEAD
0.870=0017
0.153:0.00
3.218
8.657
24

---

Expert Complexity. We conducted an ablation study to evaluate the effect of expert complexity on the performance of the IMC algorithm. The study involved varying the number hidden layers in the single-head expert architecture while assessing the IMC algorithm's s performance on the Franka kitchen task using the cumulative success rate and entropy. The results, presented Figure were obtained using the hyperparameters specified in the main manuscript. Mean and standard deviation were calculated across different seeds. Our findings demonstrate positive correlation between expert complexity and achieved performance.
Success rafe
entropy
2
6
hidden layer
hiddle layer
Figure 9: Expert Complexity Ablation: Evaluation of the IMC algorithm on the Franka kitchen task
with varying numbers of hidden layers in the single-head expert architecture.
E.3 Component Utilization
We further analyze how IMC utilizes its individual components. Specifically, assess the en- tropy of the weights assigned to these components, denoted as H(2) and defined as H(z) = p(2) log p(2). Maximum entropy occurs when the model allocates equal importance to all components to solve task, which implies that p(z) 1/Ng. Conversely, the model relies solely on single component, the entropy H(z) equals zero. We conduct the study the Franka kitchen task, evaluating it through cumulative success rates and entropy. The results are shown Figure T0a We computed the mean and standard deviation based on data collected from 5 different seeds. Our findings reveal that IMC exhibits substantial component utilization even when dealing with a large number of components, denoted as Nz.
E.4 Number of Components Sensitivity
To investigate how the algorithm responds to changes in the number of components Nz, we per- formed comprehensive ablation study. Figure Tob shows the success rate of IMC the Franka kitchen task using varying number of components. Our findings indicate that the algorithm's 's performance remains stable across different values of Ng. This robust behavior signifies that the algorithm can adeptly accommodate varying numbers of components without experiencing substan- tial performance fluctuations. Hence, opting for higher number of components primarily entails increased computational costs, without leading to overfitting issues.
succoss rate
H(2)
3.5
IMC
1/N:
0
2.5
100
150
200
100
200
N₂
N₂
(a)
(b)
Figure 10: Figure 10a shows the component utilization of IMC: The entropy of IMC's mixture weights p(z) close to the entropy of uniform distribution 1/N, indicating that the algorithm uses all components for solving task, even for a high number of components N₂.
25

---

E,5 Curriculum Pacing Sensitivity
To examine the algorithm's sensitivity to the curriculum pacing parameter 7), we conducted an ablation study. Figure [1 presents the results obtained using 30 components for the obstacle avoidance and table tennis tasks, while maintaining the remaining hyperparameters as listed in the main manuscript. For the Franka kitchen task, we analyzed the cumulative success rate and entropy across varying numbers of completed tasks. The mean and standard deviation were calculated across different seeds. Our findings reveal that the optimal value for is dependent on the specific task. Nevertheless, the algorithm exhibits stable performance even when ry values differ by an order of magnitude.
84
84
0.4
02
02
(a) Obstacle avoidance
(b) Table tennis
100
(c) Franka kitchen
Figure 11: Curriculum Pacing Sensitivity: Sensitivity analysis the IMC algorithm for perfor- mance metrics in the obstacle avoidance, table tennis, and Franka kitchen tasks, considering varying curriculum pacing (7)) values. The results illustrate the mean and standard deviation across 5 different seeds.
26


Reviews:
[{"review_id": "fd0YMZWUJa", "reviewer": "Reviewer_kVDY", "summary": "The paper proposes a learning protocol that learns multiple policies (\"skills\"), distribution over skills, and a per-skill priority over experience buffer. This is achieved by maximizing a variational lower bound of a certain averaged regularized KL distance. Namely, the objective is a sum of two terms. The first term is a KL distance between the per-skill priority over previously seen actions and a policy, both conditioned on the skill and previously seen observation, and averaged over them. A second term is an entropy of the joint distribution over experience and skills, which acts as a regularizer, forcing it to be as diverse as possible.\n\nIn the experiment section, the skill policies share a common backbone and output a mean of an isotropic Gaussian distribution. The paper tests the method over several tasks, such as Obstacle Avoidance, Block Pushing, Franka Kitchen, or Table Tennis. The approach compares (mostly) favorably against seven baseline methods.", "questions": "Consult the Weaknesses section.", "limitations": "The paper includes a brief limitations section. What could also be mentioned is that the method requires setting several important hyperparameters (see the review), some of which require non-trivial knowledge about data. The method also was tested on continuous tasks, so a natural question (limitation or future research) would be to ask about performance on discrete domains, such as combinatorial puzzles like chess or video games like Atari benchmark.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper proposes a simple objective to train multiple policies at once, such that together they cover multiple modes in the dataset. The results show that the method (mostly) performs better than the considered baselines.", "weaknesses": "The empirical part could be improved. Namely:\n* Performance metrics should be defined in the main body of the paper:\n\t* There could be a dedicated section for that purpose. This would improve the exposition and allow more discussion about the method in Sections 5.1-5.4.\n\t* This could also make reading the results (e.g., Table 1) more accessible. For instance, the definition of entropy varies across environments, as shown in Appendix C, which makes the numbers incomparable and possibly draw wrong conclusions.\n\t* The discussion on the range of entropy values should be moved from the Appendix to the main body. It provides grounding for the numbers in Table 1.\n\t* Additional place for this and a more in-depth discussion of the results could be achieved by moving Section 2.1 and Figure 1 to the Appendix (it is a well know property of KL). Similarly, some parts of Section 3 could be moved to the Appendix.\n* The paper does not provide numbers on how the algorithm mixes between skills, e.g., the entropy of the distribution over components ($p(z)$).\n* It seems that each environment required a different setup of the method. This is a limiting factor for the method. The paper, however, does not guide the potential user of the method in choosing the relevant hyperparameters.  For example, estimating how many modes are in the data can be a non-trivial task, and consequently, setting the number of components ($N_z$) selected for each experiment may be non-trivial. The paper has no information regarding the actual values of $N_z$ chosen for each experiment.\n* Descriptions in Sections 5.1-5.4 are mainly technical and deal with the setup. The discussion about the results of IMC is limited to one or two sentences that do not add more information than Table 1 or Figure 6. A reader would expect that most of such a section would provide real insights about the method (e.g., what skills were learned, how they were acquired during training, where all modes were discovered, how a distribution that mixes skills looks, etc.).\n* Figure 6: Shouldn't we expect that the performance is an increasing function of the number of components (the more modes covered, the higher the objective)?\n* There is no mention of what values $\\eta$ were chosen.\n* Other issues:\n\t* Table 1: The description is not self-explanatory. What is the selected number of components? What versions of entropy are used? What is the setup of experiments? Etc.\n\t* Sections 5.1, 5.2, and 5.4 refer to Table 13, which is not present in the paper. Most likely, it should refer to Table 1.\n\t* Figure 6 precedes Figure 5.\n\nThe quality of the technical part of the paper could be improved and more clearly agitated. In particular, \n* Section 2.2: \n\t* There is no definition of $\\mathcal O$, $\\mathcal A$, $p$, $z$.\n\t* Is $z$ a continuous or discrete random variable? From the context of the following sections, it would seem that the latter.\n* Section 3.1:\n\t* It seems more clear to write $p(o, a)$ instead of $p(\\mathcal D)$.\n\t* In equation (2), there should be $\\mathcal H(p(\\mathcal D))$ (or better $\\mathcal H(p(o,a))$, see the item above). Additionally $\\mathbb E_{p(\\mathcal D)}$ could be more transparent if written as $\\mathbb E_{o,a\\sim p(\\mathcal D)}$.\n\t* The formulas in lines 80, 85, and 86 should be better justified. There is only a cryptic comment in line 81 about optimization in an alternating fashion, which is not justified, nor a suitable reference is given.\n\t* The description in lines 87-93 is relatively informal and does not refer to the formulas in lines 80, 85, and 86. Additionally, the text uses colloquial terms such as \"representational capacity of the policy\", \"capacity [..] is exhausted\", or alludes to the convergence of curriculum, which was not proved.\n\t* There is no definition of $\\mathcal D_n$ (lines 85, 86, 119, 120, 124, 129, 130, etc.). Why not just write $(o_n, a_n)$?\n\t* In the proof of Proposition 3.1 (Appendix A.1.), the formulas read $p^*(o)$ where the Proposition refers to $p^*(z)$. A similar comment refers to other proofs in Appendix A.\n* Section 3.2\n\t* There is no description of the objective $J(\\psi)$ and its lower bound, showing how the individual terms promote desirable behavior. \n\t* In equation (3), it seems that the entropy term should be equal to $\\mathcal H(p(o, a, z))$; otherwise, equation (4) seems not to be valid.\n\t* It seems that in equation (4) and in the definition of $R_z$, there should be $q(z|o,a)$ (in place of $q(z|o)$ and $q(z|\\mathcal D)$, respectively). Additionally, it would make sense to define $R_z$ as a function of $(o, a)$ (instead of $\\mathcal D$).\n\t* Similar comments apply to Sections 3.3-3.5.\n\nEdit: After the Authors' rebuttal, I have raised the score (4->6).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "In order to adhere to the character limit we will jointly address most of the concerns regarding the empirical part and those of the technical part.\n\n**Regarding concerns on the empirical part:**\n\nWe agree with the reviewers that moving the definition of the entropy for the different experiments to the main body of the paper increases readability and makes the results more accessible. To that end, we moved Section 2.1 (explanation of KL properties ), Figure 4, and some of the algorithmic details (Section 3.5) to the Appendix. Additionally, we improved the discussion of the results and highlighted unique aspects of different environments instead of repeating similar conclusions. \nFurthermore, we moved parts of the explanation about the hyperparameters of IMC (which encompasses all design choices incl. $\\eta$ and $N_z$ for all experiments) from Appendix C.2 to Section 5. We also made Table 1 self-explanatory by elaborating on the setup and on the metrics used. Moreover, we improved the result discussion by highlighting unique aspects of the different tasks and our method.\n\n> The paper does not provide numbers on how the algorithm mixes between skills ff.\n\nFirst, we want to clarify potential confusion about the relationship between components $z$ and skills learned by the policy. For the following discussion, we assume that the reviewer refers to a skill as a sequence of actions that lead to a successful outcome. Each $z$ employs a curriculum to specialize on a subset of the data. Depending on the model complexity of the expert, this subset could encompass multiple skills or only part of a skill. Hence, there is no one-to-one correspondence between skills and $z$. Therefore, there is no need for knowing the number of skills or modes a-priori. We used $N_z=50$ for all experiments which empirically worked well. To clarify this further, we added a figure to the supplementary PDF that visualizes the curriculum.\n\nWe believe that reporting the entropy $\\mathcal{H}(p(z))$ provides valuable insights on how many of the components are used to solve a task. We will add an ablation study showing $\\mathcal{H}((p(z))$ for various $N_z$ to the appendix.\n\n>Figure 6: Shouldn't we expect that the performance is an increasing function of the number of components ff.\n\nThe performance metrics presented in Figure 6, namely the success rate and entropy, differ from the optimization objective $J(\\psi)$. While we do observe empirically that adding more components to the model tends to increase $J(\\psi)$, the same does not necessarily hold true for the performance metrics.\n\nThe reason behind this observation lies in the fact that as we add more components to the model, the overall model complexity increases. Consequently, the model becomes more susceptible to overfitting on the training data. We conjecture that this causes the fluctuations in the success rate in Figure 6a. \n\n**Regarding concerns on the technical part:**\n\nWe appreciate the reviewers' feedback regarding the notation for the curriculum $p(\\mathcal{D})$. We understand the potential confusion that might arise from this notation. The intention behind using this notation was to emphasize that the curriculum represents a categorical distribution over data points rather than a continuous joint distribution between actions and observations $p(o,a)$.\n\nUpon careful consideration and in light of the existing MoE literature, which commonly employs $(o,a)$ to denote responsibilities as $q(z|o_n,a_n)$, we find the suggested notation to be more aligned and clear. Therefore, we gladly adopt the notation $(o,a)$ in our work. Moreover, we defined $R_z$ as a function of $(o,a)$ and replaced $\\mathcal{D}_n$ with $(o_n,a_n)$.\n\n>- The formulas in lines 80, 85, and 86 should be better justified ff.\n>- [...] or alludes to the convergence of curriculum, which was not proved\n\nWe add a sketch of the convergence proof of the objective in line 80 as well as $J(\\psi)$ to the 'global' comment of the rebuttal (please note that we used the old notation for the curriculum to make it accessible to other reviewers). We add a detailed proof to the appendix in the final version of the paper.\n\nWe agree with the reviewers and 1) explain the equations 2) add references and 3) replace colloquial terms with technical statements. For 1) and 2) we refer to the global comment. Regarding 3):\n- The statement 'capacity [...] is exhausted' refers to the iteration where the optimization in line 80 reaches a fixed point in $\\theta$\n- The term 'representational capacity' refers to the model complexity. The statement 'samples that lie within the representational capacity' refers to samples where the expert is able to achieve high log-likelihood values\n\n>- There is no description of the objective $J(\\psi)$ and its lower bound, showing how the individual terms promote desirable behavior.\n\n- We focussed on the description of the per-component objective $J_z$. We believe that this explanation of the objective is the most intuitive, as $J_z$ is similar to the single expert objective (Section 3.1) with an additional term $\\log q(z| o, a)$. We will add a description explaining why $\\log q(z| o, a)$ promotes the claimed behavior.\n\n>[...] the method requires setting several important hyperparameters ff.\n\nWe argue that the only hyperparameter that needs to be tuned is the curriculum pacing (or entropy scaling) $\\eta$. We promote future research in an automatic tuning of $\\eta$, similar to the approach proposed in [1].\n\n\n\n> [...] a natural question [...] would be to ask about performance on discrete domains ff.\n\nWe will add discrete domains as promising avenue for future research.\n\nWe thank the reviewers for noticing errors and notational shortcomings in the paper. We have addressed them and made the necessary corrections. Moreover, the valuable feedback has immensely contributed to the improvement and credibility of our research. \n\n[1] Haarnoja, Tuomas, et al. \"Soft actor-critic algorithms and applications.\" "}, {"review_id": "VAvwManNgS", "reviewer": "Reviewer_EzYZ", "summary": "This paper proposes a curriculum based approach for imitation learning. Overall, the imitation learning problem is posed as a conditional density estimation problem. Given the multi-modal nature of underlying data, this paper proposes to learn a curriculum based mixture of expert policy. Intuitively, each expert is only responsible for learning a subset of the training data and learns the policy for this subset using reverse KL. Experimentally, the proposed approach is verified in 4 different environments and compared against both common and recently proposed approaches.", "questions": "Overall the approach is broadly similar to an EM approach with mixture of policies. As noted, the main difference being that the proposed approach uses a curriculum weighted optimization objective to assign weights to samples instead of directly using the variational distribution. While algorithmically, this difference is not large in terms of empirical results the EM based approaches lags behind the proposed approach quite a lot (especially for the obstacle avoidance task). Do the authors have an intuition why the EM approach suffers so much in comparison?", "limitations": "Limitations are discussed", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents an interesting and grounded approach for learning from multimodal data distribution. The overall idea of using a curriculum to weight data samples based on how well they are represented under the expert policy seems to be generally useful. The approach is shown to work in diverse settings and seems competitive with recent results.", "weaknesses": "*Baseline results:* Overall, the proposed approach provides very little significant advantage over simpler baselines. For instance, in Figure 4 and Figure 5 we can see that for most tasks (Pusing, Tennis, Kitchen), a diffusion model based approach (DDPM) is highly competitive to the proposed approach (success rate difference is less than 0.05) across all tasks. Only in the Obstacle avoidance task, does DDPM approach suffer although CVAE still performs quite well (unclear why DDPM performs poorly here). \n\n*Benchmark tasks:* While the paper tries to evaluate the proposed approach on multiple datasets, most of the tasks/datasets are not commonly used across multimodal tasks (only Franka-Kitchen is the commonly used dataset). Given the large set of recent works in this area it would be much better to evaluate on tasks which other recent works evaluate. For instance, most recent works evaluate on RoboMimic dataset [1] and the block pushing task from IBC [2]. Both of these task suites have human demonstrations available and since the underlying data distribution is highly important for these proposed approaches, reusing datasets from previously proposed approaches will allow for a much fairer comparison.\n\n[1] Robomimic: https://github.com/ARISE-Initiative/robomimic\n[2] Florence et al. Implicit Behavior Cloning\n\n--- Post Rebuttal ---\n\nI don't see any issues with the paper the provided code also looks reasonable and reproduces the results so I would update my recommendation to Weak Accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The paper presents an interesting and grounded approach for learning from multimodal data distribution. ff.\n\nWe thank the reviewers for their feedback on our paper. We are delighted to hear that the overall idea of using a curriculum to weight data samples is well-received. We would like to address address the mentioned weaknesses / questions:\n\n> Baseline results: Overall, the proposed approach provides very little significant advantage over simpler baselines. For instance [...] a diffusion model based approach (DDPM) is highly competitive to the proposed approach ff.\n\nWe appreciate the reviewers' comment and the opportunity to discuss the advantages of our approach in comparison to other baselines. Our method indeed offers distinct benefits over alternative techniques, such as DDPM.\n\nOne significant advantage lies in the inference efficiency of IMC. DDPM, due to its sequential inference procedure, suffers from longer inference times, which can be impractical for applications with real-time constraints. This limitation becomes especially pronounced when deploying models to real-world systems. In contrast, IMC's inference process requires just two forward passes: one through the inference network for sampling $z$ and another through the corresponding expert. This efficiency is particularly advantageous compared to DDPM's requirement of $N_{\\text{diffusion steps}}$ forward passes. Furthermore, we observered that our method requires less than $1/10$th of the training time that is required to train DDPM. We thank the reviewers and will include an ablation study on inference and training times in the camera-ready version of the paper. \n\nRegarding the term 'simpler baselines', we are uncertain about the reviewers' defintion of 'simple'. We understand that baselines, including DDPM, are far from being simple in terms of their design choices and intricacies. DDPM involves non-trivial decisions concerning parameters like the number of diffusion steps, variance scheduling, time embedding, and employs additional code-level optimizations such as using an exponential moving average for the parameters. The same holds for other baselines such as normalizing flows, energy based models and behavior transformers. \n\n> Only in the Obstacle avoidance task, does DDPM approach suffer although CVAE still performs quite well (unclear why DDPM performs poorly here). \n\nWe conjecture that DDPM's subpar performance can be attributed to the limited amount of data available for the obstacle avoidance task. IMC, in contrast, appears to circumvent this challenge due to its composition of 'simpler' Gaussian experts, which appear to generalize better on little data.\n\nRegarding CVAE: While CVAE has a high sucessrate, it suffers from low entropy values, i.e., is not able to cover multple modes in the data distribution. This can also be seen in Figure (3). We've empirically observed that a trade-off often exists between achieving high success rates and maintaining adequate entropy levels across most models (see Table 1). However, IMC stands out as an exception by simultaneously achieving high success rates and commendable entropy values. \n\n\n> Benchmark tasks: [...] most recent works evaluate on RoboMimic dataset [1] and the block pushing task from IBC [2]. Both of these task suites have human demonstrations available ff.\n\nWhile we agree that the RoboMimic datasets are excellent for evaluating a models' performance in complex manipulation tasks, we opted not to use them since a) the tasks within the RoboMimic datasets (lift, can, square, tool hang and transport) lack multimodality, except for the multimodality evident in human demonstrations and b) they do not provide suitable metrics to evaluate a models' ability to cover different modes present in the dataset. \n\nRegarding the block pushing task from IBC: Unfortunately, this dataset is not recorded by humans but uses a scripted policy (Quote IBC paper, Section 4: [...]We evaluate implicit (EBM) and explicit policies on both variants, trained from a dataset of 2,000 demonstrations using a scripted policy that readjusts its pushing direction if the block slips from the end effector.[...]). We found that this task can be solved by simple methods such as behavior cloning (see IBC paper Table 3, MSE). Therefore, we decided to rebuild the task and use human-recorded data which significantly increases the diffuculty due to the inherent versatility in human demonstrations. Furthermore, we want to highlight that we've acknowledged the resemblance between our task and the one introduced in the IBC paper. We provided an explanation for our decision to redevelop the task, as outlined in Appendix C.1.2. \n\nWe carefully designed new tasks and metrics and recorded data to circumvent these shortcommings of existing benchmarks such as Robomimic and intend to publish them, as well as additional tasks, in a separate work. This endeavor aims to establish a benchmark for versatile imitation learning, focusing on quantifying a model's capacity to acquire diverse skills.\n\n\n> Overall the approach is broadly similar to an EM [...]. Do the authors have an intuition why the EM approach suffers so much in comparison?\n\nWe believe that the primary reason for the relatively poorer performance of EM, compared to IMC, lies in the inherent requirement of EM to account for all data points during training. This constraint forces EM-trained models to handle outliers or challenging samples that might lead to suboptimal log-likelihood outcomes. In contrast, IMC's curriculum-based approach enables the model to focus on samples where experts perform well (in terms of log-likelihood), allowing them to disregard outliers or poorly performing instances, resulting in the observed performance boost.\n\nWe extend our gratitude to the reviewers for dedicating their time and effort to evaluating our work. Their valuable comments and suggestions helped enhancing and fortifying the quality of our manuscript."}, {"review_id": "dw0SIcasEQ", "reviewer": "Reviewer_tAUb", "summary": "The authors study a method, Information Maximizing Curriculum (IMC), that performs behavioral cloning by having the model selectively choose a learned weighing of the demonstration data for which the model is best at predicting (via minimizing the reverse KL divergence). To avoid the mode-seeking behavior of this approach, the authors extend the method to make use of K such model components, leading to a mixture of experts (MoE) approach, whereby each component selectively models the distribution in this way, while maximizing their joint information projection over the dataset (via simultaneously maximizing the entropy of the MoE distribution). Experiments in two robotics  simulators show it outperforms several other baselines based on generative models and basic MoE methods trained via expectation-maximization and backpropagation.", "questions": "- This study primarily focuses on continuous control robotics simulators. Do you have a sense of whether IMC can be made to work in discrete control settings, e.g. Atari, chip design, web navigation.\n- Are there cases where maximizing mode coverage is disadvantageous? Often demonstration datasets contain a mix of approximately optimal and very suboptimal trajectories. Could IMC lead to learning MoE that perform worse in these settings?", "limitations": "This work only focuses on fairly simple continuous control tasks. Scaling this to more complex imitation learning settings (e.g. from pixels or controlling a much more complex action space) as well as discrete control settings would add confidence in the utility of this approach. Relatedly, demonstrating success on these more challenging settings would be a great way to highlight why IMC is an important improvement over the Li, et al (2023) work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is overall well written. The method and its motivations are clearly communicated.\n- The authors compare to several important baselines, spanning generative models, energy-based models, and MoE methods, and show strong performance improvements.", "weaknesses": "The main weakness I see is that this work seems almost identical to Li, et al (2023), which proposes largely the same method. Moreover, this prior work looks at a very similar set of environments as this paper. While the authors cite this prior work, they do not directly compare to it. Given the extremely close similarity between the two methods, it seems important to compare to this work to justify the contribution in this paper, which in a sense, is an extension of the method in Li, et al (2023) by incorporating neural networks.\n\nI see two main ways to show improvement over this prior work: The authors can either (1) show their method outperforms the approach in Li, et al (2023) on the tasks studied, or (2) Demonstrate clearly how IMC can scale to environments in which the method of Li, et al (2023) cannot, thereby clearly justifying the strengths of this extension. I imagine neither of these aims is too difficult, but such a comparison seems sorely missing, given the strong similarity between the two works.\n\nLastly, I find it odd that the authors couch their method as “curriculum learning,” ignore the field of active learning altogether, and proceed to imply that their method is novel in not requiring an a priori difficulty metric for each datapoint. The authors should relate their method to active learning, which is a mature field of study, and most active learning methods can be viewed as “curriculum learning” without any a priori notion of task difficulty.\n\n### References\n\nMaximilian Xiling Li, Onur Celik, Philipp Becker, Denis Blessing, Rudolf Lioutikov, and Gerhard Neumann. Curriculum-based imitation of versatile skills. arXiv preprint arXiv:2304.05171,\n2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The paper is overall well written. ff.\n\nWe thank the reviewers for acknowledging our contribution and are committed to addressing your questions and concerns.\n\n> The main weakness I see is that this work seems almost identical to Li, et al (2023), which proposes largely the same method.\n\nWe agree with the reviewers that the method proposed by Li, et al (2023) (ML-Cur) and our work seem very similar. However, we want to point to some of the differences:\n\n- ML-Cur assumes Gaussian curricula. The parameters of these distributions must be updated in every iteration of the algorithm and involve expensive matrix inversions. The gating distribution has limited flexibility due to the Gaussianity assumption. In contrast, IMC uses non-parametric curricula $p(\\mathcal{D}|z)$ and trains a gating network ($g_{\\phi}(z|\\mathcal{D})$) once after training which is more efficient and allows for highly non-linear partitioning of the observation space.\n- ML-Cur does not admit mini-batch updates for the model parameters. Therefore, the method does not scale to large datasets. Conversely, IMC accommodates mini-batch updates (refer to Section 3.5), enabling effective scalability.\n- ML-Cur has to update the mixture weights $p(z)$ in every iteration. IMC has implicit updates due to Proposition 3.1.\n- ML-Cur uses linear conditional Gaussian distributions to represent the experts. In contrast, IMC allows for arbitrary experts where exact likelihood computation is possible. In particular, we used non-linear conditional Gaussian distributions in our experiments.\n- ML-Cur is evaluated on *episodic* tasks, i.e., where the model has the learn a mapping from a context to movement primitive parameters which represent the whole trajectory. A much more common setting is where the model has to learn a mapping from observations to actions (*step-based*). In that case, a trajectory is formed by performing multiple steps in the environment of a given task. In our work, 3 out of the 4 tasks consider the latter setup.\n\n> Moreover, this prior work looks at a very similar set of environments as this paper.\nThe tasks used in the work of Li, et al (2023)\n\nDespite the similarity in appearance of the environments used in the work of Li, et al and our work, there exist major differences:\n\n- Their tasks do not require complex manipulations of objects in contrast to the Block Pushing task as well as Franka Kitchen used in our work.\n- They consider input spaces with very low dimensions ($<4$) whereas our work considers up to $30$ (Franka Kitchen)\n- They focus on learning from a few data points ($<5000$) while our method looks at datasets with up to $463k$ samples (Block Pushing).\n- Their environments require little generalization capabilities of the models (this can be seen from the highly competitive results achieved by the k-nearest neighbor algorithm). Our environments are much more complex, see e.g. the low performance of behavior transformers (BET) and denoising diffusion models (DDPM) on Obstacle Avoidance, Block Pushing, or Franka Kitchen.\n- They do not provide performance metrics for quantifying how versatile the learned policy is. In contrast, we provide such a metric for all environments except for the table tennis task.\n\n> While the authors cite this prior work, they do not directly compare to it. ff.\n\nWe agree with the reviewers that the comparison to the work of Li, et al (ML-Cur) is missing. Therefore, we added quantitative and qualitative comparisons between IMC and ML-Cur in the 'global comment' of the rebuttal and in the accompanying PDF file. We thank the reviewers and will include these results in the final version of the paper.\n\n> I see two main ways to show improvement over this prior work: The authors can either (1) show their method outperforms the approach in Li, et al (2023) on the tasks studied, or (2) Demonstrate clearly how IMC can scale to environments in which the method of Li, et al (2023) cannot ff.\n\nWe thank the reviewers for their suggestions. We believe that by including the comparison to ML-Cur (see last comment) in the paper we demonstrate (1) that our method significantly outperforms ML-Cur on the tasks studied and (2) show ML-Cur is not able to scale to the environments used in our work.\n\n> Lastly, I find it odd that the authors couch their method as “curriculum learning,” ignore the field of active learning altogether. ff.\n\nWe agree that we should relate our method to active learning. To that end, we will include a section in Chapter 4, elaborating on the commonalities and differences between our work and active learning.\n\n> This study primarily focuses on continuous control robotics simulators. Do you have a sense of whether IMC can be made to work in discrete control settings, e.g. Atari, chip design, web navigation.\n\nAbsolutely! While this study primarily focuses on continuous control robotics simulators, our algorithm is applicable for training mixture of experts models whenever the likelihood of the experts can be defined and optimized, including the discrete control setting.\n\n> Are there cases where maximizing mode coverage is disadvantageous? ff.\n\nYes, we believe there are cases where maximizing mode coverage can be disadvantageous. However, this is a problem of divergence minimization in general and we believe that other methods for learning MoE could lead to even worse performances, as they aim to cover all data even if the model complexity is not sufficient (property of the forward KL). There are dedicated fields of study such as offline reinforcement learning [1], that address problems associated with learning from suboptimal data.\n\n[1] Levine, Sergey, et al. \"Offline reinforcement learning: Tutorial, review, and perspectives on open problems.”\n\nWe sincerely appreciate the time and effort the reviewers have invested in assessing our work. Their comments and suggestions have significantly contributed to refining and strengthening the manuscript. "}, {"review_id": "icpbwE9vKu", "reviewer": "Reviewer_kzo3", "summary": "This paper proposes an imitation learning method IMC that can model multi-modal behaviors. IMC avoids the mode-averaging issue with an objective similar to reverse-KL. To cover all modes in the dataset, IMC further introduces a mixture model with multiple components, each focusing on different data distribution it specializes to. The mixture model is optimized with the EM algorithm. The authors provide extensive experiments over simulated control environments and demonstrate IMC's superior modeling abilities.\n", "questions": "1. How sensitive is the performance of IMC with respect to different choices of $\\eta$?\n2. Can IMC be scaled up to scenarios with high-dimensional input?\n", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. IMC is a well-motivated method for multi-modal density estimation and provides an elegant reverse KL-based solution.\n2. The experiments in low-dimensional control environments are extensive. IMC is compared against major generative models (with maximum likelihood objective) and addresses the mode-averaging issue better.\n3. The visualization of learned trajectories clearly demonstrates the learned modes of different methods and the natural mode-ignorance ability of IMC.\n ", "weaknesses": "The presentation in the experiment section could be improved. Currently, the subsections for different environments repeat mostly the same conclusions that IMC achieves high success rates and covers diverse modes. I think it is better to emphasize anything special for different environments or different baselines.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. IMC is a well-motivated method for multi-modal density estimation and provides an elegant reverse KL-based solution.\n>  2. The experiments in low-dimensional control environments are extensive. IMC is compared against major generative models (with maximum likelihood objective) and addresses the mode-averaging issue better.\n> 3. The visualization of learned trajectories clearly demonstrates the learned modes of different methods and the natural mode-ignorance ability of IMC.\n\n\nWe thank the reviewers for their positive feedback.  We are glad the reviewers recognize our contribution in addressing the mode-averaging problem. We now aim to thoroughly address the concerns raised by the reviewers.\n\n> The presentation in the experiment section could be improved. Currently, the subsections for different environments repeat mostly the same conclusions that IMC achieves high success rates and covers diverse modes. I think it is better to emphasize anything special for different environments or different baselines.\n\nWe are grateful to the reviewers for their valuable suggestion. We agree that the content in the experiment section currently provides limited additional insights beyond what's presented in the tables and figures within the paper. Consequently, we are committed to enhancing the experiment section by placing a stronger emphasis on distinct characteristics of various environments.\n\n> How sensitive is the performance of IMC with respect to different choices of $\\eta$? \n\nTo address your question thoroughly, we have included a comprehensive analysis of IMC's performance sensitivity in the supplementary material of the paper (Appendix E.2). In that section, we discuss the impact of various values of $\\eta$ on the model's performance on the Obstacle Avoidance, Table Tennis and Franka Kitchen task.\n\n> Can IMC be scaled up to scenarios with high-dimensional input?\n\nIndeed! IMC can be effectively scaled up to scenarios with high-dimensional input. A viable approach to achieve this scalability is by utilizing an encoder, such as a Convolutional Neural Network (CNN) encoder, to map the high-dimensional inputs (such as images) to a lower-dimensional space before passing them to the inference network or experts.\n"}]

Author rebuttal:
We express our gratitude to the reviewers for their valuable suggestions and constructive feedback. Here, we post additions that might be of interest for all reviewers:

We have included a proof sketch, outlining the convergence of the algorithm proposed in our work, providing a more thorough understanding of its theoretical foundations.

Additionally, we have incorporated a comprehensive comparison between our proposed approach and the work introduced by Li et al. (2023) [1].

Lastly, we added a supplementary PDF file which contains
1. further quantitative and qualitative comparisons to the work of Li et al.,
2. a visual demonstration of the curricula of IMC for a different number of components on the obstacle avoidance task,
3. a figure that accompanies the convergence proof and visualizes the lower bound over training iterations. 


We firmly believe that these additions contribute to elevating the overall quality and depth of our work. Once again, we thank the reviewers for their insights, which have played an instrumental role in refining our manuscript. We encourage the reviewers to reach out without hesitation if they have any additional questions or concerns. 



## IMC Convergence Proof (Sketch)
**Singe Expert Objective** 
$f(p(\mathcal{D}), \theta) = \mathbb{E}_{p(\mathcal{D})}[ \log p(a|o;\theta)] + \eta \mathcal{H}(\mathcal{D})$.
We perform coordinate ascent on $f$ which is guaranteed to converge to a stationary point if updating each coordinate results in a monotonic improvement of $f$ [2]. For fixed expert parameters $\theta$ we can find the unique $p(\mathcal{D})$ that maximizes $f$ [3] (see Section 3.1) and hence we have
$f(p(\mathcal{D})^{(i)}, \theta) \geq f(p(\mathcal{D})^{(i-1)}, \theta),$
where $i$ denotes the iteration.
Under suitable assumptions ($f$ is differentiable, its gradient is $L$-Lipschitz, and the learning rate $\alpha$ is chosen such that the descent lemma [4] holds), it holds that 
$
f(p(\mathcal{D}), \theta^{(i)}) \geq f(p(\mathcal{D}), \theta^{(i-1)}),
$
when updating $f$ using gradient ascent. Hence, we are guaranteed to converge to a stationary point of $f$.

**Mixture of Experts Objective** $J(\psi)$. 
To show that $J(\psi)$ converges to a stationary point, that is $\nabla_{\psi}J(\psi)=0$ we only have to show that $L(\psi^{(i)}, q) \geq L(\psi^{(i-1)}, q)$ as we tighten the lower bound in every E-step [5]. We again perform a coordinate ascent on  $L(\psi^{(i-1)}, q)$. In order to prove convergence we show that we have monotonic improvement on $L$ in every coordinate. It can easily be seen that $L$ is a maximum entropy objective with respect to $p(z)$ and $p(\mathcal{D}|z)$. 
Hence, we can find the unique $p(z)$ and $p(\mathcal{D}|z)$ that maximize $L$ and thus have monotonic improvement. Noting that $q$ is not dependent on $\theta_z$, we can show monotonic improvement by using the same argument as for the single expert objective (the objective for $\theta$ is equal to the per-component objective for $\theta_z$).

**Additional Note.** While stochastic gradient ascent doesn't ensure strictly monotonic improvement in $L$ for $\theta_z$, our empirical observations (see accompanying PDF figure) reveal that $L$ indeed tends to increase monotonically in practice.

## Comparison: IMC vs. ML-Cur

We tested ML-Cur on the experiments used in the paper. We tested ${10, 30, 50}$ components. Furthermore, we did a hyperparameter sweep for the entropy scaling $\alpha$. We report the average performance $\pm$ one standard deviation across 10 seeds:

|  | Obstacle Avoidance |  |
| --- | --- | --- |
|  | success rate | entropy |
| IMC | $0.855 \pm 0.053$ | $0.930 \pm 0.031$ |
| ML-Cur | $0.454\pm 0.223$ | $0.035\pm 0.024$ |

|  | Block Pushing |  |  |
| --- | --- | --- | --- |
|  | success rate | entropy | distance error |
| IMC | $0.521 \pm 0.045$ | $0.654 \pm 0.041$ | $0.120 \pm 0.014$ |
| ML-Cur | $0.000 \pm 0.000$ | $0.000 \pm 0.000$ | $0.408 \pm 0.030$ |

|  | Table Tennis |  |
| --- | --- | --- |
|  | success rate | distance error |
| IMC | $0.870 \pm0.017$ | $0.153 \pm 0.007$ |
| ML-Cur | $0.836 \pm 0.020$ | $0.181 \pm 0.011$ |

For the Franka Kitchen task, we report the average performance for 1-4 tasks solved in the brackets.

|  | Franka Kitchen |  |
| --- | --- | --- |
|  | success rate | entropy |
| IMC | $[0.996, 0.969, 0.884, 0.626]$ | $[0.619, 1.037, 1847, 2.147]$ |
| ML-Cur | $[0.394, 0.000, 0.000, 0.000]$ | $[0.118, 0.000, 0.000, 0.000]$ |

In the accompanying PDF, we have incorporated supplementary quantitative and qualitative comparisons to further enhance the comprehensiveness of our analysis.

The results indicate the ML-Cur fails in the step-based setting (i.e., where the policy has to map from observations to actions, in contrast to the episodic tasks considered in Li, et al, where the policy maps from low-dimensional contexts to movement primitive parameters). 
We hypothesize that linear experts and Gaussian curricula cause errors to accumulate across steps, leading to failure. This insight elucidates ML-Cur's struggles in manipulation tasks requiring precise actions.


[1] Li, et al. Curriculum-based imitation of versatile skills. arXiv preprint arXiv:2304.05171, 2023.

[2] Boyd, Stephen P., and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.

[3] S. Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review.
arXiv preprint arXiv:1805.00909, 2018.

[4] Avriel, Mordecai. "Nonlinear programming." Mathematical Programming for Operations Researchers and Computer Scientists. CRC Press, 2020. 271-367.

[5] Bishop, Christopher M., and Nasser M. Nasrabadi. Pattern recognition and machine learning. Vol. 4. No. 4. New York: springer, 2006.

Meta-review: The paper proposes a novel method for learning versatile robot skills that exhibit multiple modes. A curriculum based mixture of expert policy is used to focus on subsets of the training data. The approach is evaluated in simulated robotics tasks and compared against baselines.

The tackled problem is interesting and timely, and the method shows favorable results in the experiments. The reviewers initially also pointed out some shortcomings including: writing issues, missing and unclear details, too limited experimental evaluations (baselines and tasks), missing theoretical backing, and connection to related work.

The authors provided additional results and convincing arguments in the rebuttal phase. Those managed to address all (major) concerns of the reviewers, and they are now all recommending accepting this paper.
Decision: Accept (poster)