SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Information Maximizing Curriculum: A Curriculum-Based Approach for Imitating Diverse Skills
Denis Blessing  $*\dagger$  Onur Celik†§ Xiaogang Jia†‡ Moritz Reuss† Maximilian Xiling Li† Rudolf Lioutikov† Gerhard Neumann†§
† Autonomous Learning Robots, Karlsruhe Institute of Technology  
‡ Intuitive Robots Lab, Karlsruhe Institute of Technology  
§ FZI Research Center for Information Technology
Imitation learning uses data for training policies to solve complex tasks. However, when the training data is collected from human demonstrators, it often leads to multimodal distributions because of the variability in human actions. Most imitation learning methods rely on a maximum likelihood (ML) objective to learn a parameterized policy, but this can result in suboptimal or unsafe behavior due to the mode-averaging property of the ML objective. In this work, we propose Information Maximizing Curriculum, a curriculum-based approach that assigns a weight to each data point and encourages the model to specialize in the data it can represent, effectively mitigating the mode-averaging problem by allowing the model to ignore data from modes it cannot represent. To cover all modes and thus, enable diverse behavior, we extend our approach to a mixture of experts (MoE) policy, where each mixture component selects its own subset of the training data for learning. A novel, maximum entropy-based objective is proposed to achieve full coverage of the dataset, thereby enabling the policy to encompass all modes within the data distribution. We demonstrate the effectiveness of our approach on complex simulated control tasks using diverse human demonstrations, achieving superior performance compared to state-of-the-art methods.
Equipping agents with well-performing policies has long been a prominent focus in machine learning research. Imitation learning (IL) [1] offers a promising technique to mimic human behavior by leveraging expert data, without the need for intricate controller design, additional environment interactions, or complex reward shaping to encode the target behavior. The latter are substantial advantages over reinforcement learning techniques [2, 3] that rely on reward feedback. However, a significant challenge in IL lies in handling the multimodal nature of data obtained from human demonstrators, which can stem from differences in preferences, expertise, or problem-solving strategies. Conventionally, maximum likelihood estimation (MLE) is employed to train a policy on expert data. It is well-known that MLE corresponds to the moment projection [4], causing the policy to average over modes in the data distribution that it cannot represent. Such mode averaging can lead to unexpected and potentially dangerous behavior. We address this critical issue by introducing Information Maximizing Curriculum (IMC), a novel curriculum-based approach.
In IMC, we view imitation learning as a conditional density estimation problem and present a mathematically sound weighted optimization scheme. Data samples are assigned curriculum weights, which are updated using an information projection. The information projection minimizes the reverse
*Correspondence to denis.blessing@kit.edu
KL divergence, forcing the policy to ignore modes it cannot represent [4]. As a result, the optimized policy ensures safe behavior while successfully completing the task.
Yet, certain modes within the data distribution may remain uncovered with a single expert policy. To address this limitation and endow the policy with the diverse behavior present in the expert data, we extend our approach to a mixture of expert (MoE) policy. Each mixture component within the MoE policy selects its own subset of training data, allowing for specialization in different modes. Our objective maximizes the entropy of the joint curriculum, ensuring that the policy covers all data samples.
We show that our method is able to outperform state-of-the-art policy learning algorithms and MoE policies trained using competing optimization algorithms on complex multimodal simulated control tasks where data is collected by human demonstrators. In our experiments, we assess the ability of the models to  $i$  ) avoid mode averaging and ii) cover all modes present in the data distribution.
Our approach heavily builds on minimizing Kullback-Leibler divergences as well as mixtures of expert policies. Hence, we will briefly review both concepts.
2.1 Moment and Information Projection
The Kullback-Leibler (KL) divergence [5] is a similarity measure for probability distributions and is defined as  $D_{\mathrm{KL}}(p\| p^{\prime}) = \sum_{\mathbf{x}}p(\mathbf{x})\log p(\mathbf{x}) / p^{\prime}(\mathbf{x})$  for a discrete random variable  $\mathbf{x}$ . Due to its asymmetry, the KL divergence offers two different optimization problems for fitting a model distribution  $p$  to a target distribution  $p^{*}$  [4], that is,
The M-projection - or equivalently maximum likelihood estimation (MLE) [6] - is probability forcing, meaning that the model is optimized to match the moments of the target distribution, causing it to average over modes that it cannot represent. In contrast, the I-projection is zero forcing which leads the model to ignore modes of the target distribution that it is not able to represent. The I-projection can be rewritten as maximum entropy problem, i.e.,
$$
\underset {p} {\arg \max } \mathbb {E} _ {p (\mathbf {x})} [ \log p ^ {*} (\mathbf {x}) ] + \mathcal {H} (\mathbf {x}). \tag {1}
$$
Using this formulation, it can be seen that the optimization balances between fitting the target distribution and keeping the entropy  $\mathcal{H}(\mathbf{x}) = -\sum_{\mathbf{x}}p(\mathbf{x})\log p(\mathbf{x})$  high.
2.2 Mixtures of Expert Policies
Mixtures of expert policies are conditional discrete latent variable models. Given some observation  $\mathbf{o} \in \mathcal{O}$  and action  $\mathbf{a} \in \mathcal{A}$ , the marginal likelihood is decomposed into individual components, i.e.,
$$
p (\mathbf {a} | \mathbf {o}) = \sum_ {z} p (z | \mathbf {o}) p (\mathbf {a} | \mathbf {o}, z).
$$
Here,  $\mathcal{O}$  and  $\mathcal{A}$  denote the observation and action space respectively, and  $z$  stands for the discrete latent variable that indexes distinct components within the mixture.
The gating  $p(z|\mathbf{o})$  is responsible for soft-partitioning the observation space  $\mathcal{O}$  into sub-regions where the corresponding experts  $p(\mathbf{a}|\mathbf{o},z)$  approximate the target density. Typically the experts and the gating are parameterized and learned by maximizing the marginal likelihood via expectation-maximization (EM) [7] or gradient ascent [8]. In order to sample actions, that is,  $\mathbf{a}'\sim p(\mathbf{a}|\mathbf{o}')$  for some observation  $\mathbf{o}'$ , we first sample a component index from the gating, i.e.,  $z^{\prime}\sim p(z|\mathbf{o}^{\prime})$ . The component index selects the respective expert to obtain  $\mathbf{a}'\sim p(\mathbf{a}|\mathbf{o}',z')$ .
3 Information Maximizing Curriculum
In this section, we propose Information Maximizing Curriculum (IMC), a novel algorithm for training mixtures of expert policies. We motivate our optimization objective using a single policy. Next, we generalize the objective to support learning a mixture of experts policy. Thereafter, we discuss the optimization scheme and provide algorithmic details.
3.1 Objective for a Single Expert Policy
We propose an objective that jointly learns a curriculum  $p(\mathbf{o}, \mathbf{a})$  and a parameterized policy  $p_{\theta}(\mathbf{a}|\mathbf{o})$  with parameters  $\theta$ . The curriculum is a categorical distribution over samples of a dataset  $\{( \mathbf{o}_n, \mathbf{a}_n ) \}_{n=1}^N$ , assigning probability mass to samples according to the performance of the policy. To allow the curriculum to ignore samples that the policy cannot represent, we build on the I-projection (see Equation 1). We, therefore, formulate the objective function as
$$
\tilde {J} (p (\mathbf {o}, \mathbf {a}), \boldsymbol {\theta}) = \mathbb {E} _ {p (\mathbf {o}, \mathbf {a})} [ \log p _ {\boldsymbol {\theta}} (\mathbf {a} | \mathbf {o}) ] + \eta \mathcal {H} (\mathbf {o}, \mathbf {a}), \tag {2}
$$
which is optimized for  $p(\mathbf{o}, \mathbf{a})$  and  $\theta$  in an alternating fashion using coordinate ascent [9]. We additionally introduce a trade-off factor  $\eta$  that determines the pacing of the curriculum. For  $\eta \to \infty$  the curriculum becomes uniform, exposing all samples to the policy and hence reducing to maximum likelihood estimation for  $\theta$ . In contrast, if  $\eta \to 0$  the curriculum concentrates on samples where the policy log-likelihood  $\log p_{\theta}(\mathbf{a}|\mathbf{o})$  is highest. The objective can be solved in closed form for  $p(\mathbf{o}, \mathbf{a})$  (see Appendix B), resulting in
$$
p ^ {*} (\mathbf {o} _ {n}, \mathbf {a} _ {n}) \propto p _ {\boldsymbol {\theta}} (\mathbf {a} _ {n} | \mathbf {o} _ {n}) ^ {1 / \eta}.
$$
Maximizing the objective w.r.t  $\theta$  reduces to a weighted maximum-likelihood estimation, that is,
$$
\pmb {\theta} ^ {*} = \underset {\pmb {\theta}} {\arg \max} \sum_ {n} p (\mathbf {o} _ {n}, \mathbf {a} _ {n}) \log p _ {\pmb {\theta}} (\mathbf {a} _ {n} | \mathbf {o} _ {n}).
$$
The optimization is repeated until reaching a local maximum, indicating that the curriculum has selected a fixed set of samples where the policy attains high log-likelihoods  $\log p_{\theta}(\mathbf{a}|\mathbf{o})$ . Proposition 3.1 establishes convergence guarantees, the details of which are elaborated upon in Appendix A.1.
Proposition 3.1. Let  $\tilde{J}$  be defined as in Equation 2 and  $0 < \eta < \infty$ . Under mild assumptions on  $p_{\theta}$  and the optimization scheme for  $\theta$ , it holds for all  $i$ , denoting the iteration index of the optimization process, that
$$
\tilde {J} (p (\mathbf {o}, \mathbf {a}) ^ {(i + 1)}, \boldsymbol {\theta} ^ {(i + 1)}) \geq \tilde {J} (p (\mathbf {o}, \mathbf {a}) ^ {(i)}, \boldsymbol {\theta} ^ {(i)}),
$$
where equality indicates that the algorithm has converged to a local optimum.
The capacity to disregard samples where the policy cannot achieve satisfactory performance mitigates the mode-averaging problem. Nevertheless, a drawback of employing a single expert policy is the potential for significantly suboptimal performance on the ignored samples. This limitation is overcome by introducing multiple experts, each specializing in different subsets of the data.
3.2 Objective for a Mixture of Experts Policy
Assuming limited complexity, a single expert policy is likely to ignore a large amount of samples due to the zero-forcing property of the I-projection. Using multiple curricula and policies that specialize to different subsets of the data is hence a natural extension to the single policy model. To that end, we make two major modifications to Equation 2: Firstly, we use a mixture model with multiple components  $z$  where each component has its own curriculum, i.e.,  $p(\mathbf{o},\mathbf{a}) = \sum_{z}p(z)p(\mathbf{o},\mathbf{a}|z)$ . Secondly, we employ an expert policy per component  $p_{\theta_z}(\mathbf{a}|\mathbf{o},z)$ , that is paced by the corresponding curriculum  $p(\mathbf{o},\mathbf{a}|z)$ . The resulting objective function is given by
$$
J (\psi) = \mathbb {E} _ {p (z)} \mathbb {E} _ {p (\mathbf {o}, \mathbf {a} | z)} [ \log p _ {\boldsymbol {\theta} _ {z}} (\mathbf {a} | \mathbf {o}, z) ] + \eta \mathcal {H} (\mathbf {o}, \mathbf {a}), \tag {3}
$$
where  $\psi$  summarizes the dependence on  $p(z)$ ,  $\{p(\mathbf{o},\mathbf{a}|z)\}_{z}$  and  $\{\theta_z\}_{z}$ . However, Equation 3 is difficult to optimize as the entropy of the mixture model prevents us from updating the curriculum of each component independently. Similar to [10], we introduce an auxiliary distribution  $q(z|\mathbf{o},\mathbf{a})$  to decompose the objective function into a lower bound  $L(\psi ,q)$  and an expected  $D_{\mathrm{KL}}$  term, that is,
$$
J (\psi) = L (\psi , q) + \eta \mathbb {E} _ {p (\mathbf {o}, \mathbf {a})} D _ {\mathrm {K L}} (p (z | \mathbf {o}, \mathbf {a}) \| q (z | \mathbf {o}, \mathbf {a})), \tag {4}
$$
with  $p(z|\mathbf{o},\mathbf{a}) = p(\mathbf{o},\mathbf{a}|z)p(z) / p(\mathbf{o},\mathbf{a})$  and
$$
L(\boldsymbol {\psi},q) = \mathbb{E}_{p(z)}\Big[\underbrace{\mathbb{E}_{p(\mathbf{o},\mathbf{a}|z)}[R_{z}(\mathbf{o},\mathbf{a})] + \eta\mathcal{H}(\mathbf{o},\mathbf{a}|z)]}_{J_{z}(p(\mathbf{o},\mathbf{a}|z),\boldsymbol{\theta}_{z})}\Big] + \eta \mathcal{H}(z),
$$
with  $R_{z}(\mathbf{o}_{n},\mathbf{a}_{n}) = \log p_{\theta_{z}}(\mathbf{a}_{n}|\mathbf{o}_{n},z) + \eta \log q(z|\mathbf{o}_{n},\mathbf{a}_{n})$ , allowing for independent updates for  $p(\mathbf{o},\mathbf{a}|z)$  and  $\pmb{\theta}_{z}$  by maximizing the per-component objective function  $J_{z}(p(\mathbf{o},\mathbf{a}|z),\pmb{\theta}_{z})$ . A derivation can be found in Appendix B.1. Since  $\mathbb{E}_{p(\mathbf{o},\mathbf{a})}D_{\mathrm{KL}}(p(z|\mathbf{o},\mathbf{a})||q(z|\mathbf{o}))\geq 0$ ,  $L$  is a lower bound on  $J$  for  $\eta \geq 0$ . Please note that the per-component objective function  $J_{z}$  is very similar to Equation 2, with  $J_{z}$  including an additional term,  $\log q(z|\mathbf{o},\mathbf{a})$ , which serves the purpose of preventing different curricula from assigning probability mass to the same set of samples: Specifically, a component  $z$  is considered to 'cover' a datapoint  $(\mathbf{o}_n,\mathbf{a}_n)$  when  $q(z|\mathbf{o}_n,\mathbf{a}_n)\approx 1$ . Since  $\sum_{z}q(z|\mathbf{o}_{n},\mathbf{a}_{n}) = 1$ , it follows that for other components  $z^{\prime}\neq z$  it holds that  $q(z^{\prime}|\mathbf{o}_{n},\mathbf{a}_{n})\approx 0$ . Consequently,  $\log q(z^{\prime}|\mathbf{o}_{n},\mathbf{a}_{n})\to -\infty$ , implying that  $R_{z'}(\mathbf{o}_n,\mathbf{a}_n)\rightarrow -\infty$ . As a result, the other curricula effectively disregard the datapoint, as  $p(\mathbf{o}_n,\mathbf{a}_n|z')\propto \exp R_{z'}(\mathbf{o}_n,\mathbf{a}_n)\approx 0$ .
We follow the optimization scheme of the expectation-maximization algorithm [7], that is, we iteratively maximize (M-step) and tighten the lower bound (E-step)  $L(\psi, q)$ .
3.3 Maximizing the Lower Bound (M-Step)
We maximize the lower bound  $L(\psi, q)$  with respect to the mixture weights  $p(z)$ , curricula  $p(\mathbf{o}, \mathbf{a} | z)$  and expert policy parameters  $\theta_z$ . We find closed form solutions for both,  $p(z)$  and  $p(\mathbf{o}, \mathbf{a} | z)$  given by
$$
p ^ {*} (z) \propto \exp \left(\mathbb {E} _ {p (\mathbf {o}, \mathbf {a} | z)} \left[ R _ {z} (\mathbf {o}, \mathbf {a}) / \eta \right] + \mathcal {H} (\mathbf {o}, \mathbf {a} | z)\right), \text {a n d} \tilde {p} (\mathbf {o} _ {n}, \mathbf {a} _ {n} | z) = \exp \left(R _ {z} (\mathbf {o} _ {n}, \mathbf {a} _ {n}) / \eta\right), \tag {5}
$$
where  $\tilde{p}(\mathbf{o}_n, \mathbf{a}_n | z)$  are the optimal unnormalized curricula, such that holds  $p^*(\mathbf{o}_n, \mathbf{a}_n | z) = \tilde{p}(\mathbf{o}_n, \mathbf{a}_n | z) / \sum_n \tilde{p}(\mathbf{o}_n, \mathbf{a}_n | z)$ . However, due to the hierarchical structure of  $L(\psi, q)$  we implicitly optimize for  $p(z)$  when updating the curricula. This result is frequently used throughout this work and formalized in Proposition 3.2. A proof can be found in Appendix A.2.
Proposition 3.2. Let  $p^*(z)$  and  $\tilde{p}(\mathbf{o}, \mathbf{a}|z)$  be the optimal mixture weights and unnormalized curricula for maximizing  $L(\psi, q)$ . It holds that
$$
p ^ {*} (z) = \sum_ {n} \tilde {p} (\mathbf {o} _ {n}, \mathbf {a} _ {n} | z) / \sum_ {z} \sum_ {n} \tilde {p} (\mathbf {o} _ {n}, \mathbf {a} _ {n} | z).
$$
The implicit updates of the mixture weights render the computation of  $p^*(z)$  obsolete, reducing the optimization to computing the optimal (unnormized) curricula  $\tilde{p}(\mathbf{o}, \mathbf{a}|z)$  and expert policy parameters  $\theta_z^*$ . In particular, this result allows for training the policy using mini-batches and thus greatly improves the scalability to large datasets as explained in Section 3.5. Maximizing the lower bound with respect to  $\theta_z^*$  results in a weighted maximum likelihood estimation, i.e.,
$$
\boldsymbol {\theta} _ {z} ^ {*} = \arg \max  _ {\boldsymbol {\theta} _ {z}} \sum_ {n} \tilde {p} (\mathbf {o} _ {n}, \mathbf {a} _ {n} | z) \log p _ {\boldsymbol {\theta} _ {z}} \left(\mathbf {a} _ {n} | \mathbf {o} _ {n}, z\right), \tag {6}
$$
where the curricula  $\tilde{p} (\mathbf{o}_n,\mathbf{a}_n|z)$  assign sample weights. For further details on the M-step, including derivations of the closed-form solutions and the expert parameter objective see Appendix B.2.
3.4 Tightening the Lower Bound (E-Step)
Tightening of the lower bound (also referred to as E-step) is done by minimizing the expected Kullback-Leibler divergence in Equation 4. Using the properties of the KL divergence, it can easily be seen that the lower bound is tight if for all  $n \in \{1, \dots, N\}$ $q(z|\mathbf{o}_n) = p(z|\mathbf{o}_n,\mathbf{a}_n)$  holds. To obtain  $p(z|\mathbf{o}_n,\mathbf{a}_n)$  we leverage Bayes' rule, that is,  $p(z|\mathbf{o}_n,\mathbf{a}_n) = p^*(z)p^*(\mathbf{o}_n,\mathbf{a}_n|z)/\sum_z p^*(z)p^*(\mathbf{o}_n,\mathbf{a}_n|z)$ . Using Proposition 3.2 we find that
$$
p (z | \mathbf {o} _ {n}, \mathbf {a} _ {n}) = \tilde {p} (\mathbf {o} _ {n}, \mathbf {a} _ {n} | z) / \sum_ {z} \tilde {p} (\mathbf {o} _ {n}, \mathbf {a} _ {n} | z).
$$
Please note that the lower bound is tight after every E-step as the KL divergence is set to zero. Thus, increasing the lower bound  $L$  maximizes the original objective  $J$  assuming that updates of  $\theta_z$  are not decreasing the expert policy log-likelihood  $\log p_{\theta_z}(\mathbf{a}|\mathbf{o},z)$ .
3.5 Algorithmic Details
Convergence Guarantees. Proposition 3.3 establishes convergence guarantees for the mixture of experts policy objective  $J$ . The proof mainly relies on the facts that IMC has the same convergence guarantees as the EM algorithm and that Proposition 3.1 can be transferred to the per-component objective  $J_{z}$ . The full proof is given in Appendix A.1.
Proposition 3.3. Let  $J$  be defined as in Equation 4 and  $0 < \eta < \infty$ . Under mild assumptions on  $p_{\theta_z}$  and the optimization scheme for  $\theta_z$ , it holds for all  $i$ , denoting the iteration index of the optimization process, that
$$
J (\boldsymbol {\psi} ^ {(i + 1)}) \geq J (\boldsymbol {\psi} ^ {(i)}),
$$
where equality indicates that the IMC algorithm has converged to a local optimum.
Stopping Criterion. We terminate the algorithm if either the maximum number of training iterations is reached or if the lower bound  $L(\psi, q)$  converges, i.e.,
$$
\left| \Delta L \right| = \left| L ^ {(i)} (\boldsymbol {\psi}, q) - L ^ {(i - 1)} (\boldsymbol {\psi}, q) \right| \leq \epsilon ,
$$
with threshold  $\epsilon$  and two subsequent iterations  $(i)$  and  $(i - 1)$ . The lower bound can be evaluated efficiently using Corollary 3.2.1.
Corollary 3.2.1. Consider the setup used in Proposition 3.2. For  $p^*(z) \in \psi$  and  $\{p^*(\mathbf{o}, \mathbf{a} | z)\}_{z} \in \psi$  it holds that
$$
L (\psi , q) = \eta \log \sum_ {z} \sum_ {n} \tilde {p} (\mathbf {o} _ {n}, \mathbf {a} _ {n} | z).
$$
See Appendix A.3 for a proof.
Inference. In order to perform inference, i.e., sample actions from the policy, we need to access the gating distribution for arbitrary observations  $\mathbf{o} \in \mathcal{O}$  which is not possible as  $p(z|\mathbf{o},\mathbf{a})$  is only defined for observations contained in the dataset  $\mathbf{o}$ ,  $\mathbf{a}$ . We therefore leverage Corollary 3.2.2 to learn an inference network  $g_{\phi}(z|\mathbf{o})$  with parameters  $\phi$  by minimizing the KL divergence between  $p(z|\mathbf{o})$  and  $g_{\phi}(z|\mathbf{o})$  under  $p(\mathbf{o})$  (see Appendix A.4 for a proof).
Corollary 3.2.2. Consider the setup used in Proposition 3.2. It holds that
$$
\min  _ {\phi} \mathbb {E} _ {p (\mathbf {o})} D _ {\mathrm {K L}} \big (p (z | \mathbf {o}) \| g _ {\phi} (z | \mathbf {o}) \big) = \max  _ {\phi} \sum_ {n} \sum_ {z} \tilde {p} (\mathbf {o} _ {n}, \mathbf {a} _ {n} | z) \log g _ {\phi} (z | \mathbf {o} _ {n}).
$$
Once trained, the inference network can be used for computing the exact log-likelihood as  $p(\mathbf{a}|\mathbf{o}) = \sum_{z} g_{\phi}(z|\mathbf{o}) p_{\theta_z}(\mathbf{a}|\mathbf{o}, z)$  or sampling a component index, i.e.,  $z' \sim g_{\phi}(z|\mathbf{o})$ .
Mini-Batch Updates. Due to Proposition 3.2, the M- and E-step in the training procedure only rely on the unnormalized curricula  $\tilde{p}(\mathbf{o},\mathbf{a}|z)$ . Consequently, there is no need to compute the normalization constant  $\sum_{n}\tilde{p}(\mathbf{o}_{n},\mathbf{a}_{n}|z)$ . This allows us to utilize mini-batches instead of processing the entire dataset, resulting in an efficient scaling capability for large datasets. Please refer to Algorithm 1 for a detailed description of the complete training procedure.
Imitation Learning. A variety of algorithms in imitation learning [1, 11] can be grouped into two categories: Inverse reinforcement learning [12, 13], which extracts a reward function from demonstrations and optimizes a policy subsequently, and behavioral cloning, which directly extracts a policy from demonstrations. Many works approach the problem of imitation learning by considering behavior cloning as a distribution-matching problem, in which the state distribution induced by the policy is required to align with the state distribution of the expert data. Some methods [14, 15] are based on adversarial methods inspired by Generative Adversarial Networks (GANs) [16]. A policy is trained to imitate the expert while a discriminator learns to distinguish between fake and expert data. However, these methods are not suitable for our case as they involve interacting with the environment during training. Other approaches focus on purely offline training and use various policy representations such as energy-based models [17], normalizing flows [18, 19], conditional variational autoencoders (CVAEs) [20, 21], transformers [22], or diffusion models [23, 24, 25, 26, 27, 28]. These
Figure 1: Behavior learning environments: Visualization of the obstacle avoidance task (left), the block pushing task (middle), and the table tennis task (right).
models can represent multi-modal expert distributions but are optimized based on the M-Projection, which leads to a performance decrease. Recent works [29, 30] have proposed training Mixture of Experts models with an objective similar to ours. However, the work by [29] requires environment-specific geometric features, which is not applicable in our setting, whereas the work by [30] considers linear experts and the learning of skills parameterized by motion primitives [31]. For a detailed differentiation between our work and the research conducted by [30], please refer to Appendix D.1.
Curriculum Learning. The authors of [32] introduced curriculum learning (CL) as a new paradigm for training machine learning models by gradually increasing the difficulty of samples that are exposed to the model. Several studies followed this definition [33, 34, 35, 36, 37, 38, 39]. Other studies used the term curriculum learning for gradually increasing the model complexity [40, 41, 42] or task complexity [43, 44, 45, 46]. All of these approaches assume that the difficulty-ranking of the samples is known a-priori. In contrast, we consider dynamically adapting the curriculum according to the learning progress of the model which is
Algorithm 1 IMC training procedure