SlowGuess/NeurIPS_2023_OCR huggingface dataset:
architecture. Moreover, we consider an additional scaling factor  $(\beta)$  for the KL regularization in the lower bound objective of the VAE as suggested in [61].
Table 6: CVAE Hyperparameter. The 'Value' column indicates sweep values for the obstacle avoidance task, the block pushing task, the Franka kitchen task and the table tennis task (in this order).
PARAMETER	SWEEP	VALUE
HIDDEN LAYER	{4,6,8,10,12}	8,10,4,4
HIDDEN UNITS	{50,100,150,200}	100,150,100,100
LATENT DIMENSION	{4,16,32,64}	32,16,16,16
DKL SCALING (β)	[10-3,102]	1.641,1.008,0.452,0.698
LEARNING RATE	-	10-3
MAX. EPOCHS	-	2000
BATCHSIZE	-	512
Implicit Behavior Cloning (IBC). IBC was proposed in [17] and uses energy-based models to learn a joint distribution over inputs and targets. Following common practice we parameterize the model as neural network. Moreover, we use the version that adds a gradient penalty to the InfoNCE loss [17]. For sampling, we use gradient-based Langevin MCMC [57]. Despite our effort, we could not achieve good results with IBC. A list of hyperparameters is shown in Table 7.
Behavior Transformer (BET). Behavior transformers were recently proposed in [22]. The model employs a minGPT transformer [62] to predict targets by decomposing them into cluster centers and residual offsets. To obtain a fair comparison, we compare our method to the version with no history. A comprehensive list of hyperparameters is shown in Table 8.
Table 7: IBC Hyperparameter. The 'Value' column indicates sweep values for the obstacle avoidance task and the table tennis task (in this order). We do not get any good results for the block push task and the Franka kitchen task.
PARAMETER	SWEEP	VALUE
HIDDEN DIM	{50,100,150,200,256}	200,256
HIDDEN LAYERS	{4,6,8,10}	4,6
NOISE SCALE	[0.1,0.5]	0.1662,0.1
TRAIN SAMPLES	[8,64]	44,8
NOISE SHRINK	-	0.5
TRAIN ITERATIONS	-	20
INFERENCE ITERATIONS	-	40
LEARNING RATE	-	10-4
BATCH SIZE	-	512
EPOCHS	-	1000
Table 8: BET Hyperparameter. The 'Value' column indicates sweep values for the obstacle avoidance task, the block pushing task, the Franka kitchen task and the table tennis task (in this order).
PARAMETER	SWEEP	VALUE
TRANSFORMER BLOCKS	{2,3,4,6}	3,4,6,2
OFFSET LOSS SCALE	{1.0,100.0,1000.0}	1.0,1.0,1.0,1.0
EMBEDDING WIDTH	{48,72,96,120}	96,72,120,48
NUMBER OF BINS	{8,10,16,32,50,64}	50,10,64,64
ATTENTION HEADS	{4,6}	4,4,6,4
CONTEXT SIZE	-	1
TRAINING EPOCHS	-	500
BATCH SIZE	-	512
LEARNING RATE	-	10-4
D.1 Connection to ML-Cur
This section elaborates on the differences between this work and the work by [30], as both studies leverage curriculum learning paired with Mixture of Experts to facilitate the acquisition of diverse skills. However, it's essential to discern several noteworthy distinctions between our approach and theirs:
Firstly, Li et al. primarily focus on linear policies and a gating distribution with limited flexibility, constraining the expressiveness of their learned policies. In contrast, our approach allows for the use of arbitrarily non-linear neural network parameterizations for both, expert policies and gating.
Another divergence lies in the handling of mini-batches during training. While our algorithm accommodates the use of mini-batches, Li et al.'s method does not support this feature. The ability to work with mini-batches can significantly enhance the efficiency and scalability of the learning process, especially when dealing with extensive datasets or intricate environments.
Additionally, Li et al.'s evaluation primarily centers around relatively simple tasks, that do not require complex manipulations, indicating potential limitations in terms of task complexity and applicability. In contrast, our work is designed to address more intricate and challenging environments, expanding the range of potential applications and domains.
Lastly, it's worth noting that Li et al. specifically focus on the learning of skills parameterized by motion primitives [31]. In contrast, our framework offers the versatility to encompass various skill types and parameterizations, broadening the scope of potential applications and use cases.
D.2 Connection to Expectation Maximization
In this section we want to highlight the commonalities and differences between our algorithm and the expectation-maximization (EM) algorithm for mixtures of experts. First, we look at the updates of the variational distribution  $q$ . Next, we compare the expert optimization. Lastly, we take a closer look at the optimization of the gating distribution.
The EM algorithm sets the variational distribution during the E-step to
$$
q (z | \mathbf {o} _ {n}) = p (z | \mathbf {o} _ {n}, \mathbf {a} _ {n}) = \frac {p _ {\boldsymbol {\theta} _ {z}} (\mathbf {a} _ {n} | \mathbf {o} _ {n} , z) p (z | \mathbf {o} _ {n})}{\sum_ {z} p _ {\boldsymbol {\theta} _ {z}} (\mathbf {a} _ {n} | \mathbf {o} _ {n} , z) p (z | \mathbf {o} _ {n})}, \tag {7}
$$
for all samples  $n$  and components  $z$ . In the M-step, the gating distribution  $p(z|\mathbf{o})$  is updated such that the KL divergence between  $q(z|\mathbf{o})$  and  $p(z|\mathbf{o})$  is minimized. Using the properties of the KL divergence, we obtain a global optimum by setting  $p(z|\mathbf{o}_n) = q(z|\mathbf{o}_n)$  for all  $n$  and all  $z$ . This allows us to rewrite Equation 7 using the recursion in  $q$ , giving
$$
q (z | \mathbf {o} _ {n}) ^ {(i + 1)} = \frac {p _ {\boldsymbol {\theta} _ {z}} (\mathbf {a} _ {n} | \mathbf {o} _ {n} , z) q (z | \mathbf {o} _ {n}) ^ {(i)}}{\sum_ {z} p _ {\boldsymbol {\theta} _ {z}} (\mathbf {a} _ {n} | \mathbf {o} _ {n} , z) q (z | \mathbf {o} _ {n}) ^ {(i)}},
$$
where  $(i)$  denotes the iteration of the EM algorithm. The update for the variational distribution of the IMC algorithm is given by
$$
q (z | \mathbf {o} _ {n}, \mathbf {a} _ {n}) ^ {(i + 1)} = \frac {\tilde {p} (\mathbf {o} _ {n} , \mathbf {a} _ {n} | z) ^ {(i + 1)}}{\sum_ {z} \tilde {p} (\mathbf {o} _ {n} , \mathbf {a} _ {n} | z) ^ {(i + 1)}} = \frac {p _ {\boldsymbol {\theta} _ {z}} (\mathbf {a} _ {n} | \mathbf {o} _ {n} , z) ^ {1 / \eta} q (z | \mathbf {o} _ {n} , \mathbf {a} _ {n}) ^ {(i)}}{\sum_ {z} p _ {\boldsymbol {\theta} _ {z}} (\mathbf {a} _ {n} | \mathbf {o} _ {n} , z) ^ {1 / \eta} q (z | \mathbf {o} _ {n} , \mathbf {a} _ {n}) ^ {(i)}}.
$$
Consequently, we see that  $q(z|\mathbf{o}) = q(z|\mathbf{o}\mathbf{a})$  for  $\eta = 1$ . However, the two algorithms mainly differ in the M-step for the experts: The EM algorithm uses the variational distribution to assign weights to samples, i.e.
$$
\max  _ {\boldsymbol {\theta} _ {z}} \sum_ {n = 1} ^ {N} q (z | \mathbf {o} _ {n}) \log p _ {\boldsymbol {\theta} _ {z}} (\mathbf {a} _ {n} | \mathbf {o} _ {n}, z),
$$
whereas IMC uses the curricula as weights, that is,
$$
\max  _ {\boldsymbol {\theta} _ {z}} \sum_ {n = 1} ^ {N} p (\mathbf {o} _ {n}, \mathbf {a} _ {n} | z) \log p _ {\boldsymbol {\theta} _ {z}} \left(\mathbf {a} _ {n} | \mathbf {o} _ {n}, z\right).
$$
This subtle difference shows the properties of moment and information projection: In the EM algorithm each sample  $\mathbf{o}_n$  contributes to the expert optimization as  $\sum_{z} q(z|\mathbf{o}_n) = 1$ . However, if all curricula ignore the  $n$ th sample, it will not have impact on the expert optimization. Assuming that the curricula ignore samples that the corresponding experts are not able to represent, IMC prevents experts from having to average over 'too hard' samples. Furthermore, this results in reduced outlier sensitivity as they are likely to be ignored for expert optimization. Lastly, we highlight the difference between the gating optimization: Assuming that both algorithms train a gating network  $g_{\phi}(z|\mathbf{o})$  we have
$$
\max _ {\boldsymbol {\phi}} \sum_ {n} \sum_ {z} q (z | \mathbf {o} _ {n}) \log g _ {\boldsymbol {\phi}} (z | \mathbf {o} _ {n}),
$$
for the EM algorithm and
$$
\max _ {\boldsymbol {\phi}} \sum_ {n} \sum_ {z} \tilde {p} (\mathbf {o} _ {n}, \mathbf {a} _ {n} | z) \log g _ {\boldsymbol {\phi}} (z | \mathbf {o} _ {n}),
$$
for IMC. Similar to the expert optimization, EM includes all samples to fit the parameters of the gating network, whereas IMC ignores samples where the unnormalized curriculum weights  $\tilde{p} (\mathbf{o}_n,\mathbf{a}_n|z)$  are zero for all components.
E Algorithm Details & Ablation Studies
E.1 Inference Details
We provide pseudocode to further clarify the inference procedure of our proposed method (see Algorithm 2).
Algorithm 2 IMC Action Generation
1: Require: Curriculum weights  $\{\tilde{p} (\mathbf{o}_n,\mathbf{a}_n|z)\mid n\in \{1,\dots,N\} ,z\in \{1,\dots,N_z\} \}$    
2: Require: Expert parameter  $\{\theta_z|z\in \{1,\dots,N_z\} \}$    
3: Require: New observation o\*   
4: if not parameter_update then   
5:  $\phi^{*}\gets \arg \max_{\phi}\sum_{n}\sum_{z}\tilde{p} (\mathbf{o}_{n},\mathbf{a}_{n}|z)\log g_{\phi}(z|\mathbf{o}_{n})$    
6: parameter_update  $\leftarrow$  True   
7: end if   
8: Sample  $z^{\prime}\sim g_{\phi^{*}}(z|\mathbf{o}^{*})$    
9: Sample a'  $\sim p_{\theta_z}(\mathbf{a}|\mathbf{o}^*,z')$    
10: Return a'
E.2 Expert Design Choices
Distribution. In our mixture of experts policy, we employ Gaussian distributions with a fixed variance to represent the individual experts. This choice offers several benefits in terms of likelihood calculation, optimization and ease of sampling:
To perform the M-Step for the curricula (Section 3.3), exact log-likelihood computation is necessary. This computation becomes straightforward when using Gaussian distributions. Additionally, when Gaussian distributions with fixed variances are employed to represent the experts, the M-Step for the experts simplifies to a weighted squared-error minimization. Specifically, maximizing the weighted likelihood reduces to minimizing the weighted squared error between the predicted actions and the actual actions.
The optimization problem for the expert update can be formulated as follows:
$$
\boldsymbol {\theta} _ {z} ^ {*} = \underset {\boldsymbol {\theta} _ {z}} {\arg \max} \sum_ {n} \tilde {p} (\mathbf {o} _ {n}, \mathbf {a} _ {n} | z) \log p _ {\boldsymbol {\theta} _ {z}} (\mathbf {a} _ {n} | \mathbf {o} _ {n}, z) = \underset {\boldsymbol {\theta} _ {z}} {\arg \min} \sum_ {n} \tilde {p} (\mathbf {o} _ {n}, \mathbf {a} _ {n} | z) \| \boldsymbol {\mu} _ {\boldsymbol {\theta} _ {z}} (\mathbf {o} _ {n}) - \mathbf {a} _ {n} \| _ {2} ^ {2}.
$$
This optimization problem can be efficiently solved using gradient-based methods. Lastly, sampling from Gaussian distributions is well-known to be straightforward and efficient.
Parameterization. We experimented with two different parameterizations of the Gaussian expert means  $\mu_{\theta_z}$ , which we dub single-head and multi-head: For single-head, there is no parameter sharing between the different experts. Each expert has its own set of parameters  $\pmb{\theta}_{z}$ . As a result, we learn  $N_{z}$  different multi-layer perceptrons (MLPs)  $\pmb{\mu}_{\theta_z}:\mathbb{R}^{|\mathcal{O}|}\to \mathbb{R}^{|\mathcal{A}|}$ , where  $N_{z}$  is the number of mixture components. In contrast, the multi-head parameterization uses a global set of parameters  $\pmb{\theta}$  for all experts and hence allows for feature sharing. We thus learn a single MLP  $\pmb{\mu}_{\theta}:\mathbb{R}^{|\mathcal{O}|}\rightarrow \mathbb{R}^{N_z\times |\mathcal{A}|}$ .
To compare both parameterizations, we conducted an ablation study where we evaluate the MoE policy on obstacle avoidance, table tennis and Franka kitchen. In order to have a similar number of parameters, we used smaller MLPs for single-head, that is, 1 - 4 layers whereas for multi-head we used a 6 layer MLP. The results are shown in Table 9 and are generated using 30 components for the obstacle avoidance and table tennis task. The remaining hyperparameters are equal to the ones listed in the main manuscript. For Franka kitchen, we report the cumulative success rate and entropy for a different number of completed tasks. We report the mean and standard deviation calculated across 10 different seeds. Our findings indicate that, in the majority of experiments, the single-head parameterization outperforms the mutli-head alternative. Notably, we observed a substantial performance disparity, especially in the case of Franka kitchen.
Table 9: Expert Parameterization Ablation: We compare IMC with single- and multi-head expert parameterization. For further details, please refer to the accompanying text.
ARCHITECTURE	OBSCTACLE AVOIDANCE	TABLE TENNIS	FRANKA KITCHEN
SUCCEEDRATE (↑)	ENTROPY (↑)	SUCCEEDRAT(↑)	DISTANCE ERR. (↓)	SUCCEEDRAT(↑)	ENTROPY (↑)
SINGLE-HEAD	0.899±0.035	0.887±0.043	0.812±0.039	0.168±0.007	3.644±0.230	6.189±1.135
MULTI-HEAD	0.855±0.053	0.930±0.031	0.870±0.017	0.153±0.007	3.248±0.062	4.657±0.312
Expert Complexity. We conducted an ablation study to evaluate the effect of expert complexity on the performance of the IMC algorithm. The study involved varying the number of hidden layers in the single-head expert architecture while assessing the IMC algorithm's performance on the Franka kitchen task using the cumulative success rate and entropy. The results, presented in Figure 9, were obtained using the hyperparameters specified in the main manuscript. Mean and standard deviation were calculated across 5 different seeds. Our findings demonstrate a positive correlation between expert complexity and achieved performance.
Figure 9: Expert Complexity Ablation: Evaluation of the IMC algorithm on the Franka kitchen task with varying numbers of hidden layers in the single-head expert architecture.
E.3 Component Utilization
We further analyze how IMC utilizes its individual components. Specifically, we assess the entropy of the weights assigned to these components, denoted as  $\mathcal{H}(z)$  and defined as  $\mathcal{H}(z) = -\sum_{z}p(z)\log p(z)$ . Maximum entropy occurs when the model allocates equal importance to all components to solve a task, which implies that  $p(z) = 1 / N_z$ . Conversely, if the model relies solely on a single component, the entropy  $\mathcal{H}(z)$  equals zero. We conduct the study on the Franka kitchen task, evaluating it through cumulative success rates and entropy. The results are shown in Figure 10a. We computed the mean and standard deviation based on data collected from 5 different seeds. Our findings reveal that IMC exhibits substantial component utilization even when dealing with a large number of components, denoted as  $N_z$ .
E.4 Number of Components Sensitivity
To investigate how the algorithm responds to changes in the number of components  $N_{z}$ , we performed a comprehensive ablation study. Figure 10b shows the success rate of IMC on the Franka kitchen task using a varying number of components. Our findings indicate that the algorithm's performance remains stable across different values of  $N_{z}$ . This robust behavior signifies that the algorithm can adeptly accommodate varying numbers of components without experiencing substantial performance fluctuations. Hence, opting for a higher number of components primarily entails increased computational costs, without leading to overfitting issues.
Figure 10: Figure 10a shows the component utilization of IMC: The entropy of IMC's mixture weights  $p(z)$  is close to the entropy of a uniform distribution  $1 / N_{z}$  indicating that the algorithm uses all components for solving a task, even for a high number of components  $N_{z}$ .
E.5 Curriculum Pacing Sensitivity
To examine the algorithm's sensitivity to the curriculum pacing parameter  $\eta$ , we conducted an ablation study. Figure 11 presents the results obtained using 30 components for the obstacle avoidance and table tennis tasks, while maintaining the remaining hyperparameters as listed in the main manuscript. For the Franka kitchen task, we analyzed the cumulative success rate and entropy across varying numbers of completed tasks. The mean and standard deviation were calculated across 5 different seeds. Our findings reveal that the optimal value for  $\eta$  is dependent on the specific task. Nevertheless, the algorithm exhibits stable performance even when  $\eta$  values differ by an order of magnitude.
(a) Obstacle avoidance
(b) Table tennis
(c) Franka kitchen
Figure 11: Curriculum Pacing Sensitivity: Sensitivity analysis of the IMC algorithm for performance metrics in the obstacle avoidance, table tennis, and Franka kitchen tasks, considering varying curriculum pacing  $(\eta)$  values. The results illustrate the mean and standard deviation across 5 different seeds.