SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To determine which arms (tasks) should be included to the meta-training iteration  $k$ , we formulate the arm benefit score estimation for each candidate arm  $\mathcal{T}_{k,i} \in \Omega_{\mathrm{task}}^{(k)}$ . The estimated benefit score consists of two parts: (1) the estimated arm reward of choosing this task based on existing knowledge (i.e., exploitation); (2) and the exploration score for the future potential benefit (i.e., exploration). Inspired by recent advances in neural bandits [9], we introduce two separate neural networks,  $f_{1}(\cdot ;\pmb{\theta}_{1})$  and  $f_{2}(\cdot ;\pmb{\theta}_{2})$ , to estimate the arm reward and exploration score respectively. The exploitation network  $f_{1}(\cdot ;\pmb{\theta}_{1})$  aims to learn the mapping  $h(\cdot)$  from arm contexts (i.e., meta-parameters) to rewards, while the exploration network  $f_{2}(\cdot ;\pmb{\theta}_{2})$  aims to learn the uncertainty of reward estimations as the exploration criterion. Different from conventional bandit models, e.g. [9], that works on static arm contexts given by the environment, our design alternatively leverages the evolving information from both the task (arm) side and meta-parameters side, across meta-training iterations. In addition, we consider the dynamics of meta-optimizations for a more comprehensive modeling of the exploration aspect, and the details will be introduced later. Here, given a candidate arm  $\mathcal{T}_{k,i} \in \Omega_{\mathrm{task}}^{(k)}$ , its estimated benefit score  $\widehat{s}_{k,i}$  is formulated as
$$
\widehat {s} _ {k, i} = \alpha \cdot \widehat {r} _ {k, i} + \widehat {e} _ {k, i} = \alpha \cdot f _ {1} \left(\boldsymbol {\chi} _ {k, i} ^ {q}; \boldsymbol {\theta} _ {1} ^ {(k - 1)}\right) + f _ {2} \left(\left[ \nabla_ {\boldsymbol {\theta}} f _ {1} \left(\boldsymbol {\chi} _ {k, i} ^ {s}\right); \nabla_ {\boldsymbol {\theta}} f _ {1} \left(\boldsymbol {\chi} _ {k, i} ^ {q}\right) \right]; \boldsymbol {\theta} _ {2} ^ {(k - 1)}\right) \tag {7}
$$
where  $\alpha \in (0,1]$  is the exploration coefficient to balance exploitation and exploration. Notice that  $f_{2}(\cdot ;\pmb{\theta}_{2})$  will take the concatenated gradient of  $f_{1}(\cdot ;\pmb{\theta}_{1})$  w.r.t. both arm contexts  $\chi_{k,i}^{s},\chi_{k,i}^{q}$  as the input, represented by  $[\nabla_{\pmb{\theta}}f_{1}(\chi_{k,i}^{s});\nabla_{\pmb{\theta}}f_{1}(\chi_{k,i}^{q})]$ . And the output will be the exploration score estimation  $\hat{e}_{k,i}$ . To obtain  $\nabla_{\pmb{\theta}}f_{1}(\chi_{k,i}^{q})$ , we also calculate  $f_{1}(\chi_{k,i}^{q};\pmb{\theta}_{1})$  and run the back-propagation. Afterwards, we choose the top- $B$  arms with the highest estimated benefit scores  $\widehat{s}_{k,i},i\in \mathcal{N}$ , as the chosen task batch  $\Omega_k\subset \Omega_{\mathrm{task}}^{(k)}$  (line 10, Algo. 1).
Design Intuition. First, recent advances of neural Contextual Bandits [54, 9, 38] have shown that the uncertainty of reward estimations is directly related to the gradients of the estimation model. Therefore, we leverage an exploration module  $f_{2}(\cdot;\theta_{2})$  to directly learn this unknown relationship. Second, since  $D_{k,i}^{s}, D_{k,i}^{q}$  are from the same data distribution  $\mathcal{D}_{\mathcal{T}_{k,i}}$ , if these two gradients  $\nabla_{\theta}f_{1}(\boldsymbol{\chi}_{k,i}^{s}), \nabla_{\theta}f_{1}(\boldsymbol{\chi}_{k,i}^{q})$  are distinct, the reason can be: (1) the variance of the data distribution  $\mathcal{D}_{\mathcal{T}_{k,i}}$  is high (due to the potentially noisy or difficult task); or (2) the gradients of  $f_{1}(\cdot;\theta_{1})$  tend to change significantly when adapting to task  $\mathcal{T}_{k,i}$ . In both cases, it can be harder for  $f_{1}(\cdot;\theta_{1})$  to accurately predict the arm reward  $r_{k,i}$ , and the meta-model can fail to properly adapt to the task  $\mathcal{T}_{k,i}$ . In this case, we apply the concatenated gradients w.r.t. both arm contexts as the input of  $f_{2}(\cdot;\theta_{2})$ , in order to provide the information for  $f_{2}(\cdot;\theta_{2})$  to evaluate exploration scores.
Network Architecture and Parameter Initialization. Here, we consider  $f_{1}(\cdot ;\pmb{\theta}_{1}),f_{2}(\cdot ;\pmb{\theta}_{2})$  to be two  $L$ -layer fully-connected (FC) networks with network width  $m$ , while  $\pmb{\theta} = \{\pmb{\theta}_1,\pmb{\theta}_2\}$  refer to their trainable parameters. For their randomly initialized parameters  $\pmb{\theta}^{(0)} = \{\pmb{\theta}_1^{(0)},\pmb{\theta}_2^{(0)}\}$ , the weight matrix entries for the first  $L - 1$  layers are drawn from the Gaussian distribution  $N(0,2 / m)$ , while the entries of the last layer ( $L$ -th layer) are sampled from  $N(0,1 / m)$ .
Remark 3 (Reducing Input Complexity). The input of  $f_{1}(\cdot ;\theta_{1})$  is the arm context  $\chi_{k,i}^{q}$ , whose dimensionality is the number of meta-parameters  $p$ . A similar situation also exists for the exploration
network  $f_{2}(\cdot ;\theta_{2})$ . Inspired by the idea of learning dense low-dimensional representations with Convolutional Neural Networks (CNNs) (e.g., [44]), we apply the average pooling approach to approximate original inputs for reducing the running time and space complexity in practice. To show its effectiveness, we will apply this approach on BASS for all the experiments in Section 6.
4.3 Formulating Arm Rewards and Exploration Scores
Different from the conventional neural bandit algorithms [53, 54, 9] where the reward is provided by the environment oracle, we need to carefully design the arm rewards to reflect the arm benefit in terms of the meta-model's generalization ability. Analogous to task batch rewards (Eq. 4), we formulate the single arm reward  $r_{k,i} = h(\Theta^{(k - 1)}[\mathcal{T}_{k,i}]) = 1 - \mathbb{E}_{\mathcal{T}\sim \mathcal{P}(\mathcal{T}),\boldsymbol {x}\sim \mathcal{D}_{\mathcal{T}}}\left[\mathcal{L}\big(\boldsymbol {x};\mathcal{I}(\mathcal{T},\Theta^{(k - 1)}[\mathcal{T}_{k,i}])\big)\right]$  for arm  $\mathcal{T}_{k,i}$ . Since it is impractical to calculate the arm reward by enumerating over  $\mathcal{P}(\mathcal{T})$ , we sample a batch of validation tasks  $\Omega_k^{\mathrm{valid}}$  to derive the unbiased reward approximation, denoted by
$$
\widetilde {r} _ {k, i} = 1 - \frac {1}{| \Omega_ {k} ^ {\text {v a l i d}} |} \sum_ {\mathcal {T} ^ {\text {v a l i d}} \in \Omega_ {k} ^ {\text {v a l i d}}} \mathcal {L} \left(D _ {\mathcal {T} ^ {\text {v a l i d}}} ^ {q}; \mathcal {I} \left(\mathcal {T} ^ {\text {v a l i d}}, \boldsymbol {\Theta} ^ {(k - 1)} \left[ \mathcal {T} _ {k, i} \right]\right)\right). \tag {8}
$$
Here, we adopt the single-step inner-loop optimization [19, 40] to derive  $\mathcal{I}(\mathcal{T}_{k,\widehat{i}},\Theta^{(k - 1)}[\mathcal{T}_{k,i}])$  in Eq. 8, in order to save the computational cost in practice. Under the few-shot settings, the computation of arm rewards is efficient, since the support set is generally small for inner-loop optimization. The approximation error here can be bounded by the concentration inequality, as the validation tasks  $\Omega_{k}^{\mathrm{valid}}$  are sampled from  $\mathcal{P}(\mathcal{T})$ .
On the other hand, to formulate the exploration score  $e_{k,i}$  (i.e., the label for  $f_{2}(\cdot;\pmb{\theta}_{2})$ ), we consider two separate exploration objectives: (1) the prediction uncertainty for the exploitation module  $f_{1}(\cdot;\pmb{\theta}_{1})$ , which is  $r_{k,i} - f_{1}(\pmb{\chi}_{k,i}^{q};\pmb{\theta}_{1})$ ; (2) the validation loss of the meta-model, which represents the difficulty of adapting to  $\mathcal{T}_{k,i}$ , inspired by the "task difficulty measurer" in Curriculum Learning [52, 50]. As a result, with  $\mathcal{L}_{k,i} = \mathcal{L}(D_{k,i}^{q};\Theta_{k,i}^{(J)})$  being the validation loss of arm  $\mathcal{T}_{k,i}$ , we formulate the exploration score as  $e_{k,i} = \alpha \cdot (r_{k,i} - f_{1}(\pmb{\chi}_{k,i}^{q};\pmb{\theta}_{1}^{(k-1)})) + (1-\alpha)\cdot \mathcal{L}_{k,i}$ . Analogously, with the approximated reward  $\widetilde{r}_{k,i}$  (Eq. 8), we calculate the exploration score approximation by
$$
\widetilde {e} _ {k, i} = \alpha \cdot \left(\widetilde {r} _ {k, i} - f _ {1} \left(\boldsymbol {\chi} _ {k, i} ^ {q}; \boldsymbol {\theta} _ {1} ^ {(k - 1)}\right)\right) + (1 - \alpha) \cdot \mathcal {L} _ {k, i}. \tag {9}
$$
Here, the exploration coefficient  $\alpha \in (0,1]$  (in Eq. 7) is also used to balance our two exploration objectives, which are (1) prediction uncertainty  $r_{k,i} - f_1(\cdot ;\pmb{\theta}_1)$ : if the exploitation model  $f_{1}(\cdot ;\pmb{\theta}_{1})$  is under-estimating the arm reward, leading to the positive residual  $r_{k,i} - f_{1}(\cdot ;\pmb{\theta}_{1})$ , we will have a high exploration score to enhance the exploration for this arm; otherwise, when  $r_{k,i} - f_{1}(\cdot ;\pmb{\theta}_{1})$  is negative, it indicates an excessively high estimation, which will alternatively lower the exploration score to compensate for the over-estimation. With a higher  $\alpha$  value, our exploration strategy will focus more on the behavior of the exploitation model  $f_{1}(\cdot)$ ; (2) the difficulty of task adaptation (i.e., validation loss)  $\mathcal{L}_{k,i}$ : if the current meta-model does not generalize well to arm  $\mathcal{T}_{k,i}$ , the validation loss  $\mathcal{L}_{k,i}$  will be high, which will also lead to a high exploration score. In this way, our formulation considers two different exploration objectives as well as the dynamics of meta-optimizations (base on concatenated network gradients), for a refined exploration strategy.
4.4 Updating Bandit Scheduler Parameters
After updating the meta-parameters (Line 14, Algo. 1, Remark 2) with tasks  $\Omega_{k} = \{\mathcal{T}_{k,\hat{i}}\}_{\hat{i}\in \hat{\mathcal{N}}_{k}}$  we proceed to update the parameters of BASS (Line 15, Algo. 1). Recall that  $f_{1}(\cdot ;\pmb{\theta}_{1})$  tries to learn the reward mapping function  $h(\cdot)$ , and  $f_{2}(\cdot ;\pmb{\theta}_{2})$  aims to learn the exploration score. Given the selected arms  $\Omega_{k}$ , with  $\eta_1^\theta ,\eta_2^\theta \in \mathbb{R}^+$  being the learning rates, we apply the GD and quadratic loss to update the parameters of BASS, denoted by  $\pmb{\theta}_{1}^{(k)} = \pmb{\theta}_{1}^{(k - 1)} - \eta_{1}^{\pmb{\theta}}\cdot \nabla_{\pmb{\theta}_{1}}\big(\frac{1}{B}\sum_{\pmb{k},\hat{i}\in \Omega_{k}}\big|f_{1}(\pmb{\chi}_{k,\hat{i}}^{q};\pmb{\theta}_{1}^{(k - 1)}) - \widetilde{r}_{k,\hat{i}}\big|^{2}\big)$ ,  $\pmb{\theta}_{2}^{(k)} = \pmb{\theta}_{2}^{(k - 1)} - \eta_{2}^{\pmb{\theta}}\cdot \nabla_{\pmb{\theta}_{2}}\big(\frac{1}{B}\sum_{\pmb{k},\hat{i}\in \Omega_{k}}\big|f_{2}\big([ \nabla_{\pmb{\theta}}f_{1}(\pmb{\chi}_{k,\hat{i}}^{s});\nabla_{\pmb{\theta}}f_{1}(\pmb{\chi}_{k,\hat{i}}^{q})]\big);\pmb{\theta}_{2}^{(k - 1)}\big) - \widetilde{e}_{k,\hat{i}}\big|^{2}\big)$ . We refer to Eqs. 8-9 for calculating the approximated arm reward  $\widetilde{r}_{k,\hat{i}}$  and exploration score  $\widetilde{e}_{k,\hat{i}}$ .
Recall that in each iteration  $k \in [K]$ , we receive candidate arms (tasks)  $\Omega_{\mathrm{task}}^{(k)} = \{\mathcal{T}_{k,i}\}_{i \in \mathcal{N}}$ , and each arm  $\mathcal{T}_{k,i}$  is associated with two context vectors  $\boldsymbol{\chi}_{k,i}^s, \boldsymbol{\chi}_{k,i}^q$ . For the sake of analysis, we normalize these
two contexts such that  $\| \boldsymbol{\chi}_{k,i}^{s}\|_{2} = \| \boldsymbol{\chi}_{k,i}^{q}\|_{2} = 1$ , and set the exploration coefficient  $\alpha = 1$ . Following the existing work [47, 48], we let the meta-model  $\mathcal{F}(\cdot ;\Theta)$  be a  $L_{\mathcal{F}}$ -layer FC network with Gaussian Initialization, with the network width  $m_{\mathcal{F}}$ . Note that our results can also be generalized to other network architectures, such as CNN and ResNet [22], based on the analysis of over-parameterized neural networks [11, 48, 3]. For the theoretical analysis, we adopt Sigmoid activation for  $f_{1}$  and ReLU for  $f_{2}$ , in order to make  $f_{1}$  Lipschitz smooth under over-parameterization settings. Then, we draw trained parameters of BASS with  $\{\pmb{\theta}_1^{(k)},\pmb{\theta}_2^{(k)}\} \sim \{\widetilde{\pmb{\theta}}_1^{(\tau)},\widetilde{\pmb{\theta}}_2^{(\tau)}\}_{\tau \in [k]}$ . Here, starting from the randomly initialized parameters  $\{\pmb{\theta}_1^{(0)},\pmb{\theta}_2^{(0)}\}$ , each parameter pair  $\{\widetilde{\pmb{\theta}}_1^{(\tau)},\widetilde{\pmb{\theta}}_2^{(\tau)}\},\tau \in [k]$  is separately trained on past arm rewards  $\{r_{\tau',\widehat{i}}\}_{\tau' \in [\tau], \widehat{i} \in \widehat{\mathcal{N}}_{\tau'}}$  and exploration scores  $\{e_{\tau',\widehat{i}}\}_{\tau' \in [\tau], \widehat{i} \in \widehat{\mathcal{N}}_{\tau'}}$  with  $J_{\theta}$ -iteration GD. Next, similar to existing neural bandit works (e.g., [54, 9, 53]) and the works on meta-model convergence analysis (e.g., [47, 48]), we have the following separateness assumption.
Assumption 5.1 ( $\rho$ -Separateness). After  $K$  meta-training iterations, for every pair of arm contexts  $\mathcal{X}_{k,i}^{q}, \mathcal{X}_{k',i'}^{q}$  with  $k, k' \in [K]$  such that the corresponding arms  $\mathcal{T}_{k,i} \in \Omega_k \wedge \mathcal{T}_{k',i'} \in \Omega_{k'}$ , if  $(k,i) \neq (k',i')$ , we have  $\| \mathcal{X}_{k,i}^{q} - \mathcal{X}_{k',i'}^{q} \|_2 \geq \rho$  where  $0 < \rho \leq \mathcal{O}\left(\frac{1}{L}\right)$ .
The assumption above is mild because of two main reasons: (1) since  $L$  is manually chosen (e.g.,  $L = 2$ ), we can easily satisfy the condition  $0 < \rho \leq \mathcal{O}\left(\frac{1}{L}\right)$  as long as no two arm contexts are identical; (2) since the meta-parameters  $\Theta^{(k)}, k \in [K]$  are constantly changing, the corresponding arm contexts will also be distinct across different meta-training iterations. Additional discussions on this assumption are in Appendix Section B. With standard settings of over-parameterized neural networks [47, 3, 11] and the definition of regret  $R(K)$  in Eq. 5, we have the following Theorem 5.2.
Theorem 5.2. Define  $\delta \in (0,1)$ ,  $0 < \xi_1, \xi_2 \leq \mathcal{O}(1 / K)$ ,  $\xi_f = \max\{\xi_1, \xi_2\}$ ,  $0 < \rho \leq \mathcal{O}(1 / L)$ ,  $c_\xi > 0$ ,  $\xi_L = (c_\xi)^L$ . Suppose the network width  $m \geq \Omega\bigl(Poly(K, L, N_{task}, \rho^{-1}) \log(1 / \delta)\bigr)$ ;  $m_{\mathcal{F}} \geq \Omega\bigl(Poly(K, L_{\mathcal{F}}, N_{task}) \cdot \log(1 / \delta)\bigr)$ . Then, let the learning rates be  $\eta_1, \eta_2 = \Theta\bigl(\frac{m_{\mathcal{F}}^{-1}}{Poly(K, N_{task}, L_{\mathcal{F}})}\bigr)$ ;  $\eta_\theta^1, \eta_\theta^2 = \Theta\bigl(\frac{\rho \cdot m^{-1}}{Poly(K, N_{task}, L)}\bigr)$ .  $J_\theta = \Theta\bigl(\frac{Poly(K, N_{task}, L)}{\rho \cdot \delta^2} \cdot \log\bigl(\frac{1}{\xi_1}\bigr)\bigr)$ . Following Algo. 1, with probability at least  $1 - \delta$ , the  $K$ -round  $R(K)$  of BASS could be bounded by
$$
R (K) \leq \mathcal {O} \left(\frac {1}{\sqrt {K}} \left(\sqrt {2 \xi_ {f}} + \frac {3 L}{\sqrt {2}} + \left(1 + 2 \gamma_ {1}\right) \sqrt {2 \log (K / \delta)}\right)\right) + \mathcal {O} \left(\frac {\xi_ {L} ^ {2} K J B \sqrt {L _ {\mathcal {F}}}}{\sqrt {m _ {\mathcal {F}}}}\right) + \gamma_ {m} \tag {10}
$$
where  $\gamma_{m} = \mathcal{O}\left(\frac{1}{m^{1 / c_{\gamma}}}\right),c_{\gamma} > 1$  , and  $\gamma_{1} = \mathcal{O}(1)$  with sufficient network width  $m$  of BASS.
The proof is presented in Appendix Section D. Here, the first term on the RHS is scaled by the  $1 / \sqrt{K}$  term, which means the regret bound will shrink along with more iterations  $K$ . The second term on the RHS is scaled by  $1 / \sqrt{m_{\mathcal{F}}}$ , which makes it a diminutive term under the over-parameterization settings. Since the network depth  $L$  of BASS is a small integer (we apply  $L = 2$  for experiments in Section 6),  $\xi_L$  will also be a relatively small constant. Meanwhile,  $\gamma_m$  will also decrease significantly with increasingly large network width  $m$  of BASS. In contrast, with a convex loss function (e.g.,  $L_2$  loss or cross-entropy loss) and the same over-parameterization settings, the regret upper bound of the uniform sampling strategy [19, 40] can possibly scale up to 1 for the worst-case scenario, and the upper bound will not decrease with more iterations  $K$  (Appendix Lemma D.13). Alternatively, BASS works under the bandit settings, by directly measuring the meta-model performance difference w.r.t. the chosen task batch and the optimal one. With more iterations  $K$ , BASS tends to make more accurate scheduling decisions, which makes our regret bound possible. For the existing works, [51] prove that they can improve the optimization landscape with the assumed correlation of task difficulties and meta-model generalization ability. [13] show that their self-paced strategy can improve the model robustness when facing noisy training tasks. Different from previous works, we provide the performance guarantee for the proposed BASS under the neural bandit framework.
In this section, we compare BASS against seven strong baselines, including: (1) Uniform Sampling; non-adaptive self-paced methods and task schedulers (2) SPL [26], (3) Focal-Loss (FOCAL) [30], (4) DAML [29], (5) GCP [31], (6) PAML [24]; and the adaptive task scheduler (7) ATS [51]. Since GCP is not originally compatible with our problem settings and can only work with classification
problems, we properly adapt it by choosing the tasks with the highest probabilities, and apply it on the classification data sets. ANIL [40] is adopted as the backbone meta-learning framework. Due to page limit, we include the complementary experiments (e.g., parameter study for  $\alpha$ , effects of different levels of task skewness), and the configurations to Appendix Section A.
6.1 Real Data Sets with Noisy Meta-Training Tasks
We adopt Drug [36], Mini-ImageNet (M-ImageNet) [42] and CIFAR-100 (CIFAR) [25] data sets under the few-shot learning scenario. Similar to [51], we apply classification accuracy as the evaluation criterion for the Mini-ImageNet and CIFAR-100 data sets, and consider the squared Pearson coefficient for the Drug data set. For each meta-learning iteration  $k \in [K]$ , the learner is given a candidate pool of 10 tasks (i.e.,  $|\Omega_{\mathrm{task}}^{(k)}| = N_{\mathrm{task}} = 10$ ), and it will need to choose a batch of  $B = 2$  tasks as the training tasks  $\Omega_k$  for this iteration. For the Mini-ImageNet and CIFAR-100 data sets, we consider half of the meta-training tasks are perturbed by the label flipping noise [21], where the chance of a label being flipped is  $\epsilon \in [0,1]$ . As the Drug data set stands for a regression problem, we draw the label noise from the Gaussian distribution  $N(0,\epsilon^2)$ . The experimental settings are under 1-shot or 5-shot, 5-Way (for classification data sets) learning scenario with the noise level  $\epsilon = 0.5$ . The experiment results are shown in Table 1 and Figure 3. For the average ranking column, we exclude results from the M-ImageNet (1) setting, since BASS and the baselines tend to train a meta-model performing "random guessing". More discussions are in the next paragraph.
Table 1: Results on real data sets [data set (shot); results ± standard deviation]. For 1-shot M-ImageNet, all the methods end up with an invalid meta-model performing "random guessing".
Algo. \ Data	Drug (1)	Drug (5)	M-ImageNet (1)	M-ImageNet (5)	CIFAR (1)	CIFAR (5)	Avg. Rank
Uniform	0.210±0.013	0.220±0.001	0.201±0.002	0.301±0.025	0.234±0.029	0.526±0.011	4.4
SPL	0.244±0.008	0.236±0.004	0.203±0.002	0.240±0.018	0.200±0.002	0.367±0.039	5.0
FOCAL	0.222±0.024	0.223±0.003	0.200±0.000	0.316±0.029	0.231±0.024	0.485±0.006	4.4
DAML	0.146±0.009	0.177±0.003	0.201±0.001	0.310±0.016	0.247±0.003	0.414±0.025	5.4
GCP	N/A	N/A	0.201±0.001	0.282±0.016	0.243±0.007	0.508±0.009	6.0
PAML	0.192±0.020	0.205±0.009	0.199±0.001	0.218±0.013	0.199±0.004	0.316±0.022	7.2
ATS	0.230±0.002	0.237±0.014	0.201±0.001	0.334±0.053	0.257±0.048	0.515±0.015	2.4
BASS	0.242±0.012	0.245±0.006	0.198±0.004	0.351±0.012	0.272±0.025	0.553±0.008	1.2
Here, BASS can generally outperform the baselines by directly learning the mapping from the metaparameters to the arm rewards, as well as balancing the exploitation and exploration. ATS also achieves good performance as it adaptively learns the correlation between the task adaptation difficulty and task scheduling, which proves that it is necessary to apply the adaptive scheduling strategy instead of staying with a fixed protocol. Meanwhile, BASS can generally train the meta-model more efficiently (Figure 3), leading to good performances at the early stage of
meta-training. In particular, for the Mini-ImageNet data set under the 1-shot, 5-way settings, all the algorithms fail to train an effective meta-model. Here, under the 5-way classification scenario, all the methods will likely generate a meta-model performing "random guessing" (around  $20\%$  accuracy). In this case, utilizing Ensemble Inference techniques [12, 16] can help alleviate this problem, and we include further discussion in the case study (Subsec. 6.3).
Figure 3: Accuracy results (5-shot,  $\epsilon = 0.5$ ). BASS can achieve a good performance at early meta-training stage.
6.2 Effects of the Skewed Task Distribution
The skewed task distribution  $\mathcal{P}(\mathcal{T})$  commonly exists in real-word cases. For instance, consider an animal image classification data set where each class (i.e., task) corresponds to one
Table 2: CIFAR-100 with skewed task distribution (5-shot, 5-way / 10-way).
Data\Algo.	Uniform	SPL	ATS	BASS
5-way	0.375±0.009	0.279±0.002	0.382±0.007	0.408±0.008
10-way	0.264±0.047	0.165±0.007	0.283±0.039	0.320±0.021
kind of animals. In this case, felid classes can be considered as "frequent" tasks in the task distribution due to their large quantity and strong mutual correlations, compared with "tail" tasks like kangaroo classes. In this case, paying insufficient attention to the "tail" classes can impair the
generalization performance of the trained meta-model. Thus, to investigate the effects of when the task distribution  $\mathcal{P}(\mathcal{T})$  is skewed, we randomly choose some tasks from  $\mathcal{P}(\mathcal{T})$ , and assign them with higher sampling probabilities (weights). This corresponds to the situation when  $\mathcal{P}(\mathcal{T})$  is skewed, so that sampling from  $\mathcal{P}(\mathcal{T})$  will likely lead to similar tasks. Here, with the CIFAR-100 data set, we sample 10 tasks and assign them with higher sampling probabilities (weights) (5 tasks with  $10\%$ , 5 tasks with  $5\%$ ), while the rest of the tasks equally share the remaining  $25\%$  probability.
Figure 4: Average weights of chosen meta-training tasks. The testing accuracy vs. iterations needed. BASS can actively exploring for "tail" tasks, and requires much fewer iterations for the same performance (as few as  $\sim 1/3$  of baselines' iterations).
From Table 2, our proposed BASS maintains the best performance due to its adaptive scheduling strategy and the ability of balancing exploitation and exploration. On the other hand, the baselines are unable to adjust their meta-training strategies towards exploration, when the meta-model has already well adapted to the "frequent" tasks. This can possibly lead to the sub-optimal performance of the meta-model. In particular, based on Figure 4, when facing a skewed task distribution, BASS can actively explore the "tail" tasks, after sufficiently exploring the "frequent" tasks. BASS is also able to achieve good performances with fewer meta-training iterations.
6.3 Case Study: BASS-aided Ensemble Inference