Interval Bound Interpolation for Few-shot Learning with Few Tasks
Authors: Shounak Datta
Sankha Subhra Mullick
Anish Chakrabarty
Swagatam Das
Venue: Submitted to ICLR 2023 (2023)
Keywords: meta-learning
metric learning
task interpolation
Interval Bound Propagation

Abstract: Few-shot learning aims to transfer the knowledge acquired from training on a diverse set of tasks to unseen tasks from the same task distribution, with a limited amount of labeled data. The underlying requirement for effective few-shot generalization is to learn a good representation of the task manifold. This becomes more difficult when only a limited number of tasks are available for training. In such a few-task few-shot setting, it is beneficial to explicitly preserve the local neighborhoods from the task manifold and exploit this to generate artificial tasks for training. To this end, we introduce the notion of interval bounds from the provably robust training literature to few-shot learning. The interval bounds are used to characterize neighborhoods around the training tasks. These neighborhoods can then be preserved by minimizing the distance between a task and its respective bounds. We then use a novel strategy to artificially form new tasks for training by interpolating between the available tasks and their respective interval bounds. We apply our framework to both model-agnostic meta-learning as well as prototype-based metric-learning paradigms. The efficacy of our proposed approach is evident from the improved performance on several datasets from diverse domains in comparison to recent methods.

Under review as a conference paper at ICLR 2023
INTERVAL BOUND INTERPOLATION FOR FEW-SHOT LEARNING WITH FEW TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
Few-shot learning aims to transfer the knowledge acquired from training on a diverse set of tasks unseen tasks from the same task distribution, with limited amount of labeled data. The underlying requirement for effective few-shot gen- eralization is to learn good representation of the task manifold. This becomes more difficult when only a limited number of tasks are available for training. In such a few-task few-shot setting, beneficial to explicitly preserve the local neighborhoods from the task manifold and exploit this to generate artificial tasks for training. To this end, we introduce the notion of interval bounds from the provably robust training literature to few-shot learning. The interval bounds are used to characterize neighborhoods around the training tasks. These neighbor- hoods can then be preserved by minimizing the distance between a task and its respective bounds. We then use novel strategy to artificially form new tasks for training by interpolating between the available tasks and their respective interval bounds. We apply our framework to both model-agnostic meta-learning as well as prototype-base -based metric-learning paradigms. The efficacy of our proposed approach is evident from the improved performance on several datasets from diverse domains in comparison recent methods.
INTRODUCTION
Few-shot learning problems deal with diverse tasks consisting of subsets of data drawn from the same underlying data manifold along with associated labels. The joint distribution of data and corresponding labels which governs the sampling of such tasks often called the task distribution (Finn et al. 2017 Yao et al. 2022). Consequently, few-shot learning methods attempt to leverage the knowledge acquired by training on a large pool of such tasks to easily generalize unseen tasks from the same distribution, using only few labeled examples. We hereafter refer to the support of the task distribution as the task manifold which is distinct from but closely-related to the data manifold associated with the data distribution. Since the unseen tasks are sampled from the same underlying manifold governing the task distribution, we should idcally learn a good representation of the task manifold by preserving the neighborhoods from the high-dimensional manifold in the dover-dimensional feature embedding (Tenenbaum et al. 2000 Roweis & Saul 2000) Van der Maaten & Hinton 2008). However, the labels associated with a task can define any arbitrary partitioning of the data. Therefore, we can attempt to preserve the neighborhood for a task by simply conserving the neighborhoods for the corresponding subset of the data manifold in the feature embedding learned by the few-shot learner. This facilitates effective generalization to new tasks using a limited amount of labeled data by only updating the classifier as the learned feature embedding would likely require very little adaptation. However, cxisting few-shot learning methods lack an explicit mechanism for achieving this. Further, real-world few-shot learning scenarios like rare disease detection may not have the large number of training tasks required for effective learning, due to various constraints such as data collection costs, privacy concerns, and/or data availability in newer domains (Yao et al. 2022). In such scenarios, few-shot learning methods are prone to overfit the training tasks, thus limiting the ability to generalization to unseen tasks. Therefore, in this work, we develop a method to explicitly constrain the feature embedding in an attempt to preserve neighborhoods from the digeesiinns task manifold and to construct artificial tasks within these neighborhoods in the feature space, to improve the performance when limited number of training tasks are available.

---

Under review as a conference paper at ICLR 2023
The proposed approach relies on characterizing the neighborhoods from the high- -igh-dimensional task manifold and propagating them through the network with the intent to preserve the task neighborhood in the feature space. We achieve this by employing the concept of interval bounds from the provably robust training literature (Gowal et al. 2019, Morawiecki et al. 2020), i.e. the axis-aligned bounds for the activations in each layer, obtained using interval arithmetic (Sunaga 1958). Concretely, as shown in Figure 1 we first define a small e-neighborhood for each few-shot training task and then use Interval Bound Propagation Gowal et al. 2019) to obtain the bounding box around the mapping of the corresponding neighborhood in the feature embedding space. We then explicitly attempt to preserve the e-neighborhoods by minimizing the distance between a task and its respective interval bounds in addition to optimizing the few-shot classification objective. We further devise mechanism to construct the artificial tasks by interpolating between task and its corresponding IBP bounds. is important to notice that this setup is distinct from provably robust training for few-shot learning in that we do not attempt to minimize (or calculate for that matter) the worst-case classification loss.
Task manifold
Learned embedding
Lye
Distribution p(T)
CLB
Sample Task
T
fg
fa
+Cex
Learner
Figure I: Illustration of the proposed interval bound propagation-aided few-shot learning setup (best viewed in color): We use interval arithmetic to define a small e-neighborhood around a training task Ti sampled from the task distribution p(T). IBP is then used obtain the bounding box around the mapping of the said neighborhood in the embedding space faz given by the first S layers of the learner fo. While training the learner fo minimize the classification loss LC on the query set DJ, we additionally attempt to minimize the losses LLB and LUB, forcing the c-neighborhood to be compact in the embedding space as well.
(a)
(b)
Figure 2: Interval hound-based task interpolation (best viewed in color): (a) Existing inter-task interpolation methods create new artificial tasks by combining pairs of original tasks (blue ball). However, depending on how flat the task-manifold embedding is at the layer where interpolation is performed, the artificial tasks may either be created close the task-manifold (green cross) or away from the task-manifold (red box). (b) The proposed interval bound-based task interpolation creates artificial tasks by combining an original task with one of its interval bounds (yellow ball). Such artificial tasks are likely to be in the vicinity of the task manifold as the interval bounds are forced to be close the task embedding by the losses LLB and LUB-
Various methods have been proposed to mitigate the few-task few-shot problem using approaches such as explicit regularization (Jamal & Qi, 2019: Yin et al.) 2019), intra-task augmentation (Lec) al. 2020 Ni et al. 2021| Yao et al. 2021) ,. and inter-task interpolation to construct new artificial tasks Yao al. (022). While inter-task interpolation has been shown to be the most effective among these existing approaches, it suffers from the limitation that the artificially created tasks may be generated away from the task manifold depending
the curvature of the feature embedding space, as there is no natural way to select pairs of task which are close to each other on the manifold (Figure 2(a]p. The interval bounds obtained using IBP, the other hand, likely be close to the original task embedding as we explicitly minimize the distance between a task and its interval bounds. Thus, using them for interpolation is likely keep the generated tasks close the manifold (Figure 2(b).
2

---

Under review as a conference paper at ICLR 2023
In essence, the key contributions made in this article advance the existing literature in the following ways: (1) In Section 4.1 we present for the first time, a novel method to synergize few-shot learning
with interval bound propagation (Gowal et al. 2019) as to explicitly lend the ability to preserve task neighborhoods in the feature embedding space of the few-shot learner.
(2) In Section 4.2
propose the interval bound-based task interpolation technique which can create new tasks (as
opposed to augmenting each individual task (Lee et al. 2020; Ni et al.) [021]] Yao et al. 2021 ) by
interpolating between task sampled from the task distribution and its interval bounds. (3) Unlike
existing inter-task interpolation methods that require paired tasks for interpolation (Yao et al. 2022
our framework can generate new tasks from only a single task. This allows the proposed framework
to be seamlessly integrated with existing few-shot learning paradigms.
In Section 5 we empirically demonstrate the effectiveness of our proposed approach on both gradient- based meta-learning and prototype-based metric-learning on few-task real-world datasets from multiple domains, in comparison to the recent prior methods. Finally, we make concluding remarks and also discuss future scopes of research in Section 6
RELATED WORKS
Few-shot learning: The aim of few-shot learning is to generalize to new tasks using only a few examples Wang et al. (020)) through three major strategies. First, one can augment the tasks at the data level (Hariharan
Girshick 2017). Second, the hypothesis space can constrained at the model level (Snell et al. (2017). Third, the hypothesis search strategy at the algorithm level can be improved (Finn et al. [207]) The problem of few-task learning can be even more difficult when there is a scarcity of training tasks in a few-task scenario. We take the route of Yao et al. (2022) to offer a novel task augmentation strategy that can work in conjunction with both algorithm- meta-learning well as model-level metric-learning methods.
Provable robust training of neural networks: A way to build robust neural networks is to find a differentiable upper bound on the verifiable violation of specifications. Such upper bounds then be directly optimized alongside the original loss (Mirman et al. 2018 Raghunathan et al.) 2018: Wong let al. 2018). IBP (Gowal et al. 2019) follows this direction by exphiitly minimizing the worst- loss inside the e-neighborhood of an input for an arbitrary network with some architectural constraints. repurpose IBP characterize the e-neighborhood to learn better representation such that the generalization to new tasks by a few-shot learner becomes easier. Moreover, the bounds of the e-neighborhood obtained through IBP gives us direct way to construct new artificial tasks when the number of available tasks is scarce.
However, in work, instead of building robust networks,
Manifold learning: Traditional methods like ISOMAP Tenenbaum et al., 2000 LLE (Roweis Saul 2000), t-SNE (Van der Maaten & Hinton| 2008), etc. aims to represent high- high-dimensional data in lower-dimensional space while preserving the local neighborhoods through manifold learning (Abukmeil et al. 202T). Recent manifold learning approaches mostly employ generative neural networks such as deep belief network (Lee et al. 2009), variational auto-encoders (Connor et al.
2021 Kumar& & Poole, 2020), flow-based approaches (Brehmer & Cranmer [2000] Caterini et al. 2021). etc. In a similar spirit, we repurpose IBP to define e-neighborhoods for few-shot learning tasks and constrain the learned feature embedding preserve the said neighborhoods.
Task augmentation: To train on datasets with a limited number of tasks, some works directly impose regularization on the few-shot learner (Jamal & Qil 2019 Yin et al., 2019). Another line of work performs data augmentation on the individual tasks {Lee et al. 2020 Ni et al. 2202]] Yao et al.) 2021). Finally, a third direction is to employ inter-task interpolation to mitigate task scarcity ( (Yao al.) 2022). Our approach is similar to the third category in that we directly create new artificial tasks. But, we also differs from all of the chove-mentionel" methods in that we neither undertake intra-task augmentation nor inter-task interpolation.
3 PRELIMINARIES
In few-shot learning problem, we deal with tasks T/ ~ p(T). Each task T, is associated with dataset D. (X,, Y,), that we further subdivide into a support set Dj- = (X;, Y,") = {(x),, )1 and query set D (XY, (xx",, , Given a learning model fo. where 0 denotes the model parameters, few-shot learning algorithms attempt to learn 0 to minimize the loss on the query
3

---

Under review as a conference paper at ICLR 2023
set D.d for each of the sampled tasks using the data-label pairs from the corresponding support set Di Thereafter, the trained model fo and the support set Dj for new tasks T, can be used to perform inference on the corresponding query set D In the following subsection, we discuss gradient- meta-learning while the prootype-based metric-learning is detailed in Appendix [A]
Gradient-based meta-learning: In gradient-based meta-learning, the aim is to learn initial param- eters 0* such that typically small number of gradient update steps using the data-label pairs in the support set Dj results in model fo. that performs well on the query set of task Ty. During the meta- meta-training stage, first, a base learner is trained on multiple support sets Dj, and the performance of the resulting models fø; is evaluated on the corresponding query sets Dj. The meta-learner param- eters 0 are then updated such that the expected loss of the base learner on query sets is minimized. In the meta-testing stage, the final meta-trained model fa- is fine-tuned on the support set Dj for the given test task T to obtain the adapted model foy that can then be used for inference on the corresponding query set DJ. Considering Model-Agnostic Meta-Lean Learning (MAML) (Finn et al. 2017) as an example, the bi-level optimization of the gradient-based meta-learning is formutated as:
0- arg min (TT() [C(f. D]) where di
170 VeL(fe: Di).
(1)
while t/ denotes the inner-loop learning rate used by the base learner to train on Dj for task T,, and C is the loss function, which is usually the cross-entropy loss for classification problems:
Cc = BT I- Σ, log Ix,,,
(2)
A key requirement for effective few-shot generalization to new tasks for both gradient-based meta- learning and prototype-baser metric-learingg to learn good embedding of the high- dimenssinl manifold characterizing the task distribution p(T), i.c. the task manifold. Ideally, the learned embedding should conserve the neighborhoods from the high-dimensional task manifold (Tenenbaum et al. 2000; Roweis & Saul 2000). Hence, in the following subsection, we discuss Interval Bound Propagation (IBP) (Gowal et al. 2019)) that can be employed to define a neighborhood around task.
Interval bound propagation: Let us consider a neural network fo consisting of a sequence of transformations hy. (1 E (1,2..... ,)) for each of its L layers. We start from an initial input Zg x to the network along with lower bound 2, (c) = x 1 and upper bound Zn(e) = x + 1e for an e-neighborhood around the input x. In each of the subsequent layers 1 E 21. ... , of the network, we get an activation 2 14 (2(-1) IBP uses interval arithmetic to obtain the corresponding axis-aligned bounds of the form (c) I 2; I Z2(c) on the activations for the I-th layer. Given the specific differentiable transformation h4, interval arithmetic yields corresponding differentiable lower and upper bound transformations (c) = hy (c), (c)), and (c(c) h4 (c), (c)) as described in Appendix C This ensures that each of the coordinates (c) and (t, of (c(c) and (e) respectively, satisties the conditions:
min (E) (0)) (1 15 (()
max (T and (e, Zi-1 (e) 15 166)
e e.
where e, is the standard c-th basis vector. Further extending to multiple layers, such as fas having the first S layers of fe. the individual transformations 4 and l for 1 E {1,2, ... S} can be composed to obtain the corresponding functions Las and Jgs. such that 25(e) = Los (Zo. e), and
(c(c) (zo: c).
4 PROPOSED METHOD
Our proposed method aims enable the learner fa learn feature embedding that attempts to preserve the e-neighborhoods in the task manifold. Therefore, in the following subsection, we describe the notion of an e-neighborhood for a training task T; using TBP and show how they can be preserved to aid in few-shot learning problems.
4.1 FEW-SHOT LEARNING WITH INTERVAL BOUNDS
Consider the network fo = fee-s o fos where S (W L) is a user-specified layer number that demarcates the boundary between the portion sas of the model that focuses feature representation
4

---

Under review as a conference paper at ICLR 2023
and the subsequent portion far-s responsible for the classification. For a given training task Tú, the Euclidean distances between the embedding fos (xx,p) for the query instances and their respective interval bounds Los (x c) and go (x) <) is measure of how well the c-neighborhood is preserved in the learned feature embedding:
1 Σ (x²,) - + and Ng
LLB
e)
(4)
CUB Ng E (x],)) - Jos (x) e)
(5)
To ensure that the small c-neighborhoods get mapped to small interval bounds by the feature em- bedding fas, we can minimize the losses LLB and LUB in addition to the classification loss LCE in (2. Notice that the losses LLB and LUB are never used for the support instances x
Attempting to minimize naive sum of the three losses can cause some issues. For example, weighing the classification loss Loe too high essentially reduces the proposed method to vanilla few-shot learn- ing. On the other hand, assigning very high weights to the interval losses LLB and/or LU may diminish learnability as the preservation of e-neighborhoods gets precedence over classification performance. Moreover, such static weighting approaches are not capable of adapting to (and consequently mitigating) situations where one of the losses comes to unduly dominate the others. Thus, we minimize a convex weighted sum L of the three losses:
cE
uB
LB
MM /
Mny
30.4
www. Aa
0.2
2E+4
4E+4
6E+4
Stops
Figure 3: Dynamic weights for MAML+IBP on minilm- ageNet when 7 is set I for ease of visualisation.
((t) (CE(t) + wLB(t)
+ wuB(t)
(6)
where denotes the current training step and we is the weight for the corresponding loss Le, e E {CF, I.B, U B} the 1-th training step,
which is dynamically calculated based on softmax across the current values of the three losses:
(xp(Lc(t)/n)
(e = E Ee'E(CE,LB,UB (x)(Ce
The hyperparameter 7 controls the relative importance of the losses. If any of the losses become too large, the dynamic weighing scheme strives to restore balance by assigning very high weightage to the concerned loss, thus prioritizing its minimization over that of the other losses. The changes in the dynamic weights over training steps for IBP-aided MAML (hereafter called MAML+IBP) using "4-CONV" network (Vinyals et | (0166) on the minilmageNet dataset (Vinyals al. 2016) is illustrated in Figure 3 We can observe that while there is clear ordering to the magnitude of the weights (and therefore the corresponding losses) throughout the entire training run, the weights are in fact able adapt changes in loss values to maintain the status quo among the different losses.
Motivating results: Ta- ble [ demonstrate the ef- fect of employing IBP-aided training for MAML using "4-CONV" network. Apart from vanilla MAML, we consider three other regu- larized variants of MAML, (1) MAML+SN that applies Spectral Normalization (Miy- ato et 2018) to the feature embedding through the first S
Table 1: Effect of IBP on MAML for minilmageNet and tieredIm- ageNet datasets in terms of 5-way 1-shot Accuracy and intra-task compactness
Algorithm
Accursey
I-NN distance
minilmapeNet
tieredlmageNet
minilmageNet
direddImapNNet
MAML Fine et 2017
48.20-t
$1.67: 1.81%
0.97-20.02
0:98:20:022
90::::11
45.3612 1.05%
1.38220.04
1.41:20000 0.04
MAMLAGI.
48. 13:70= 0.97%
01.90:20.99%
0.96::
0.98220002
MAML-ULLBI
49 43:1-09900
0.911.9911
0.94:20:02
0:97-20:07
91..7611188%
54.36:1..00%%
0.900:20.02
0.96220002
layers, (2) MAML+GL that uses the distance between the original query set and its perturbed (by
additive Gaussian noise) version as an extra loss, and (3) MAML+ULBL that considers the distance
between the upper and lower interval bounds as additional loss (Morawiecki et al. 2020) (further
details in Appendix F ..
We see that MAML+IBP achieves higher 5-way I-shot classification
accuracy than the 5 contenders on the minilmageNet and tieredl (Ren et al. 2018) datasets.
5

---

Under review as a conference paper at ICLR 2023
Moreover, we also illustrate that the feature embedding learned by IBP-aided training exhibits better intra-task compactness in terms of the mean Euclidean distances from the nearest neighbor in the same class for 100 query instances from 600 tasks, in the feature space characterized by fas. Recent works (Ni et al. 2021 Yao et al. 2022) have shown that augmenting the training data with artificial tasks can improve performance domains with limited amount of tasks. Therefore, while IBP-aided training improves the performance of vanilla MAML (as well as other baselines, detailed in Appendix F33. we are particularly interested in the added advantage that it lends by facilitating the generation of artificial tasks within the neighborhoods defined by the interval bounds.
4.2 INTERVAL
TASK INTERPOLATION
Since minimizing the additional losses LLB and LUB is expected to ensure that the e-neighborhood around a task is mapped to a small interval in the feature embedding space, artificial tasks formed within such intervals are naturally expected to be close to the task manifold. Therefore, we create additional artificial tasks by interpolating between an original task and its corresponding interval bounds (i.e., either the upper or the lower interval bound). In other words, for a training task Ti, a corresponding artificial task T,' is characterized by a support set D = ((H: yi, } in the embedding space. The artificial support instances Hi,, are created as:
H (1 Jas (x(,) + (1 Ls)D (X9, + VAdATOS (xf,r:)) 6);
(8)
where i denotes the class to which x belongs, A E [0, 1] is sampled from a Beta distribution Beta(a, _9), and the random choice of nh {0, 1} dictates which of the bounds is chosen randomly for each class. The labels y for the artificial task remain identical to that of the original task. The
query set DD for the artificial task is also constructed analogously. We then minimize the mean of the additional classification loss L'e for the artificial task TV' and the classification loss LC for the original task T. for query instances (also the support instances in case of meta-learning). As a reminder, the losses LI.8 and LU are also additionally minimized for the query instances. The complete IBP- task interpolation or Interval Bound Interpolation (IBI) training setup is illustrated in Figure 4 in Appendix B Since IBI does not play any part during the testing phase, the testing recipe remains identical to that of vanilla few-shot learning. The detailed pseudocode of MAML+IBI (along with the IBI variant of ProtoNet) can be found in Appendix B
The data X; (i 1,2, .) for tasks Tr can be thought of as i.i.d. obser- vations from a marginal distribution Px defined on a compact subset X' of Rd (d 2 1), paired with corresponding Y; drawn from the marginal distribution lY. The map fox is bestowed with the task of producing lower-dimensional representation of the input X. Let us denote the embedding space by H R* given that d. The spaces X' and H are endowed with lg norm for simplicity and conformity to our convention. One may observe that fos h1 lag o ................................ o hg. where in general hz(z) a (A₁z + b;) given that A, E Rd411 xd, and b Rd == 1,... S.S The function o denotes the activation (such as ReL.U), applied component-wise. Evidently, in our notation di d and ds+1 = k. With this setup, we proceed to undertake the theoretical analysis of our approach. Please find the detailed proof's in Appendix C
Theoretical analysis:
Definition (Perturbation). Given any X1 E X, an s-perturbation corresponding to x1 is the set of points x (s) X such that [|x x21 E, Vx2 E xx){); 0.
For the particular choice of the l2 norm, Definition 1 |chraatterizes e-perturbation as a hollow ball of radius = evd around a given point.
Lemma 1 (Lipschitz networks ensure bounded IBP), Let x and x be E-perturbations of x ~ Px for an E > 0 (i.e. X. x x())). Given that the activation is Lipschitz continuous (such as ReLU) with constant Ca > 0, there exists a constant D = D(c,; ; At, A2, ",,,," ,A;E)) such that Los (x, and J gs (x, E)) will at most be an é-perturbed version of fas (x), where é ē= =D.
The minimization objective function of IBI can be rephrased as L LCE + WwILLB + W2Cun where W1, W2 2 0 are Lagrangian multipliers. The forthcoming result, however, relies on the constrained formulation of the objective, given as min( subject to LLB t1 and Lu S tz. where ti t2 2 0. This is motivated by the fact that the constrained formulation yields solutions upper bounding the ones obtained using its Lagrangian counterpart (Boydd& & Vandenberghe, 2004), Chapter

---

Under review as a conference paper at ICLR 2023
5]. Lemma I implies that the two losses (LUB and LLB) appearing in the constraints can always be made arbitrarily small, depending upon E. As such, in the constrained regime, the remaining problem is to show that the multi-task sample classification loss can indeed be dealt with.
Theorem 1 (Gencralization bound). Let I' be the joint distribution of (fos (fos(X), supported on H x R. In the multi-task regime, let 1 denote the set of tasks, each consisting of N samples. Define R(N, [Z]) E ((xx,Y))~<i()) (1)))tt))CCC(ffos> (CCc((foss(HH) Y3)] and R
BT E(x,,)))T,(CC (aarllCoofffa((feeee (X))). Nj)]. For a bounded loss function LCE R x R -
[0, al 2 0), the neural Setwork-inducced map fai-s is such that /V faa-s (+) < oc, we ensure:
log 2111
log
R(N, [T]) R 12--5 2 S-1 22 21 +2)
+
VN
vI
|
holds with probability at least 1 8, where ā = A(E
5 EXPERIMENTS
The experiments are conducted few-task few-shot image classification datasets, viz. a subset of the minilmageNet dataset called minilmageNet-S (Yao et al. 2022). and Iwo medical images datasets namely DermNet-S (Yao et al., 2022), and ISIC (Codella al. 2018
Experiment protocol:
ei al. 2020). We begin our experiments with a few analyses yses and ablations to better understand the properties of our proposed method. We then empirically demonstrate the effectiveness of our proposed IBI method on the gradient-based meta- -earning method MAML (Finn et al. 2017) as well metrie-learner ProtoNet (Snell et al. 2017) to show that IBI can seamlessly integrated with multiple few-shot learning paradigms. For our experiments, we employ the commonly used "4-CONV" network (Vinyals et al., 2016) as well as the larger ResNet- 12 network (Lee et al.) 2019) to demonstrate the scalability of the proposed method (further details on scalability in Appendix We perform 5-way I-shot and 5-way 5-shot classification on all the above datasets (except ISIC where we 2-way classification problems, similar to Yao et al. (2021), due to the lack of sufficient training classes). Further discussion on the datasets and implementation details of IBI along with the choice of hyperparameters can be found in the Appendix.
as the prototype-base
Ablation studies on task interpolation:
Table 2: Ablation on task interpolation strategies in terms of mean Accuracy and average median distance between original and interpolated tasks over 600 tasks on minilmageNet-S (mIS), ISIC and DermNet-S (DS).
We undertake an abla- tion study highlight the importance of generating artificial tasks using IBP bound- based interpolation by comparing IBI with (1) inter-task interpolation on im-
Algocitim
Accuracy
Average median distance
mlS
ISIC
DS
miS
ISCC
DS
MAML -inter-askk interpolation imuge space
40 10.90%
53.25%
48.30%
N/A
NIA
NIA
MAME AMte--ntr-tdck interpalation after fg
41 41.00%
61.33%
47.43%
3.08
1.23
2.99
DAML-IBB-WC.
41. 41.566%
64.75%
48.90%
NA
NA
MAML+UILRR -Inteeeentn interpolation after fgg
40.35 40.37%
64.91%
48.23%
3,10
0.97
2.83
MAME- IIPPGAA (Imago Space)
41. 839
62.67%
48. 18.83%
NA
NA
NA
MAML-IBP-GA (after fas)
1..66%
63.75%
NA
NA
IMI.T Yan al. 2023
41 58%58%
61.79%
48.03%
3.24
1.36
3.05
ages, (2) inter-task
MAML-IBI withoor
1nd C losses
35.2696
45.04%
41.50%
N/A
N/A
N/A
interpolation in the
MAML- (ouns)
42.20%
68.58%
49.13%
2.74
0.60
2.65
feature embedding learned by fg. (3) Worst-Case Loss (WCL) on the e-neighborhood (Gowal et al.; 2019) along with IBP losses, (4) tase interpolation while minimizing ULBL (Morawiecki al. 2020), (5) Gaussian
The Average median distance caleniated with teanares after the third block for all cases.
perturbation (GA) in the image space with IBP losses, (6) Gaussian noise noise-based perturbation in
the feature embedding space Jas with IBP losses, (7) MLTI Yaoo al. [022), which performs MixUp (Zhang et al.) (017) at randomly chosen layers of the learner, and (8) IBP bound- based interpolation without minimizing the LUB and LLB while only optimizing LcE (more results in Appendix F.3) We perform the ablation study on 5-way 1-shot classification with the "4-CONV" network on minilmageNet-S ISIC, and DermNet-S. From Table 2 we observe that IBI performs best in all cases. Moreover, inter-class interpolation at the same fixed layer S as IBI and at randomly selected task-specific layers in MLTI shows worse performance, demonstrating the superiority of the proposed interval bound-based interpolation mechanism. Further, it is interesting observe that IBI,
7

---

Under review as a conference paper at ICLR 2023
when performed without minimizing the LUB and LLB. performs the worst. This behavior is not unexpected as the neighborhoods are no longer guaranteed to be preserved by the learned embedding in this case, thus potentially resulting in the generation of titt-ff-mnifold artificial tasks.
To further check whether the tasks generated by IBI indeed follow the distribution, we undertake an additional comparison based on the similarity of the artificial tasks with the corresponding original tasks. Concretely, we define the distance between a task and its artificial counterpart as the median of the pairwise distances between the corresponding data instances in the two tasks. If an artificial task is created by combining two tasks, a la MLTI (Yao et al. 2022). we consider the smaller of the two median distances. We observe from Table 2 that the average median distance over 600 tasks is smaller for the proposed method compared to MLTI, as well as inter-task interpolation in the feature embedding learned by fā- This indicates that the tasks generated by IBI are more likely to lie close to the original task distribution.
Importance of dynamic loss weighting: To validate the usefulness of softmax- based dynamic weighting of the three losses for both IBP and IBI, we first find the aver- age weights for each loss in a dynamic weight run and then plug in the respective values as static weights for new runs. All experiments in Table B are conducted on the minilmageNet-S dataset. From the up- per half of T we can see that the three average weights are always distinct with a definite trend in that LCE gets maximum importance followed by LUB while LLB contributes very little the total loss L. This may be due to the particular "4-CONV" architecture used in this study which employs ReLU activations, thus implicitly limiting the spread of the lower bound (Gowal et al. 2019). Further, the average weights of IBP and IBI are similar for particular learner highlighting their commonalities, while they are distinct over different learners stressing their rearner-depender behavior. Further, in the lower half of Table B we explore the effect of using static weights well the transferability of the loss weights across learners. In all cases, the softmax-basse dynamic weighting outperforms static weighting, thus demonstrating the importance of dynamic weighting.
Table 3: Average loss weights for the two proposed methods. and a comparison of the static weighting and dynamic weighting versions, including transferability of static weight values across variants.
MAML-IBP
MAML+BI
Avarge of dynamic loss weights calculated for TBP and TBI
#CB 5
0,5600
0.8658
0.1369
0,1314
0.0029
0.0027
Anmmey of algoritfums with different weight choices.
Dynamic weighting
0.799%
42.20-10.825%
average weights for MAML-IBP Static average weights for MAML+1BI
40.55-+0.8116
N/A
NIA
4..72-10.79%
For evaluating the few-shot classification performance of IBI in few-task situations, we compare against the eggularizationbbasee meta-learning methods TAML (Jamal & Qi 2019). Meta-Reg (Yin et al. 2019). and Meta-Dropon (Lee et al. 2020) for MAML. We also compare against data augmentation-baatt
Results on few-task few-shot classification problems:
methods like MetaMix (aao et al.
202T), Meta- (Ni et al.) 202T), and MLTI (aa et al. 2022 for both MAML and ProtoNet. The results in Table 4 show that in keeping with the observation Table ! IBP without task interpolation can improve upon the corresponding baselines. The incorporation of IBPed task interpolation in
IBI generally improves the results even further. Overall, we observe that both IBP and IBI outperform the other competitors, with the largest gains being observed for the ISIC dataset.
Cross-domain transferability analysis: The minilmageNet-S and DermNet-S datasets both allow 5-way 1-shot classification. Moreover, minilmageNet-SS contains images from natu- ral scenes while DermNet-S consists of med- ical images. Therefore, we undertake cross- domain transferability study in Table 5 We summarize the Accuracy values obtained by source model trained on DermNet-S but tested on minilmageNet-S and vice-versa (denoted DS - mIS and mIS + DS, respectively). We can see that in most cases the IBP variant able to improve upon the corresponding baseline. Fur- ther, the interpolation- based methods, i.e. MLTI
Table 5: Transferability comparison of MAML and ProtoNet, with their MLTI, IBP, and IBI variants in terms of Accuracy over 600 tasks.
Algovihms
Accuracy
MAME
33.40%
MAML+MLII Yao al. 2022
30.00%
36.74%
MAML-IBP (OUnS)
27.06%
33.90%
MAML-IBI (ouns)
30.23%
36.21%
ProtoNet
28. 76%
34,03%
ProtaNet +MLII Yoo
2012
35,4616
PLotaNet+BBP (oua))
24.13%6
ProtoNet-tBl (ours)
30.32%
35.63%
ProtoNet implementtion to per You et
2023
8

---

Under review as a conference paper at ICLR 2023
Table 4: Performance comparison of the proposed method with baselines and contending algorithms in terms of mean Accuracy over 600 tasks.
Backbonc
Algoritim
minilnageeNe-S
DermNet-S
Network
1-shot
5-4hot
1-shot
5-shot
1-shot
S-shot
MAML
Finn al.
2017
Yau et
X02
38.2716
52.14%
57.59%
65.24%
43.47%
60.561
MAML Mel Kox
Yin
bo al. 2022
38.351%
51.74%
58.57%
68.45%
45.01%
60.92%
TAML (Jmmil
2019
of
38.70%
52.75%
58.39%
66.09%
45.73%
61.14%
58.40%
67.32%
MAML-MenaMii
et PUM AA54
39.13%
50.14%
60.34%
69,47%
46.81%
63.52%
MAMI AMenn-Maxtig Maxby poraeii
XMM 100
2022
39.28%
53.00%
58.68%
69.16%
46 10%
62.64%
MAML+METI Yao
41.58%
55.22%
61.79%
70.69%
48.00%
64.55%
4-CONV
MAML+IBP (omrs)
41.30%
$4.569
64.91%
78,75
48, 33%
63.334
MAML-IBI
42.201
55.23%
68.88%
79.75%
49.13%
65.43%
ProtoNet (Snell et 2017
Yio of
2123
36.26%
50.7256
58, 56%
66.25%
44 44.21%
60.33%
ProtoNet Stolt
40.209
53.166
58%
75.25%
46.86%
62.05%%
ProtoNet "MIINEX
29.67%
53.10%
60.59%
70.12%
47.71%
62.68%
ProtoNe! -Meto-Maupp Nu
NI4 Tiki et
2002
30,805%
53.355 35%
59 666
68.97%
46.06%
60.97%
ProtoNet" -MLTI Yoo
41.36%
55.3455
62.82%
71.52%
49.38%
19%1
ProioNes+1BP (oum))
41.46%
55.00%
70.75%
81.01%
48 66%
67.26%
ProtoNen-IRI (ours)
43.30%
55.73%
70.25%
81.16%
51.13%
65.933%
MAML Finn er 2017 Yeo et al.
2122
40.02%
52.56/
59.41%
67.66%%
47.58%
63.1396
MAML
62.06%%
12.1856
51.40%
64.82%
MAML-
ct.l.
53.92%
72.04%
50.82%
64.24%
MAMLAMETT Yao
43.35%
54.89%
02. 16%
73,5099
52.03%
65.1244
MAML+1BP (omrs)
43.50%
55.135
73.9116
50.40%
(5.400%
ResNet-12
MAMI AIRI (ouns)
43.90 %
57,00% %
63. 2.5%
75.66 %
52.10%
66.50%
ProtoNet"
al
40.966
61.91%
72.97%
64.61%
ProioNet (Seel 01
42.601%
55.00%
63.01%
75.91%
50.66%
65.40%
ProtoNat "MebAMi
al
2022
42.95%
56. 95.9%%
65.55%
78.336
S1. 18%
66.80%
ProtoNei" -MetaMaraa
77.62%
50.96/G
ProtoNet" -MLTT Yoo
202
44.08%
57. 14%14%
66.02%
79.15%
52.01%
67.28%
ProtoNetrIBP (ours)
51.33%
67.57%
ProtoNet+IBl (ouns)
45.33%
58.23%
66.75%
$1.83%
52.53%
68.00%
ProtoNet implomentation per
and IBI are able to further improve performance, with IBI achieving the best performance in most cases, thus showing that IBI training can improve cross-domain transferability.
6 CONCLUSION AND FUTURE WORKS
In this paper, we attempt to explore the utility of IBP beyond its originally usage for building and verifying classifiers that are provably robust against adversarial attacks. In summary, we identify the potential of IBP to conserve a neighborhood from the input image space to the learned feature space through the layers of a deep neural network by minimizing the distances of the feature embedding from the two bounds. This can be effective in few-shot classification problems to obtain feature embeddings where task neighborhoods are preserved, thus enabling easy adaptability to unseen tasks. Further, since interpolating between training tasks and their corresponding IBP bounds can yield artificial tasks with a higher chance of lying on the task manifold, we exploit this property of IBP to prevent overfitting to seen tasks in the few-task scenario. The resulting IBI training scheme shown be effective in both the meta-learning and metric-learning paradigms of few-shot learning.
We demonstrate in our results that IBI can be effectively scaled to relatively large networks like ResNet- 12 as we typically only need to apply IBP a few initial layers (see Appendix (E) However, this still adds extra computational cost (see Appendix for a comparative study) which scales linearly with the number of layers subjected to IBP. Therefore, and computational cost, a future direction of research may be to investigate the applicability of more advanced provably robust training methods that yield more efficient and tighter bounds (Lyu et al. 2021). Moreover, few-shot learners can also be improved with adaptive hyperparameters (Baik et al. 2020), feature reconstruction (Lee & Chung 2021). knowledge distillation (Tian et al., 2020). embedding propagation (Rodríguez et al., 2020 etc. Thus, it may be interesting to observe the performance gains from these orthogonal techniques when coupled with IBL. However, this may not be a straightforward endeavor, given the complex dynamic nature of such frameworks.
limit the additional complexity

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We have included the pseudo-codes and PyTorch based Python inplementation for the proposed method in Appendices B and G respectively. The deseription of all datasets used in this study along with other key implementation details is available in Appendices D E The hyperparameter settings for different algorithms along with their tuning strategy are listed Appendix
[ For the theoretical analyses, complete proofs provided in Appendix C A copy of the code is available in the Appendix G while the same can also found at atttpss: anonymous 4open. science/
r/mamm1 1ipam bi-DD72/
REFERENCES
Mohanad Abukmeil, Stefano Ferrari, Angelo Genovese, Vincenzo Piuri, and Fabio Scotti. A survey of unsupervised generative models for exploratory data analysis and representation learning. Acm computing surveys (csur), 54(5):1-40, 2021.
Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. In International Conference on Machine Learning, pp. 232-241. PMLR, 2019.
Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lee. Meta- learning with adaptive hyperparameters. Advances in Neural Information Processing Systems, 33: 20755-20765, 2020.
Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference Learning Representations, 2019.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Johann Brehmer and Kyle Cranmer. Flows for simultancous manifold learning and density estimation. Advances in Neural Information Processing Systems, 33:442-453, 2020.
Anthony L Caterini, Gabriel Loaiza-Ganem, Gcoff Pleiss, and John Cunningham. Rectangular flows for manifold learning. Advances in Neural Information Processing Systems, 34:30228-333 30244,, 2021.
Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp. 168-172. IEEE, 2018
Marissa Connor, Gregory Canal, and Christopher Rozell. Variational autoencoder with learned latent structure. In International Conference on Artificial Intelligence and Statistics, pp. 2359-2367.
PMLR, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
leee, 2009.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1133 PMLR, 2017.
Sven Gowal, Krishnamurthy (Dj) Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. Scalable verified training for provably robust image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hierarchical bayes. In International Conference on Learning Representa- fions, 2018.
10

---

Under review as a conference paper at ICLR 2023
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In Proceedings of the IEEE international conference on computer vision, pp. 5018-3027) 2017.
Muhammad Abdullah Jamal and Guo-Jun Qi. Task agnostic meta-learning for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp.
11719-11727, 2019.
Abhishek Kumar and Ben Poole. On implicit regularization vacs. International Conference on Machine Learning, pp. 5480-5490 PMLR, 2020.
Dong Hoon Lee and Sae- Young Chung. Unsupervised embedding adaptation via early-stage feature reconstruction for few-shot classification. In International Conference on Machine Learning, pp. 6098-6108. PMLR, 2021.
Hac Beom Lee, Taewook Nam, Eunho Yang, and Sung Ju Hwang. Meta dropout: Learning to perturb latent features for generalization. In International Conference on Learning Representations, 2020.
Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings the 26th annual international conference on machine learning, pp. 609-616, 2009.
K Koonooon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10657-10665, 2019.
Xiaomeng Li, Lequan Yu, Yueming Jin, Chi-Wing Fu, Lei Xing, and Pheng-Ann Heng. Difficulty- aware meta-learning for rare disease diagnosis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 357-366. Springer, 2020.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to learn quickly for 2017.
few-shot learning. arXiv preprint
Zhaoyang Lyu, Minghao Guo, Tong Wu, Guodong Xu, Kehuan Zhang, and Dahua Lin. Towards evaluating and training verifiably robust neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4308-4317, June 2021.
Ron Meir and Tong Zhang. Generalization error bounds for bayesian mixture algorithms. Journal of Machine Learning Research, 4(Occ):839-86) 2003.
Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable abstract interpretation for provably robust neural networks. In International Conference on Machine Learning, pp. 3378-3586. PMLR, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
Paweł Morawiecki, Przemyslaw Spurek, Marek Šmieja, and Jacek Tabor. Fast and stable interval bounds propagation for training verifiably robust models. In ESANN 2020 proceedings, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, 2020.
Renkun Ni, Micah Goldblum, Amr Sharaf, Kezhi Kong, and Tom Goldstein. Data augmentation for meta-learning. In International Conference on Machine Learning, pp. $152-8161 PMLR, 2021.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.00990 2018
Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se- Youn Yun. [BOIL]] Towards representation change for few-shot learning. In International Conference on Learning Representations, 2021.
Viraj Prabhu, Anitha Kannan, Murali Ravuri, Manish Chaplain, David Sontag, and Xavier Amatriain. Few-shot learning for dermatological disease diagnosis. In Proceedings of the 4th Machine Learning for Healthcare Conference, volume 106, pp. 532-552, PMLR, 2019.

---

Under review as a conference paper at ICLR 2023
Aditi Raghunathan, Jacob Steinhardt, and Perey Liang. Certified defenses against adversarial examples. arXiv preprint arXiv:1801.0934 2018.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. 2018.
arXiv preprint
Pau Rodríguez, Issam Laradji, Alexandre Drouin, and Alexandre Lacoste. Embedding propagation: Smoother manifold for few-shot classification. In European Conference on Computer Vision, pp. 121-138. Springer, 2020.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290655000:2323-23333 2000.
Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In International Conference on Learning Representations, 2018.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
Teruo Sunaga. Theory an interval algebra and its application to numerical analysis. RAAG memoirs, 2:29 46, 1958.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 29065500)2331-2223, 2000.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In European Conference on Computer Vision, pp. 282- Springer, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9111), 2008.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances neural information processing systems, 29, 2016.
Yaqing Wang, Quanming Yao, James Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
Stephan Wojtowytsch and Weinan E. On the banach spaces associated with multi-layer relu networks: Function representation, approximation theory and gradient descent dynamics. arXiv preprint arXiv:2007.15623, 2020.
Eric Wong, Frank Schmidt, Jan Hendrik Metzen, and J Zico Kolter. Scaling provable adversarial defenses. Advances in Neural Information Processing Systems, 31, 2018.
Huaxiu Yao, Long- Kai Huang. Linjun Zhang, Ying Wei, Li Tian, James Zou, Junzhou Huang, et al. Improving generalization in meta-learning via task augmentation. In International Conference on Machine Learning, pp. 11887-11897 PMLR, 2021.
Huaxiu Yao. Linjun Zhang, and Chelsea Finn. Meta-leamnig with fewer tasks through task interpola- tion. In International Conference on Learning Representations, 2022.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. arXiv preprint 19Xiv:1912..33360 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, arXiv preprint 17000 2017.
12

---

Under review as a conference paper at ICLR 2023
A PROTOTYPE-BASED METRIC-LEARNING
Metric-based few-shot learning aims obtain a feature embedding of the task manifold suitable for non-parametric classification. Prototype-based metric-leariing specifically Prototypical Network (ProtoNet) (Snell al., 2017), assigns query point to the class having the nearest (in terms of Euclidean distance) prototype the learned embedding space, Given the model fe and task Ti, first compute class prototypes {c*)k=I as the mean of (x(,), for the instances x belonging to class k:
1 No E
C₁
(x vt Jep fo(
where Drt C D represents the subset of N, support samples from class Given sample x from the query set, the probability ply,,- = k|x?,) of assigning it to the k-th class is calculated using the distance function d(...) between the representation fa (x(,))) and the prototype
exp(-d([(xx)) .).c)))))
plul, = k0) En e(p(- cw)
(10)
Thereafter, the parameters 0 for the model Se can be trained by minimizing cross-entropy loss (2) In the testing stage, each query sample xy is assigned to the class having the maximal probability, i.e. y), arg max ply),r k) k)x
B ALGORITHSS OF MAML AND PROTONET COUPLED WITH IBP AND IBI
The following Figure illustrates a schematic diagram for the training of IBP and IBI variants.
for
LcE
x T
y
Softmax
LUB
CLB
C
weighting
Figure 4: Interval bound propagation- based few-shot training (best viewed in color): For each query data-label pair (x, y) in given training task T, we start by defining a e-neighborhood - 1e, x le] around x. The bounding box [f g- (x, e), 7 (x, <)] around the embedding fos (x) after the first $ layers of the learner is found using IBP. In addition to the classification loss LCE, we also minimize the losses LLB and LUB which respectively measure the distances of fas (x) to f (x, e) and
Fø. (x,x)) A softmax across the three loss values is used to dynamically calculate the convex weights for the losses, so as to prioritize the minimization of the dominant loss(es) at any given training step. For IBP-based interpolation, artificial tasks T/ are created with instances H' formed by interpolating both the support and query instances with their corresponding lower or upper bounds. The mean of the classification loss LoE for the T, and the corresponding extra loss L'g for T is minimized.
The steps for MAML+IBP/BI and ProtoNet=IBBP/III are respectively presented in Algorithm T Tand 2 Please consult the main paper for various notations and equations used in the algorithms.
Remark 1. The way in which the training support set DD informs the loss calculation on the corresponding query set DD differs between the MAML and ProtoNet variants. While limited number of training steps on the support set is undertaken to obtain the model fa. where the loss is calculated on the query set for MAML, the support set is used to calculate the prototypes (ca ) k= for the loss calculation on the query set for ProtoNet.
13

---

Under review as a conference paper at ICLR 2023
Algorithm IBP/IBI for MAML training
Requires: Task distribution p(T), batch size B, learning rates 70o and 7/, interval coefficient e.
1: Randomly initialize the meta-learner parameters 0.
2: while not converged do
3: Sample batch of tasks from the distribution p(T).
4: For IBI, randomly sample an index 1 S m S B to perform the interpolation.
5: for all E 21, [3 do
6:
Initialize base learner to meta-leamer state.
7: Sample support set D of data-label pairs (. (7,r) from task Tr.
))m= Calculate the classification loss Lce using fo(x),) and yi.- if m then
Y7,
8:
9:
10:
Generate interpolated support and query instances Hi and Hi using (8) Calculate classification loss f'vE using for s sHH and yi Set Lcn = (CCC + l'cg).
11:
13:
end if
14:
Update base learner parameters to -00000
15:
Sample query set D, of data-label pairs {(x] y y from task Te
16:
Calculate the classification loss Loe with fa. (x. and y
17:
Calculate LL and LUB respectively using and if then
18:
19:
Calculate classification loss C'e using fac-s HHY and yi,,-
20:
Set LCe = (fCE + f'cg).
21:
end
22:
Calculate L by accumulating Cce, LLB and LUB using 6
23:
end for
24:
Update meta-learner parameters 0 = 0 m D VoL.
25:
end while
Algorithm 2 IBP/IBI for ProtoNet training
Requires: Task distribution p(T), learning rate 7,, interval coefficient e.
1: Randomly initialize the learner parameters 0.
2: while not converged do
3: For IBI, randomly select if interpolation is to be performed.
4: Sample support set Dj of data-label pairs (E, from task T.
5: Calculate the features fae (x+,r) and find the prototypes {cr KK km using 9).
6: if interpolation be performed then
7:
Generate interpolated support and query instances H and H using 8).
8:
Calculate features fq--s (He') k.r and find prototypes (ck 1K end if
9:
10:
Sample query set D of data-label pairs ((x) (x",,.
1 from task Tg
Calculate the loss LoE using (10) and (2)
12:
Calculate LLB and LUB using (4) and 5
if interpolation be performed then
Calculate classification loss f'g with for s (H).). (ch and yř, by 10) and (2)
14:
Set LCF (CoE L'CF)).
16:
end if
Calculate L by accumulating Leg, LLB and LUB using 6 Update learner parameters / = VoL.
18:
end while
14

---

Under review as a conference paper at ICLR 2023
C DETAILED THEORETICAL ANALYSIS
Interval bound propagation for networks with afline layer: Let us assume a network f with L layers where the 0-th layer denotes the initial input. Let us also consider a layer S L that is not the O-th input layer. The O-th layer of f takes the input along with its perturbed counterparts as shown in Section in the main paper. If at the end of 1-th layer the activation, upper bound, and lower bound are respectively 11-1, R1-1 and - If the 1-th layer performs affine transformation (such as a convolutional, fully connected, batch normalization, etc.) followed by a monotonic activation function (such as ReLU, sigmoid, tanh, etc.) i.c. zi o (Aiz:11 + by), then as per Gowal et al. (2019), we can calculate the interval bounds for the subsequent /-th layer as follows:
2. = a(m v/1,
(11)
2, = 0(4 + 1/2).
(12)
where 1/1 = =AAihh-1 and μrI = A + bj given pI-1 = -+ and 1-
Lemma 1 (Lipschitz networks ensure bounded IBP). Let X and x be s-perturbations of x ~ Px for an e > 0 (i.e. x.X E x(e)). Given that the activation is Lipschitz continuous (such as ReLUU)
with constant Co > 0, there exists a constant D = D(co A1, A2, ... -As;e) such that f as (x, s)
and fos (x, will at most be an é-perturbed version of fes (xx) where é =D.
Proof. Given that X1 E X
[x1)(x1) - hi (x2) = (| + - (
S <c A1 - -x2)||
(13)
< co [oe|Allllxx - X21l
where||AA.l= = supix |=1 I1 A|x|I The inequality 13 is due to the Lipschitz continuity of a. Commonly used activation functions, such as ReLU, tend to satisfy this condition. In particular, for ReLU, Co = 1. As such, the map h2 also turns out to be Lipschitz continuous. A similar argument also proves that h1, continuous with accompanying constant (c, A) where A = max 4114.
2.... S all follow the same trait. As result, fas also becomes Lipschitz
The recurrence relation of extremities in IBP, as suggested by Gowal et al. (2019), can be written as:
[x, = { (Ai + +|Ai))
2 [ (x))) + (Ai (|ri))
2 Lo (xx5) + b
(Ai (IA,) [x, + {Ai + |||1]]
and f (x, = 0
(x))
+bb bz).
2
2
where the |. opperaoor results in a matrix with all elements replaced by their corresponding absolute values, and 1 = 1,2, ..., S. Thus,
N (x,e) E) - for (x0)l
(Ai +|Ai1) sol- (x,,)
(Ai (Ai-IAil) Los (xx,) + +bi
(X) +
0
2
(Ai + +|AiI)
+ (Ai (lAIl)
<cn
Lor (x,,) -Asorr
2
(Ai +|AII) (x,) -ov-((xi) + (Ai (lAl))
- [
=0
2
(Ar + -1Ar))
[1A,l - Ai) (fo- Lor Ix. (xx,6)
<ep
((x,8) Sor-1 S0-1(x))
<co
Ai +|Ad|
(x, 6)- - (x))
Ari - AA
[|(50-1(()) - (x.
2
2
15

---

Under review as a conference paper at ICLR 2023
Observe that, in particular for 1 1
Ai + +Ail]
A1l A
S ca
| x||
+
||x x
A1 |1|
A
A
||
Cge
E1 say,
2
2
iee, the deviation in the first layer can be made arbitrarily small based on 8. The quantity Jo (x) Loi (x, e) can be shown to be upper bounded using similar argument. In other words, both [x, E) and La (x, e) are at most E1-perturbed from fe (x). By the method induction we > (As
eventually get a D D(co:
for which the lemma holds.
Theorem 1 (Generalization bound). Let P be the joint distribution of (fos (X)) supported on H x R. In the multi-task regime, let 1 denote the set of tasks, each consisting of N samples. Define R(N, [T] ((x, (C (H and R (T ((xx,)))),, (aaellCC(ffsssseeeeee (X,)), Yj)]. For a bounded loss function LCE R X R + [0, a] 2 0), the neural nenwork-induced map fur-s is such that fac--(()) < oc, ensure:
PRON,11) R -8 /2ll1/221000 22 +2)
1
1
log( 2
log( 2
+
VN
vIT
holds with probability at least 8, where Ā A(E,A).
Proof. Before beginning with the proof we point out that, based on Definition I given >> 0 and X, any x x(=) can be written as x x ((E) For example, in the simplest case, 77=) can be function in the family #c1. Thus, in case of IBI, the Los (x,, E) and Fas (X,, E) can both be expressed Jos (xx) + v)() with corresponding (ε)). In essence Hद- (1 A)Sos (x,) + A(Sos (x,) + (()). where A E [0, 1]. Now, we can observe that,
for (H;) faa-3 (11 (1- + > [ + n(t)]
= Sau-s (Jos Xn((E)
fau-s (x;)) + AV (xa)(x)) 7(E),
(14)
where 77(6) RR being mentioned in lemma
We obtain 14p by using the Taylor expansion of
fet-s up the first order. Given that fal (-) < co, the second term XV fat-s (fas n())
can be made arbitrarily small. The higher-order terms the expansion all follow suit, which justifies their omission. Now,
LoElfe s (H)),)) JHXR Cce(fol s s(x), (x, 8)
N [Co((oo-s(HH) (H;), Lcn(Soe-s (xxx))))
xf - LCE(JoL-s(x),)) wdlP v)
JHRR
1=1
- LorlJor-s wdPp (15)
(F
N
16

---

Under review as a conference paper at ICLR 2023
Since networks use ReLU activation, the map induced by fat-s can be shown to be continuous. Given is compact the output space also becomes compact. Restricted to such a space, the cross- entropy loss Lox (similarly, regularized cross-entroe coss-entropy loss) turns out to be Lipschitz continuous. Consequently,
[CEE(feL 8 (.)
(fos(xx)), s callfor (H)) fai-s (fes (xx)) A),
(16)
where eg > 0 is the Lipschitz constant associated with LCg. Without loss of generality we can construct the map for such that fo =1 & 1. Now, in case there are tasks involved, namely (TT) /1 [= (i.e., the multi-task regime), the population risk turns out to be
R. = pTT B(,,)
fee (Jo--s (X,s
(T) Lx,,, (EE (for (HH)
We are interested in observing the deviation of the same from the realized risk. In other words,
[R(N,, R| - |R(N, (1) J J
(17)
where J == (TT) B(x,)) KY)T, (eE (Jen-s (H2). Yj) and P is the empirical counterpart of the task distribution. Using Jensen's inequality, (i) can be upper bounded by
[COE (for (HH)).)) Exx,,,, [CE (for s(H,),
BT Eix,
<A + ET B ((x,)) [CE (for s (H))) E(x [OE (for s {H,). x2)
(18)
where we utilize arguments 15 and 16 reach
18).
Using the union bound based on | tasks on
top of Corollary 3.14 of Wojtowytsch & B we can show that the second term in the right-hand
(2020)
side of (18) becomes 3 24-5+1
(og(2x++2)
+ a
2 log( 2
with probability at least 6.
N
N
To put a deterministic upper bound on (ii) let us first define the class of functions
g = 0=f0: gg) = E (x,s())))= [AEE (Jo--s (H), y)] for-s E EWL-s
where W is the function space induced by networks with L S hidden layers 2020 Let us now calculate the Rademacher complexity of the class functions g:
Wojtowytsch &
I
[ Σ
Rad (g, (T)
Eg sup gEG
(T.
Ee sup geg
E.ET.
[CEE (H),s((H))))
I1 LoL (for-s (H),
< Ey, E. sup gEg
(19)
< CT,TE,
sup for
Σ Eifoc s (H)
(20)
SEWL-B
2 log (2x 2)
$ 22226 941
(21)
[1|
where 20 is due to the Lipschitz property pippetit of Lor( y) [Lemma 26.9 of Shalev- Shwartz & Ben-David 2014) Theorem of Meir & Zhang 2003)). We arrive (2T) using lemma 3.13 of Wojtowytsch
17

---

Under review as a conference paper at ICLR 2023
& E 2020). The inequality (19) is based the fact that supuet/ [X)] - Elsup wel [(
given the expectation exists for the class of functions 24 and random variable X.
Thus obtain the deterministic bound on (ii) given by
E (x,, [COE (for (H,)) BTe-p(T)
[FoE (for a (HI))
2 Log 2xx + 2)
log(()
[T]
that holds with probability at least 6. The bounds on (i) and (ii) together prove the theorem.
D DETAILS OF DATASETS USED IN THIS STUDY
minilmageNett: The minilmageNet dataset (Vinyals al. 2016) is commonly used subset of ImageNet (Deng et al. 2009) for evaluating few-shot classifiers. The dataset contains a total 100 classes each containing 600 images of resolution 84 x 84 x 3. Following the directives of Vinyals iet al. (2016) from the total 100 classes 64 are kept in the Training set, 16 are retained for Validation, and the rest of the 20 classes are used for testing.
miered In (Ren et al., 2018) the authors proposed a new larger subset of ImageNet (Deng al. 2009) for addressing the limitations of minilmageNet. In minilmageNNtt it is not ensured that the classes used for training are distinct from those contained the Test set. Evidently, this contains the risk of information leakage and may provide fair evaluation of the few-shot classifier. As a remedy |een et al. (2018) proposed to go higher in the class hierarchy in ImageNet. This enables rieredl use higher-level categories in the Training, Validation, and Test sets maintaining significant diversity between the three. In essence, total of 608 ImageNet leaf-level classes are considered that can categorized into 34 groups. Among these 34 higher-level groups, are used for training, 6 are kept for validation, and the rest are included in the Test set.
minilmageNet-S:: This dataset is created by only using subset of the original minilmageNet Training set for training the few-shot learner in few-task scenario Yao et al. (2022).
Training Classes: a03017168, n07697537, a02108915, =02113712, n02120079, n04509417, n02089867, n03888605, n04258138, n03347037, n02606052, n06794110
Validation and Test sets are kept as same as those used in minilmageNeet.
DermNet-S: DermNet-S Yao al. 2022) is subset of the "Dermnet Skin Disease Atlas" publicly available at wilhtp:/ //www de renett.... com/ The dataset after discarding the duplicates contains more than 22,000 medical images spread across 625 classes of dermatologica diseases. Following preprocessing suggested by Prabhu et al. (2019) the authors of (Yao et al. (022) created DermNet-S by first extracting the 203 classes containing more than 30 images. Then from the long-tailed data distribution of the 203 disease classes, the top 30 larger classes kept for training while the smaller 53 bottom classes are considered for meta-testing. The images are resized to 84 x 84 x 3 to match the resolution of minilmageNet. We follow the same dataset construction strategy in our case. Moreover, we use random classes not included in the Training or Test set as the Validation set. The complete list of classes in the Training and Test are listed follows:
Training Classes: Seborrheic Keratoses Ruff, Herpes Zoster, Atopic Dermatitis Adult Phase, Psoriasis Chronic Plaque, Eczema Hand, Seborrheic Dermatitis, Keratoacanthoma, Lichen Planus, Epidermal Cyst, Eczema Nummular, Tinea (Ringworm) Versicolor, Tinea (Ringworm) Body, Lichen Simplex Chronicus, Scabies, Psoriasis Palms Soles, Malignant Melanoma, Candidiasis large Skin Folds, Pityriasis Rosea, Granuloma Annulare, Erythema Multiforme, Seborrheic Keratosis Irritated, Stasis Dermatitis and Ulcers, Distal Subunqual Onychemycosis, Allergic Contact Dermatitis, Psoriasis, Molluscum Contagiosum, Acne Cystic, Perioral Dermatitis, Vasculitis, Eczema Fingertip
18

---

Under review as a conference paper at ICLR 2023
Testing Classes: Warts, Ichthyosis Sex Linked, Atypical Nevi, Venous Lake, Erythema Nodosum, Granulation Tissue, Basal Cell Carcinoma Face, Acne Closed Comedo, Scleroderma, Crest Syndrome, Ichthyosis Other Forms, Psoriasis Inversus, Kaposi Sarcoma, Trauma, ym Lightt Eruption, Dermagraphism, Lichen Sclerosis Vulva, Pseudomonas, Cutaneous Larva Migrans, Psoriasis Nails, Corns, Lichen Sclerosus Penis, Staphylococcal Folliculitis, Chilblains Perniosis, Psoriasis Erythrodermic, Squamous Cell Carcinoma Ear, Basal Cell Carcinoma Ear, Ichthyosis Dominant, Erythema Infectiosum, Actinic Keratosis Hand, Basal Cell Carcinoma Lid, Amyloidosis, Spiders, Erosio Interdigitalis Blastomycetica, Scarlet Fever, Pompholyx, Melasma, Eczema Trunk Generalized) Metastasis, Warts Cryotherapy, Nevus Spilus, Basal Cell Carcinoma Lip, Enterovirus, Pseudomonas Cellulitis, Benign Familial Chronic Pemphigus, Pressure Urticaria, Halo Nevus, Pityriasis Alba, Pemphigus Foliaceous, Cherry Angioma, Chapped Fissured Feet, Herpes Buttocks, Ridging Beading
ISIC: Following Yao al. (2022) for "ISIC 2018: Skin Lesion Analysis Towards Melanoma Detection" (Codella et al. 2018 Li et al. 2020) we select the third task where 10,015 medical images are categorized into seven classes based on lesion types. We first resize the images to 84 84 to match the minilmageNet resolution. Then among the seven classes in the ISIC dataset, we select the classes containing higher number of samples for training while considering the rest for meta-testing as per the directives of Yao et al. (2022). Since there are only 4 classes in the Training set setting the number of ways to 2 results in six possible class combinations
task. This in consequence offers tuning random classes are used as Validation set
an extreme few-task scenario. For following the cetasssvladation-aaseeee approach employed in (Yao et al. 2022). The list of classes in the Training and the Test sets are listed as follows:
Training Set: Nevus, Melanoma, Benign Keratoses, Basal Cell Carcinoma
Testing Set: Dermatofibroma, Pigmented Bowen's s, Vascular
E IMPLEMENTATION DETAILS
Scheduling of In their paper Gowal et al. (2019) recommended starting with initial perturbation eo 0 and gradually increasing it to the intended perturbation over the training steps. In our case, we follow similar approach for scheduling the value of perturbation e, at the t-th training step. We have observed that a fast increase in perturbation usually slows down training while a very slow increment fails to aid the learner. From extensive experimentation with various scheduling techniques such as linear, cosine, etc., we have found that the following strategy works well in practice. If the maximum allowed number of training steps set to T then for the step f the perturbation 6, calculated as:
if 1 [0.9T]
(22)
0.97
In essence, we linearly increase 6 starting from up c over 90% of the maximum training steps T and keep it fixed at for the remainder of the training.
Frequency of interpolation for IBI variants: Performing IBP bound- based interpolation for every task during training may not be beneficial and may instead mislead the learner. For MAMI., we have seen that performing interpolation once in every batch of B tasks aids the training process. In the case of ProtoNet, we have found that performing IBP bound-basee interpolation with a 25% probability results in the best outcome.
Modifications to network architecture: We have used two networks for our experiments namely "4-CONY" and "ResNet-12" The "4-CONY" network can be seamlessly integrated with IBI for both MAML and ProtoNet. This network consists of 4 blocks, each having convolution, batch normalization, max pooling, and ReLU in sequential order. IBI can be performed after any one of the blocks. The ResNet-12 network also consists of blocks where block (except the first one)
19

---

Under review as a conference paper at ICLR 2023
receives inputs from (1) the output of the preceding block, and (2) the input of the preceding block through a skip connection. While the idea of applying IBI after any of the blocks seems appealing. the presence of skip connections may hinder a straightforward integration of IBI in this case. To understand how ResNet-12 can be customized to accommodate MAML+1IB (and consequently MAML+IBI) we undertake ablation study on the minilmageNet-S dataset in a 5-way 1-shot classification problem as described in Table 6 We can observe that in our initial hyperparameter tuning experiment, MAML+IBP can not match the performance of vanilla MAML on ResNet-12. Moreover, the performance gap increases as IBP is applied deeper into the network. This may be explained by the fact that the interval bounds become gradually loose as they progress through the network. Thus, with increasing depth, the magnitude of the bound losses (especially LU as the ReL.U activations prevent LL from becoming too large) will largely outscale the classification loss and consequently affect convergence (see Remark 2). Applying IBP after only the first block still fails to achieve parity with the baseline because IBP induces a distortion in the feature space, due to its regularization effect. While the sequential part of the blocks after IBP can adapt to this distortion due to their complexity, the simpler skip paths can not do so. Hence, the effect of the distortion keeps propagating to the deeper blocks via skip connections. To aid the network in such a situation, we investigate three approaches to modify the skip connection immediately after the block(s) subjected to IBP, viz. (1) remove the skip connection for the subsequent block (2) introduce additional layers in the skip connection for the subsequent block to make it deeper and more complex (3) use skip after one or more of the initial sub- (looklsk(()) (consisting sequentially of one convolution, one batch normalization, and one ReLU layer) of the next block. Among the three approaches, we empirically found that IAM (consequently MAML++II) performs best when the skip connection starts after the second sub-block in block 2. Due to the comparatively powerful learning strategy of ProtoNet, no such modifications to ResNet-12 12 are necessary for ProtoNet+IBI.
Table 6: Ablation study of ResNet-11 modifications for MAML+IBP on minilmageNet-S in terms of mean Accuracy over 600 tasks with 95% confidence interval.
Algorithm
IBP position
Accuracy
MAMI.
None (Baseline)
40.02++0.78%
MAML+IBP
after block 4
21.24:10.54%
MAML+IBP
after block
23.77=+00.99
MAML+IBP
after block 2
29. 29.62:+:06254
MAML+IBP
after block
37.95++0.83%
MAML+IBP
after block with no-skip at block 2
37.81++0.855
MAML+IBP
after block 1 with deeper skip at block 2
38.54++0.81 %
MAML+IBP
after block 1 with skip and output combination at block 2
40.50.10.833
MAML+IP
after block with skip after I sub-block in block 2
42.18.10.8244
MAML+IBP
after block with skip after 2 sub-blocks in block
43.50-L %
IScalability of IBI] IBP (and consequently IBI) requires the propagation of the two interval bounds along with the input data. This introduces computational overhead, especially in deeper networks. However, in practice even in a deeper network, we may not need to perform IBP except in the initial few layers, as the bound losses will otherwise overwhelm the classification loss and consequently impact convergence. To demonstrate this, we plot the losses (up to 5000 training steps for the ease of visualization) in the following Figure 5S] for MAML+IBI using a ResNet- 12 network for 5-way 1-shot mini ImageeeaeNet-S classification, when IBP is applied up to blocks 1-4. We can see that the three losses have comparable scales only when IBP is applied after block 1. In all other cases, LUB heavily dominates the total loss. But, due to its sheer magnitude, the optimizer is unable to minimize it. Thus, in practice, IBP should only be limited to a few initial layers in deeper networks. Consequently, IBI easily scales to deeper networks despite the computational overhead.
Remark 2.
Remark 3. To show that IBP and IBI variants are well-scalable as their vanilla counterpart we list the actual training costs in the following Table 7 in terms of the average time in seconds to execute single training step of the algorithm. All the experiments are performed in the same environment using a RTX 3090 GPU. From Table 7 we can observe that, in the case of MAML, the IBI and IBP variants only takes about 40%-70% 70% additional time when "4-CONV" is used. The difference in cost reduces further if ResNet- 12 is used as the backbone. This is expected we only need to apply
20

---

Under review as a conference paper at ICLR 2023
LSS wtth upoe troadd
Log W1Th Deee SHAES
TeX
10
1000
20037
3000
4050
1000
2000 2000
6000
1993
2033 2003
1928
2033 3933
Stops
(a)
(b)
(c)
(d)
Figure 5: In the four plots above of losses against training steps the Blue, Green, Red, and Magenta lines respectively denote IBI applied after blocks 1.2.3, and in ResNet-12 without any additional modifications. (a) Plot of LUB in log scale for case of visualization. (b) Plot of LLB. (c) Plot of Ceg. (d) Plot of C in log scale for of visualization.
IBP in the first few layers of ResNet- 12 to gain its full advantage. For ProtoNet, the increment in computational cost for the proposed techniques is observed to be slightly more compared to that of MAML.
Table 7: Actual computational cost in seconds for IBP and IBI variants MAML and ProtoNet with "4-CONV" and ResNet-12 12 backbone.
4-CONV
ResNet-12 12
Algorithm
1-shot
5-shot
I-shot
5-shot
MAML
0.244
0,432
1.408
3.742
MAML + IBP (ours)
0,407
0.615
1.994
4.324
MAMI. + IBT (ours)
0,412
0.616
2.001
4.326
ProtoNet
0.067
0.073
0.075
0.091
ProtoNet + IBP (ours)
0.129
0.144
0.196
0,221
ProtoNet + IBI (ours)
0.133
0.156
0.202
0,233
F HYPERPARAMETERS USED IN IBP AND IBI
F.1 NAMES AND FUNCTIONS HYPERPARAMETER
The following Table g describes the hyperparameters used the vanilla MAMI, MAML+IBP, and MAML+IBI.
The following Table describes the hyperparameters used in the vanilla ProtoNet, ProtoNet+IBI, and III
F.2 HYPERPARAMETER SEARCH SPACE AND TUNING
For hyperparameter tuning, we employ a grid search. In Table 10 we list the search spaces for each of the hyperparameters used MAML+IBP and MAML+IBI. Moreover, in Table [T we also detail the search spaces for each of the hyperparameters used in ProtoNet+IBP and IIL For all other learners used in Tables and 2 the main paper, the results are either taken from corresponding article or reproduced using the originally recommended hyperparameter settings.
In Tables 12 and 13 we report the optimal dataset-specifi hyperparameters for MAML+IBP and MAML+IBI BI. Similarly, Tables 14 and 15 detail the optimal dataset-specific hyperparameter choices for ProtoNet+IBP and PBotoNet+IBI.
For Table 3 in the main paper, the methods using static weights share the same hyperparameter settings with their dynamic weighted counterpart except for 7, which is not used for the static weight For Table 4 in the main paper, all the MAML variants the same settings vanilla MAML. Further, for all the different interpolation strategies Beta distribution is used with the choices of a and B matching those of the MIML+IBI settings.
21

---

Under review as a conference paper at ICLR 2023
Table 8: Descriptions of hyperparameters used in vanilla MAMI., IAML+IBP, MAML+IBI
Hyperarameter name
Hyperparameter description
Hyperparameters used in MAML
Meta-shots
Number of shots the query set in the training phase.
Inner loop iterations
Number of iterations of the inner loop during training on support set. Learning rate for SGD in the inner loop during training support set. Learning rate for ADAM in the meta-leamer update during training. Batch size task during training.
Inner loop learning rate r7o
Meta-step size 7
Meta-batch
Meta-iterations
Number of training steps.
Evaluation iterations
Number of fine-tuning steps on support set during meta-testing.
Additional hyperparameters introduced in MAML+IBP
Interval coefficient e Softmax coofficient y
Perturbation required for IBP.
Controls the relative importance of the three losses used in MAML+IBP during softmax based weighting in training phase.
Layer
layer in the network where IBP losses will be calculated.
Additional hyperparameters introduced in MAML+IBI
and 8
Hyperparameters associated with the Beta distribution required for performing IBP bounds based interpolation.
Table 9: Descriptions of hyperparameters used in vanilla ProtoNet, ProtoNet+IBP, ProtoNet+IBI
Hyperarameter name
Hyperparameter description
Hyperparameters used in MAML
Number of ways in training
Traditional ProtoNet al. (017) usually considers higher num- ber of ways during training.
Meta-shots
Number of shots in the query set in training phase.
Meta-step size
Learning rate for ADAM in the learner update during training.
Meta-iterations T
Number of training steps.
Distance metric
Choice of distance measure, Euclidean or Cosine.
Additional hyperparameters introduced in ProtoNet+IBP
Interval coefficient
Perturbation required for IBP.
Softmax coefficient y
Controls the relative importance of the three losses used in ProtoNet+IBP during softmax- weighting in training phase. A layer in the network where IBP losses will be calculated.
Layer
Additional hyperparameters introduced ProtoNet+IBB
and
Hyperparameters associated with the Beta distribution required for performing IBP mounds-based interpolation.
F.3 FULL RESULTS
Contenders in Motivating Example: For the contenders in Taee settings are as follows:
1. MAML+SN on fas: This variant of MAML applies Spectral Normalization (Miyato et al.. 2018) up to the S-th layer of the "4-CONV" network. Here similar to the MAML+IBP the value of is set to 3.
2. MAML+SN fo: Here Spectral Normalization is applied the full network.
3. MAML+GL: In this variant, we calculate Gaussian regularization loss instead of IBP. Here we send the query set along with its perturbed version and attempt to minimize their norm after the S-th layer alongside Log. The extra loss LGE can be expressed as follows:
N₂
LeL
(xs - (x) c)
22

---

Under review as a conference paper at ICLR 2023
Table 10: Grid search space of hyperparameters used in vanilla MAMI., IAML+IBP, MAML+IBI
Hyperarameter name
Hyperparameter search space
Hyperparameters used in MAML
Meta-shots
Set to 15 following Finn Finn et al 20177)
Inner loop iterations
Set to 5 following Finn al. 2017).
Inner loop learning rate ryo
Set 0.01 following hinn et al (2017
Meta-step 7
Set 0.001 following Finn [20)
Meta-batch B
Set 4 following Finn al. (2017)
Mcta-iterations
Set to 60000 for minimageNet and ticredImageNet following Finn et al.
(2017). Set to 50000 for minilmageNet-S, DermNet-S, and ISIC following
Yao et al. (2022).
Evaluation iterations
Set 10 following Finn et al. (217)).
Additional hyperparameterrs introduced in MAML+IBP
Interval coefficient
Scarched in the set {0.05, 0.1, 0.2).
Softmax coefficient y Layer
Scarched the set {0.01, 1, 10).
For "4-CONV" learner containing 4 blocks of Convolution, Batch nor- malization, Max pooling, and ReLU, is searched at the block level in the set {1, 2, 3, 3.1J. For example, $ = 2 means IBP losses are calculated after the second block. For the "ResNet-12"" network the ablation study in Appendix provides the optimum choice of S.
Additional hyperparameters introduced in MAML+IBI
and 8
Search space contains three pairs of choices (0.1, (0.25, 1). and (0.5, 0.5) where tuple contains the value of and in order.
Table 11: Grid search space of hyperparameters used in vanilla ProtoNet, ProtoNet+IBPP ProtoNet+IBI
Hyperarameter name
Hyperparameter search space
Hyperparameters used in ProtoNet
Number ways in training
Set to for minilmageNet and Nieredlmageeet following (nnll et al (2017),
Set to 5 for minilmageNet S and DermNet-S. S, and 2 for ISIC as the benefit of training using higher ways cannot he leveraged in the few-task scenario
Yao et al. (2022).
Meta-shots
Set 15 following Snell al. (201
Meta-step size v)
Set to 0.001 following Snell et al, (2017
Meta-iterations T'
Set to 20000 for magelme and ueredImageNet following Snell et al.
2017). Our implementation ProtoNet, unlike Yao al. (202). does not require an additional hyperparameter B, analogous MAML, for IBP IBI training. Thus, for minilmageNet-S,, DermNet-S, and ISIC also we set
to 20000.
Distance metric
Set to Euclidean following Snell al. (011)).
Additional hyperparameters introduced in ProtoNet+IBI
Interval coefficient e Softmax coefficient y Layer
Searched in the set {0.05, 0.2}.
Searched in the set {0.01, 10).
For the "4-CONV" learner containing 4 blocks of Convolution, Batch nor- malization, Max pooling, and ReLU", is scarched the block level in the set (1, 3, 4]. For example, S = 2 means IBP losses are calculated after the second block. For the "ResNct-12" network the ablation study in Appendix (npenddxx/providdde provides the optimum choice of S.
Additional hyperparameters introduced in ProtoNet+IBI a and
Search space contains three pairs of choices (0.1, 1). (0.25, 1). and (0.5, 0.5) where a tuple contains the value of and 3 in order.
23

---

Under review as a conference paper at ICLR 2023
Table 12: Optimal hyperparamter setting for MAML+IBP, MAML+IBI in 1-shot settings when "4-CONV" network is used.
Hyperarameter
Hyperparameter settings for datasets
minilmageNet
tieredImageNet
minilmageNet-S
DermNet-S
ISIC
Additional hyperparameters introduced MAMI
0.1
0.05
0.2
0.05
0.1
0.1
0.1
0.1
3
3
3
3
Additional hyperparameters introduced in MAML+IBI
and |
(0.25, 1)
(0.25, 1)
(0.5, 0.5)
(0.5, 0.5)
(0.25, 1)
Table 13: Optimal hyperparamter setting for MAML+IBP, MAML+IBI 5-shot settings when "4-CNN" network is used.
Hyperarameter
Hyperparameter settings for datasets
minilmageNet
tieredlmageNet minilmageNNet-S
DermNet-S
ISIC
Additional hyperparameters introduced in MAML+IBP
0.1
0.05
0.2
0.05
0.1
0.1
0.1
0.1
0.1
S
3
3
3
3
Additional hyperparameters introduced in MAML+IBI
a and |
(0.1, I)
(0.1, I)
(0.5, 0.5)
(0.5, 0.5)
(0.25, 1)
Table 14: Optimal hyperparamter setting for ProtoNet+IBP, ProtoNet+IBI in 1-shot settings when *4 44CONV" network is used.
Hyperaaammetrr
Hyperparameterr settings for datasets
minilmageNet
tieredImageNet
minilmageNet-S
DermNet-S
ISIC
Additional hyperparameters introduced in ProtoNet+IBP
0.05
0.05
0.1
0.1
0.05
3
1
Additional hyperparameters introduced in ProtoNet+IBI
a and 8
(0.1. 1)
(0.25. 1)
(0.5.
(0.25, 1)
(0.1. 1)
where Ç (0, or), and the standard deviation is scheduled similar with starting from 0 and slowly increasing to e/2.
4. MAML+ Following Morawiecki et al. (2020) we replace the two bound losses with single one that calculates the distance between the upper and lower interval bounds. The loss LULBL in this case can written as:
N,
CULBL =
Σ
11 (x),r; - (x", 12
Ng
The full version of Table D is provided in the following Table
Comparison of IBP with other few-shot learners: As contending meta-learning algorithms, choose the vanilla MAML along with notable meta-leamers such as Meta-SGD Li et al. (2017). Reptile Nichol al. 2018). LLAMA Grant et al. (018)) R2-D2 Bertinetto al. (009)) and BOIL.
24

---

Under review as a conference paper at ICLR 2023
Table 15: Optimal hyperparamter setting for ProtoNet+IBP, PBotoNet+ in 5-shot settings when "4-CONV" network is used.
Hyperarameter
Hyperparameter settings for datasets
minilmageNet
tieredImageNet minilmageNet-S
DermNet-S
ISIC
Additional hyperparameters introduced ProtoNet+IBP
0.05
0.05
0.1
0.05
1
i
1
Additional hyperparameters introduced in ProtoNet+IBI
and |
(0.1, I)
(0.1, I)
(0.5, 0.5)
(0.5, 0.5)
(0.25, 1)
Table 16: Optimal hyperparameter settings for MAML+IBP/IBI and ProtoNet+1BP/IBB when "ResNet- 12" used as the network.
Datasets
Learner
Shots
Parameter
minilmageNet-S
DermNet-S
ISIC
MAML+IBP
and and 5
0.1
0.1
0.1
MAML+IBP
0.1
0.1
0.1
MAML+IBP
and 5
S*
1
1
1
MAML+IBI (Additional)
and 5
a and
(0.1, 1)
(0.1, 1)
(0.1, 1)
ProtoNet+IBP
and
0.05
0.05
0.05
ProtoNet+IBP
and 5
0.1
0.1
0.1
ProtoNet+IBP
1 and 5
S*
1
1
1
ProtoNct+IBI (Additional)
and 5 a and
(0.1, 1)
(0.1, 1)
(0.1, 1)
:: Set as per Appendix E with necessary modifications.
Table 17: Effect of IBP on MAMI. for minilmageNet and mageed datasets in terms 5-way --shot Accuracy and intra-task compactness. This is the full version of Table
Accuracy
1-NN distance
Algorithm
minilmageNet
ticredImageNet
minilmageNet
ticredImageNet
MAML (Finn et al.
2017)
48.70:11.55%
51.67
0.97110.02
0.98:10.02
MAML+SN on Jas
44.90=t1.12%
45.26=11005%
1.38:00.04
1.41
MAML+SN fe
42.83-10.94%
43.06=:0.66%
1.52=0004
1.53=+0.04
MAML+GL
48.70= 0.97%
51.90=00.8%%
0,96:10.02
0.98+0.02
MUML+ULBL
49.43+t0.90%
51.67 %
0.94+00.22
0.97-10.02
MAML+IBP (ours)
50.76.+0.83%
5..86%00.8%
0.90:10.02
0.96:10.02
(2021) Moreover, considering the regularizing effect of IBP and IBI, also include meta-learners such as TAML Jamal & Qi (2019), Meta-Reg Yin et al. 2019). and Meta-Dropout Lee et al. (2020) which employ explicit regularization. We further include data augmentation-relian learners such as MetaMix Yao et al. (202T), Meta-Maxup Ni et al. (20TT) as well as the inter- task interpolation method MLTI Yao al. (022). case of metric-leamenss we compare against the vanilla ProtoNet in addition to other notable methods like MatchingNet Vinyals et al. (2016). RelationNet(Sung et al. (2018), IMP Allen al. (2019), and GNN GNN Satorras & Estrach Estrach (2018). also compare against ProtoNet coupled with data augmentation methods such as MetaMix, Meta-Maxup and MLTI, as done in Yao et al. (2022). While Yao et al. (2022) had to modify the training strategy of the canonical ProtoNet to accommodate the changes introduced by MetaMix. Meta-Maxup, and MLTI, the flexibility IBP and IBI imposes no such requirements. We summarize the findings in Table 18 We can observe that either IBP IBI or both achieve better Accuracy than the competitors in all cases. The slightly better performance of IBP with ProtoNet seems to imply that IBP-based task interpolation is often unnecessary for ProtoNet when large number of tasks is available.
Oh et al.
25

---

Under review as a conference paper at ICLR 2023
Table 18: Performance comparison of the two proposed methods with baselines and competing algorithms on minilmageNet and tieredImageNet datasets. The results are reported in terms of mean Accuracy over 600 tasks with 95% confidence interval.
Datasel
Learner type
Algorilim
I-shat
5-shol
MAML (Fion
2017
48.70=11.5%
63.11=00911%
Meta-SGD (L
2017
50.47=
64.03=00.44%
Reptile
Niciiol e
2018
49.97:1:03288
65.99:00588%
LLAMA Urail
2015
49.40-10.84%
R2-D2 Bertnnllio nl.
21719)
49.50.1:0.00%
65.40:10.20%%
TAMI, (Jamal 2019 Yao
2022
46.40:1:0.875
63. 63.26%10.68%
BOIL et
20211
49.61 16%
66.45:t0.37%
Meta-learners
MAML+Mea- Keg Yin
2019
Yao
2022
47.0221-0.7%%
63. 63.19:10.69%%
MAML-Mesa-Drop Los
Yao
2022
47.47=0.81%
64. 117711
MAML+MeiaMii ( 2041
47.81=:0.7885
64.22:10.68%
MAML+Meia Maxup [Nr
Yad
2022
47.68.10.7999
63.511...758
MAMI.+MI.TI Yao ml 2022
48.62:1:0.766
64.65.10.70%
minilmageNes
MAML+IBP (ours)
50.76:
67. 13:20.8119
MAMI.+IRI (ours)
52.16:1:0.845
67.56:10.86
MatchingNeo Vinyals et
2016
04.44-0777%
53.31=-0.35%
RelationNet (Sung et
50.44:1-0882%
05.52=00.70%
IMP Allen all [019)
49 49.60.1:0.800
68. 0.810
GNN
Estrach
2018
49.02:1.0.88%
63.50010..849
ProtoNet (Snell (anell et al 2017
49.42:1:0.78%
68.20110.666%
Metric learners
ProtoNet +MeraMix Yao
2027
ProtoNet +Meta-Maxinn (N of al
AX1 Yao
2022
47.33-t:0.794
04.43=#0.69%
ProtoNet" +MLTI Yao ot
48.11:+:081%%
65 65.22::0.70%
ProtoNet-1BP (ours)
50.48:10.83%%
68.33:10.79%
ProtoNet-IBI (ours)
51.79:t0.81
6..46%0..9%%
MAML (Finn al.
51.67411.811
70. 0..00%
Mcta-SGD (L
48.97=00.21%
0.21%
Reptile Nichol el 2018
49.97:1:0.22%
65.99%30
Meta learners
BOIL (Oh
49.35.10.26%
19.37.10.123
tirredlmageNet
MAML+IBP (ours)
54.36:10.80%%
0.77%:
MAML+IBI (ours)
54.16.10.79%
71.00:1.0.444
MatchingNet [Vinyals et RelationNer (Sung ct
2016
54.02:1:0.79%
70. 0..%%
54.48:10..33%
71.82
Metric-leammers
ProtoNet (Smll
53.31::0.20%
72.69:20.745
ProtoNet-IBP (ours)
81%
0.826
ProtoNet-1BI (ours)
55.16:10.77%
74.96:20.82%
ProtoNet implomentation per Yao 2022
Notes on contenders used in Table 2 The extra parameter settings required for the contenders in Table are as follows:
1. MAML+WCL. Here given a task its worst-case loss in the e-neighborhood (Gowal al.) 2019) is added with the original loss. In essence, this acts similar to augmentation with the worst-case logits. We tune the relative contribution of the original task and the worst-case task to the final Log following the recommendations made by (Gowal et al. 2019).
2. MAML+GA (image space): Here the original task is perturbed with Gaussian noise to form the augmented task in the image space. The noise sampled from Gaussian with mean and standard deviation a = c/2. The value of is scheduled similarly to c.
3. MAML+GA (at fos feature space): Here the embedding of the original task after fos is perturbed with Gaussian noise. Similar to the image space, the mean of the normal distribution used for sampling noise can be set to O. However, finding a good a may not be straightforward the for feature space is continuously updating. In our implementation, we take a as half of the median distance between the original task and its bounds over MAML+IBI run.
The full version of Table 2 is detailed in Table 19 Moreover, the full version Table in the main paper is
The full version of Table 4 in the main paper is
provided here across Tables 20 and 21 presented in Table 22
26

---

Under review as a conference paper at ICLR 2023
Table 19: Full version of T performance comparison of MAML+IBI against 11 augmentation strategies, the 5-way 1-shot setting. The results are reported in terms of mean Accuracy over 600 tasks along with the 95% confidence intervals.
Algorithm
mlS
ISIC
DS
MAML+ interpolation in image space
40.90=0.86%%
55.25-t1.5888
48.30-+0.81
MAML+Inter interpolation after fgs
41.00=:0.83%
61.33:1.52%%
47.43
MAML+WCL
41.56=0.88%
66.83+11644%
48.20:10.81%
MAML+ULBL+WCL
41.27=0.84%
63.5011.48%
48.43310.80%
MACL+IBP+WCL
41.56=10.85%
64.75+1..61%
48.90-10.83%
MAML+UUBLL+nnrraaee Interpolation
40.37=0.80%
64.91+1.45%
48. 48.23++0.77%
MAML+GA (Image Space)
41.33=0.85%
63.25+1.68%%
47.67++0.66%
MAML+IBP+GA (Image Space)
41.83=0.82%
62.67+1.59%
48 08.83%0..82%
MAML+IBP+GA (after (2))
41.66.10.84%
6..75
47.60.10.82%
MAML+MLTI] Yao et al. 2022
41.58:10.72%
61.79
48.03-10.80%
MAML+IBI without Lus and LLB losses
35.26=10.79%
48.94.1..366
41.30-10.8198
MAML+IBI (Ours)
42.20-10.822
08.58:10.934
49.13
Table 20: Full results for MAML variants minilmageNNt-S, DermNet-S, and ISIC in Table of the main paper. All results are reported in terms of Accuracy over 600 tasks along with 95% confidence level.
Algorithm
4-CONV
ResNet-12
5-way I-shot
S-way 5-shot
5-way I-shot
5-way 5-shot
minilmageNet-
MAML Finn et al.
2017
38.27
52.14:1:0.55%
40.02=20.78%
52.56++0.85%
MAML
MAMM-MMla
Meta Ker
2019
Yau
2032
38.75%
51.74:10.68%%
TAMI. Jamal & Qi AT19 Tao al
A124
38.70:10.77%
52.75
MAML-MMeaaDDupuu Drupoul
52.53-10.6995
MAML-MetaMix Yaa et 202
39.43-10.775
54.14.10.73%
0..26-10.756
54.65 80..77
MAML+Mera-Maxipp Nr al
Yao
39.28-40.7764
53.02:1-0722
41.97-40.788
53,92-40
MAML+MLTI Yao et
2012
41.58=0.725
55.22=00.76%
43.35=00900%
54.89+00888%
1AML+1BP (ours)
41.30-20.794:
54. 54.36.10.811 8
43.50=10.864
55.13:10.909
MAML+IBI (ours)
42.20-10.82%
$5.23:10.81% %
43.90:10.90%
57.00=0.888%
ISIC
MAML Finn al
2017
17.591
68.24.10.77%
59.41 11.8%
67.66.11.929
MAML Miera-K K3y
ein al. 2019
Yao
2022
58.97.0..44%
68.45:1081%
2019
MAMI
Mela- Impoii Le
MI Van
2022
58.40-4 1.02%
67.32-10.29%
MAML+MeraMix Yao al
2021
60.34-
60.47--0.00
62.06
72.18 1.75%
MeML+Meta-Maall (N
2021
Yau
2022
58. $8.68=20.886
69.16=10.61%
61 61.4411811
12.044::1.999
MAML+MITI Yao el
2022
61.79 1.00%
70.69 0..688
62.16 1.88%
73.56 1.82%
MAML+IBP (ours)
64.91
78. 78.75:1:0.944
64.50:1148%
73.91 1001::142200 42%
(ours)
68.58:1
79.75.10.91%
63.25:1
DermNet-S
MAML
Finn et al
2017
43.47.00.3%%
60.56::0.744
47.58-10.93%
63. 83:
MAML
2019
45.01 20.83%%
60.92:10.699
TAMI.
Jamal & X01 Yáo or
2024
45.73-00.84%
61.14:1.072%%
MAML-AMita-Droooul (Log
00 Yao
2022
44.30=0.8845 0.84%
60.86:2:07339
MAML+MetaMix Yao
46.81 0.811
63.52:1:0.39%
01.90
64.82-10.87%
MAML+Meta Maxup (Ni
alal.
Yan el
16.10:11 0.82%
62.64:10.72%
50.82.100 0.85%
64.24 64.24 10886%
MAML+MI.TI (Yao
48.03 79%
64.55:10:748
52.03-10.9004
65.12: 0.88%
(ourx)
63.33.10.84'%
65.40.10.899
MAML.1III (ours)
49.13-1-.500%
65.43-10.799%
52. 10-80.87%%
66.50=0.9299
27

---

Under review as a conference paper at ICLR 2023
Table 21: Full results for ProtoNet variants minilmageNNe--S, ISIC, and DermNet-S in Table of the main paper. All results are reported in terms of Accuracy over 600 tasks along with 95% confidence level.
Algorithm
+-OONV
ResNet-12
5-way
5-way
5-way
5-way
minillaageNot-S
ProtoNet Snell et
2017
Yao et al
2022
36.26:: 70%
50.72.10, 70%
ProtoNet Snall
40.70=+0.7%%
53.16=0.77%
40.96+0075%%
55 05.00=0.866%
ProtoNet +MclaMix Yau al.
202
2022
39.67:1:0.11%
03.10:00.74%
42.95::0.87%
56, 06.95=0088%%
ProtoNet"
42.68.10.7888
ProtoNet" +MI.TI (Ya et al.
41.76::0.75%
55.34-10.74%
44.08:10.839
57. 1.990.
ProtoNet+IBP (ours)
41.46.10.791%
55.00
43.33.1...22
1..90%
ProtoNet+1BI (ours)
43.30:10.81
0.803
45.33-10.85
08.23%
ISIC
ProtoNet" Snell al.
2017
Yao
2022
18.56=#1.01%
66.25 0.996
ProtoNer Snall et
2017
65.58:20.919
75.25 90%
61.91 1.9%
75.91 = 92%
ProtoNet
Yt
2022
60.58+11.7%
70.12=00.44%
6
78.33=1110
ProtoNet +Mela-Mukup Ni
2021
Tab al. 2022
59.66.1 1.6.::11.134
68.97 0.83%
64.17:1:1885
77.62:1 1.86%
ProtoNet" (Ya
71.52.1 0.89%
ProtoNet+1BP (ours)
0..5%%
81.01 0.93%
66.66:1:1.522
81.00::110 81.00:11.49% 1.49%
ProtoNet-IBI (ours)
70.25.10.91%
$1.16.10.9499
66.75.1 1.63%
81.83 1.58%
DemnNet-8
ProtoNet" et cl.
2017
Yao et al.
2022
44.21=:075%%
60.33=0.70%%
ProtoNet Sacll et
06.86:10..77%
62.03 0.793
48.65510.8851
65.40:10.81%
ProtoNet #MelaMEx Yab
2029
47.71.10.834
62.68 0.7.1.
51.18
66.80.10.831
ProtoNet"
06.06
50.96:10088%%
0.85%
ProtoNei +MLTI (Ya et al
A122
49.38.10.85%
65.19 0.7.3%%
52.01 1.0.93%
67.28 0.87%
ProtoNet+IBP (ours)
48.006:20.81%
6..26:10.84%
51.33::-0,9118
67. 67.57:20.885
ProtoNet+IBI (ours)
0..13
65.93 0.82%
$2.53 0,944
68.00.1 0.888
ProtoNet implementation per Yao et
2022
Table 22: Full result for dabeeinjssn transferability comparison of MAML and ProtoNet, with their MLTI, IBP and IBI variants. All results are reported terms of Accuracy over 600 tasks along with the 95% confidence intervals. Here, A - B indicates the model trained dataset A is tested on dataset B.
Algorithms
Accuracy
DermNet-S
minilmugeNet-
minilmageNets
MAML
25.06.10.799
33.40:10.775
MAML+MLII MAML-IBP (ours) (ours)
30.03+1..88%
36.74:10.649
27.06:10.78%
33. 33.90-1-.881
36.21:10.844
ProtoNet ProtoNet +ML.II
28.76:10.82%
34.03-10.80'$
30,06 10.56%
35.46-10.6399
ProtoNet-IBBP (ours)
29 10.
34.13: 82%
ProtoNet+IBl (ours)
30.32-00.8%%
35.63:20.8339
ProtoNet implementation as per
Yao et al.
2022
28

---

Under review as a conference paper at ICLR 2023
G CODES
The following code can be used to run MAML, MAML+IBP, and MAML+IBI Please put the data in folder with the dataset name. The data should be in ".jpg" image format typically of resolution 84 x 84. The data expected to be arranged in train, validation, and test folders each containing folders for individual classes. The main code is kept inside the "sre" folder while the helper code can reside outside. The "main" function resides in the "eun.leamer.pp"..
Helper code: image data process.py
import
import random
from PIL. import Image
import numgy up import
def read, dataset dita_dir, Trs.preseecc-TTree:
image datasel.
if val presence is Truc:
else replet..eed..llasees path. join (data_dir, for U"raia", "vall, "rest"))
return tuple( read classes path, path. join dir, x)) for in Dirain", "rear"s "reat"))
classes (dir. path):
Road the class directories is a valtet directory.
8 Images should be "jpp" format.
return (ImgeProcesChassos path path.juin juin (dit.path, ()) for us. (istdir(dir.paat)
class ImagePracosaClas::
Loading and using the image dataset.
To use these APIs, you should prepare a directory that contains three mbbdirrctooiis: train, and val.
..init_. (xelf, dir.path):: self. dir. = dir-path xelf. Jeache (}
def sample(self. num_imagos):
Sample (as pytorch tenzor) from the class.
names [f
as listdir (self, (salt.dir.att dir.path) enndwwith jips ips"))
randum. shuffle (names)
names Image
images append ( self rread.im .image (name))
return images
def read..
name):
For reading images and trausformations necessary.
Image resolution
81284,
if name in self..cache:
relurn self ceache [nmee]
with opentos. join
dir.path
"eb") in-file:
img Image. open iflle resize 84)), 'RGD")
img up, array (imp), astype "flaa322" " Oxff
("feea("
img sap. rollaxis (img. 2. 0)
self sache [name]] turch. (ensor(ing) return read..-raaddmmaeee (uame)
Main code: run learner.py
import random
Import
import sys
import nimpy
import
import argparse
from img_date sateddaa_ppaaeee pracess Import read, dataset
datetime import datetime
from copy import despeopy
Trom models import Medea mo.evveddddll import bulk_evaluate import
from are learners import Learner
ergument_parser
Get an argament parser for training script.
29

---

Under review as a conference paper at ICLR 2023
parser argparse ( ( (foraattec.laasseergaar.eee hll.ss/ggass.gggggggtttllittttoo
aarser..dd.argumenn --dataaet help- nmme_of=dataset defaul)=Ness)
parser.add. add.argument "-algorichm' help= nume nfme-of-alggriibee default=None)
help- "andommmeedd" defaaltw0, type=int)
parser.
add.arumemmm.. '-order". aepp-coter.-00-MMMMaaa----aaa
defauli-None, type-int)
aatser.add.argumm. ("cumennn--ccasees help= mumbbr..f..lasse..pr......
Jefault=None.. typesint)
parser. sisser.ddd.argmment "-sshots" help- mumeer....cammiesse.......
default=Nene. type=int)
argument "-meta-shots" help='shots.fer.aeuu default-Nose type-int)
defsult=None. type-int)
parser..dd.aad addserredddargumennt -inaer-iters "
addser.add.argumenn --eerrian---aaaa rate heepp="inetrooop.eeaiii_____.. default=None, type-float)
default-None. type-fleat)
parser.ade add argument! -meta-stp
addter.eed.rrgmmentt mmeta-batch hetp--/ebb-taaiiing--bosss.e
default-None. type-int)
aasser.add.argumeent -meta-iters aage-mmesettsaainn-leeettttsss default=None. type-int)
parser. argument! addeereend
-eval-iters" cetpessviaaiioo.nnrriieeeioo
defaultzNone. type-int)
parser.add... add.arredd.armmmnn evall hetp::/eaalnttooo-aapppss default-Nous. defauttt-Me type-int)
-val-intezvaal "ttpp="vvaaattoo.iittrvlldde
None type-int)
sedier..dd-arggamoe -taaaaaattaiiiteell----pppp /arps duriples.ddiii---ii during--traininn defanlt-None, type-iat)
parser. -ibp-pps'......p help--1Bf..eegalbrroodd....
default-Ol. type-float)
parser. add argument! - -sottmax-temm help= tttppsssottaaxiemmerruuuuu defauls=None, type-fleat)
parser. " http:iiooooo........oo...oeeea
parser. add....ucnd...mm chheckpoiis.... hetpe-//oodaseedcccchsspssosooo setion=
default-False)
default=Noss)
parser.add argumenti -estt-iirrs help- mamber owt.sr.......l...... evalluaatinnns defauls=None, type-int)
parser. add.argament: beetaaa-""
default-None. type-fleat)
parser. add -ebta--
default=Nose, type-float)
parser.ade argumenti -mixup' help='ses se... mixpp. task
store trae
parser, add.argumentt"" libp-- help- namber-layer....
oetee -t....oo...r...../////. default-Nose, type-int)
return parser
model.kwargs kwarga parsed-args):
# Porameterz used for initialiging the fearner.
return {
args.leerring "meta_atep step_itte_ parsed_args step
update Ir": parsed args learning rate
'beta parsed args,
h": parsed args,
"softmas temp': parsed_args. temp
# Paramsters used for traiziag.
{
"order:: parsed_orgs order
"num "num_classes": parsed.args classes.
"nam_shats"" parsed_args.
shotoshots'': pags mhta_thots. "inner_iters" parsed. args, innersiters... "mera,
b bate_batc__ieee parsed_args meta,batch.
'meta..ters" prgs meta.iters.
'eval_inner.iters "s parsed.args args eval iters.
cvat_interval"" parsed_args intervnterval
eeval_inierval
parsed_args, wtt...eeal.innerrl...om..
"ippeepsioon' parsed.args ibp_eps.
"mixup" parged_arge mixup. "lyers' '; args. wtw..r..p.pp.err..... ibp_layers
def evaluate kwargs (parsed.args):
Parameters used evaluation multiple tarks.
return {
'num num-classes': war.darrgs....s....... "nam.shots"" parsed.args. shots.
eeval_inner_iee pagsed,args aval_iters.
numm.samples" pagsed.args. saal.
argument.parser (). (arse.args
torch. menual seed)
random, seed seed args.seed) seed)
random. (aegs seed)
according DATA.DIR "Thome/CuseNamme Dtr.euueeeeeeeeceeooiiineretoooe >/data? args. datasel
Create directory
results and ixiriate logging.
path eus www.paah..jinni.. (DATA.AA.. "val')):
val.presence True
print(" Validation..se.... aalidtion.eet.ib.pceemm.
clse:
vallpreeence print itt.iation..e...s........ Sxiting.")
exit()
datetime ("eddd S
30

---

Under review as a conference paper at ICLR 2023
ousput-folder args. args.dataset args. algorithm + "-ostput.folder." time.atrin
output.file output.folder + '/" "log-" time-string + .Ix1
us.att.existtttoo existstouupu..flder ma.kedii (outptt.foider)
with openooutpt..iile "at") as fp:
print( (f for vars(args). items (). file=ip)
device norch.deviee ("cuda")
Instuntiate the datazet. train.set. val.set. test.set (BA., val-presence)
* Iastuctiate the Iearner model-NetworMModel clasee)
learner Learser model, device. r=model.kwargs(arg.. (arge)
training evaluation per nerd.
args eviy.evatttooo False:
train trainnee train set, val set. output file. output folder. setrain kaargs
clse:
args checkpoint
checkpoin
print( Evaluating....
res file output,foldes 'P" "eest_pefformance time streestring "..txt"
with open(res file "a+")
print( Evalulation maalalation.cccckoonnt...
args. chg.cchccpooist file-fp)
checkpoint model torch laad chac(args.checkkoinnt
Iocation-'cude "cniiinn""cnd011111
learner lead,state_dictt learner. meta optim. (eceeeeeeekooiiaa (checkpoiat, stade
dict modell optim state
train aacuuacccuray. val_accuracy. test.necuracy UI
U. UI.
for in rangelargs. iters
train_div evalaeval
val,ace, dal bulk, evaluate (learner. val.ser, revaluuaee (argss test aec, test div = balk evaluate (laaantee rest set, evevalnate (warga(arge)
arairacce appendi val accuracy append aee) test_aceurae,
trais_enf. eppped (train_div) val aalpend val.div) tost.enf appendi test.div)
open(res.file... 'st')
print ('Tst. "Festt.iee ietrttiin io" str(ii 1), filezfp)
print( (Trrain_accureey. --- str(train_accune accurney [-1]] "_+--"" parctrain_enf[1]] file-fp)
print( (VValidation aaccuracy
str(val. [-[]] ".t-u"" str strival.en([-]]) filezfp)
print( (aant"TTst aceart- : str(leet [-I]] "Wt-""" str(test,ef[[-]] "'"' filezfp)
save.path = ooldet_folder 'y' "resalts" '.ppx" train.accuracy
(crain_
val.acouracy up. array avaluccuracy
test.accuracy- array (cest_accuracy)
irain.enf up array (rrin.een)) val.enf array (val.enf). test,enf up.
pp.eaaez tran..ceurreeyttrin..aaaa val.acouracyeval.eee res.aacoracyyiees.eeoo traih,coaiideseettniinn Val.ooffidescesvl...n
test.confidensesttst.ccf]
Model: "src/models.py"
Import torch
import torch. as
import functional
import numpy up
# regular 4-COW artwork Network
def (self. k.way):
Feitialier network layers.
super super Modelr self). init (
self conel Conv2d(3, kernel.iizest stride=l, padding=(), 10)
batchb
batcht BatchNorm2a644, track ransing stats-False)
31

---

Under review as a conference paper at ICLR 2023
ceif.conv2 na. Conr2d(b,, 64, kerael.size=3,, stride=l. padding=(1, 11)
mn Buchhorr221644 backkrunning.siats-le
conv3 aa. Conv2d(64, kerael-size=3,, stridesl. padding=(1,
self. batch3 mn Bucchoorm2d664
self. convd Co.22262,,, 64, kernel size=3, stride=l. padding=(1,
batch4 BatchNorm22
self. linl on Linear( k.way]
forward(self.
Jorward function only for reference.
(F. max (sel
(ellleeeee.....e000 2)]
F.relu mox-pool2d self. batch2 (elh2 (elff.cov2(())) 2)]
F.relu max peel selff batch3(
(off.conv3())) 2)1
max-poal2d batch4 co)x4(x)).
x-e. 6445-5)
self (X)
return
def functional feactioaal.forward tself
weight diet
layer.index-Nonc
mixap flag=None.
way-None.
beta_a-None.
functional farnard that will actually be used for all requiremenis. only [unctionalz explicitly weights be passer. juactionals can use regular layer function
furm required.
robust Trae
y. robust None. Filse
Block
[orward
weight diet fietleenee l'eaevl weight"], weight dier boavt, stride=l, paddisg=(I, D), rohusterobuxt)
rohust_ norm (
weight diet batchh, weight"], diet [batcht bias 'J, robusterobust)
F.msx,pool2d(s, kizeel_ stride=2) (x)elu(
if layer_inda
y. x.
mxx mixup-flag, kwaa, beta.a, bela.b)
Rlock
robust.cony [orward( (
weight,diet ['eon22 weight'], weightddee( (oonn2 bias", stride=l, paddieg=(I, (), robusterobust)
rohust,batch,, aar.haaarmm.f
weight diet batch2 weight"), ', diet bbbtch2,, bias "I. robusterubus:)
mxxxp.ool22xxx kizeel stride=2) F.relu(x)
if layer_index = mixra.. mixup_flag, k.way, beta.a. beta.b)
Block
robust.conv. forward
weight. ''env33. weight']. weight_diet( (eaedeeeeeooonaaoaeeee 3onn3, stride=l, padding=(1, D), robustarobuxt)
rathst_batch_ne
x. diet batch3, weight'), deit batch3. bias robustzrobust) F.maxxpuol2d(x. sizeel stride=2) F.relu(x)
if layer_index x. robust = intra.class.mixupc. mixup_flag k.way, beta.a. beta.b)
Block
x robust.conv_foret forwardd
weight. diet convd weight'], weight.deetf conn4, board, stride=l, paddisg=(1. D. robustsrobuxt)
robust_batch. ford.frrardd
weight.dict batché, weight'), deig bbbtcch. bias robustzrobuse) -f.mxxppol2d(s. sizeel, stride=2) F.relu(x)
if lnder_index
y. x. robust intra.class.mixupt. mixap.flag, kwwa,, betasa, beta.b)
classer.
x-1.. 64e5-3)
x F.lineart weight-weight.diet" lill weight!. bias=weight.diot| lial, bias
y.
32

---

F. out o, Under review as a conference paper at ICLR 2023
def robust.covvforwardii.. weight. bias. strido. padding, rebust):
Convolution fanction that propagate isterval bounds. if robust is False:
Regalar convolution
P.covvddix. weight. bias. stride. padding) return
6 Convolation propogating interval bounds. shape (0]/73
iaput-p = xizb.size] iaput.o xlb.size:2a b.aizee iaput
size:l
u (iaput-p + input-n))2 (iaput input m)/2
out_a F.coav2dru. weight, bias. stride.. - padding)
out r F.conv2dir. absc weight), None. stride.. padding)
ont.a F.coav2d(inpat.o. weight, bias, stride, padding)
return cat(f
rl. 0)
def robust batch norm fxrward weight, bias, robust
normalicution function
propagate interval boands.
if rohuat is False:
Regalar batch sormafization.
batchhh norm(x, running mean-None. runniag var=None, weight-weight, bina-bias, training-Tree)
return
Batck nocmalization propogating interval bounds.
b_size shapape (03/13
le-5
iaput_p xl:b_size] input =1th size:2eb size]
impos_n =izerh
Equivatent ispur meon (fO. 3))
mean input_o. (0 1), contignous (). view(
input.o shape [I]. -1).mean (1)
inpur_a. transpose
contiguons ().
input.o.shhae(
var(1, Fabiased-False)
# wlesent-wite suttiplier multiplier torch. rsqrt(var eps) multiplier multiplier weight
offset (-mulkiplier
bias
multiplier- multiplier unsqueeze (0), unsqueese (3)
offset = offset. unsqueeze unsuueee (0), unsqueeze (2), (nsqueeze
Becauze scale might negative, u (input_p input_nn/2
apply strategy lincer
(inpot_p input_m)/2
multiplier)
out.r torch mul(r, torch abs abs(multiplier)) out.o -orcch input.o. multiplier) + offset
tarch. cati[our_e
out.r.
def intra mixup_flag, kwwy, beln... beta.b);
Perform
interpolation
robust False
busize y.shape [0]/33
yl: bliieeize]
= b_siees
= b.szsize bixbsiiee
mixup-flag
num.shots busize k/a.
mixup-params rapeat wandom. beta(beta.a. beta.b, k.way), num.shots)
mixup-purams lensor paixup.parmms. diypestorch. flaat). ciew(b.size ty.
rand.ex. ext reprettat(np random. randint(O. 2. kaway). num shotshots). rand.ext tarch teaser
dtype=torch. float). view(b.size
devioc)
mixup sarmm.p mixup-params rand_ext.c = rand_ext
parauppparamsccoo
33

---

Under review as a conference paper at ICLR 2023
mixup-paren www.paaaaaarddeetteeeeee- mikup-parmm--rnd..e.....
torch catilo. ulo return y.
return rabust
Learner: "sre/earners..p"
import
import torch
import torch. mn fanctional as
import optim
import numpy up
import torch. optim import despenpy
collections import OrderedDiet
class Learner:
Base Learner class for MAM, and IBP/IB! varients.
def ixelf.. model. device. update.lr. meta.step.size. beta.a. beta.b, softmax.temp)
Initialigation
self.device = device
self, model, tolself device)
meta.optim
(arameters)). lx=meta.step.size))
self, update_Ir update.lr
self. self.betan bela.a. self. beta_b beta,a, beta.b self softmaxx.nxm suftmax.temp
def train.step (self.
dataset.
order
aum_classes.
num.shots..
mela.shots
inner.iters...
meta.batch.size.
ibp_epsilon..
mixup.
iby-layers):
Training function for MAlL MAMLiBB,, and MAML-BI learners.
For record keeping
spser.Joss.rec. lower.loss.ree, task.loss.ree.
Trilgers RAMAE and vazianis If required. graph.
True. True
create.graph. retain.graph Palse, False
self. meta.optim zero-grad()
if mixep is Trae:
random.tat task randdmmom... rand(O, meta.batch.size)
Iterate
meta-barch
for lask in range! msta.batch.sizee:
weith OrderedDiet(sell pammc...rrmmeterss)
triia set, test set split train test(
mample.miii.ce datsse num num.elasses. wum.-hotssmett.shhtss. Dsst-tBots-meta.thoee
Support
inputs labels (ip(+train.set)
iaputs torch.stack www.k ). selff www...deviiiee device)
lahels torch. (labels) (oelf device)
# Fix ordering
labels. sort index tarch. sort(lahels)
iapots inpers
inputs cst torch cat iipppts- wps.iipppppio........... inputs, inpats-ibp ipputt-iip-essiton]] epailon]. 0)
a Fost adaptation steps for range
if mixup Tree and task ind random lask: MAMEA1R logits net. functional. suactional.forwarrd inpuuss,cat weight. ihp layers. mixup, num classes, self, beta a. self, beta b)
busize logits [0]// logits logits [:b size]
34

---

Under review as a conference paper at ICLR 2023
logits.uto
lust = (B. ww..cre.eeetopp.. (legits.o. labels)
cross.entropy (logits.alo. labels
For NEUL and MAML-IBP
logits self net. Lacciooal.orw..rllapptt fast.weight,
None False. None. None None)
fast.loss cross.entrooyy labels)
fost-gradients
grad(fast.iost, fest. values(), sraate-graphhccceee--g...
fast_weight OrderedDiet(
self. update grad
((name, parom), grad param) west. itemm.......)..... (eldgrdddenss)
# Query set
inputs. labels xip(stest.set)
iaputs (tack tolself, device)
labels torch.tenserr ). to to device)
# ordering
sort
lart(labels)
inputs - inputs inrx_index]
layers
Youilla
logits self. wwt.ee-ffcctionae (inputs weight,
None, False Note, Nose, None)
total,loss F.eross,entropy (logits.. labels) less rec task total lass ree total loss ree ree + total loss loss, item
(temm)
and MAM--18
iaputs_cat torch. cat cat iInpatsssip iupptssibpp.cpallo.. inpots, inpats ibgets--b.. 0)
if mixap N8MBA1I
task_ind random_tesk:
ihp estimate logits celf. nor funetional forward inpats_eat. fost_weight, ibp_layers. mixup, aum_classes, self. beln.b)
self.
busize logits. shape (03/22 logits_o logits b.size] logits_alo logiis [h_siee]
task loss = (F. cros.rrss.eee. entrsssee
labels)
F. cnotss_enntooy (logits.elo, labels))/2
For MUM-IIB
ibp thp.estimate. logits celf.ne ner, funetional,
inputs.cal. fest_wiight, ibp_layers, False. None, None None)
rask_loss croos_eenyoopp (logits, labels).
Find the propagated boundy iibp_estimate.. shape [0]/13
ibp_estimato.u = ibp_estimate [: b_size]
ibp.estimate.o -ibpessimate bbssize:rbb.iizz ibp.estimate.l ibp_estimate b_size:]
Calculate S(methcal(L) (L) (UBB} and (L)rhhal(()-((LL)))
upper_losx lsss (ibp.essibsstimmmte ibp.estimate.o) lower_loss wtt.oolossip..etiinnee
ibp.estimate_o)
Dynomic weighting of loszer
concat.loss torch. task.loss. unsqueeze(0). upper.loss unsqueeze lower.loss unsqueeze
weights F. lotal.luss torch conceuaat..sss weights)
softmax.temp. dim=0)
Record keeping
loser_ = upper.loss.ree + upser item lower.loss.rec lower.loss.ree lower.loss tusk_loss.ree task.los.rec task-less. ilem lotal_luss.rec = loss.re + cotal item
toial_loss buckward butwandretihhggpppereu....c
Averaging the less over meta batcher for purams in self, net parameiers O): params. grad params. grad meta.batch.size
apdate the meto learner parameters self. meta-optim step self. meta.opuu..eroo.fe
35

---

Under review as a conference paper at ICLR 2023
upper_loss..re upper.loss-red mmettabatchh.iize lower_loss.Iec /
resk.luss.rec task lask.loss.rec batcha.
total.loss..ec totallloss..ee hot.oss.reemmtabacccc....o. beiiabbatch.size
return upper.loss.rec lower.loss.res, lask.loss.rev. loss..loss.rec
def vvaluate(self.
dataset.
aum-clasees.
num shots
inner.iters):
Ron single evaluation of the model.
Preserve trained model.
old-state deepeapy net, state.dist
fast_weight (steredbict(self net, samcd.prrameterss)
train-set.
sppit.traiin (swtp-sammle..mm mini. (dataset, num.classes, num.shots
Support sef
inpots. labels zip(-train.sett
inputs -Loorch. stack disselffdevice
[iorcch.eensor to.to(self. device)
Fast adapration
range
logits self.ne net. functioala forward (inputs. fast, wesghttight.
None, False. None, None. None)
fast, fast gast.geadeents
(logits. labels).
autograd grad loss weight.
fast, weight OrderedDict(
param self
param)
((aame, param). grad param) zip weight. (t)ms(), fast. fast-gradient gradients
Query
inpots. labels xip(-test_set)
inputs -Hoorch.stack stack(inputs Co.ssef..device.
labels (torch. tensor (lahels }). to(self device)
Inference
logils self.net. fanctional forward (inputs. weight
None, False. Nose, None, None)
lest.preds (F. softmax (logits, dim=l)) (dm=ix)
Arewacy
num correct torch eretest.preds. labels sm)(()
network original state for rafety. self, net.. soad_state_dico (old
relurn num.correct.c ()
csample.mii caimple_miiiddaaeee (dttaset nam,elasses, num,shots)
Sample few thot tayk from dataret.
shuffled = list (datase)
random shuffle (s (shuffled)
for class.idx. class.obj in caumerate num_classes ]): sample class.abj. wtw....mmpleemmm..ll
yield (sample, class,idx)
def minii.bacchi (xamples, batchaiiee. num.batches].
Generate mini-earcker from some data.
samples (sample))
batch.count
while True:
randum. shuffle(samples
sample samples:
cur.batch. append (sample)
(samellll
if len batch < baich.size:
continue
yield cur.batch
baich.count += baich.count num.batches:
def -splt.. trait.trriaccoo taapllls shotssh
Split few-shor train ten set.
train.set [ist
in train.set)
36

---

Under review as a conference paper at ICLR 2023
range(test.ssooss
for label in labels:
for 1. item in enumerate(train.set) if train-setli]
label:
lest.set (itm)
break
len IndexBrror(" o0w.a/ag...aaaate...............
len (labes)))) shots:
return train.sel. test.set
Training function: "sre/train "sreltrain.mode..pp""
Import
impurt numpy up
import torch
from
Jearners import Learner
(learner.
train_sel.
val. .iet.
model.outpu.filessome
model_save.patiee
order=None.
sum.classesaNome.
num.shotsaNone.
meta_shots-Non::
inner_iters=Nume
nets.batch.aizeeNNon
meta-iters-None,
eval-inner.lleriaNoss
eval_interval Nonnerrval-Noe....
eval.interval.smmplees
ibp.epxilon=Nome.
mixep=False.
lbp.layers=Nome):
dataser.
araincac val.ascuracy [l.
Juser..us..tore. lower.loss.store lask.less.store. loss.. = [I (]. D.
Loop training steps.
for in range (mge@meta.ilerstil
Find value interval coefficient. cur.ibp.epsilos eps.scheduler(). meia.iters. ibp.epsilon)
Train the learner for step. upper.loss, lower.lost, lask.loss. total.loss learner train.step (trais..st. arderworder.
rum.clasee=-mmmlaase.
num.shotseaum.shos..
aat.shoossemel.shouu
aner-iters=inner.itee
batch size=meta batch
th..epalloo-cur.....ppiiee mixap-miuup,
ibp layers-ibp layers)
Record a upper.loss) spper_loss.store
lower store append lewer loss)
tosk.loss.stoo append loss) total loss append loss)
If & ivterinterval
Perform intermediate evalxation. total.correct
for in raugeleval interval sample]: coralcc correct learner evaluate (arauatteerriin... (train.set..
num.elasess-nmm.cln
itersweval isers)
train aww.racyccccc.....
invaa_ineerval.so sample num classes
save.path model save.paah "/intermediate." str(i) '_model pt"
torch save((" model mate learner net. state dier(), mera_optim,state save path)
meta,optim. state_dict()).
total_correet D
range (eval interval sample): total correct total- correct learner avalnate(val.
num sum_classss-naa classesssss....... num www..hho-uuumheeee ianer Herraeeva inner iters)
val_aceurayy append
(exal_interval.commle num_elasses
with open( model outpat file., 'a4") as fp:
print( Baathhsdiuute staiirrriistt & (1.
train_aceuracy [-]], val.aceuracy [-I]), file=fp)
37

---

Under review as a conference paper at ICLR 2023
Intermediate record keeping. res-save-path = model.save.path + "/ "atermeediate.aeuurrsiss.s app
loss.save..alle = saveu. + + "intermechates.lossee losses...p
(ree uniin.cccurcc--n.. wacccuracc-np.rrree. aurp.arrayy atray(train.accurrr... val. cal.aceeracysnp..r arryy (val.ae
savez path losse.lssssnp..oo (apper loss store).
lower-loss-np.arce (rw.eriiree...ssstttrr. toss.loss--p. array (task.kkk..ooo
Lotal.toss-pp.arct atray (taa(tttallossssttreee
def eps.echedulrrt,, meta.iters. ibp-epsiton):
Schedule the value of interval coefficient.
iters return (1 [mera (mera-itersaD) 4))) ibp.epsilon
return 1bp epsilon
Inference function: "sre/eval model.ppp
Helpers for evaluating
Import numpy learnera import Learner
def bulk, evalunte clearner. dataset s.m_classes-5,
shotsw3, eval inner iters-10, -00.s00ppes-100000:
Fralunting =
on set tasks.
for in samples):
total_correet classes-uss classes. num sin.shhtssummhhhe shooss
inner Hterseeval inner iters)) ))
accurrcccss array (total_correct) claselasses
lest accurncy rotal securacies sum() num somples
enst up. toal
For confidence interval 6,95% cacors
test.enf t.score (tert.enf / (up, sgrt
testaaccurayy, test.enf
38


Reviews:
[{"review_id": "q3SHNJ-uoiV", "reviewer": "Reviewer_Ww7U", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose a solution for few-task few-shot learning. Their solution is based on two steps:\nFirst they take the task and use interval bounds to estimate two other losses (if the task was in the neighborhood manifold) one upper bound and one lower bound. Then during optimization, they look at the loss of the task and make sure it should be minimized plus the feature representation of the model for the task should be close to the feature representation of the interval lower and upper bound feature representation. They furthermore take the softmax of the three losses and minimize it which is an interesting idea. \n\nSecond, they create artificial tasks from the latent space of the model. Since the upper and lower bounds are in proximity of original task, we can interpolate the features of original task with any of these bounds to create artificial tasks.", "summary_of_the_review": "Based on the above comments, I vote for borderline reject. ", "strength_and_weaknesses": "Strength:\nWell-structured\nNice idea of looking into intervals during training\n\nWeakness\nThe experiments do not compare with baselines to show the strength of the method. For example, what if instead of using lower and upper bounds for intervals, I just add some noise to the feature representation? That would be an interesting baseline to test against. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "W7Mt751IcR", "reviewer": "Reviewer_yeNC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors use interval-bound propagation to work with few-shot learning problems. Interval arithmetic was used to model the manifold of data devoted to task interpolation. Finally, the authors apply interval architecture to model-agnostic meta-learning and prototype-based metric-learning paradigms.", "summary_of_the_review": "The paper is very interesting, but I see some problems in the motivation of using Interval arithmetic without worst-case classification loss. In such scenario, we can use simpler objects than intervals. \n", "strength_and_weaknesses": "1. Related work section is poor. It does not describe a few-shot learning method. The chapter on manifold learning is poor. There exist many flows, NKK methods, and VAE-based method, which was even mentioned. \n2. Equation (4) is misleading. It is not trivial to understand what exactly mean I_i^s(\\epsilon.)  \n3. The idea of keeping small interval bounds was described in the paper: https://www.esann.org/sites/default/files/proceedings/2020/ES2020-57.pdf and should be mentioned in the paper.  \n4. It is unclear why we add losses deducted to the center of the interval and its two borders $L_{LB}$, $ L_{UB}$. We can use the only distance between borders.\n5. According to Tab 1. the authors claim, \"We see that MAML+IBP outperforms vanilla MAML in terms of 5-way 1-shot classification accuracy on the miniImageNet and tieredImageNet (Ren et al., 2018) datasets.\"\nIt is too strong since MAML has a significant variance on miniImageNet, and MAML+IBP is very close. It looks like a slight difference. Maybe the authors should use some statistical comparison of the models.\n6. It is possible to see worst-case accuracy instead of interpolation. Does it steal work well? \n7. If we do not use worst-case accuracy, we do not want any guarantees. Why we work with relatively complex intervals, we can simply work with a probability distribution (for example, Gaussian ones). Is it possible to use Gaussians instead of intervals?\n8. Why are experiments presented on miniImageNet-S instead of miniImageNet? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NfYubyJzRP", "reviewer": "Reviewer_9giy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The notion of interval bounds is used to enhance few-shot learning, especially in the situation where only few tasks are in the training set.  Interval bound propagation (IBP) through network layers and the use of the modified training loss ensure that the model outputs stay close to the interval bounds in the embedding space of initial layers. This improves the chances for the artificially formed tasks to remain on the task manifold. A theorem is established that guarantees that artificially created (via IBP interpolation) support sets are essentially as good as those from the actual distribution. Experiments on both MAML and prototypical few-shot learning show improved performance over baselines.", "summary_of_the_review": "A reasonably good paper overall. Although the IB and IBP method/concept is not new, the application to few-shot few-task learning is interesting and insightful. A theoretical analysis that reflects the effect of bound losses would have been better, as they are in the core of the proposed method. ", "strength_and_weaknesses": "Strengths: The authors propose an interesting and effective solution to few-shot, few-task learning. A theoretical analysis is provided. Ample experimental results are given validating the proposed method. Figures 1 and 2 are very helpful in understanding the proposed ideas.\n\nWeaknesses: \n\nIt is a bit difficult to fully understand the descriptions of interval bound propagation without any background on related topics. It would be better to have more detailed insights, explanations, discussions on this and add them to the Related Works or Preliminary section.\n\nThe theoretical analysis seems to say nothing about the algorithm in terms the efficacy of minimizing L_{LB} + L_{UB}; the analysis focuses only on CE loss by making the remaining terms arbitrarily small (also lacking any insights into the effectiveness of the proposed task interpolation). The generalization error with bounded loss function is somewhat obvious from stat theory point of view.\n\nNot necessarily weaknesses but some questions:\n\n1. In Table 4, why does +IBI perform worse than +IBP in some cases even with the augmented tasks?\n\n2. If the authors are concerned about whether the new task would be on the task manifold or not, then why don’t they consider just interpolating between the task and the neighboring one, where the neighboring task lies within the epsilon ball centered from $f_{\\theta^{S}}(x_{i,r}^s)$?\n\n3. Applying IBP boosts the performance even without task interpolation, which I found intriguing. Could the authors provide more insights into how preserving the neighborhoods can assist the model to generalize well on unseen tasks?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Gvy-X0ZYki", "reviewer": "Reviewer_NpKQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose to extend the idea of interval bounds from the provably robust training literature to few-shot classification and introduce task-level interval bounds based regularization for the training of few-shot classification models (MAML + protonets). \nTo further improve the model robustness under the challenge of insufficient training tasks, the authors introduce interval bounds interpolation to synthesize more plausible augmented tasks to improve the task diversity for training. ", "summary_of_the_review": "More discussions are expected to clarify the contribution of this paper on a task level. Please refer to the Strength And Weaknesses question for details. ", "strength_and_weaknesses": "**Strength**\n\nThe challenge of training a few-shot learning model with few tasks is indeed a very interesting direction as the small number of tasks leads to a poor approximation of the task distribution. And interpolating in the task space is clearly a promising way of addressing this challenge. \n\nThe idea and the implementation are clearly presented. \n\n** Weaknesses**\n\n1. Task manifold\n\nMy primary concern regarding this paper is the definition of 'task manifold'. While manifold is a well-studied concept, task manifold seems new to me, and the connection between tasks manifold and data manifold remains very much unclear to me after reading this paper. \nSpecifically,  according to the definition of $\\epsilon$-neighborhood of a task, I do not see a clear difference between the interval bounds of a task and the interval bounds of a collection of samples. And based on the little difference, the 'task manifold' in this paper seems almost identical to data manifold to me. And this is also confirmed by the realization of the method, which is basically achieved by injecting **sample-wise** noise. \nIn this case, what is the contribution of this paper to the study of task-level few-shot learning? \n\nWhile I appreciate the theoretical analysis provided by the authors, I believe the analysis is still very much based on samples instead of tasks. \n\n\n2. Cost\n\nThe discussion of the limitation of the proposed method on cost is appreciated. However, I believe it is still necessary to provide in the experiment section comprehensive comparisons of the training cost between the proposed method and the other methods. \n\nBased on the cost issue, it might be necessary to discuss other alternatives. For example, can adopting spectrum normalization achieve similar results by simply promoting Lipschitz continuity?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject