On sensitivity of meta-learning to support data
Year: 2021

Abstract:
Meta-learning algorithms are widely used for few-shot learning. For example, image recognition systems that readily adapt to unseen classes after seeing only a few labeled examples. Despite their success, we show that modern meta-learning algorithms are extremely sensitive to the data used for adaptation, i.e. support data. In particular, we demonstrate the existence of (unaltered, in-distribution, natural) images that, when used for adaptation, yield accuracy as low as 4% or as high as 95% on standard few-shot image classification benchmarks. We explain our empirical findings in terms of class margins, which in turn suggests that robust and safe meta-learning requires larger margins than supervised learning.

heading: 1 Introduction
text: Meta-learning, or learning to learn [29], is the problem of training models that can adapt to new tasks quickly, using only a handful of examples. The problem is inspired by humans’ ability to learn new skills or concepts at a rapid pace (e.g. recognizing previously unknown objects after seeing only a single example [20]). Meta-learning has found applications in many domains, including safety-critical medical image analysis [23], autonomous driving [35], visual navigation [43] and legged robots control [40]. In this paper, we investigate the vulnerabilities of modern meta-learning algorithms in the context of few-shot image classification problem [25], where a meta-learner needs to solve a classification task on classes unseen during training using a small number (typically 1 or 5) of labeled samples from each of these classes to adapt. Specifically, we demonstrate that the performance of modern meta-learning algorithms on few-shot image recognition benchmarks varies drastically depending on the examples provided for adaptation, typically called support data, raising concerns about its safety in deployment.
Despite the many empirical successes of artificial intelligence, its vulnerabilities are important to explore and mitigate in our pursuit of safe AI that is suitable for critical applications. Sensitivity to small (possibly adversarial) perturbations to the inputs [13], backdoor attacks allowing malicious model manipulations [6], algorithmic biases [2] and poor generalization outside of the training domain [18] are some of the prominent examples of AI safety failures. Some of these issues have also been studied in the context of meta-learning, e.g. adversarial robustness [45, 11, 44] and fairness [37]. Most of the aforementioned AI-safety dangers are associated with adversarial manipulations of the train or test data, or significant distribution shifts at test time. In meta-learning, prior works demonstrated that an adversary can create visually imperceptible changes of the test inputs [11] or the support data [44, 30] causing meta-learning algorithms to fail on various few-shot image recognition benchmarks. In this work we demonstrate adversary-free and in-distribution failures specific to meta-learning. Sohn et al. [39] studied a similar problem in the context of semi-supervised learning.
A distinct feature of meta-learning is the adaptation stage where the model is updated using a scarce amount of labeled examples from a new task. In practice, these examples need to be selected and presented to a human for labeling. Then a meta-learner adapts and can be used to classify the
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
remaining images without a human. Currently, meta-learning algorithms choose support examples for labeling at random. Dhillon et al. [8] noted that the standard deviation of accuracies computed with random support examples may be large. We demonstrate that if the support data is not carefully curated, the performance of the meta-learner can be unreliable. In Figure 1 we present examples of unaltered support images (i.e. they are not modified adversarially or in any other way) representative of the corresponding task (in-distribution) that lead to vastly different performance of the meta-learner after adaptation. Support examples causing poor performance are not prevalent, however, they are also not data artifacts as there are multiple of them. The existence of such examples might be concerning when deploying meta-learning in practice, even when adversarial intervention is out of scope.
Our main contributions are as follows:
1. We present a simple algorithm for finding the best and the worst support examples and empirically demonstrate the sensitivity of popular meta-learning algorithms to support data.
2. We demonstrate that a popular strategy for achieving robustness [22] adapted to our setting fails to solve the support data sensitivity problem.
3. We explain the existence of the worst case examples from the margin perspective suggesting that robust meta-learning requires class margins more stringent than classical supervised learning.

heading: 2 Meta-learning approaches
text: Meta-learning approaches are typically categorized into model-based [34, 26, 33], metric-based [17, 41, 38] and optimization-based methods [10, 28, 14]. We refer to Hospedales et al. [15] for a recent survey of the area. In this paper we mostly focus on the optimization-based method popularized by the Model Agnostic Meta Learning (MAML) [10] bi-level formulation (metric-based prototypical networks [38] are also considered in the experiments). Let {Tn}ni=1 be a collection of n tasks Ti = {Ai,Di} each consisting of a support Ai and a query (or validation) Di datasets. MAML
bi-level optimization problem is as follows [10]:
min ✓
1
n
nX
i=1
`(✓i, ✓;Di),
such that ✓i = argmin ✓i
`(✓i, ✓;Ai), i = 1, . . . , n. (2.1)
The first line of equation 2.1 is typically minimized using gradient-based optimization of ✓ and is called meta-update. The second line is the adaptation step and its implementation differs depending on whether the fine-tuned ✓i are treated as parameters of the end-to-end neural network or as the “head” parameters, i.e. the last linear classification layer. We refer to Goldblum et al. [12] for an empirical study of the differences between the two adaptation perspectives.
MAML [10] and Reptile [28] are examples of algorithms that fine-tune all network parameters during the adaptation. Instead of solving the argmin of ✓i exactly they approximate it with a small number of gradient steps. For one gradient step approximation equation 2.1 can be written as
min ✓
1
n
nX
i=1
`(✓ ↵ @`(✓;Ai)
@✓ ;Di) (2.2)
for some step size ↵. MAML directly optimizes equation 2.2 differentiating through the inner gradient step, which requires expensive Hessian computations. Reptile introduces an approximation by-passing the Hessian computations and often performs better.
Another family of meta-learning algorithms considers ✓ as the neural network feature extractor parameters shared across tasks and adapts only the linear classifier parametrized with ✓i, that takes the features extracted with ✓ as inputs. The advantage of this perspective is that the adaptation minimization problem is convex and, for many linear classifiers, can be solved fast and exactly. Let a(✓,A) denote a procedure that takes data in A, passes it through a feature extractor parametrized with ✓, and returns ✓i, i.e. optimal parameters of a linear classifier using the obtained features to predict the corresponding labels. Then equation 2.1 can be written as
min ✓
1
n
nX
i=1
`(✓, a(✓,Ai);Di). (2.3)
This approach requires a(✓,A) to be differentiable. R2D2 [4] casts classification as a multi-target ridge-regression problem and utilizes the corresponding closed-form solution as a(✓,A). Lee et al. [21] propose MetaOptNet, where a(✓,A) is a differentiable quadratic programming solver [1], and implement it with linear support vector machines (SVM) and ridge-regression. Prototypical networks [38] is a metric-based approach that can also be viewed from the perspective of equation 2.3. Here a(✓,A) outputs class centroids in the neural-feature space and the predictions are based on the closest class centroids. Last-layer adaptation approaches, especially MetaOptNet, outperform full-network approaches on most benchmarks [21, 12].
Both adaptation perspectives have a weakness underlying our study: their test performance is based on an optimization problem with large number of parameters and as little as 5 data points (1-shot 5-way setting). To understand the problem, consider the last-layer adaptation approach: even when the feature extractor produces linearly separable representations, the dimension is large (for example, Lee et al. [21] utilize a ResNet-12 architecture with 2560-dimensional last layer features that we adopt in our experiments) making the corresponding linear classifier extremely sensitive to the support data. In Section 3 we demonstrate the problem empirically and provide theoretical insights in Section 4.

heading: 2.1 Meta-learning benchmarks
text: Before presenting our findings, we discuss the meta-learning benchmarks we consider. Meta-learning algorithms are often compared in a few-shot image recognition setting. Each task typically has five unique classes, i.e. 5-way, and one or five examples per class for adaptation, i.e. 1-shot or 5-shot. Classes that appear at test time are not seen during training. Few-shot learning datasets are typically the derivatives of the existing supervised learning benchmarks, e.g. CIFAR-100 [19] and ImageNet [7]. Few-shot problem is setup by disjoint partitioning of the available classes into train, validation and test. Tasks are obtained by sampling 5 distinct classes from the corresponding pool and assigning
a subset of examples for adaptation and a different subset for meta-update during training or accuracy evaluation for reporting the performance.
CIFAR-FS [4] is a dataset of 60000 32⇥32 RGB images from CIFAR-100 partitioned into 64, 16 and 20 classes for training, validation and testing, respectively. FC-100 [31] is also a derivative of CIFAR-100 with a different partition aimed to reduce semantic overlap between 60 classes assigned for training, 20 for validation, and 20 for testing. MiniImageNet [41] is a subsampled, downsized version of ImageNet. It consists of 60000 84⇥84 RGB images from 100 classes split into 64 for training, 16 for validation, and 20 for testing.

heading: 3 Finding adaptation vulnerabilities
text: In this section we study the performance range of meta-learners trained with a variety of algorithms: MAML [10], Meta-Curvature (MC) [32], Prototypical networks [38], R2D2 [4], and MetaOptNet with Ridge and SVM heads [21]. At test time a meta-learner receives support data and its performance is recorded after it adapts. Typical evaluation protocols sample support data randomly and report average performance across tasks. While this is a useful measure for comparing algorithms, safetycritical applications demand understanding of potential performance ranges. We demonstrate that there are support examples the lead to vastly different test performances. To identify the worst and the best case support examples for a given task we perform an iterative greedy search.
Let X = {xkm} k2[K] m2[M ] be a set of potential support examples for a given task. K is the number of classes, M is the number of examples per class (we assume same M for each class for simplicity), and [M ] = {0, . . . ,M 1}. Let D be the evaluation set for this task. In a J-shot setting, let Z = {zkj } k2[K] j2[J] be a set of indices of the support examples (z k j 6= z k j0 for any k, j 6= j
0 and zkj 2 [M ] for any k, j). Denote R(Z,X ,D) a function computing accuracy of a meta learner on the query set D after adapting on A = {xk
zkj } k2[K] j2[J] . Finding the worst/best case accuracy amounts to finding
indices Z minimizing/maximizing R(Z,X ,D). We solve this greedily by updating a single index zkj at a time, holding the rest of Z fixed, iterating over [K] and [J ] multiple times. We summarize the procedure for finding the worst case accuracy in Algorithm 1 (the best case accuracy is analogous).1 Due to the greedy nature, this algorithm finds a local optima, but it is sufficiently effective as we see in the following section.
Algorithm 1 Finding the worst case support examples Input: trained meta-learner, potential support examples X to search over, query data D. repeat
Initialize indices Z = {zkj } k2[K] j2[J] randomly. for j 2 [J ], k 2 [K] do zkj argminzkj R(Z,X ,D), z k j 2 [M ] \ {z k j0}j0 6=j
end for
until R(Z,X ,D) stops decreasing Output: support examples indices Z = {zkj } k2[K] j2[J]
Visualizing the worst case support search In Figure 2 we visualize a single iteration of Algorithm 1 on a 1-shot task from the CIFAR-FS dataset. This is a 5-way task with “snail”, “red pepper”, “bed”, “plain”, and “telephone” as classes. In round 1, i.e. k = 0, of iteration 1 we start with a random example per class and evaluate post-adaptation accuracy on the query data D for each potential support “snail”, i.e. {x0m}m2[M ]. Here the first line corresponds to “snail” indexed with m = 0, i.e. x00, and post-adaptation accuracy of 73.5%. We select a “snail” support image corresponding to the worst accuracy of 53.8% and proceed to round 2, i.e. k = 1, of iteration 1. In round 2 we repeat the same procedure for the corresponding class “red pepper” (again the first line corresponds to “red pepper” indexed with m = 0, i.e. x10), holding the support “snail” image selected previously and
1This is a simple approach, however we found it faster and more efficient then a more sophisticated continuous relaxation using weights of the potential support examples with sparsity constraints.
support images for other classes fixed. The algorithm finds the worst case support “red pepper” image corresponding to the post-adaptation accuracy of 37%. Then the algorithm proceeds analogously to rounds 3, 4, and 5 of iteration 1, resulting in a combination of support examples corresponding to 2.9% post-adaptation accuracy. This is already sufficiently low, however on some tasks and higher-shot settings it may be beneficial to run additional iterations. On iteration 2, the algorithm will again go through all the classes starting from the support examples found on iteration 1 instead of random ones. In our experiments we always run Algorithm 1 for 3 iterations. In Appendix B we empirically study the convergence of the algorithm justifying this choice.

heading: 3.1 Performance range results
text: We summarize the worst, average and best accuracies of six meta-learning algorithms on three benchmark datasets (see Section 2.1 for data descriptions) in 1-shot, 5-shot, and 10-shot setting in Tables 1, 2, and 3. All meta-learners are trained using code from the authors or more modern meta-learning libraries [3] (see Appendix A for implementation and additional experimental details). For evaluation we randomly partition each class in each task into 400 potential support examples composing X and 200 query examples composing D (all datasets have 600 examples per class). To compute average accuracy we randomly sample corresponding number of support examples per class; for the best and the worst case accuracies we use Algorithm 1 to search over support examples in X .
Our key finding is the large range of performances of all meta-learning algorithms considered in 1-, 5-, and 10-shot settings. Prototypical networks have slightly better worst-case 5-shot accuracy on CIFAR-FS, but it has large variance and is likely due to our algorithm finding poor local optima on one of the tasks. We also see no differences between meta-learners adapting end-to-end, i.e. MAML and MC, and those adapting only the last linear classification layer, i.e. R2D2 and MetaOptNet. Goldblum et al. [12] showed that the last-layer adaptation methods produce good quality linearseparable embeddings. One could expect such methods to be less sensitive to support data, however, as we discuss in Section 4, linear separability is not sufficient in the few-shot learning setting. 10-shot worst-case accuracies are not as poor (despite mostly remaining worse than random guessing), but
we expect that with more support data available, the gap would narrow.2 Finally, we note that the best-case accuracy is significantly better, especially in the 1-shot setting.

heading: 3.2 Worst case support examples are not artifacts
text: We have demonstrated that it is possible to find support examples yielding poor post-adaptation performance of a variety of meta-learners. It is also important to understand the nature of these worst-case examples: are they data artifacts, i.e. outliers or miss-labeled examples, or realistic images that could cause failures in practice? We argue that the latter is the case.
1. In Figure 1 we presented several examples of the worst-case support images on CIFAR-FS: they are correctly labeled and appear representative of the respective classes. Inspecting the images on the left closer we note that it is often not easy to notice visually that such support examples could result in a poor performance without significant expert knowledge of the dataset. Only the cellphone image labelled “telephone” and grey-scale image labeled “plain” seem potentially problematic. On the other hand, “snail” and “bed” images all appear normal. On the right figure, majority of the images also appear reasonable.
2. 10-shot setting should be a lot more resilient to outliers, however Algorithm 1 continues to be successful in 10-shot setting, finding ten different examples per class leading to accuracy slightly better than a random predictor as shown in Tables 1, 2, and 3.
3. In Figure 3 we present histograms of accuracies visualizing the first iteration over classes of Algorithm 1 in 1-shot learning on CIFAR-FS. The right most histogram corresponds to post-adaptation accuracies for different choices of support image for class 0 and random choices for classes 1-4. The subsequent histogram is for different choices of support images for class 1, where image for class 0 is chosen with Algorithm 1 and classes 2-4 are random, and analogously for the remaining three histograms. We see that the lower accuracy tails of the first two histograms contain multiple worst-case support examples in the corresponding classes. By the third histogram, the range of accuracies is below 50% for all possible support examples in the corresponding classes.
4. In Figure 4 we present histogram of accuracies of 3991 unique combinations from CIFARFS of 1-shot support examples evaluated by Algorithm 1 throughout 3 iterations. There are 3335 distinct sets of 5 examples each with less than 50% post-adaptation accuracy.
We present analogous analysis for other meta-learners and datasets in Appendix C, where we also conclude that worst-case adaptation examples are realistic and can cause malfunctions in practice.

heading: 3.3 Improving support data robustness with adversarial training
text: The issue of robustness has been studied in many contexts in machine learning, and adversarial vulnerability of deep learning based models has been explored extensively in the recent literature [13, 5, 22]. While the support data sensitivity of meta learners presented in this work is a new type of non-robustness, it is possible to approach the problem borrowing ideas from adversarial training [22]. The high-level idea of adversarial training is to use a mechanism exposing non-robustness to guide the data used for model updates. For example, if a model is sensitive to perturbations in the inputs, for an incoming batch of data we find the worst-case (by maximizing the loss) perturbations to the inputs and use the perturbed inputs to updated model parameters. To converge, adversarial training needs to find model parameters such that the mechanism exposing non-robustness can no-longer damage the performance. Adversarial training has theoretical guarantees for convex models [42] and has been shown empirically to be successful in defending against adversarial attacks on deep learning models [22].
We use adversarial training scheme in an attempt to achieve robustness to support data in meta learning. Specifically, we use Algorithm 1 to find worst-case adaptation examples during training instead of using random ones as in standard training. The result is quite intriguing: Tables 4 and 5 summarize performance of adversarially (in a sense of support data) trained meta-learners on train and test tasks. In most cases, adversarial training converged, i.e. we are no longer able to find detrimental worst-case support examples with Algorithm 1 on the training tasks, however we observe
2With enough support data, last-layer methods should succeed as their embeddings are linearly separable.
no improvements of the worst-case accuracy on the test tasks. Our experiment demonstrates that support data sensitivity in meta-learning is not easily addressed with existing methods and requires exploring new solution paths.

heading: 4 Meta-learning margin analysis
text: Meta-learners, specifically those only adapting the last layer linear classifier, produce embeddings that appear (approximately) linearly separable even when projected into two dimensions as shown in Figure 5 (we used Multidimensional Scaling [24] to preserve the relative cluster sizes and distances between clusters). In the supervised learning context this would be considered an easy problem for, e.g., linear SVM as in the MetaOptNet-SVM meta-learner. The problem, however, is that in the supervised learning context we typically have sufficient data, while in the few-shot setting meta-learners are restricted to as little as a single example per class in a high-dimensional space.
Lets take another look at Figure 5. These are Multidimensional Scaling [24] projections of the embeddings obtained with the MetaOptNet trained on CIFAR-FS. Embeddings in (a) and (c) correspond
(a) Train task embeddings (standard training) (b) Test task embeddings (standard training) (c) Train task embeddings (adversarial training) (d) Test task embeddings (adversarial training)
Figure 5: Projected embeddings of MetaOptNet-SVM for a train and a test task query data from CIFAR-FS. We compare standard and adversarial training discussed in Section 3.3. Points are colored with their labels. Highlighted points are the worst-case support examples selected with Algorithm 1.
to a query data from a train task for standard and adversarially trained models, and embeddings in (b) and (d) to a query data from a test task for the corresponding models. All embeddings appear well-clustered, however embeddings in (c) have the largest separation and the smallest within-class variance. This wouldn’t make a significant difference in supervised learning with enough data, but makes a big difference for meta-learning: when using Algorithm 1 to find the worst-case support examples (highlighted in the figures), the corresponding accuracies are 1.8% for (a), 2.90% for (b), 99.4% for (c), and 5.70% for (d). We see that although all embeddings are well-separated3, only (c) is robust to support data selection. We present the projected embeddings for a variety of meta-learners on the FC100 dataset in Appendix D. As we discuss next, robust meta-learning, in addition to vanilla linear-separability, requires features with bigger class separation and lower intra-class variances.
In the following theorem, we show that as long as the class embeddings are sufficiently separated, the probability of any two points sampled from the classes leading to a max-margin classifier with large misclassification rate is exponentially small. We note the high degree of separation (linear in embedding dimension) necessary to guarantee robustness to the choice of support data. This suggests unless the class embeddings are well-separated, the resulting meta-learning algorithm will be sensitive to the choice of support data. Theorem 4.1. Consider a (binary) Gaussian discriminant analysis model:
X | Y = 1 ⇠ N(µ, 2Id),
X | Y = 0 ⇠ N( µ, 2Id).
3For comparison, the accuracies of the corresponding supervised learning problems (i.e. linear classifiers trained using embeddings of all 400 per class potential support examples X , rather than a single example per class) are 99.6% for (a) and 94.3% for (b).
As long as µ = (d+ p 2dt+ 2t) for some t > 0, then the max-margin classifier between two points sampled independently from each cluster has misclassification rate at most ( d p 2dt 2t) with probability at least (1 e t)2.
Proof. Consider a (binary) Gaussian discriminant analysis model:
X | Y = 1 ⇠ N(µ, 2Id),
X | Y = 0 ⇠ N( µ, 2Id).
Define the core of the clusters as the sets
C1 , {x 2 Rd | kx µk2  r }, C0 , {x 2 Rd | kx+ µk2  r }.
It is a tedious geometric exercise to show that as long as µ > 2r , then the midpoint of any pair of points (x1, x0) 2 C1 ⇥ C0 falls outside C1 [ C0. Further, the hyperplane
H , {x 2 Rd | (x1 x0)Tx = 12 (kx1k22 kx0k22)} bisects C0 and C1 for any choice of x1, x0. This implies the risk of any max-margin classifier constructed from x1 and x0 has misclassification error rate at most 2 ( r), where is the N(0, 1) CDF. We note that the probability of a pair of independently sampled points x1 ⇠ N(µ, 2Id) and x0 ⇠ N( µ, 2Id) falling in C1 and C0 respectively is F 2d(r), where F 2d is the CDF of a 2 d random scalar. By picking r = d+ p 2dt+ 2t for some t > 0, the probability of x1 2 C1 and x0 2 C0 is at least (1 e t)2.

heading: 5 Conclusion
text: We studied the problem of support data sensitivity in meta-learning: the performance of existing algorithms is extremely sensitive to the examples used for adaptation. Our findings suggest that when deploying meta-learning, especially in safety-critical applications such as autonomous driving or medical imaging, practitioners should carefully check the support examples they label for the meta-learner. However, even when the data is interpretable for a human, e.g. images, recognizing potentially detrimental examples could be hard as we have seen in our experiments.
In our experiments, we considered popular few-shot image classification benchmarks. We note that meta-learning has also been applied to data in other modalities such as language understanding [9] and speech recognition [16]. We expect our conclusions and Algorithm 1 for finding the worst-case support data to apply in other meta-learning applications, however, an empirical study is needed to verify this.
Going forward, our results suggest that robustness in meta-learning could be achieved by explicitly encouraging separation and tighter intra-class embeddings (at least in the context of last-layer adaptation meta-learners). Unfortunately, the adversarial training approach, while successful in promoting robustness in many applications, fails to achieve robustness of meta-learners to support data. In our experiments, adversarial training achieved well-separated and tight intra-class embeddings resulting in robustness on the train tasks (i.e., tasks composed of classes seen during training), but failed to improve on the test tasks. Our findings demonstrate that new approaches are needed to achieve robustness in meta-learning.
Finally, we note that our results provide a new perspective on a different meta-learning phenomenon studied in prior work. Setlur et al. [36] and Ni et al. [27] studied the importance of the support data during training. Setlur et al. [36] quantified the impact of the support data diversity, while Ni et al. [27] considered the effectiveness of the support data augmentation, and both concluded that meta-learning is insensitive to the support data quality and diversity. In our experiments with a variation of adversarial training in Section 3.3, where support data during training is selected based on Algorithm 1 to find worst-case support examples, we achieved significant improvements in terms of the worst-case accuracies on the train tasks. Thus, the conclusion is different from [36, 27], i.e. the support data used during training has an impact on the resulting meta-learner from the perspective of sensitivity studied in our work.
Acknowledgments and Disclosure of Funding
This note is based upon work supported by the National Science Foundation (NSF) under grants no. 1916271, 2027737, and 2113373. Any opinions, findings, and conclusions or recommendations expressed in this note are those of the authors and do not necessarily reflect the views of the NSF.

Review summaries:
This paper empirically finds that popular meta-learning algorithms are very sensitive to the selections of the support images. The author shows that six meta-learning algorithms can achieve testing accuracy within a wide accuracy (worst case accuracy is extremely poor) given different support samples. They also find that this problem cannot be trivially solved by adversarial training. 

This paper is studying the performance of several meta-learning approaches depending on the support images used for meta-test tasks.  First, the authors propose a simple greedy algorithm that select for each class in a given task, the image giving the worst/ the best performance. With this algorithm, they show that popular meta-learning approaches have a very high variance in their performance and that the worst case scenario is often worse than a random guess. The authors argue that the worst tasks found by their algorithm are not artifacts in the distribution, and can realistically appear.  Then, they apply their algorithm to train the meta-learning methods in an adversarial way, by learning on the worst cases found by their algorithm. Even though the adversarial training improves the worst-case performance on the meta-training tasks, there are no improvements on the worst-case accuracy on meta-testing tasks. Finally, the authors argue that the worst-case performance are better when the task embeddings are sufficiently separated. 

The paper studies the sensitivity of meta-learning to the support data. The paper presents an algorithm for finding the best and worst support examples relative to the test accuracy, and demonstrates empirically the sensitivity of several meta-learning methods to the support data. The paper explored using  adversarial robustness strategies, however, while these methods seem to work on the training tasks, they fail to generalize on new test tasks. The paper explores the sensitivity of meta-learning algorithm to the support data from the margin perspective, arguing that higher margins and interclass tightness leads to more robust meta-learning methods.