Uniform Sampling over Episode Difficulty
Authors: Sebastien Arnold
Guneet Singh Dhillon
Avinash Ravichandran
Stefano Soatto
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: episode sampling
few-shot learning
meta-learning

Abstract: Episodic training is a core ingredient of few-shot learning to train models on tasks with limited labelled data. Despite its success, episodic training remains largely understudied, prompting us to ask the question: what is the best way to sample episodes? In this paper, we first propose a method to approximate episode sampling distributions based on their difficulty. Building on this method, we perform an extensive analysis and find that sampling uniformly over episode difficulty outperforms other sampling schemes, including curriculum and easy-/hard-mining. As the proposed sampling method is algorithm agnostic, we can leverage these insights to improve few-shot learning accuracies across many episodic training algorithms. We demonstrate the efficacy of our method across popular few-shot learning datasets, algorithms, network architectures, and protocols.


Uniform Sampling over Episode Difficulty
Sébastien M. R. Arnold Guneet S. Dhillon Avinash Ravichandran Stefano Soatto
'University of Southern California, PUniversity of Oxford, 3 "Amazon Web Services, "University of California, Los Angeles
seb. arnolduusc. edu, guneet dhillon@stats ox. ac. uk,
ravinash@amazon.c com, soattos@amazon.co com
Abstract
Episodic training is core ingredient of few-shot learning to train models on tasks with limited labelled data. Despite its success, episodic training remains largely understudied, prompting us to ask the question: what is the best way to sample episodes? this paper, we first propose method approximate episode sampling distributions based their difficulty. Building on this method, we perform an extensive analysis and find that sampling uniformly over episode difficulty out- performs other sampling schemes, including curriculum and easy-/hard-mining As the proposed sampling method is algorithm agnostic, we can leverage these insights improve few-shot learning accuracies across many episodic training algorithms. We demonstrate the efficacy of our method across popular few-shot learning datasets, algorithms, network architectures, and protocols.
1 Introduction
Large amounts of high-quality data have been the key for the success of deep learning algorithms. Furthermore, factors such as data augmentation and sampling affect model performance significantly. Continuously collecting and curating data is resource (cost, time, storage, etc.) intensive process. Hence, recently, the machine learning community has been exploring methods for performing transfer-learnngg from large datasets to unseen tasks with limited data.
A popular genre of these approaches is called meta- learnig few-shot approaches, where, in addition to the limited data from the task interest, a large dataset disjoint tasks is available ppra-ntraining..
These approaches are prevalent in the area of computer vision (31 and reinforcement learning (6. key component of these methods is notion of episodic training, which refers to sampling tasks from the larger dataset for training. By learning to solve these tasks correctly, the model can generalize to new tasks.
However, sampling for episodic training remains surprisingly understudied despite numerous methods and applications that build on it. To the best of our knowledge, only handful of works (69 55
33 explicitly considered the consequences of sampling episodes. In comparison, stochastic 46 and mini-batch [4] sampling alternatives have been thoroughly analyzed from the perspectives of
optimization [1 ( information theory [27 91 and stochastic processes
[68 [66, among many
others. Building similar understanding of sampling for episodic training will help theoreticians and practitioners develop improved sampling schemes, and is thus of crucial importance to both.
In this paper, we explore many sampling schemes to understand their impact on few-shot methods. Our work revolves around the following fundamental question: what is the best way to sample episodes? Our focus will be restricted image classification the few-shot learning setting where "best" is taken to mean "higher transfer accuracy of unseen episodes" and leave analyses and applications to other areas for future work.
Equal contributions; work done while at Amazon Web Services.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Contrary to prior work, our experiments indicate that sampling uniformly with respect to episode difficulty yields higher classification accuracy a scheme originally proposed to regularize metric learning (62) To better understand these results, we take closer look at the properties of episodes and what makes them difficult. Building on this understanding, we propose method approximate different sampling schemes, and demonstrate its efficacy on several standard few-shot learning algorithms and datasets.
Concretely, we make the following contributions:
We provide detailed empirical analysis of episodes and their difficulty. When sampled randomly. we show that episode difficulty (approximately) follows normal distribution and that the difficulty of episode is largely independent of several modeling choices including the training algorithm, the network architecture, and the training iteration.
Leveraging our analysis, we propose simple and universally applicable modifications the episodic sampling pipeline to approximate any sampling scheme. We then use this scheme thoroughly compare episode sampling schemes including easy/hard-mining curriculum learning, and uniform sampling and report that sampling uniformly over episode difficulty yields the best results.
Finally, we show that sampling matters for few-shot classification as it improves transfer and state-of-tee of-the-art [64 algorithms on
accuracy for diverse of popular [53] 20 15
standard and cross-domaie benchmarks.
2 Preliminaries
2.1 Episodic sampling and training
We define episodic sampling as subsampling few-shot tasks (or episodes) from larger base dataset 77]. Assuming the base dataset admits generative distribution, we sample an episode in two steps First, we sample the episode classes C, from class distribution p(C,); second, we sample the episode data from data distribution p(z.y) y C,) conditioned on C, This gives rise the following big-iikeldood for model lo parameterized by A:
[()) E [loglo(T)] )~)(+)
where q(T) is the episode distribution induced by first sampling classes, then data. In practice, this expectation is approximated by sampling a batch of episodes each with their set TO of query samples. To enable transfer unseen classes, it is also common include small 73 of support samples to provide statistics about т. This results in the following Monte-Carke estimator:
(())
~
Lag xx Ts),
(2)
B
TEB
(x,w))TQ
where the data in TQ and Ts are both distributed according to p(x, y Cy). In few-shot classification, the n-wayy k-shot setting corresponds to sampling n = |C,l classes, each with A: = Iral support data points. The implicit assumption in the vast majority of few-shot methods that both classes and data are sampled with wniform probability - but are there better alternatives? We carefully examine these underlying assumptions in the forthcoming sections.
2.2 Few-shot algorithms
We briefly present few representative episodic learning algorithms. A more comprehensive treatment of few-shot algorithms is presented in Wang et al. (59] and Hospedales et al. [24]. A core question in few-shot learning lies in evaluating (and maximizing) the model likelihood lg. These algorithms can be divided in two major families: gradient-based methods, which adapt the model's parameters the episode; and metric-based methods, which compute similarities between support and query samples learned embedding space.
Gradient-based few-shot methods are best illustrated through Model-Agnostic Meta-Learning [[5] (MAML). The intuition behind MAML is to learn set of initial parameters which can quickly
We use the notation for a probability distribution and its probability density function interchangeably.

---

specialize to the task at hand. To that end, MAML computes la by adapting the model parameters 0 via one (or more) steps of gradient ascent and then computes the likelihood using the adapted parameters 0' Concretely, we first compute the likelihood po x) using the support set rs. adapt the model parameters, and then evaluate the likelihood:
Ly 2, TS) = por x) s.t. = aV
Σ (H.y)ETS
log po (y/ x),
where a > is known as the adaptation learning rate. A major drawback training with MAML lies in back-propagating through the adaptation phase, which requires higher-order gradients. To alleviate this computational burden, Almost No Inner Loop [4]] (ANIL) proposes to only adapt the last classification layer of the model architecture while tying the rest of the layers across episodes. They empirically demonstrate little classification accuracy drop while accelerating training times four-fold.
Akin to ANIL, metric-based methods also share most of their parameters across tasks; however, their aim is learn metric space where classes naturally cluster. To that end, metric-based algorithms learn a feature extractor do parameterized by 0 and classify according to a non-parametric rule. A representative of this family is Prototypical Network [53] (ProtoNet), which classifies query points according to their distance to class prototypes- - the average embedding of a class in the support set:
exp (-d(oo(x) ob))
lo I. T8) Eyec,, exp (dd(po(z).d))
S.I.
Σ
K
(x-Y)ETS
where d(.,-) is distance function such as the Euclidean distance or the negative cosine similarity, and o is the class prototype for class c. Other classification rules include support vector clustering [32
neighborhood component analysis (30], and the earth-mover distance [67].
2.3 Episode difficulty
Given an episode r and likelihood function lo, we define episode difficulty to be the negative log- likelihood incurred that episode:
Dr (T) log (o(T),
which is a surrogate for how hard it is to classify the samples in To correctly, given lo and Ts. By definition, this choice of episode difficulty is tied the choice of the likelihood function lg-
Dhillon et [TT use a similar surrogate as a means systematically report few-shot performances. We use this definition because it is equivalent to the loss associated with the likelihood function lo on episode T, which is readily available at training time.
Methodology
In this section, we describe the core assumptions and methodology used in our study of sampling methods for episodic training. Our proposed method builds on importance sampling [21] (IS) which found compelling for three reasons: (i) IS is well understood and solidly grounded from theoretical standpoint, (ii) IS is universally applicable thus compatible with all episodic training algorithms, and (iii) IS is simple to implement with little requirement for hyper-parameter tuning.
Why should we care about episodic sampling? A back-of-the the-theelloopee calculation suggests that there are on the order of 10 different training episodes for the smallest-scall experiments in Section 5 Since iterating through each of them infeasible, we ought to express some preference over which episodes to sample. In the following, we describe method that allows us to specify this preference.
3.1 Importance sampling for episodic training
Let us assume that the sampling scheme described in Section 2.1 induces distribution q(T) over episodes. We call it the proposal distribution, and assume knowledge of its density function. We wish
FFor a base dataset with N classes and K onput-output pairs per class, there are total of (M) episodes that can be created when sampling & pairs each from n classes.
(K) possible
3

---

Algorithm 1: Episodic training with Importance Sampling
Input: target (p) and proposal (ạ) distributions, likelihood function l0, optimizer OPT.
Randomly initialize model parameters 0. repeat
Sample a mini-batch B of episodes from ((T).
for each episode in mini-batch B do
Compute episode likelihood: (T).).
Compute importance weight: (((r) g(T))
p(r)
end for Aggregate: (()) + ErEs (T(r) log (T).
Compute effective sample size ESS
Update model parameters: +- OPT( (10)
ESS(B)
until parameters have converged.
to estimate the expectation in Eq. T when sampling episodes according to target distribution p(T) of our choice, rather than ((7). To that end, we simply re-weights the observed values for a given episode T by w(T) = a(r)*
importance sampling estimator which p(r) the ratio of the target
and proposal distributions:
E [log [o
= E T~a(())
[w((T log [o()]
r-p(-)
The importance sampling identity holds whenever (()) has non-zero density over the support of p(T). and effectively allows us to sample from any target distribution p(T).
A practical issue of the IS estimator arises when some values of w(T) become much larger than others; in that case, the likelihoods (T( associated with mini-batches containing heavier weights dominate the others, leading to disparities. To account for this effect, average in the Monte-Carle estimate of Eq. (2) by the effective sample size ESS( 29 34)
replace the mini-batch
1 ESS(B)
(ErEB w(T))² s.t. ESS(B) ΣtES w(T)²
[log la(r)] (-p())
Σ
(o()
(3)
where B denotes mini-batch of episodes sampled according to q(T). Note that when w(T) is constant, we recover the standard mini-batch average setting ESS(BB |B]. Empirically, we observed that normalizing with the effective sample size avoided instabilities. This method is summarized in Algorithm
3.2 Modeling the proposal distribution
priori, we do not have access to the proposal distribution q(7) (nor its density) and thus need to estimate it empirically. Our main assumption is that sampling episodes from 4(T) induces normal distribution over episode difficulty. With this assumption, we model the proposal distribution by this induced distribution, therefore replacing ((T) with (f4, (r) μu, oa) where μu, o² are the mean and variance parameters. As we will see in Section 5.2 this normality assumption is experimentally supported various datasets, algorithms, and architectures.
We consider two settings for the estimation of la and a²: offline and online. The offline setting consists of sampling 1, 000 training episodes before training, and computing a² using model pre-trained on the same base dataset. Though this setting seems unrealistic, i.e. having access to pre-trained model, several meta-learning few-shot methods start with pre-trained model which they further build upon. Hence, for such methods there is no overhead. For the online setting, we estimate the parameters on-the-fly using the model currently being trained. This is justified by the analysis in Section 5.2 which shows that episode difficulty transfers across model parameters during training. We update our estimates of μu, ²² with exponential moving average:
<- Ap + (1 x)04 (T) and o² + 2o + (1 2)(2, (7) =n)2
where [0, 1] controls the adjustment rate of the estimates, and the initial values of μ,, d² are computed in warm-up phase lasting 100 iterations. Keeping À 0.9 worked well for all our

---

experiments (Section 5p. We opted for this simple implementation as more sophisticated approaches like West [6 yielded little no benefit.
3.3 Modeling the target distribution
Similar to the proposal distribution, we model the target distribution by its induced distribution over episode difficulty. Our experiments compare four different approaches, all of which share parameters n4 o² with the normal model of the proposal distribution. For numerical stability, we truncate the support of all distributions [N- 2.58a, f + 2.58r], which gives approximately 99% coverage for the normal distribution centered around
The first approach (HARD) takes inspiration from hard negative mining [5T]. where we wish sample only more challenging episodes. The second approach (EASY) takes a similar view but instead only samples easier episodes. We can model both distributions follows:
(() I 14, l + 2.58o)
(HARD)
and
(()),(T) 1 /z 2.58a, 11)
(EASY)
where U denotes the uniform distribution. The third (CURRICULUM) is motivated by curriculum learning (2), which slowly increases the likelihood of sampling more difficult episodes:
(M4 (T) pe. pe.
(CURRICULUM)
where is linearly interpolated from 2.58o to + 2.58c as training progresses. Finally, our fourth approach, UNIFORM, resembles distance weighted sampling 62 and consists of sampling uniformly over episode difficulty:
((Qig(T) 100--2258,, 2.58o, μ 2.58g).
(UNIFORM)
Intuitively, UNIFORM can be understood uniform prior unseen test episodes. with the intention of performing well across the entire difficulty spectrum. This acts as a regularizer, forcing the model to be equally discriminative for both easy and hard episodes.
4 Related Works
This paper studies task sampling in the context of few-shot 36 14 and meta-learning
56
Few-shot learning.
This setting has received a lot of attention over recent years 58
43
Broadly speaking, toite--o-thh-aree art methods can be categorized in two major families: metric- based and gradient-basse based.
Metric-based methods learn a shared feature extractor which is used to compute the distance be- tween samples in embedding space 53
30 The choice of metric mostly differentiates
one method from another; for example, popular choices include Fuclidean distance [53]. negative
cosine similarity (20), support vector machines [32 set-to-set functions (64]. or the earth-mover
distance (67].
Gradient-based algorithms such as MAML 15, propose an objective to learn a network initialization that can quickly adapt to new tasks. Due to its minimal assumptions, MAML has been extended to probabilistic formulations 22 o5 to incorporate learned optimizers implicit [16] or explicit [40]
and simplified avoid expensive second-order computations [3 42 In that line of work, ANIL 41
claims to match MAML's performance when adapting only the last classification layer thus greatly reducing the computational burden and bringing gradient and metric-based methods closer together.
Sampling strategies. Sampling strategies have been studied for different training regimes. Wu al. (22] demonstrate that "sampling matters" in the context of metric learning. They propose to sample a triplet with probability proportional to the distance of its positive and negative samples, and observe stabilized training and improved accuracy. This observation was echoed by Katharopoulos and Fleuret 27 when sampling mini-batches: carefully choosing the constituent samples of mini- batch improves the convergence rate and asymptotic performance. Like ours, their method builds on importance sampling
[52] [12 26 but whereas they compute importance weights using the magnitude
5

---

of the model's gradients, we use the episode's difficulty. Similar insights were also observed in reinforcement learning, where Schaul et al. 48 suggests a scheme to sample transitions according to the temporal difference error.
Closer work, Sun et al. 55 present hard-mining scheme where the most challenging classes across episodes are pooled together and used to create new episodes. Observing that the difficulty of class is intrinsically linked to the other classes in the episode, Liu al. [33] propose mechanism to track the difficulty across every class pair. They use this mechanism to build a curriculum 2 63
of increasingly difficult episodes. In contrast these two approaches, our proposed method makes use of importance sampling to mimic the target distribution rather than sampling from it directly. This helps achieve fast and efficient sampling without any preprocessing requirements.
Experiments
We first validate the assumptions underlying proposed IS estimator and shed light on the properties of episode difficulty. Then, we answer the question we pose in the introduction, namely: what is the best way sample episodes? Finally, we ask if better sampling improves few-shot classification.
5.1 Experimental setup
We review the standardized few-shot benchmarks and provide detailed description in Appendix A
Datasets. We use two standardized image classification datasets, Mini-ImageNNt 58 and Tiered- ImageNet (45), both subsets of ImageNet [10]. Mini-ImageNet consists of 64 classes for training, 16 for validation, and for testing, we use the class splits introduced by Ravi and Larochelle [43].
Tiered-ImageNy contains 608 classes split into 351, 97, and 160 for training, validation, and testing, respectively.
Network architectures, We train two model architectures. A 4-layer convolution network conv(64), Vinyals et al. (58) with 64 channels per layer. And ResNet-12, 12-layer deep residual network (23] introduced by Oreshkin et al. [39]. Both architectures use batch normalization 25 after every convolutional layer and ReLU as the non-linearity.
Training algorithms. For the metric-based family, we ProtoNet with Euclidean 53 and sealed
negative cosine similarity measures (20). Additionally, we use MAML [15 and ANIL 41 as representative gradient-based algorithms.
We tune hyper-parameee for each algorithm and dataset to work well across different few-shot settings and network architectures. Additionally, we keep the hyper-parameters the same across all different sampling methods for a fair comparison. We train for 20k iterations with mini-batch of size 16 and 32 for Mini-ImageNec and Tiered-ImageNet respectively, and validate every 1k iterations on 1k episodes. The best performing model is finally evaluated on 1k test episodes.
5.2
Understanding episode difficulty
All the models in this subsection are trained using baseline sampling as described in Section 2.1 iee. episodic training without importance sampling.
5.2.1 Episode difficulty is approximately normally distributed
We begin our analysis by verifying that the distribution over episode difficulty induced by ((T) is approximately normal. In Fig. we use the difficulty of 10k test episodes sampled with q(T). The difficulties are computed using conv(64), trained with ProtoNet and MAML on Mini-ImageNNet for 1-shot 5-way classification. The episode difficulty density plots follow bell curve, which naturally modeled with a normal distribution. The Q-Q plots, typically used to assess normality, suggest the same the closer the curve the identity line, the closer the distribution is to normal.
Finally, we compute the Shapiro- -Wilk test for normality (50), which tests for the null hypothesis that the data is drawn from normal distribution. Since the valuee for this test is sensitive to the

---

0.8
2.0
Sample quantiees
A Algorithim Prototyoical Network [cosine)
1.5
Aigorithm
0.4
Protatypical Network (cosine)
1.0
0.0
2
Dificcltty
Thecretical quantiles
Figure Episode difficulty approximately normally distributed. Density (left) and Q-Q (right) plots of the episode difficulty computed by conv(64), 's on Mini- ImageNet (1-shot 5-way), trained using ProtoNets (cosine) and MAML (depicted in the legends). The values are computed over 10k test episodes. The density plots follow bell curve, with the density peak in the middle, which quickly drops-off on either side of the peak. The Q-Q plots close to the identity line (in black). The closer the curve the identity line, the closer the distribution to normal. Both suggest that episode difficulty distribution can be normally approximated.
sample size we subsample 50 values 100 times and average rejection rates over these subsets. With 0.05, the null hypothesis is rejected 14% and 17% the time for Mini Mini-Im and Tiered- Ieeeed-mageeNet respectively, thus suggesting that episode difficulty can be reliably approximated with normal distribution.
5.2.2 Independence from modeling choices
By definition, the notion of episode difficulty is tightly coupled the model likelihood le (Section 2.3 and hence to the modeling variables such as learning algorithm, network architecture, and model parameters. We check if episode difficulty transfers across different choices for these variables.
We are concerned with the relative ranking of the episode difficulty and not the actual values. this end, we will use the Spearman rank-order cor- relation coefficient, a parammaeetrric measure of the monotonicity of the re- lationship between two sets of values. This value lies within [-1; 1]. with 0 implying no correlation, and +1 and -1 implying exact positive and nega- tive correlations, respectively.
Training algorithm. We first check the dependence on the training algo- rithm. We use all four algorithms to train conv(64), 's for 1-shot 5-way classification on Mini- ImageNe,,,,,, then compute episode difficulty over 10k test episodes. The Spearman rank- order correlation coefficients for the difficulty values computed with re- spect to all possible pairs of training algorithms are 0.65. This positive correlation is illustrated in Fig. and suggests that an episode that is diffi- cult for one training algorithm is very likely to be difficult for another.
1.
1.8
Iesshppial Ndwwer
MAN
ANL
Figure 2: Episode difficulty transfers across training algo- rithms. Scatter plots (with regression lines) of the episode difficulty computed 1k ImageNe test episodes (1-shot 5-way) by conv(64)a's trained using different algorithms. The positive cor- relation suggests that an episode that difficult for one training algorithm will be difficult for another.
Next analyze the dependence the network architecture. We trained conv(64), and ResNet-12's using all training algorithms for Mini-ImageNett 1-shot 5-way classi-
Network architecture.
For large sample size, the values reliable they may detect trivial departures from normality.
7

---

Protstypical Network
MAHL
Prosstestcll Neww coulnc
NANL
2,00
1,75
2.00
1.75
1.50
1.75
1:50
Epreedo tepo
1.25
1.50
1.35
Easy
Hard
1.00
1.25
1.00
0.75
1.0
2.0
1.5
5950
15:00 20003 Algortfmm
20005
5035
Algorfam
20003 25000
22000
RecMee-11
Reshwes12
Figure 3: Episode difficulty transfers across network architectures. Scatter-plots (with regres- sion lines) the episode difficulty computed by conv(64), and ResNet-12's trained using different algorithms. This is computed for 10k 1-shot 5-way test episodes from Mini-ImageNet. We observe strong positive correlation between the computed values for both network architectures.
Figure 4: Episode difficulty is transferred across model parameters during training. We select the 50 casiest and hardest episodes and track their difficulty dur- ing training. This is done for conv(64),'' 's trained on Mini- ImageNet (1-shot 5-way) with different algorithms. The average difficulty of the episodes decreases over time, until convergence (vertical line), after which the model overfits. Additionally, easier episodes remain easy while harder episodes remain hard, indicating that episode diffi- culty transfers from one set of parameters to the next.
fication. We compute the episode difficulties for 10k test episodes and compute their Spearman rank-order correlation coefficients across the two architectures, for a given algorithm. The correlation coefficients are 0.57 for ProtoNet (Euclidean), 0.72 for ProtoNet (cosine), 0.58 for MAMI, and 0.49 for ANIL. Fig. 3 illustraaess this positive correlation, suggesting that episode difficulty is transferred across network architectures with high probability.
Model parameters during training. Lastly, study the dependence model parameters during training. We select the 50 easies/ and 50 hardest episodes, ie., episodes with the lowest and highest difficulties respectively, from 1k test episodes. We track the episode diffieulty for all episodes over the training phase and visualize the trend in Fig. E for conv(64)4's trained using different training algorithms Mini-ImageNet (1-shot 5-way). Throughout training, easy episodes remain easy and hard episodes remain hard, hence suggesting that episode difficulty transfers across different model parameters during training. Since the episode difficulty does not change drastically during the training process, we can estimate it with a running average over the training iterations. This justifies the online modeling of the proposal distribution in Section 3.2
Table 1: Few-shot accuracies on benchmark datasets for 5-way few-shot episodes in the offline setting, Mean accuracy and 95% confidence interval computed over 1K test episodes. The first row in every scenario denotes baseline sampling. Best results for fixed scenario are shown in bold and the t indicates matching improving over baseline sampling.
Mini- Miae-Ema
Tiered-ImageNet
c6nv(6)1)a
ResNet-12
ResNet112
1-shot (%)
5-shot (%)
I-shot (%)
5-shot (%)
1-shot (%)
5-shot (%)
ProtoNet (cosine)
50.03-+0.61
52.85 01.56++0.53
60.01-0.73 62.11+
72.75++0.59
+ EASS
49.60:0.611
53.35-10.33* 65.17
63.55
60.03110.55
74.65
+ HARD
49.01=t0.60
66.45-10.50
52.65-:0.63*
70.15=t0.51*
55.44=0.72
75.97=0.555*
+ CURRICULUM
49.38=+0.61
64. 0..53t
53.21+0065t
65.89=0.52*
60.37=10.76
75.32:10.88
+ UNIFORM
50.07:10.59
66.52
54.27
70.85
60.27
7..36....4
5.3 Comparing episode sampling methods
We compare different methods for episode sampling. To ensure fair comparisons, we the offline formulation (Section 3.2) so that all sampling methods share the same pre-trained network (the network trained using baseline sampling) when computing proposal likelihoods. We compute results over 2 datasets, 2 network architectures, 4 algorithms and 2 few-shot protocols, totaling in 24 scenarios. Table presents results ProtoNet (cosine), while the rest are in Appendix D
We observe that, although not strictly dominant, UNIFORM tends to outperform other methods as it is within the statistical confidence of the best method in 19/2 scenarios. For the 5/24 scenarios
8

---

Table 2: Few-shot accuracies on benchmark datasets for 5-way few-shot episodes in the offline and online settings. The mean accuracy and the 95% confidence interval are reported for evaluation done over 1k test episodes. The first row in every scenario denotes baseline sampling. Best results for fixed scenario are shown in bold. Results where sampling technique is better than or comparable to baseline sampling are denoted by i. UNIFORM (Online) retains most of the performance of the offiin formulation while being significantly easier to implement (online competitive in 15/24 scenarios vs 16/24 for offline).
Mini-ImageNet
Tiered-Imagele
c6nv(64),
ResNet-12
ResNet-12
1-shot (%)
5-shot (%)
1-shot (%)
5-shot (%)
1-shot (%)
5-shot (%)
ProtoNet (Euclidean)
49.06=0060
6..2
49.67=00.4
59.10=0.733 6..45=+0.51
06595=0.56
+ UNIFORM (Offline)
18.19=00.62
66.73=+0:52'
53.94=0.63'
70.79=0.49"
58.63+0,76'
78.52
UNIFORM (Online)
48.39:10.62
67.86110.50'
52.97:1.644'
0..63
59.6711..70'
78.73
ProtoNet (cosine)
50.03.10.61
61.56110.33
52.85110.64
62.11110.52
60.01
72.75.1..59
+ UNIFORM (Offline)
50.07.10.591
66.33110.52
54.2710.65'
70.8510.511
60.2710.75
78.36:10,54*
UNIFORM (Online)
50.06:+0.61
65.99:+0.521
53.90=0.63"
68.78+00.11
1
61.37+0072
77.81+0..6''
MAML
46.88=0.60
55.16=0.55
49.92++0.65
63.93++0.59
55.37=+077
72.93=0.60
+ UNIFORM (Offline)
46.67.10.63/
62.09-0.555
52.65-0.655'
66.7611..57
54.58110.77
72.00-10.66
UNIFORM (Online)
46.70.10.61
61.62:10.54
51.17=0.681
65.63
57.15:10.74
71.67-0.67
ANIL
46.59=0.60
63.47+0.55
49.65++0.65
59.51+0.56
54.77++0.76
69.28+0067
UNIFORM (Offline)
46.93-1-0.21
6..75=0.60
09.56=00621
6..72
54.15
0.644
UNIFORM (Online)
46.82:10.63
62.63=0.59
49.82=00688'
55.18:10.74' 6..51=0.62
69.751
where UNIFORM underperforms, it closely trails behind the best methods. Compared to baseline sampling, the average degradation of UNIFORM is -0.83% and at most -1.44% (ignoring the standard deviations) in 4/24 scenarios. Conversely, UNIFORM boosts accuracy by much as 8.74% and on average by 3.86% (ignoring the standard deviations) in 13/24 scenarios. We attribute this overall good performance to the fact that uniform sampling puts uniform distribution prior over the (unseen) test episodes, with the intention of performing well across the entire difficulty spectrum. This acts regularizer, forcing the model be equally discriminative for easy and hard episodes. If we knew the test episode distribution, upweighting episodes that are most likely under that distribution will improve transfer accuracy [13.. However, this uninformative prior is the safest choice without additional information about the test episodes.
Second best is baseline sampling it is statistically competitive on 10/24 scenarios, while EASY, HARD, and CURRICULUM only appear among the better methods in 4, 4, and 9 scenarios, respectively.
5.4 Online approximation of the proposal distribution
Although the offline formulation is better suited for analysis experiments, it is expensive as requires a pre-training phase for the proposal network and 2 forward passes during episodic training (one for the episode loss, another for the proposal density). In this subsection, we show that the online formulation faithfully approximates offline sampling and can retain most of the performance improvements from UNIFORM. We take the same 24 scenarios as in the previous subsection, and compare baseline sampling against offline and online UNIFORM. Table 2 reports the full suite of results.
We observe that haseline is statistically competitive on 9/24 scenarios; on the other hand, offline and online UNIFORM perform similarly on aggregate, as they are within the best results in 16/2: 24 and 15/24 scenarios respectively. Similar to its offline counterpart, online UNIFORM does better than or comparable to baseline sampling in 21 out of 24 scenarios. On the 3/24 scenarios where online UNIFORM underperforms compared to baseline, the average degradation is -0.92%, and at most -1.26% (ignoring the standard deviations). Conversely, in the remaining scenarios, it boosts accuracy by as much as 6.67% and on average by 2.24% (ignoring the standard deviations). Therefore, using online UNIFORM, while computationally comparable to baseline sampling, results in a boost in few-shot performance; when it underperforms, it trails closely. We also compute the mean accuracy difference between the offline and online formulation, which is 0.07% 0.35 accuracy points. This confirms that both the offline and online methods produce quantitatively similar outcomes.

---

Few-shot accuracies for 5-way cross-domain few-shot episodes after training on Mini-ImageNet. The mean accuracy and the 95% confidence interval are reported for evaluation done over 1k test episodes. The first row in every scenario denotes baseline sampling. Best results for a fixed scenario are shown in bold.
Table 3:
CUB-200
Describable Textures
ResNet-12
conv(64),
1-shot (%)
5-shot (%)
1-shot (%)
5-shot (%)
ProtoNet (cosine)
38.67:10.60
49.75
32.09-0.45
38.44.10.41
+ UNIFORM (Online)
40.55+0060
56.30=0.55
33.63=0.47
43.28++0.44
MAML
35.80:10.56
45.16.10.22
29.47-0.46
37.85110.47
UNIFORM (Online)
37.18110.55
46.58 0.58
31.84-0.4
40.81110.44
5.5 Better sampling improves cross-domain transfer
To further validate the role of episode sampling as a way to improve generalization, we evaluate the models trained in the previous subsection episodes from completely different domains. Specifically, we use the models trained on Mini- Mini-ImageNet with baseline and online UNIFORM sampling to evaluate on the test episodes of CUB CUB-200 [60], Describable Textures (8). FGVC Aircrafts [35], and VGG
Flowers [38]. following the splits of Triantafillou et al. [57]. Table 3 displays results for ProtoNet (cosine) and MAML on CUB-20 and Describable Textures, with the complete set of experiments available in Appendix(F] Out of the 64 total cross- scenarios, online UNIFORM does statistically better in 49/ /64 scenarios, comparable in 12/64 scenarios and worse in only 3/64 scenarios. These results further go to show that sampling matters in episodic training.
Table 4: Few-shot accuracies on benchmark datasets for 5-way few-shot episodes using FEAT. The mean accuracy and the 95% confidence interval are reported for evaluation done over 10k test episodes with ResNet- 12. The first in every scenario denotes baseline sampling. Best results for fixed scenario shown in bold. UNIFORM (Online) improves FEAT's accuracy in 3/1 scenarios, demonstrating that sampling matters even for state-of-the-ane art few-shot methods.
Mini-ImageNNtt
Tiered-ImageNee
1-shot (%)
5-shot (%)
1-shot (%)
5-shot (%)
FEAT UNIFORM (Online) 66.27-+0.20
66.02.10.20
81.1710.14 01.54+0.14
84.26.10. 7.25
70.61=0.23 84.42++0.16
5.6 Better sampling improves few-shot classification
The results in the previous subsections suggest that online UNIFORM yields a simple and universally applicable method to improve episode sampling. To validate that state-of-the-ard methods can also benefit from better sampling, we take the recently proposed FEAT (44] algorithm and augment with our IS IS-based implementation of online UNIFORM. Concretely, we use their open-source implementationf] to train both baseline and online UNIFORM sampling. We use the prescribed hyper-paree ppper-parametess without any modifications. Results for ResNet- 12 on Mini- Minn-ImmeeNet and Tiered- ImageNet are reported in Table where online UNIFORM outperforms baseline sampling on 3/4 scenarios and matched on the remaining one. Thus, better episodic sampling can improve few-shot classification even for the very best methods.
6 Conclusion
This manuscript presents a careful study of sampling in the context of few-shot learning, with an eye on episodes and their difficulty. Following empirical study of episode difficulty, propose an importance sampling-base based method to compare different episode sampling schemes. Our experiments suggest that sampling uniformly over episode difficulty performs best across datasets, training algorithms, network architectures and few-shot protocols. Avenues for future work include devising better sampling strategies, analysis beyond few-shot classification (e.g., regression, reinforcement learning), and a theoretical grounding explaining our observations.
Available at: https://github..c hctt:://githaa..oo//hhaaaaaaaaa
10

---

References
[1] S. M. R. Amold, P. Mahajan, D. Datta, I. Bunner, and K. S. Zarkias. leam2learn: A library for Mcta- Learning research. Aug. 2020. http:///:/aaxx
aba./200..222 12284
[2] Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pages 41 48, New York, NY. USA, June
2009. Association for Computing Machinery. URL htRt:tttp:////..oooo 10. 1145/1563374, 1553380
[3] L. Bertinetto, F. Henriques, P. Torr, and A. Vedaldi. Meta-lcaming with differentiable closed-form solvers. Sept. 2018. URL
[4] D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynann programming. 27(6), Jan. 1996. URL https
// /www. www..rewn www.erecgaateeeeeee.reee.eeeeeee not/
Dwn.miinnnnnnne P_ooorrammingg
[5] I. Bottou, F. F. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning. June 2016.
URL http:://arxy org/ abs/1606. 04838
[6] Botvinick, S. Ritter, J. Wang, Z. Kurth-Nelson, Blundell, and Hassabis. Reinforcement learning, fast and slow. Trends in cognitive sciences, 23(5):488-477 2019.
[7] W.-L. Chao, H.-J Ye, D.-C. Zhan, M. Camphell, and K. Q. Weinberger. Revisiting meta-learning as supervised learning. arXiv preprint arXiv:2002.00557 2020.
[8] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[9] D. Csiba and Richtárik. Importance sampling for minibatches. J. Mach. Learn. Res., 19(27)1121, 2018. URL pett:://jmlz org pttppppppp//iiiaaaaaaaaeap 241.2241tm html
[10] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hicrarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iccc, 2009.
[11] G. S. Dhillon, P. Chaudhari, A. Ravichandran, and S. Soatto. A baseline for few-shot image classification. In International Conference Learning Representations, 2019.
[12] A. Doucet, N. d. Freitas, and N. Gordon, editors. Sequential Monte Carlo Methods in Practice. Springer,
New York, NY, 2001. URL hattps://iink springer.rrngggr....mmm com/ book/10. 1007/978-1-4457-344
[13] A. Fallah, A. Mokhtari, and A. Ozdaglar. Generalization of model-agnostic meta-learning algorithms: Recurring and unseen tasks, arXiv preprint arXiv:2102.03832, 2021.
[14] L. Fei-Fei, R. Fergus, and P. Perona. One-shot learning of object categories. IEEE Trans. Pattern Anal. Mach. Intell., 28(4):594-611 Apr. 2006. URL http:: doi http:///xxdoi.ooo///////
109/TPAMI 2006. 79
[15] C. Finn, P. Abbeel, and Levine. agdel-agnostic meta- meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126- 1135. PMLR, 2017.
[16] S. Flennerhag, A. A. Rusu, R. Pascanu, H. Yin, and R. Hadsell. Mcta-I with warped gradient descent. Aug. 2019. URI http:://aaar wow..oggaaaaa....... abs/ 1909. 00025
[17] M. P. Friedlander and M. Schmidt. Hybrid Seteminnistie-Stohhaeeee methods for data fitting. SIAM J. Sci. Comput., 34(3):A1 (000:-1133--A111 A1405, Jan. 2012. |RL http////oo.ooo//////////o 10. 1117 111/100836699
[18] M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. Rezende, and M. Eslami. Conditional neural processes. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 1704-1713, Stockholmsmässann Stockholm Sweden, 2018. PMLR. URL http://proceedingg mlr, wwtpppppwssseppppppppppppppppeet wwt.rr///gaaaaelaaae.mm html
[19] G. Ghiasi, T.-Y. Lin, and Q. V. Le. Dropblock: a regularization method for convolutional networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 10750-10760, 2018.
[20] S. Gidaris and N. Komodakis. Dynamic few shot visual learning without forgetting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognation, pages 4367-4375, 2018.
[21] P. Glasserman. Monte Carlo methods in financial engineering. Springer, New York, 2004. ISBN 0387004513 9780387004518 1441918221 9781441918222.
[22] E. Grant, C. Finn, S. Levine, T. Darrell, and T. Grifliths. Recasting Gradient- tradient-Based Meta-Learning as hierarchical bayes. Jan. 2018. URI. URI.tttp: pettt:///rxxt //tp///xxvvvvv www.:ogg/aaao arg//bs//800/ 1801/180...oo 009930
[23] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[24] T. Hospedales, A. Antoniou, P. Micaelli, and others. Meta- in neural networks: A survey. arXiv preprint arXiv, 2020. https:/ // //trxi//vvv////////i//ee////aall www//bog//////eeeeeeoo abt//////////...//////////////// 05439
11

---

[25] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
[26] T. B. Johnson and C. Guestrin. Training deep models faster with robust, approximate importance sampling. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 7265-7275. Curran Associates, Inc., 2018.
[27] A. Katharopoulos and F. Fleuret. Not all samples are created equal: Deep learning with importance sampling. Mar. 2018. http:ttttt://tiiiee org/ /bw//803......... 00942
[28] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
[29] A. Kong. A note on importance sampling using standardized weights. University of Chicago, Dept. of Statistics, Tech. Rep, 348, 1992.
[30] S. Laenen and L. Bertinetto. On episodes, prototypical networks, and few-shot leaming. Dec. 2020. URL http://arxiy,
wwrggabs//2111 2012. 2b0200000
[31] B. M. Lake, T. D. Ullman, J. B. Tenenhaum, and S. J. Gershman. Building machines that learn and think like people. Behavioral and bruin sciences, 40, 2017.
[32] K. Lee, S. Maji, A. Ravichandran, and S. Soatto. Mcta-lcarning with differentiable convex optimization. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10657-
10665, 2019.
[33] C. Liu. Z. Wang, Sahoo, Y. Fang, K. Zhang, and S. C. H. Hoi. Adaptive task sampling for Meta- LetanLea July 2020. URL/tttp://arxtt org/ abs/2077 08735
[34] J. S. Liu. Metropolized independent sampling with comparisons to rejection sampling and importance sampling. Statistics and computing, 6(2): 6(2):113-119) 19, 1996.
[35] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint /30.v5. 2013.
[36] E. G. Miller, N. E. Matsakis, and P. A. Viola. Learning from one example through shared densities on transforms. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. NR.0R066,,), volume 1, pages 464-471 471 vol.1, June 2000. URL http::/dxxx doi. org/10. 10. 1109/ CVPR. 2000. 855856
[37] A. Nichol, J. Achiam, and J. Schulman, On First-Order Meta-Learning algorithms. Mar. 2018. URL http://arxiv. org/ /10s///3...aaaaa/ 02999
[38] M.-E. Nilsback and A. Zisserman. visual vocabulary for flower classification. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '06), volume 2, pages 1447-1454. IEEE, 2006.
[39] B. N. Oreshkin, P. Rodriguez, and A. Lacoste. Tadam: task dependent adaptive metric for improved few-shot learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 719-729, 2018.
[40] E. Park and J. B. Oliva. Meta-Curvature. Feb. 2019.
[41] A. Raghu, M. Raghu, Bengio, and Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. International Conference on Learning Representations, 2019.
[42] A. Rajeswaran, C. Finn, S. Kakade, and S. Levine. Meta-Learning with implicit gradients. Sept. 2019. URL /Rtp:pt:://axx org/ httg//bs/1990 1atps/999990000000
[43] S. Ravi and H. I. Larochelle. Optimization as model for few-shot learning. 2016.
[44] A. Ravichandran, R. Bhotika, and S. Soatto. Few-Shot lcarning with embedded class models and Shot-Free meta training. May 2019. http://t:///rtr http:://arxiv.orr/abb/// 1905. 043998
[45] Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel. Meta-learning for semi-supervised few-shot classification. In International Conference on Learning Representations, 2018.
[46] H. Robbins and S. Monro. A stochastic approximation method. Ann. Math. Stat., 22(3):400440) Sept. org/ euclid. aoms hooms/117779588
1951. http:ttttp:/ppooecceeeee
[47] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-Learning with Memory- Augmented neural networks. In M. F. Balcan and K. Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1842-1850, New York, New York, USA, 2016. PMLR. TRL. mlr . html
[48] T. Schaul, J. Quan, 1. Antonoglou, and D. Silver. Prioritized experience replay. Nov. 2015. URL http://arxiv, wwwprrxvvvorgg/bbe ab111111//111....1/.1 05952
12

---

[49] J. Schmidhuber. Evolationary Principles in Self-Referentic Learning. PhD thesis. 1987. URL http: //people. /pppppe..daia.......... ch/ wtth://ergee/dipll..ooooo
[50] S. S. Shapiro and M. B. Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52(3/4):591-66 611. 1965.
[51] A. Shrivastava, A. Gupta, and R. Girshick. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 761-769, 2016.
[52] P.
Smith, M. Shafi, and Hongsheng Gao. Quick simulation: a review of importance sampling techniques in communications systems. IEEE J. Sel. Areas Commun., 15(4):597-611 May 1997. URL http'
/ .o..di..oo www.ooooo......o.o 1110/99..550 585771
[53] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Proceedings of the 31tt International Conference on Neural Information Processing Systems, pages 4080-4090, 2017.
[54] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()):1929-1958 2014.
[55] Q. Sun, Y. Liu, Z. Chen, T.-S. Chua, and B. Schiele. Meta- Meta-Transfer learning through hard tasks. Oct. 2019. /tttt:///axxiiii org/ abs/1910. 03648
[56] S. Thrun and L. Pratt, editors. Learning to Learn. Kluwer Academic Publishers, Norwell, MA, USA, 1998. URL //ttps://d1. acm. org/citation. cfm?id-296635
[57] E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, U. Evci, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol, and H. Larochelle. Meta-datased A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2020. URL http:stts////otrreee// pet/forum"Id--kkgAG
[58] O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukeuoglu, and D. Wierstra. Matching networks for one shot lcarning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 3637-3645, 2016.
[59] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni. Generalizing from few examples: A survey on few-shot learning. ACM Comput. Surv., 53(3):1-34, June 2020. URL httpp:://oi..orgg///eeoo/ 1145/3386252
[60] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-ucsd birds 200.
2010.
[61] D. H. D. West. Updating mean and variance estimates: An improved method. Commun. ACM, 22(9): 532-535, Sept. 1979. ISSN 0001-0782. doi: 10. 164.149411144.359533. URL https://ddi..rrg
1145/359146.355 359155
[62] C. Wu, R. Manmatha, A. J. Smola, and P. Krähenbühl. Sampling matters in deep embedding learning. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2859-28880 1159,,267,.- Oct. 2017. URL
oto//o..11//0000 1ww19/II9IICv.. http//CCC.......... abs/2012.0311
http://dx.ce
[63] X. Wu, E. Dyer, and B. Neyshabur, When do curricula work? Dec. 2020. URL http:://///axii /ttp::///xxvvvog//////
[64] H.J. Ye, H. Hu, D.-C. Zhan, and F. Sha. Few-shot learning via embedding adaptation with set-to- set functions. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8808-8817, 2020.
[65] J. Yoon, T. Kim, O. Dia, S. Kim, Y. Bengio, and S. Ahn. Bayesian Model-Agnostic Meta- Learning. In S. Bengio, H. Wallach, H. I Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett, editors, Advances in Neurul Information Processing Systems. volume 31, pages 7332-7344 Curran Associates, Inc., 2018 URL https://proceeddnng wwwpeeeeddnnn...urpps..ee.eeeeee www.rpp..c/ppppp..ee.eeee / 2tt////1///f//
pdf
[66] C. Zhang, C. Öztireli S. Mandt, and G. Salvi. Active mini-batch sampling using repulsive point processes.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5741-577 57488 2019.
[67] C. Zhang, Y. Cai, G. Lin, and C. Shen. Deepemd: Few-shot image classification with differentiable earth mover's 's distance and structured classifiers. In Proceedings of the IEFF//VE Conference on Computer Vision and Pattern Recognition, pages 2220--1221, 2020.
[68] P. Zhao and T. Zhang. Accelorating minibatch stochastic gradient descent using stratified sampling. May 2014. Utthttt::/aaae
org/abs//400 1/////115.... 30800
[69] Y. Zhou, Y. Wang, J. Cai, Y. Zhou, Q. Hu, and W. Wang. Expert training: Task hardness aware Meta- Learning for Sew-Shot classification. July 2020. http:/ttp://arii wwwp:////g//..oo....///
13


Reviews:
[{"review_id": "uxQTTlpZXia", "reviewer": "Reviewer_KvDG", "summary": "Meta-learning (few-shot learning) algorithms, in general, equally treat all tasks for the training process. Tasks are sampled uniformly at random. This paper proposes a novel meta-learning framework that controls the task sampling probabilities according to difficulties. This paper measures the difficulties of tasks using the negative log-likelihood and leverages the difficulties to introduce different weights to tasks. The main idea of the learning framework is that the prior of the difficulties follows a normal distribution while the target sampling distribution could be uniform or some other interesting forms. Then, important sampling provides different weights to different tasks. From experiments, the uniform target distribution shows good results. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nThe paper proposes a novel task sampling framework for meta-learning, which is, indeed, a task weighting method considering task difficulties. The authors should discuss other existing task weighting methods. \n\nQuality: \nThe proposed episodic training method is intuitive, and the performance looks good.\n\nThe proposed algorithm is evaluated with ProtoNet, MAML, and ANIL. \n\nAlthough the training algorithms are well-known algorithms, they are not the state-of-the-art. Since one of the claims is that the proposed method is algorithm agnostic, it is necessary to perform experiments with recent training algorithms. \n\nThe authors only consider Mini-ImageNet and Tiered-ImageNet for the evaluation. In meta-learning research, specific data sets such as CUB and Cars are also used frequently, and cross-domain tasks are also important. More experiments with various data sets would be very useful.\n\nAlthough the performance shows great results, it is not clear why uniform sampling outperforms other sampling distributions. The choice of the negative log-likelihood as the difficulty measure also is required to be justified. What will happen when we use other difficulty measures?\n\nClarity: \nThis paper is easy to read.\n\nSignificance: \nThis paper proposes a novel episodic learning framework. The idea could be used to build other research results.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper did not properly discuss their limitation. Please refer to the main review comments. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15"}, "extra_text": {}}, {"review_id": "mvURwsxlQBm", "reviewer": "Reviewer_3S5L", "summary": "This paper investigates the effect of episode difficulty and sampling schemes on the performance of few-shot classification algorithms. It defines difficulty based on the negative log predictive likelihood of the few-shot learner and proposes to use importance sampling to mimic various sampling strategies. Experiments demonstrate that episodic difficulty is (a) approximately normally distributed, (b) tends to transfer across architectures and learning algorithms, (c) tends to be consistent over the course of training, and (d) that a uniform sampling scheme over difficulty tends to produce the best results. An online scheme for computing importance weights is also proposed.\n\n\n=== Post-rebuttal ===\n\nI have read the authors' response and other reviews and will maintain my original rating.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Episodic training underpins many few-shot learning algorithms and thus understanding it well is a key concern. For this reason, the results presented in this paper will be of broad interest to the few-shot learning community. The proposed online uniform difficulty sampling scheme is simple and therefore should be relatively easy to implement.\n\nThe paper is very well-written and easy to read. Experiments are systematic and easily interpreted. Though the performance gains from uniform difficulty sampling are not always very large, it does appear to be consistently better than other approaches.\n\nI would have liked to see a discussion of future extensions, for example training a network to directly estimate sample difficulty or possibly a policy network that outputs the importance weight to be assigned to an episode. Also it would be good to understand what the authors tried that did not work. For example, were other measures of episode difficulty such as accuracy considered? Were other sampling schemes tried and discarded?\n\nOverall I believe this is a strong paper and that others will benefit from reading it.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This work proposes a general purpose improvement to episodically trained few-shot learning algorithm and therefore the same societal implications that result from few-shot learning also apply here.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "lsEd3I-Pac", "reviewer": "Reviewer_sGro", "summary": "This paper explores episode/task difficulty in few-shot learning. The task difficulty is defined as the negative log-likelihood over the query set given the support set and model parameters. The authors find evidence that episode difficulty likely follows a normal distribution and proposes a new method, called UNIFORM, based on importance sampling. The new method achieves similar or better performance than other sampling procedures (EASY, HARD, and CURRICULUM). The paper also presents two variants of UNIFORM: offline and online. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\\+ means strength, \\- means weakness\n\n**Originality**\n\n\\+ *Interesting and novel insights into the task difficulty for meta-learners.* Although the idea of controlling episode difficulty is not very novel [47, 28], the authors go about it in a novel way and propose to use importance sampling for a faster and more efficient way of sampling episodes that is model agnostic. \n\n**Quality**\n\n\\+ *Solid evaluation and analysis.* Testing over multiple models, algorithms, and benchmarks makes for a convincing set of results.\n\n\\+ *Confirmed assumptions through experimentation.* The paper uses Q-Q plots and Shapiro-Wilk to verify the assumption that episode difficulty follows a normal distribution. \n\n\\- *Advantage of UNIFORM over other procedures is not consistent.* The tables show that UNIFORM does not always offer a clear advantage over the results, especially in the 1-shot setting. Do the authors have a theory for why the method is not as effective on the 1 shot setting?\n\n**Clarity**\n\n\\+ *Experiments are well designed, and the results are clear.*\n\n\\+ *Paper is organized and well-written.* \n\n**Significance**\n\n\\+ *Interesting solution to a relevant problem.* Episode difficulty is understudied in few-shot learning, and uniform presents a model agnostic solution to the problem of how to sample tasks during meta-learning.\n\n\n_________\n## Post-Rebuttal\n_________\n\nI have read the rebuttal and other reviews, and I increase my original rating. This paper addresses an interesting and relevant issue of episode difficulty in meta-learning and will be a particularly valuable contribution at the conference. The paper presents a solid set of evaluations and analyses, verifying assumptions (e.g. via Q-Q plots and Shapiro-Wilk tests) and reporting results across multiple benchmarks. The paper presents a novel, simple but effective method that, on the most part, boosts algorithm performance by a couple of additional accuracy points on average. Therefore, I recommend this work for the conference with a score of 8. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations not given. ", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "MIWBMirfDjm", "reviewer": "Reviewer_wemo", "summary": "This submission studies episodic sampling, which is used to sample datasets (consisting of support and query sets) for training meta-learning models. The authors define a notion of episodic difficulty (the negative log-likelihood of the query set given the support set and model parameters) and propose using importance sampling to train using several different types of episode-sampling strategies for meta-training. By using importance sampling, there is no sampling required for the target distribution but just weighting the episode loss based on the proposal and target distribution. They conduct experiments both to confirm some of their assumptions that are critical for their proposal (that episodic difficulty is normally distributed) and to show how their episodic sampling strategies compare against random episodic sampling, which is the dominant strategy used by work in this area.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: the paper considers an under-studied but important aspect of meta-learning that to my knowledge hasn't been studied in previous work. Rather than modify an existing architecture or method for meta-learning, the authors are studying an interesting aspect of the problem that most work in this area take for granted.\n\nQuality: authors conducted experiments beyond simply comparing methods on benchmarks to confirm assumptions made in the paper. These include confirming the normal distribution of episodic difficulty or that episodic difficulty is independent from modeling choice. Additionally, benchmark experiments were conducted for a variety of networks, datasets, and meta-learning methods.\n\nClarity: the paper is well-written - the ideas are presented clearly and the description of the proposed method and experiments are easy to understand.\n\nSignificance: my main concern with the paper is the experimental results. I believe the sampling strategies proposed here would be used by the community if the benefits they offer are significant. I believe the online strategy is much more feasible for use in training but I don't believe it's clear it offers a big benefit in performance. For example, in 1/3 of the scenarios presented for mini-ImageNet and tiered-ImageNet, the baseline sampling strategy is better or equivalent in terms of performance. Additionally, for the FEAT comparison, the proposed strategy offers basically the same performance as the baseline one. Thus, because of the work involved in implementing the proposed sampling strategies, I'm not sure the benefit shown is large or consistent enough for this to become something standard that is used in this research subfield.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Though not discussed, the societal impact of this work is probably similar to that of any other few-shot method.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission investigates the question of how to sample episodes for few-shot learning. By introducing a measure of episode difficulty based on the few-shot learner's NLL on the episode, the authors experiment with importance sampling to simulate different episode difficulty distributions. They observe that without any intervention, episode difficulty is normally distributed, and that a uniform distribution during meta-training yields better results.

Reviewers found the paper well-written and easy to read, and found the proposed approach original and easy to implement. Even though the improvement margins are not substantial, reviewers thought that the ideas and experiments have a broad enough appeal to the few-shot learning community to make the submission a valuable contribution. I therefore recommend acceptance.
Decision: Accept (Spotlight)