Class Imbalance in Few-Shot Learning
Authors: Mateusz Ochal
Massimiliano Patacchiola
Jose Vazquez
Amos Storkey
Sen Wang
Venue:  (2021)
Keywords: few-shot learning
class imbalance

Abstract: Few-shot learning aims to train models on a limited number of labeled samples from a support set in order to generalize to unseen samples from a query set. In the standard setup, the support set contains an equal amount of data points for each class. This assumption overlooks many practical considerations arising from the dynamic nature of the real world, such as class imbalance. In this paper, we present a detailed study of few-shot class imbalance along three axes: dataset vs. support set imbalance, effect of different imbalance distributions (linear, step, random), and effect of rebalancing techniques. We extensively compare over 10 state-of-the-art few-shot learning methods using backbones of different depths on multiple datasets. Our analysis reveals that 1) compared to the balanced task, the performances of their class-imbalance counterparts always drop, by up to $18.0\%$ for optimization-based methods, although feature-transfer and metric-based methods generally suffer less, 2) strategies used to mitigate imbalance in supervised learning can be adapted to the few-shot case resulting in better performances, 3) the effects of imbalance at the dataset level are less significant than the effects at the support set level. The code to reproduce the experiments is released under an open-source license.

Under review as a conference paper at ICLR 2021
CLASS IMBALANCE IN FEW-SHOT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Few-shot learning aims to train models on limited number of labeled samples from a support set in order to generalize to unseen samples from query set. In the standard setup, the support set contains an equal amount of data points for each class. This assumption overlooks many practical considerations arising from the dynamic nature the real world, such as -tass-imbllance. In this paper, we present detailed study of few-shot class-imbalance along three axes: dataset vs. support set imbalance, effect of different imbalance distributions (linear, step, random), and effect of rebalancing techniques. We extensively compare over 10 state-of-the-ari few-shot learning methods using backbones of different depths on multiple datasets. Our analysis reveals that 1) compared to the balanced task, the performances of their class-imbalance counterparts always drop, by up to 18.0% for optimization- based methods, although Seature-transfee and metric-based meth- ods generally suffer less, 2) strategies used to mitigate imbalance in supervised learning can adapted to the few-shot case resulting better performances, 3) the effects of imbalance at the dataset level are less significant than the effects the support set level. The code to reproduce the experiments is released under an open-source license.
INTRODUCTION
Deep learning methods are well known for their state-of the-art performances on a variety of tasks [LeCun et
2015 Russakovsky et 2015 Schmidhuber 2015). However, they often require to be trained on large labeled datasets to acquire robust and generalizable features, Few-Shot Learning
(FSL) (Chen et al. 2019 Wang et al. 2019b Bendre et al. 2020) aims reducing this burden by defining distribution over tasks, with each task containing few labeled data points (support set) and a set of target data (query set) belonging to the same set of classes. A common way to train FSL methods is through episodic mela-truining (Vinyals et al. 2017) with the model repeatedly exposed to batches of tasks sampled from task-distribution and then tested on a diffcrent but similar distribution in the meta-testing phase. The prefix "meta" is commonly used to distinguish the high- level training and evaluation routines of meta-learning (outer loop), from the training and evaluation routines at the single-task level (inner loop).
Limitations. Standard meta-training overlooks many challenges stemming from real-world dy- namics, such as class-imbalance (CI). The standard setting assumes that all classes in the support set contain the same number of data points, whereas in many practical applications, the number of sam- ples for each class may vary (Buda al.) 2018 Leevy et 20.8). Given the limited amount of data used in FSL, small difference in the number of samples between classes could already introduce significant levels of imbalance. Most FSL methods are not designed to cope with these more chal- lenging settings. Figure I exemplifies these considerations by showing that several state-of-the-ar FSL methods underperform when tested under three CI regimes (linear, step, random).
Previous work. Previous work mainly focuses on the single imbalance case or grouping several settings into one task, offering limited insights into the effects of CI on FSL and making it chal- lenging to quantify its effects (Guan et al. 2020 Triantafillou et al. 2020 Lee et al. 2019 Chen et al. 2020). A common approach to mitigate imbalance is Random-Shot meta-training (Triantafil- lou et al. 2020), which exposes the model to imbalanced tasks during meta-training. However, previous work provides little insight into the effectiveness of this procedure on the imbalanced FSL evaluation task. Furthermore, minimal work exists that investigates meta-training outcomes un- der an imbalanced distribution of classes at the (meta- dataset level, while this case is common in recent FSL applications (Ochal et al. 2020; Guan et al.) (220)) and meta-learning benchmarks (Tn- antafillou et al. 2020). The CI problem well-known within the supervised learning community,

---

Under review as a conference paper at ICLR 2021
Sshot (balanced)
4-6shot linear
1-9shot step 1minor
1-9shet random
Accuracy
50
40
20
Baseline
Baseline tune--tune
Baseline++ +
Matching
ProtoNet
ProtoNet
Relstion Net (CE)
DKT
SimpleShot
MAML
ProtoMAML
BMAML
BMAML (chaser)
(2-NN)
Net
(20-way)
-raitrraneddlll inss
metric-basen models
optimization-ban models
Figure I: Accuracy (mean percentage on 3 runs) and 95% confidence intervals on FSL methods with balanced tasks (red bars) vs imbalanced task (blue bars). Most methods perform significantly worse the imbalanced tasks, showed by the lower accuracy of the blue bars.
which has systematically produced strategies to deal with the problem, such as the popular Random Over-Sampling (Japkowicz & Stephen 2002) that aims at rebalancing minority classes by uniform sampling. While such strategies have been extensively studied on many supervised learning prob- lems, there is little understanding of how they behave with the recently proposed FSL methods in the low-data regime.
Our work and main contributions. In this paper, we provide, for the first time, a detailed anal- ysis of the CI problem within the FSL framework. Our results show that even small CI levels can introduce a sigmificant performance drop for all the methods considered. Moreover, we find that only veta-training (Triantafillou et al., 2020 Lee et al., 2019 Chen et al.] 2020) over the classical (balanced) episodic meta- veeta-trainin (Vinyals et al. [017]: while pairing the meta-training procedures with Random Over- ever-Sampling offers a substantial advantage. The experimental results show that imbalance severity at the dataset level depends on the size of the dataset. Our contributions can be summarized as follows:
a few models benefit from Random-Shor
1. A systematic, comprehensive and in-depth study of the effects of CI within the FSL frame- work along three axes: (i) dataset vs. support set imbalance, (ii) effect of different im- balance distributions (linear, step, random), (ii) effect of rebalancing techniques, such as random over-sampling and the recently proposed Random-Shor meta-training (Triantafillou et al. 2020).
2. We reveal novel insights into the meta-learning and support set adaptation capabilities to the CI regime, supported by extensive results on over 10 FSL methods with different imbalance settings, backbones, support set sizes, and datasets.
3. We provide insight into the previously unaddressed CI problem in the (traiin) dataset, showing that the effects of imbalance at the dataset level are less significant than the effects at the support set level.
RELATED WORK
2.1 CLASS IMBALANCE
In classification, imbalance occurs when at least one class (the majority class) contains higher num- ber of samples than the others. The classes with the lowest number of samples are called minority classes. If uncorrected, conventional supervised loss functions, such as (multi-class) cross-entropy, skew the learning process in favor of the majority class, introducing bias and poor generalization toward the minority class samples (Buda et al. 2018 Leevy et al. 2018). Imbalance approaches are categorized into three groups: deta-level. algorithm-leve, and hy brid. Data-level strategies ma- nipulate and create new data points equalize data sampling. Popular data-level methods include Random Over-Sampling (ROS) and Random Under-Sampling (RUS) (Japkowicz & Stephen] (2002). S200). ROS randomly resamples data points from the minority classes, while RUS randomly leaves out randomly selected portion of the majority classes to decrease imbalance levels. Algorithm-level strategies use regularization or minimization of loss/cost functions. Weighted loss is a common ap- proach where each sample's 's loss is weighted by the inverse frequency of that sample's class. Focal loss (Lin et al. 2017) is another type of cost function that has seen wide success. Hybrid methods combine more types of strategies (e.g. Two-Phase Training, Havaei et al. (2017).
2

---

Under review as a conference paper at ICLR 2021
Modeling Imbalance. The object recognition community studies class imbalance using real- world datasets or distributions that approximate real-world imbalance (Buda et al. 2018 Johnson & Khoshgoftuar 2019 Liu et al. 2019). Buda et al. 2018 note that two distributions
be used: lin- ear and step imbalance (defined in our methodology Section 3). At large-scale, datasets with many
samples and classes tend to follow a long-tail distribution (Luu et al. 2019 Salakhutdinov et al.)
2011 Reed, 2001), with most of the classes occurring with small frequency and a few classes oc- curring with high frequency. Our work primarily focuses on the tail-end of the distribution and does not consider the case of large sample size. Therefore, we do not examine the long-tail mechanisms.
2.2 FEW-SHOT LEARNING
FSL methods can be broadly categorized into metric-learning. optimization- hallucination, data-adapiation, and probabilistic approaches (Chen et al. 2019). Metric-learning approaches such as Prototypical Networks (Snell et al., 2017). Relation Networks (Sung et al., (017). the Neural Statistician (Edwards& & Storkey 2017) and Matching Networks (Vinyals et al. 2017]. learn fea- ture extractor capable of parameterizing images into embeddings, and then use distance metries to classify mapped query samples based on their distance support points. Optimization-based ap- proaches such as MAML (Finn et al.) 2017) and Meta-Learner LSTM (Ravi & Larochelle| 2016)), meta-trained to use guided optimization steps on the support set for quick adaptation. Hallu- cination or data augmentation techniques perform affine and color transformations on the support set to create additional data points (Zhang al.] 2018). Probabilistic methods use Bayesian infer- ence to learn and classify samples, for example, the recently proposed Deep Kernel Transfer (DKT) (Patacchiola al., 2020), which uses Gaussian Process at inference time. We use the term domain adaptation to represent those approaches using standard transfer-learning with a pre-training stage on large set of classes and a fine-tune stage on the support set - examples are Baseline and Base- line++ from Chen et al. (2019), and the recently proposed Transductive Dane-Tuning from Dhillon ct al. 2020p. The details of the methods used in our experiments are reported in Appendix A For completeness, it is worth mentioning Incremental Few-Shot Learning (Ren et al., 2018 Gidaris & Komodakis 2018 Hariharan & Girshick, 2017) which is an extension of FSL. It considers main- laining performance base classes imeta-trainin dataset) while incrementally learning about novel classes using limited data, typically without re- re-training from scratch on all data. Here, we focus on studying how imbalance affects the learning of novel classes only; therefore, we will not consider incremental FSL further.
2.3 IMBALANCE FEW-SHOT AND META LEARNING
Class Imbalance in the low-data regime has received some attention, although the current work is not comprehensive (Guan et al.) 2020; Triantafillou et al. 2020 Lee et al. 2019] Chen et al. 2020)). We identify that in FSI., class-imbalance occurs at two levels: the task-level and the meta-dataset level. At the task level, class-imbalance occurs in the support set or the query-set, directly affecting learning and evaluation procedures. Class imbalance at the meta-dataset level is caused by im- balanced dataset classes in one (or more) of the three data splits: meta- metta-taiiinn, meta-validation, meta-testing. This disproportion affects the distribution of tasks that a model is exposed to during meta-trane veta-training, affecting their ability to generalize to new tasks. In Figure 2 we highlight the dif- ferences between imbalanced task and imbalanced meta-dataset. Related to, but distinet from, these two class-imbalance types is lask-distribution imbalance (Lee et al.] 2019); skewed tast-disti can occur as a result of meta-datased level flass-imbalance or as a result of the task-sampling pro- cedure. In extreme cases, distriiittiiiionn imbalance can lead to -idto--distribttinn tasks during meta-evaluation. dassk-iisti imbalance has already received some attention (Lee et al. 2019
Cao al. 2 ; therefore, it will not be considered in this work.
Class Imbalance in Tasks. Triantafillou al. 2220)) uses imbalanced support sets to create a more realistic and challenging benchmark for meta- meta--carning. The authors use random-shot tasks with randomly selected classes (way) and samples (shot), which replace the balanced task during meta-training and meta-evaluation. A similar idea is explored by Lee et al. (2019), Chen et al. (220)). and Guan et al.] (2020) with the last two using a fixed number of classes (way). However, none of these works quantify the impact of class-imbalance on the FSL task nor the advantages of Random-Sho meta-training. Lee et al.] (2019) explores a small range of class-imbalanes in the support set. However, details into the effects of class-in huss-imbalance are lost when combined with task- distribution imbalance, making it challenging to attribute any performance changes caused by class- problem the support set, but their
imbalance. Chen et al. (0000) explore pure
3

---

Under review as a conference paper at ICLR 2021
Imbalanced Support Set
Imbalanced Meta-Training Dataset
Samples
8
8g
4
Samples
S
Sg
Cleszes
Classes
Task
Task
Task
Task
53
Q5
S
Q5
Dataset
Dalaset
Figure 2: The two types of imbalance settings investigated this work. Left: Imbalanced support set. Classes are balanced at the dataset level, but tasks are imbalanced by one of Z-distributions: linear (task 1), step (task 2). and random (task 3) Right: Imbalanced meta-training dataset. Classes are imbalanced at the dataset level, but all the support sets are balanced at the task level. Following standard practice in the literature, query sets are kept balanced in both settings.
analysis is limited to just two methods (their proposal and MAML). Guan et al. (2020), meta- learning is applied on aerial imagery, exploring step imbalance ranging from 5 to 140 samples per class (shot); however, only two FSL methods are compared (Prototypical Networks and their RF- MML method). Previous work provides limited insight into dlass-mbblance the task level.
Class Imbalance in the Meta-Dataset. Standard meta- meta-datasets (e.g. Mini- Mini-ImageNet) can be swapped for other domain-specinf datasets, such CUB (Wah al. 2011), VGG Flowers (Nils- back & Zisserman 2008), and others (Triantafillou et al. 2020). These datasets sometimes contain an unequal number of class samples, but previous work has never reported the effects of class- imbalance in the meta-training dataset (Guan et al. [020: Triantafillou et al. [2220] Lee et al. 2019 Chen al. 2020). We emphasize that studying the impact of imbalance at this level is important since imbalanced domain-specific meta-datasets are common in real-world applications (Guan et al., 2020 Ochal et al. 2020) and recent benchmarks (Triantafillou et al., 2020). Our work is the first to provide quantitative insights into this setting.
3 METHODOLOGY
3.1 STANDARD FSL
A standard K-shot N-way FSL classification task is detined by small support set, S ((z1,81),.. (z),,...((.,))))))) yn)} ~ D. containing N x K image-label pairs drawn from N unique classes with K samples per class ((S) K x N). The goal is correctly predict labels for a query set, D. containing different set of M samples drawn from the same N classes (i.e. Q(4) g(x))) = 0 and Q(v) = S(V)). The support set can also be referred to as sample sel and the query set as target set.
Q = @ M1), (x),
3.2 CLASS GLASS-IMBALANCER FSL
We define taaas-imbblanced FSL task Kiin-K maa--hho N-way I-distribution task. Similarly to the standard FSL task, a model is given a small support set, S ~ D and a query ser, Q ~ D, containing a different set of samples drawn from the same N classes. However, in the imbalance case, the support set contains between Kmin to Kmaz (inclusive) number of samples per class distributed according to the imbalance Z-distribution, where I E {linear, step, rondom) (Budal et al., 2018). Similarly, the query set can contain Mmén to Mmar samples per class distributed according to the I-distribution. Mmaz) for fair evaluation[] For brevity, but without loss of generality, we define imbalance I- distribution in relation to the support set (see Figure (2 as:
our experiments, we keep a balanced query set (M Mmin =
* Linear imbalance. The number of class samples, K₁, for classes i E [1..N] is defined by:
K; round (Kmin c c (i - 1) x (Kmar + + Kmin)///nn )/N/)////Nn 1)), (I)
'Tiis is standard procedure used the class-imbens imbalance literature (Buda et al. 2018), which reduces the number of variables and allows isolating the effect of imbalance. Note that an imbalanced query set would influence methods such as SCA (Antoniou& & Storkey 2019), which use the query set as an additional unlabeled set during the inner-loop. We do not consider such methods in our experiments since they assume immediate access to the query set, which limits their practical application.
4

---

Under review as a conference paper at ICLR 2021
where 0.499 for rounding purposes. For example, this means that for linear 1-9-shot 5-w task, K, E {1, 3, 5, 7,9), and for linear 4-6-shot 5-way task K, E (4, 5, 6, 6}. Step imbalance. The number of class samples, Kü, is determined by an additional variable Nmn specifying the number of minority classes. Specifically, for classes E {1..N):
K Min
if Nmin: otherwise.
K;
(2)
K Kmaz:
For example, in step 1-9-shot 5-way task with 1 minority class K; (1,9,9,9,9)
Random imbalance. The number of class samples, K2, is sampled from a uniform distribu- tion, i.c K; ~ Unif(Kminn (maz) with Kmin and Kmar inclusive. This is appropriate for the problem at hand (small number of classes), but it could be replaced by Zipf/Power Law (Reed) 2001) for a more appropriate imbalance in problems with a large number of classes.
We also report the imbalance ratio p. which is scalar identifying the level of class-imbalance; this often reported in the CI literature for the supervised case (Buda et al., 2018). We define p to be the ratio between the number of samples in the majority and minority classes in the support set:
Kmaz
K min
3.3 CLASS-IMBALANCED META DATASET
Training FSL methods involves three phases: meta- sett-training, meta-validation and meta-testing, Each phase samples tasks from different dataset, Dtrain. Dval, and Diest, respectively. A balanced dataset contains D.N classes with D. samples per class, where * E {train, val, test). However, in the real-world, datasets contain any number of samples with imbalance. For fair evaluation, control dataset imbalance according to the I-distribution described in Section [..2 with K minn Kmaz, N, Nmin changed for DKmin,, D.Kmns, D.N, D.N.... Similarly, we report the imbalance ratio In our experiments, we apply imbalance only at the meta-training stage to limit the factors of interest, but similar procedure could be used at the meta-testing and vata-vvliittion stages.
3.4 REBALANCING TECHNIQUES AND STRATEGIES
Random Over-Sampling. We apply Random Over-Sampling (ROS) and for each class- imbalanced task, we match the number of support samples in the non-majority classes to the number of support samples in the majority class, K; = ma) This means that for {linear, step), the number of samples in each class is equal to K Kmaz- We match K; to max; (K,) by resampling uniformly at random the remaining (K,)(K;) - K; support samples belonging to class and then appending them to the support set. When applying ROS with augmentation (ROS+), we perform further data transformation on the resampled supports. A visual representation a class imbalanced task after applying ROS and ROS+ is presented in the Appendix (Figure 7
Random-Shot Meta- Training. We apply Random-Shor meta-training similarly to the Standard episodic (meta-) training (Vinyals et al. 2017) but with the balanced tasks exchanged with Kmin" tasks, as defined above. We use random-distribution following pre-
Shot random-distribun vious work (Triantafillou et al. 2020 Lee et al.) 2019). since in real-world applications, the actual imbalance distribution is likely to be unknown at (meta-)ess veeeta--)vlluttoon time.
Rebalancing Loss Functions. We apply two rebalancing loss functions: Weighted Loss Buda et al. 2018) and Focal Loss (Lin et al. (0177). Both of them have been applied to the inner loop of optimization-base methods. Full details are reported in the supplementary material (Appendix
EXPERIMENTS
4.1 SETUP
Class Imbalance Scenarios and Tasks. We address two class-imbalance scenarios within the FSL
framework: 1) imbalanced support set, and 2) imbalanced meta-traite meta-training dataset. For the imbalanced support set scenario,
first focus on the very low-data range with average support set size of 25 samples (5 avr. shot). We train FSL models using Standard (episodic) meta-training (Vinyals et al. 2017)
using 5-shot 5-way tasks, as well as Random-Shot meta-training (Triantafillou et al. 2020
5

---

Under review as a conference paper at ICLR 2021
Lee al., 2019 Chen al. 2020) using 1-9shot 5-way aunom--distibunn tasks (as described in Section We pre-train baselines (i.e., Fine-Tune. I-NN, Baseline++) using descent, and then fine- tune the support or perform 1 1-NN classification. We evaluate all baselines and models using a wide range of imbalanced meta-testing tasks. In contrast to previous work, we evaluate models using additional imbalance distributions, linear and step; this allows to control the imbalance level deterministically and provide insights from multiple angles. For the imbalanced meta-dataset scenario, we vary the class distributions of the meta-training datasets, We isolate this level of imbalance by meta-trann vetaattraining and mneta-evaluation on balanced FSL tasks. All main experiments are repeated three times with different initialization seeds. Each data point represents the average performance of over 600 meta-testing tasks per run.
gradient
Additional details. We adapted range of 11 unique baselines and FSL methods: Fine-tune base- line (Pan & Yang 2010), I-NN baseline, Baseline++ (Chen et al. 2019). SimpleShot (Wang et al.) 2019a), Prototypical Networks (Snell et al.) (017), Matching Networks (Vinyals et al. [2017), )) Re-
lation Networks (Sung et al. 2017), MAML (Finn et al. 2017). ProtoMAML (Triantafillu et al.)
2020 DKT (Patacchiola et al. (2220), and Bayesian (BML(BMAMM)
(Yoon et al. (018)). Imple-
mentation details of these algorithms are supplied in Appendix A We used 4 layer convolutional
network as backhone for each model, following common practice (Chen et al. 2019). We train and evaluate all methods on MinilmageNet (Ravi & Larochelle 2016)] Vinyals et 2017), containing 64 classes with 600 image samples each. In the imbalanced meta-dataset setting, we half the Mini- ImageNet dataset to contain 300 samples per class on average, and control imbalance as described For full implementation details, see Appendix A
in section 3.3
4.2 CLASS IMBALANCED SUPPORT SET
Effect of Class Imbalance with Standard Deta-Training. Figure 1 highlights the crux of the llss-imballane problem the support set level. Specifically, the figure shows standard meta- trained FSL models (Vinyals et al. 2017) and pre-trained baselines, evaluated on the balanced 5- shot 5-way task and three imbalanced tasks. We observe that introducing even a small level of imbalance (linear 4-6-shot 5-way, of 13 algorithms, compared with the balanced 5-shot task. The average accuracy drop -1.5% for metric-based models and -8.2% for optimization based models. On tasks with a larger imbalance (1-9shot random, p = 9.0), the performance drops by an average -8.4% for metric-based models and 18.0% for optimization-
1.5) produces (
difference for 6 out
models compared to the balanced task. Interestingly, despite the additional 12 samples in the support set in 1-9shot step tasks with minority class (p 9.0), the performance drops by -5.0% on the balanced task with 25 support samples in total.
Baseline (2-NN)
Baseline (une-tune)
Baseline
Matching Net
ProtoNet
70
50
30
ProtoNet (20-way)
Relation Net (CE)
DKT
SimpleShot
MAML
s
66
Accurate
50
40
20
ProtoMAML
BMAML
BMMML (chaser)
70
Standard
Standard (ROS+ inference)
Random Shot
50
Random Shot (ROS+ at inference only!
Random RDS
30
Random Shot with ROS IROS+ at Inforence only)
Handom-Shot with ROS
3
3 Imbalance, 0 K Kmm
Figure 3: Standard episodic training (Vinyals et al. 2017) vs. random-shot episodic training Tri- antañilou et al., 2020). We explore pairing methods with Random Over-Sampling (ROS) without and with augmentation (ROS+).
-ooooverlgnning 95% confidence intervals indicate 'significant" performance difference.

---

Under review as a conference paper at ICLR 2021
Standard vs. Random-Shot Meta-Training. In Figure show the accuracy for increasing imbalance levels (p) using evaluation tasks with a linear distribution and fixed support set size (K, Ro 5) for a fair comparison. Comparing Standard and Random-Shan meta-training (solid black and solid red lines) reveals that only a few methods benefit from Random-Shot meta-training. On the balanced 5-shot task, we observe -6.0% decrease in aceuracy, caused by Random-Shot over Standard meta-training, On the imbalanced 1-9shot random task, Random-Shot offers a limited improvement over the Standard, with a significant increase in performance for only 3 out of 10 models. Those improvements include +2.5% for Relation Net, and +6.6% for BMAMLs. These results suggest that exposing FSL methods imbalanced tasks during meta-training does not automatically lead to improved performance at meta-test time. Interestingly, in an extreme imbalance case (1-21 step 4minor, Appendix E only ProtoNet and RelationNet obtained a significantly higher performance with Random-Shot (+18% compared to Standard). This suggests that the advantage may emerge from coupling Random Shot with the prototype calculation mechanism unique to those methods. The results also suggest that some models have natural robustness to imbalance: Relation Net, MatchingNet, and DKT only drop slightly compared to other methods.
Random-Shot with Random Samp-ig In Figure 3 we observe that the performances of optimization based methods such MAML and BMAML significantly improve by applying random over-sampling with augmentation (ROS+) and without augmentation (ROS). In the largest imbalance case in the graph (/=9), we observe that models using ROS+ at inference (dotted and yellow lines) improve over the Standard (solid black) by +6.7%; in particular, optimization-bn methods improve by +12.2%, fine-tune baselines by +7.4%, and metric-based by +2.8%. In the imbalanced task, the least affected model is MatchingNet only dropping -1.9% compared to the balanced task; we provide a list of top-50 performing models in Table (Appendix C.Ip. Standard (ROS+ at inference) achieves the highest average performance gains (+8.5%); tieing for second best is Random-Shov with ROS (ROS+ at inference) with +6.9% and Random-Shot with ROS+ (+6.4%). We breakdown the results by type in Appendix C.2 (Figure
Imbalance with More Shots. We explored additional settings with higher number of shots, see Figure Specifically, we train models using Random-Sin Shot meta-training with 1-29 shot and 1-49 random episodes. We then evaluate those models on imbalanced tasks with an average number of 15 shots and 25 shots, respectively. The bottom row of Figure 4 shows the difference in performance between the imbalance and balanced tasks. We observe that for the high-shot condition (right col- umn), the general model performance increases while the models are less affected by imbalance; however, the gap with respect to the balanced condition remains significant. Models achieve 55- 60% of their performance on the balanced task within first 5 avr. shots; increasing the number of shots to 15 only boosts their performance by +7% This may explain why imbalance will have an inevitable impact on small classification tasks: better performance achieved via higher numbers of support samples in the majority classes, does not offset the performance lost due to lack of samples in the minority classes. In Appendix C
we breakdown the results for each model type.
Standard Random-Shot
*o Standard (ROS+ at inference)
Random-Shot (ROS+ at inference)
Sshot [balanced]
1-9shot 1minor
Random-Shat with ROS
Random-Shot with ROS (ROS at inference)
4. Eshhot linear
1-9shot random
Random-Shat ROS+
60
Acc. Diff. to Sshot, %
avr. shot
avr. shot
avr. shot
70
58
Accuracy,
60
66
Accuracy
56
55
50
%
45
48
Conu4
Convé
ResNet10
ResNet34
Convá Convé
Re
ResNet34
Acc. Diff.
balanced,
-5
10
Figure 5: Combined average model performance against different back- bones and imbalanced tasks. Left: combined performance of all models and training scenarios. Right: relative performance w.r.t. the balanced task.
-15
9
Imbalance, p= KmauiK
Figure 4: Comparing imbalance levels via support sets of different size. Each line represents the average across all
models in each training and imbalance setting.
7

---

Under review as a conference paper at ICLR 2021
Backbones. In Figure 5 report the combined average accuracy of all models and imbalance strategies against different backbones (Conv4, Conv6, ResNet10, ResNet34). Overall, deeper back- bones seem to perform slightly better the imbalanced tasks, suggesting higher tolerance for imbalance. For instance, using Conv4 gave -8.6% difference between the halanced and the 1-9shot random task, while using ResNet10 the gap is smaller (-8%%. The performance degradation ob- served with ResNet34 is similar to that reported by [Chen et al. (2019), and is most likely caused by the intrinsic instability of meta-training routines on larger backbones. In Appendix C.3 down the results across different models and training strategies.
we break-
Precision and Recall. Looking at the precision and recall tables in Appendix C provides ad- ditional insights about each algorithm. For instance, DKT (Patacchiola al.] 2020) shows very strong performance in classes with small number of shots and well-balanced performances for higher shots. This may be due to the partitioned Bayesian one-vs-rest scheme used for clas- sification by DKT, with separate Gaussian Process for each class; this could be more robust to imbalance. BMAML, on the other hand, fails to correctly classify samples with and K samples, showing that the method has a strong bias towards the majority classes.
K- Standard (ROS+)
+ Random-Shot (ROS+)
Rebalancing Cost Functions We applied
Standard Loss)
Random-Shot (Pocal Loss)
Focal Loss (in al 2017] and Weighted Loss
Standard (Weighted Loss)
Random-Shor (Welghted Loss]
(Buda et al. 2018 2018|papoooo Japkowicz & Stephen 2002)
65
to the inner-loop of optimization-
meth-
55
60
Acc. Diff.,
ods at inference time only. Results in Fig-
50
-10
ure 6 and Appendix [ show that overall, Fo-
40
45
-15
cal Loss is not as effective as ROS+ techniques. However, ROS+ and Weighted Loss perform very similarly, suggesting a similar effect the imbalanced task. The advantage of using ROS/ROS+ is their versatility; any FSL algo- rithm can use ROS, while algorithm-level bal- ancing approaches, such Weighted Loss, do not straightforwardly extend to metric-learning methods.
35
-20
30
-25
1
2
3
4
5
6
7
8
9
4
5
8
9
Imbalance, P Kmax/Kmain
Figure 6: Combined average model performance against re-balancing strategies applied at test-time only. Left: all models and training scenarios. Right: performance w.r.t. the balanced task.
4.3 CLASS IMBALANCED META-DATASED DATASET
Imbalanced Mini-ImageNet. To induce dataset imhalance, we half the number of samples per class in Mini-ImageNNe. In table (left), we show the accuracy after training via standard episodic (meta-)training (Vinyals et al. (017)) with (balanced) 5-shot 5-way tasks. In this particular scenario, use significantly higher imbalance levels (p 19) compared to those in the previous section; despite this, we observe small, insignificant performance differences between balanced and imbal- anced conditions. In additional experiments, we further reduced the dataset size contain a total of 4800 images and 32 randomly selected classes. In Figure 10 (Appendix C.5) we observe more significant performance drop as we increase the number of minority classes. Meta-evaluating on CUB showed similar trend, with average drop of - 1..6 the most extreme imbalance set- ting: 30-510 step with 24 minority classes and = 17.0 (Appendix C.5). When we breakdown the results by model in Appendix C.3 we observe that aptimization-based approaches and fine-tune baselines have a slight advantage over the metric-basee based, most likely due to the ability to adapt during inference. Interestingly, in this setting RelationNet performs the worse with drop of -4.3% w. t. the balanced task in the most extreme setting (24 minority, Mini-ImageNNt).
Additional results. To evaluate the performance under a strong dataset shift, we evaluated Mini- ImageNet trained models on tasks sampled from CUB-200-2011. In (righ][ we observe that models are not affected at all by the imbalanced setting despite the harder scenario. In Appendix D we provide additional experiments with BTAML (Lee et al. 2019), and an analysis of the correlation between meta-dataset size and performance
DISCUSSION
FSL robustness to class imbalance. All examined FSL methods are susceptible to class im- balance, although some show more robustness (e.g., Matching Net, Relation Net, and DKT).
8

---

Under review as a conference paper at ICLR 2021
Table 1: Training on imbalanced meta-training dataset. Imbalanced distributions represent
= 19 (DKmin = 30, DKmu= 570) with step imbalance containing DNimnn = 32 minority classes (out of 64 available in the dataset). Small differences in accuracy between balanced and I Z-distributions, suggest insignificant effect of imbalance at dataset level. Left: Evaluation on the meta-testing dataset of Mini- lini-em Right: Evaluation on the meta-testing dataset of CUB.
Imbalanced Mini Maageeem lineur
Imbalanced Mini- Minii-Imageett - CUB
Imbalance
balanced
random
step
balanced
lineur
random
step
Basefine (1-NN)
42.69 gm
43.42_g04
42.15
_1.45
43.21 am
43.42gae
43.39_
Baseline (fine-tume)
51 m8
50.13
54. 16
52.47
53.19
$1.95
53.52-002
$2.68
Baseline++
48.44.a
47.18_064
51.47 in
51.88 10.m
49.38 Loos
46.83
50.48 pm
48.42.n
Matching Net
58.26
58.24gnm
58.45
56.53.20
50.92.07
51.32aam
50.77_mm
20.51
ProtoNet
60.65
59.17 Loos
60.16
58.69 Loma
51.85 12
52.06_
52.42.4m1
ProtoNer (20-way)
60.91
60.64
60.37
58.83 mome
52.80.mm
52.60
52.31
$1.33_00
Rolation Net (CE)
62.78 (a:7u
61.39 sw
62.35 10.70
$7.93
54.32
54.41 :am1
52.13
49.90
DKT
58.09 Love
$7.59 pos
57.81 1ov
55,91 1ost
54.62 jam
54.19 1a71
54.86 J072
54.44 am
SimpleShot
59.78 59,55
58.74-002
58.89 -0:11
53.16
53,46
52.88
$2.87
MAML
$4,43
14
54.97
54.30=3.m
$3,26
55.14=007
$3,96:00
ProtoMAML
51.31
54.57=0m
53,56-00
48
51.25
45.27=000
$1.64au5
Avr. Diff. to balanced
0.0
-0.1
-0.2
-0.7
0.0
-0.2
-0.3
-0.6
Optimization- based methods and fine-tune baselines suffer more as they use conventional supervised loss functions in the inner-loop which are known to be particularly susceptible to imbalance (Buda ct al. 2018 Johnson & Khoshgoftaar 2019; Japkowicz & Stephen 2002 Moreover, the problem of class imbalance persists as the backbone complexity and support set size increase. Those results suggest that current solutions will offer sub-optimal performance in real-world few-shot problems.
Effectiveness of Random-Shol meta-training, Our experiments test simple solution to class imbalance that has been popular in the meta-learning community - -Random-She meta-training Triantafillou et al. 2017 Lee et al. 2019 Chen et al. 2020 Contrarily to popular belief, our findings reveal that this method is scarcely effective when applied by itself. Extensive analysis and validation performance through epochs (see Appendix [ suggest that these results are genuine and unlikely to be the result of inappropriate parameter tuning. This finding has an important con- sequence, suggesting that robustness to imbalance cannot be obtained by the simple exposure to imbalanced tasks.
Effectiveness of re-balancing procedures. The results suggest that a simple procedure, Random Over-Samplin (ROS), quite effective in tackling class imbalance issues. Therefore, we encourage the community to include it in their evaluation as ROS is simple to implement, and it can be applied to almost any algorithm. However, ROS does not provide any particular advantage to methods the highest performance ranking levels, like Matching and DKL. This could be due to diminishing returns and should be investigated on a case-by-case basis.
Effeet of imbalance at the meta-datase level. Our results suggest that imbalance in the meta- dataset has minimal effect on the meta-learning procedure. This could result from standard episodic (meta-) sertaa)training that samples classes with equal probability and causes natural re-sampling. Likely, datasets with lower intra-class variation and larger imbalance (Liu et al.] 2019 Wang et al. 2017 Salakhutdinov et al. 2011) could produce more dramatic performance changes.
6 CONCLUSION
In this work, we have provided a detailed analysis of class-imbalance in FSL, showing that class- imbalance the support set level is problematic for many methods. We found that most metric-based models present built-in robustness to support-set imbalance, while in citimization-basee models imbalance issues can be alleviated using oversampling. In our experiments, Random-Shot meta- training provided minimal benefits suggesting that meta-learning methods do not learn to balance from random-sha shot episodes alone. Results on meta meta-dataset imbalance showed just a small negative effect, but this effect is not dramatic as with the lask-level imbalance. In future work, the insights gained with our investigation could be used to design novel few-shot methods that can guarantee a stable performance under the imbalance condition.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Antreas Antoniou and Amos Storkey, Learning to learn via Self-Critique
arXiv preprint
arXiv: 10295, 2019.
Antreas Antoniou, Massimiliano Patacchiola, Mateusz Ochal, and Amos Storkey. Defining Bench- marks for Continual Few-Shot Learning. arXiv preprint arXiv:2004. /9677, 2020.
Nihar Bendre, Hugo Terashima Marín, and Peyman Najafirad. Learning from Few Samples: A Survey. arXiv preprint arXiv:2007. 15484, 2020.
Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-259, 2018.
Tianshi Cao, Marc T Law, and Sanja Fidler. A theoretical analysis of the number of shots in few-shot learning. International Conference on Learning Representations (ICLR), 2020.
Wei Yu Chen, Yu Chiang Frank Wang, Yen Cheng Liu, Zsolt Kira, and Jia Bin Huang. A closer look at few-shot classification. International Conference on Learning Representations (ICLR), 2019.
Xinshi Chen, Hanjun Dai, Yu Li, Xin Gao, and Le Song. Learning to Stop While Learning to Predict. International Conference Machine Learning (ICML), 2020.
Zhi-Qi Cheng, Xiao Wu, Siyu Huang, Jun-Xiu Li, Alexander G. Hauptmann, and Qiang Peng- Learning to Transfer Learn. In 2018 ACM Multimedia Conference on Multimedia Conference MM 18, pp. 90-98. ACM Press, 2018.
Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A Baseline for Few-Shot Image Classification. International Conference on Learning Representations, 2020.
Harrison Edwards and Amos Storkey. Towards a Neural Statistician. International Conference on Learning Representations (ICLR), 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnosti Meta-Learning for Fast Adapta- tion of Deep Networks. International Conference on Machine Learning (ICML), 2017.
Spyros Gidaris and Nikos Komodakis. Dynamic Few-Shot Visual Learning Without Forgetting. In IEEE Conference on Computer Vision and Puttern Recognition ( (CVPR), 2018.
Jian Guan, Jiabei Liu, Jianguo Sun, Pengming Feng, Tong Shuai, and Wenwu Wang. Meta Metric Learning for Highly Imbalanced Aerial Scene Classification. In ICASSP 2020 2020 IEEE In- ternational Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 4047-4051. IEEE, 2020.
Bharath Hariharan and Ross Girshick. Low-Shot Visual Recognition by Shrinking and Hallucinating Features. IEEE International Conference on Computer Vision (ICCV), 2017.
Mohammad Havaei, Axel Davy, David Fardeeyarley,aae Antoine Biard, Aaron Courville, Yoshua Ben- gio, Chris Pal, Pierre Marc Jodoin, and Hugo Larochelle. Brain tumor segmentation with Deep Neural Networks. Medical Image Analysis, 35:18-31, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. International Conference on Machine Learning (ICML), 2015.
Nathalic Japkowicz and Shaju Stephen. The class imbalance problem: A systematic study. Intelli- gent Data Analysis, 6:429 449, 2002.
Justin M. Johnson and Taghi M. Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data 6, 2019.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521:436 444, 2015.
10

---

Under review as a conference paper at ICLR 2021
Hae Beom Lee, Hayeon Lee, Donghyun Na, Sachoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning to Balance: Bayesian Meta-Learning for Imbalanced and Tasks. International Conference on Machine Learning (ICML), 2019.
Joffrey L. Leevy, Taghi M. Khoshgoftaar, Richard A. Bauder, and Naeem Seliya. A survey on addressing high-class imbalance in big data. Journal of Big Data, 5, 2018.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to Learn Quickly for Few-Shot Learning. arXiv preprint arXiv:1707.099: 09975, 2017.
Tsung-yi Lin, Priya Goyal, Ross Girshick, Kaiming He. and Piotr Dollár. Focal Loss for Dense Object Detection. IEEE International Conference on Computer Vision (ICCV), 2017.
Ziwei Liu, Zhonggi Miao, Xiaohang Zhan, Jiayun Wang, Boging Gong, and Stella X. Yu. Large- Scale Long-Taile Tailed Recognition an Open World. IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2537-2546, 2019.
Maria Elena Nilsback and Andrew Zisserman. Automated flower classification over large num- ber of classes. Proceedings 6th Indian Conference on Computer Vision, Graphics and Image Processing, ICVGIP 2008, pp. 722-729, 2008.
Mateusz Ochal, Jose Vazquez, Yvan Petillot, and Sen Wang. A Comparison of Few-Shot Learning Methods for Underwater Optical and Sonar Image Classification. OCEANS 2020 preprint, 2020.
Sinno Jialin Pan and Qiang Yang. A Survey on Transfer Learning. IEEE Transactions Knowledge and Data Engineering, 22:1345-1359 2010.
Massimiliano Patacchiola, Jack Turner, Elliot J. Crowley, Michael O' O'Boyle, and Amos Storkey. Bayesian Meta-Learning in the Few-Shot Setting via Deep Kernels. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2020.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. International Conference on Learning Representations (ICLR), 2016.
William J Reed. The pareto, zipf and other power laws. Economics Letters, 2001.
Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard Zemel. Incremental Few-Shot Learning with Attention Attractor Networks. arXiv preprint arXiv:1810.07216. 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115:211-252, 2015.
R. Salakhutdinov, A. Torralba, and J. Tenenbaum. Learning to share visual appearance for multiclass object detection. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
Jürgen Schmidhuber, Deep learning in neural networks: An overview. Neural networks, 61:85-117, 117, 2015.
Jake Snell and Richard Zemel. Bayesian Few-Shot Classification with One-vs- -ach Polya- Gamma Augmented Gaussian Processes. urXiv preprint arXiv:2007. 2020.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical Networks for Few-shot Learning. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. Learning to Compare: Relation Network for Few-Shot Learning. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Eleni Triantafillou, Richard Zemel, and Raquel Urtasun. Few-Shot Learning Through an Informa- tion Retrieval Lens. Advances Neural Information Processing Systems (NeurIPS), 2017.

---

Under review as a conference paper at ICLR 2021
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- Dataset: A Dataset of Datasets for Learning to Learn from Few Examples. International Confer- ence on Learning Representations (ICLR), 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing Networks for One Shot Learning. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Evan Vogelbaum, Rumen Dangovski, Li Jing, and Marin Soljačić. Contextualizing Enhances Gra- dient Based Meta Learning. arXiv preprint arXiv:2007. 2020.
C. Wah, S Branson, P Welinder, Perona, and S Belongie. The Caltech-UCSD Birds-200-225 Dataset. In California Institute of Technology, 2011.
Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. SimpleShot: Revis- iting Nearest-Neighbor Classification for Few-Shot Learning. arXiv preprint 2019a
Yaqing Wang, Quanming Yao, James Kwok, and Lionel Ni. Generalizing from a Few Examples: A Survey on Few-Shot Learning. arXiv preprint arXiv: I904.050400 1, 2019b.
Yu- Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems (NeurIPS). 2017.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. Advances in Neural Information Processing Systems (NeurIPS), 2018.
Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yangqiu Song, Yoshua Bengio, and Yangqiu Song. MetaGAN An Adversarial Approach to Few-Shot Learning. Advances in Neural Information Processing Systems (NeurIPS), 2018.
12

---

Under review as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
A.1 DATASETS
For the imbalanced support set experiments, we used MinilmageNet (Vinyals al. 2017 Ravi & Larochelle| 2016) following the same version popular version as (Ravi & Larochelle 2016 Cheng et al. 2018). All meta-learning models used 64, 16, 20 classes for the meta-training D Praiin meta-validation Deat, and meta-testing Drest datasets, respectively, with each class containing 600 samples. All images are resized to 84 by 84px. For the feature-trane transfer baselines (1-NN, fine-tune, and Baseline+++), we used a conventionally partitioned training and validation datasets. Specifically, we combined Dtrain and Dent classes (i.e., 64 + 16 = 80 classes), then partitioned the samples of each class into 80% 20% split for pre-training validation, forming DD Dirain and Da val
(where D'(v) = D'(a)) ). Thus, the baseline were trained on the same number of training samples as the meta-learning methods, albeit with more classes and less samples per class. All models were
evaluated on FSL tasks sampled from Drest-
For the imbalanced meta-dataset experiments, we used two variants of MinilmageNet. In the first, referring Table 13, we halved the average number of samples per class in the meta-training dataset Drrain to allow us to introduce imbalance artificially into the dataset. In the second scenario, referring to Figure 10. we reduced the size of the meta- veta-training by a more considerable degree. Specifically, the size of the meta-training dataset was controlled to contain a total of 4800 images distributed among 32 randomly selected classes from the meta-training dataset of Mini-ImageNNe. For meta-learning methods, we kept the original 16 and 20 classes for meta-validation and meta-testing with 600 samples each. To allow as fair comparison as possible, we used the same meta-training datasets for the baselines and meta-learning models. However, the baselines used balanced validation set created from the leftover samples from the original meta- centt-ttaninng dataset.
For the imbalanced meta-dataset experiments, we also evaluated on tasks sampled from 50 randomly selected classes from CUB-200-2011 (Wah al. 2011), following the same line of work as Chen et al. (2019).
A.2 TRAINING PROCEDURE.
All methods follow a similar three-phase learning procedure: meta-training, meta- validation,, and meta-testing. During meta-training, an FSL model was exposed to 100k tasks sampled from Dtrain- After every 500 tasks, the model was validated on tasks from Doot and the best performing model was updated. At the end of the meta-training phase, the best model was evaluated on tasks sampled from Drest. The baselines (i.e., fine-tune, I-NN, Baseline++) follow a similar three- three-phase procedure but with the meta-training / meta- valta-validttion phases exchanged for conventional pre-traie pre-training / vali- dation on mini-batchar (of size 128) sampled from Dirain and Di as outlined above. In all three meta- phase's tasks, used 16 query samples per class, except for the 20-way Prototypical Net- work, where we used 5 query samples per class during meta-training to allow for a higher number of samples in the support set.
Meta-/Pre- Training Details. In the imbalanced support set setting, we meta-train FSL methods using standard episodic meta-training (Vinyals et al. (017)) using 5-shot 5-way tasks. We also explore random-shot episodic training (Lee al., 2019) using 1-9shot 5-way candom-distribution tasks (as described in section We meta-/pre- trained on 100k tasks/mini musks/mini-attches, using learning rate of 10-3 for the first 50k episodes//miii-aathhee
and 10-1 for the second half. The baselines and SimpleShot are trained using 100k balanced mini-batches with batch size of 128. All methods were meta-vale validated on 200 tasks/mini-batchie every 500 meta-training httss/mmni-batchee to select the best performing model.
Meta- Meta-Testing. The final test performances were measured on a random sample of 600 tasks. We report the average 95% mean confidence interval in brackets/errorbars, In the imbalanced support set experiments,
evaluate tasks with various imbalance levels and distributions, as specified in figures and tables. In the imbalanced meta- meta-dataset experiments, we evaluate using regular, balanced 5-shot 5-way tasks.
13

---

Under review as a conference paper at ICLR 2021
Data Augmentation. During the meta-/pre-trainie phases, apply standard data augmentation techniques, following a similar setup to Chen et al. (2019), with a random rotation of 10 degrees, scaling, random color/conttras//bightnsss jitter. dletavvalidation and meta-testing had no augmenta- tion apart from in the Random-Shot (ROS+) setting where the same augmentations were applied on the oversampled support images. All images resized 84 by 84 pixels.
A.3 BACKBONE ARCHITECTURES
All methods shared the same backbone architecture. For the core contribution of our work, we used Conv4 architecture consisting of 4 convolutional layers with 64 channels (padding 1), interleaved by batch normalization (loffe & Szegedy 2015), ReLU activation function, and max-pooling (kernel size 2, and stride 2) (Chen et al. 2019). Relation Network used max-pooling only for the last 2 layers of the backbone to account for the Relation Module. The Relation Module consisted of two additional convolutional layers, each followed by batch norm, ReLU, max-pooling).
For experiments with different backbones: Conv6, ResNet10, and ResNet34 (Chen al. 2019). Conv6 extended the Conv4 backbone to 6 convolutional layers, and max max-pooling applied af- ter each first layers. ResNet models (He et al. 2016) followed the same setup as Chen al. (2019).
For imbalanced meta-dataset and imbalanced reduced meta-dataset, we used the Conv4 model, with 32 channels instead of 64, due to less training data.
A.4 FSL. METHODS AND BASELINES
In our experiments, used a wide range of FSL methods (full details can be found in our source code):
Baseline (fine-tune) (Pan & Yang. 2010) represents classical way of applying transfer learning, where neural network is pre-trained a large dataset, then fine-tuned on smaller domain-specific dataset. The baseline's 's backbone followed a single linear classifi- cation layer with single output for each meta-training dataset class. The whole network was trained during pre-training. During meta-testing, the baseline's 's pre-trained linear layer was exchanged for another randomly initialized classification layer with outputs matching the task's number of classes (N-way). Fine-tuning was performed on the new randomly initialized classification layer using the support set S.
2. Baseline (1-NN) is another classical method of applying transfer learning but using a k- nearest neighbor classifier instead of the classification layer during meta-validation. Pre- training was performed in the same was the fine-tune baselines. During the meta-testing time, instead of fine-tuning, the model matched query samples to the nearest support sam- ple's class based on Euclidian distance.
3. Baseline++ (Chen et al. 2019) augments the fine-tune baseline by using Cosine Similarity on the last layer.
4. Matching Network (Matching Net) (Vinyals et al. 2017) uses context embeddings with an LSTM to effectively perform k-nearest neighbor in embedding space using cosine simi- larity to classify the query set.
5. Prototypical Networks (ProtoNet) (Snell al. 2017) maps images into feature space and calculates class means (called prototypes). The query samples are then classified based the closest Fuclidian distance to classes' prototype. We evaluate two models, one meta-trained like the others on 5-way episodes, and another variation trained on 20-way episodes. During 20-way meta-training, we set the class' query size to 5.
6. Relation Networks (Relation Net) (Sung et al. 2017) augment the classical Prototypical Networks by introducing a relation module (another neural network) that compares the distance instead of using Euclidian. The original method uses Mean Squared Error to minimize the relation score between samples of the same type. However, we follow work by Chen et al. (2019), use cross-entropy loss that expediates meta-training. The structure of the relation module is described in section A.3
7. DKT (formally called GPShot) proposed by Patacchiola et al. 2020 is probabilistic ap- proach that utilizes the Gaussian Processes with a deep neural network as a kernel function. We used Batch Norm Cosine distance for the kernel type.
14

---

Under review as a conference paper at ICLR 2021
8. SimpleShot (Wang al., 2019a) augments the I-NN baseline model by normalizing and centering the feature vector using the dataset's mean feature vector. The query samples are assigned to the nearest prototype's 's class according to Euclidian distance. In contrast to the baseline models, pre-training is performed on the meta-training dataset like other meta-learning algorithms, and neta-validation is used select the best model based on performance on tasks sampled from Dval.
9. MAML (Finn et al. 2017) is meta-learning technique that learns a common initialization of weights that can be quickly adapted for task using fine-tuning on the support set. The task adaptation process uses standard gradient descent algorithm minimizing Cross-Entron loss on the support set. The original method uses second-order derivates; however, due to more efficient calculation, we use the first-order MAML, which has been shown to work just as well. We set the inner-learning rate to 0.1 with 10 iteration steps. We optimize the meta-lcarner model on batches of 4 meta-training tasks. These hyperparameters were selected based on our hyperparameter fine-tuning.
10. ProtoMAML (Triantafillou et al., 2020) augments traditional first-order MAML by reini- tializing the last classification layer between tasks. Specifically, the weights of the layer are assigned to the prototype for each class's corresponding output. This extra step com- bines the fine-tuning ability of MAML and the class regularisation ability of Prototypical Networks. We set the inner-loop learning rate to 0.1 with 10 iterations. Unlike for MAML, we found that updating the meta-learner after a single meta-training task gave the best performance.
11. BAYL (BMAML) (Yoon et al. 2018) augments the MAML method by re- placing the inner loop's standard stochastic gradient descent with Bayesian gradient-based updates. BMAML uses Stein Variational Gradient Descent that is a non-parametric varia- tional inference method combining strengths of Monte Carlo approximation and variational inference. The algorithm learns to approximate posterior over the initialization parame- ters conditioned on the task support set. Yoon et al. 2018)) also adds a chaser loss, which utilizes the samples in the query set during meta-traimin
approximate the true task pos- terior. Minimization of the KL divergence between the true task posterior and the estimated task parameter posterior can be used to drive the meta-training process. We set the inner- loop learning rate to 0. with 1 inner-loop step. We found that using a higher inner-loop step number destabilized performance. We used 20 particles. The chaser loss variation used a learning rate of 0.5. Again, we found these combinations of hyperparameters to give the best results.
12. Bayesian-TAME (BTAML) (Lee et al. 2019) [Left out of the main paper body.] This method augments the MAML method with four main changes: 1) task-dependent pa- rameter initialization z, 2) task-dependent per-layer learning rate multipliers 7, 3) task- dependent per-class learning rate multiplier w. 4) meta-learned per-parameter learning rate a (similar to Meta-SGD, Li et al. (2017)) However, our results for BTAML were un- stable, suggesting a fault in our implementation. Unfortunately, we did not identify it in time for the submission, and we decided to move the method's results to the appendix. In our experiments, we explored several variations this method with various parameters (z, w., Yr o)) turned on and off, as well as different hyperparameters. We found that using meta-learning rate of 10- performed better than 10 in contrast the other models. We set the inner-learning rate to 0.1 with 10 iteration steps, and optimize the meta-learner model on batches of 4 meta- veta-training tasks. Again, we found this set up worked best in our experiments.
A.5 CLASS IMBALANCE TECHIIQUESS AND STRATEGIES
We pair FSL methods with popular data-level class-imbalance strategies:
1. Random Over-Sampling (ROS) (Jakkowizz & Stephen] 2002) without and with data augmentation (ROS and ROS+). For the augmentations we used: random sized crop- ping between 0.15 and 1.1 scale of the image, random horizontal flip, and random color/brihhness/cotrras jitter. A visualization of ROS and ROS+ is presented in Figure 7 During meta-tran training, ROS+ augmentations were applied twice: once when sampling from the meta-training dataset, and the second time during the support set resampling. This may have slightly destabilized meta-training, which would explain why sometimes Random-
15

---

Under review as a conference paper at ICLR 2021
Shot with ROS (ROS+ at inference only) achieved better performance than Candom-Shor with ROS+ Figures andd
2.
2019 Chen et al, 2020 was applied as specified in the main body of the paper (Section 3.4).
Random-Shot Meta- Teta-Training (Triantafillou et al., 2020; Lee et
3. Focal Loss (Lin et al. 2017 Focal Loss has been found to be very effective in combating the class-imbalance problem on the one-stage object detectors. We exchanged the inner- loop cross-entropy loss of optimization- focal loss with 7 and a Results are presented in Figure 6 in the main paper body and in Appendix D.1
algorithms and fine-tune baselines with the
4. Weighted Loss. Weighted loss is also commonly used to rebalance the effects of class- imbalance (Buda et al. 2018 Leevy et al. 2018). We weight the inner-loop cross-entropy loss of aptimizatiionbaasee algorithms and fine-tune baselines by inverse class frequency of support set samples. Results are presented in Figure [ in the main paper body and in Appendix D.I
Figure 7: Visualisation linear 1-5shot support sets. Left: no ROS. Middle: ROS. Right: ROS+.
16

---

Under review as a conference paper at ICLR 2021
B VERIFICATION OF IMPLEMENTATION
We implement the FSL methods in PyTorch, adapting the implementation of (Chen et al. 2019) but also borrowing from other implementations online (see individual method files in the source code for individual attribution). However, we have heavily modified these implementations to fit our imbalanced FSL framework, which also offers standard and continual FSL compatibility (Antoniou et al. 2020). We provide our implementations for ProtoMAML and BTAML for which no open- source implementation PyTorch existed as writing. To verify our implementations, we compare methods on the standard balanced 5-shot 5-way task with reported accuracy. Results are presented in 2 We see that algorithms achieve very similar performance with no less than 3% accuracy points compared to the reported performance. The discrepancies training batch for SimpleShot, different augmentation strategies for the other methods, and natural variance stemming from random initialization. We show the validation performance epochs for each method in Figure g on the next page.
be accounted for due to smaller
Table 2: Results of standard 5-shot 5-way experiments on Mini-lmageNett as achieved with our implementation compared the original (reported) accuracy and other work. Other Sources's Ac- curacies were taken from: (Chen et al. 2019). (nell & Zemel 2020). (Vogelbaum et al. 2020
Original Acc(95%CC)
Other Sources'
Our Acc (95%CI)
Acc(95%CCT)
Model
Baseline (1-NN)
39.72=a33
Baseline (fine-tune)
62.67
62.53.am
Baselinett
66.43
66.43aaa
Matching Net
62.27
55.31 +9.23
63.48
ProtoNet
64.37 20m
65.77
64.24
ProtoNet (20-way)
65.762070
68 200s
66. 00.688.nh 20.0
Relation Net (CE)
64.76=n.s
65.32+000
66.60=e.0
DKT
62.92
64.00 Lom
62.88 Loos
SimpleShot
63.74.nm
6
MAML
61.83=071
63.15
62.71
ProtoMAML
59.86-076
60.70=am)
BMAMI.
59.89
59.23
BMAMI. (chaser)
56.45_os7
59.93
17

---

Under review as a conference paper at ICLR 2021
20
50
30
Randomm-Snat (Rio))
20
Ra Romoo Shot (ROS with buga
1.0
Masc Wing Not
Troto
Protehid (20-way)
54
40
355
1000
1.00
1.50
150
200
Fpei
Eyecch
-5
40
80
50
1.00
150
200
1.00
1.50
200
1.50
200
NAVE
ProtoNANL
100
1.00
1.5
200
Fpech
Spechs
Fpecch
1.00
150
200
Figure 8: Validation performance through epochs on Standard 5-shot 5-way meta-training, and Ran- dom Shot (1- I-ho random). The shaded areas show standard deviation over three repeats on different seeds.
18

---

Under review as a conference paper at ICLR 2021
C BREAKDOWN OF RESULTS
In this section, we breakdown the results from the main body of the paper. Specifically, we provide the top-50 performing models on the imbalaned 1-9shot linear task subsection C.I The break- down of higher shots experiment (Figure 4 is provided in subsection C.2 In subsection C.3 we include the breakdown of the backbone experiment (Figure [] showing the performance by algo-
rithm type and training procedure. We provide precision and recall tables of linear 1-9shot 5-way
tasks for each meta-training procedure in subsection C.4
In section
C.5 we provide the results for
the imbalanced reduced meta-dataset (Figure TO)
C.1 TOP-50 PERFORMING MODELS ON 1-9SHOT LINEAR
Table 3: Top-50 models using different meta-training strategies, showing absolute and relative dif- ference between the balanced and the imbalanced task. Results sorted by relative difference.
5shot
1-9shot linear
Abs. Diff.
Rel.
Model
Training Method
Matching Net
Random-Shhe Shot (ROS+ at inference)
58.12
56.17
-1,9461
-0.0335301
Random- Shot with ROS (ROS+ at inference)
60.59
58. a8
-2,28958
+0.0377888
Standard (ROS+ inference)
60.26..ae
$7.90
-2.36208
-0.0919522
Relation Net (CE)
Random-Shoo Shot with ROS+
60.48 ea:1
$7.99
-2,49653
-0.0412758
ProtoNet (20-way)
Random- Shot with ROS+
61.21
58.58
-2,62778
-0.0429288
Matching Net
Random-Süi Shou with ROS
61.75
59.07
-2,67331
-0.0433091
Relation Net (CE)
Random-Shot
63.502a10
60.64297
-2.86181
-0.0 040445648
Matching Net
Random Shot with ROS+
60.05 Lozk
$7.24
-281111
0.0468096
Baseline (1-NN)
Standard
39.72
37.85+012
-187222
-0.0471337
BMAML
Random-Shol (ROS+ at inference)
49.97 go:t
47.53
-2,43611
-0. 0,0487513
Baseline (1-NN)
Random-Shol with ROS
39.90
37.95+03
-195119
-0.0489112
BMAML (chaser)
Random-Shot (ROS+ inference)
45,00
42.79:aa
-2.20833
0.0499778
Relation Net (CE)
Random Shot with ROS (ROS+ at inference)
62.97
$9.84
-3.3303
0.0497668
Random-Shoe with ROS
64.12
60.93
-3.19444
-0.0498181
DKT
Random-Shot (ROS+ inference)
62.33
59.17
-3,15886
-0.0506178
Relation Net (CE)
Random Shot (ROS+ at inference)
62.53um
59.301am
3.22361
-0.0515549
Baselinet+
Random-Shie (ROS+ at inference)
65.52
62.14
-3.37847
-0.0515628
DKT
Random Shot with ROS (ROS+ at inference)
62.40
59.181as
3,22083
-0.5161599
Baseline++
Random-Sil Shoooo with ROS (ROS+ at inference)
65.00
61.60
-3,40694
-0.0524117
Baseline (I-NN)
Random-Shot Shhot with ROS+
40.83
38.68
-2.42336
-0.0524749
PeotoNet
60.05g07
56.891aa
3.15764
-0.1555823
BMAML
Standard (ROS+ at inference)
$8.82 :o2
S5.72
33.10
-0.0527811
Random Shot ROS (ROS+ inference)
58.00
54.84194
3.15486
0.0543948
Baseline++
Standard (ROS+ inference)
65.27 :oxe
61.72
-3,55583
-0.0544764
DKT
Standard (ROS+ at inference)
61.96-u1
58.57
-3.3925
-0.054749
Random- Shot
ROS+
61.16.0m
57.71
3,44722
MAML
Random: Shot (ROS+ at inference)
50.39
47.52
-,86597
-0.0568763
BMAML (chaser)
Random-SDe Shoott--Sottooo with ROS
53.52
50.46
-3.05117
-0.057068
ProtoNet
Random: Shot (ROS+ inference)
50.67
47.76:38
-290972
-.05743003
BMAML
Random-Shot with ROS+
56.52
53.23+
-3,28403
-0.588088
Net
Landom-Shot
51.65
48.57.296
-3.08194
0.0596698
BMAML (chaser)
Random Shot with ROS (ROS+ inference)
$1.87
48.7419a
-313194
0.0603821
ProtoNet (20-way)
Random-Shot (ROS+ at inference)
58.31
54.79
-3,52153
0.0603935
Baseline (I-NN)
Sandom-Shhot with ROS (ROS+ at inference)
39,22 :3:e
36.83
-239922
- 0.0609506
MAML
Random-Shor with ROS+
54.60-a01
51.25
-3,35069
-0.0613626
Random- Shot with ROS (ROS+ at inference)
58.362271
54.70
-3663319
-0.0627707
Relation Net (CE)
Standard (ROS+ at inference)
63.89 pare
59.85
4.035
1.06331558
BMAML (chaser)
Standard (ROS+ inference)
55.40
5189:a4
3350958
-0.0633513
ProtoNet (20-way)
Rundom-Short with ROS (ROS+ at inference)
58.32
54,62
-3,70556
0.0663559
BMAML
Random-Shaa with ROS
58.98
55.23+am2
-3.75139
00.0636038
Baseline (1-NN)
Standard (ROS+ inference)
39.75
37.18294
-2,57042
-0.06466255
MAML
Standard (ROS+ at inference)
61.00.cm
$7.04
-396083
-0.0649286
ProtoNet
Random-Shov with ROS (ROS+ at inference)
54.39-aa4
50.74+am
-3,64375
0.0669961
Baseline (fine-tune)
Random-Shot with ROS+
60.46
56.12:aa
-,33819
-0.0717551
DKT
Random: Shot with ROS
63.21 1om
58.65100
4.55625
-0.0720799
Baseline (I-NN)
Random-Shot (ROS+ at inference)
29.98
37.08
-2.00117
-0.0726382
Baseline (fine-tune tune)
Random Shot with ROS (ROS+ at inference)
61.82.ar
57.03186
4,79236
-0.0775149
Standard (ROS+ at inference)
61.46-m
56.65+am
-48075
-0.0782216
ProtoMAML
Random-Shot (ROS+ at inference)
54.09
49.73
-4,55903
0.0805521
Baseline (fine-tune)
Random Kandom-Shot (ROS+ at inference)
61.631010
56.5510s4
-5.77669
+0.0823633
19

---

Under review as a conference paper at ICLR 2021
C.2
RANDOM 1- 1-29SHOT AND 1 1-49SHOT TASKS
avr. shot
75
70
Basdlres Spntard
Bresres Stodard (ISS intererce)
Drestre fandem:5he Stat
Bisdires Fando Shot (RGS offrence
Bistfres Fondo Shot with ROS
Bisdires fandom-Shot with RCS (RO5- inference only)
Basdires fundom-Shot with RCS-
Mcuk: sssed Stanoard
Meit-cored Stntaar internnos)
Sandom-Sher
Sandom-Shor (RCS- inference ony)
only
Optiiieiiln-teec Bandom
25 avr. Shot
Optmicestennteree Random (50S interence onl)
optmieesler-beee beseeddddeed-e-ee Random Shott IC5
gpininnilr-teeed Barden-Shit RDS (BDS+ infernce only
Opiminelln-teeed Sardem-Ghct-oe with 805+
80
Imbalance, p KruatKres
Figure 9: Linear imbalance by model type. Best viewed in color on a computer. The metric-based methods (represented by green-cyan lines) tend be perform better than the baselines (red-orange) and the optimization- based models (purple-pink).
20

---

Under review as a conference paper at ICLR 2021
C.3 BACKBONE EXPERIMENTS
We run additional experiments with different backbone models. In Tables and 5 we show the random 1-9 shot 5-way performance on Conv6, ResNetI0, and ResNet34. We can observe that for many of the methods, ROS still benefits the models. However, the reader should exercise caution since we used the same prrereaaramtterizztiion as for the four-layered CNN. We did not perform any hyperparameter fine-tuning on these backbones; the results would likely be higher if we allowed for longer meta-training. Some results are missing due to destabilization meta-training caused by deeper backbones.
Table 4: Standard vs Random Shot (accuracy)
Stundard ResNet10
Rundom-Shhe Shot ResNet10
Conv6
ResNer34
Conv6
ResNet34
Baseline (I-NN)
51,21
53.18
53.25
50.01 =0:0
52.24
52.49
(fine tune-tune)
14.12.
58.55 ue
59.02.m
54.62_nm
58.36100
60.54_.00
Baselinc++
51.95 s0s9
50.94:ca4
54, cms
52.23
50.99
52.91 :0.0
Matching Net
54.99
58. =3.60
60.00
53,70
56.34
ProtoNet
57.92.00
62.06
64.35
56.31
20.
59.81 2075
ProtoNet (20-way)
57.66
60.61
62.17
62.46
DKT
55.60
57.98
57.07 am
57.44_
58.77 58.77 1m
SimpleShot
54.36
60.95
60.48
54,90
61.33
55.25
MAML
52.53
57.69
52.52
46,91 =0072
52.61
ProtoMANI
53.00
53.59 96
51.22
49.72_mm
54.37 2M
52.48 2077
BMAML
39.39
48.09
46.76
BMAML (chaser)
35.81 Hour
30.90
30.27 -u37
33.03
31.62
26.41 -044
Table 5: Random Shot (ROS) vs Random Shot (ROS+) (accuracy)
Rundom-Shot (ROS)
Random- Shot (ROS with aug.)
Conv6
ResNet10
ResNer34
Conv6
ResNet10
ResNer34
Baseline (I-NN)
51.05
52.65 s0:5
46.75
50.69
Baseline (fine- tune)
55.32 uno
59.77...
54.48
59.02.00
60.48
Baselinet+
61.08 -130
57.41
58.07 cm2
56.95
54.81
55,49
Matching Net
58.32
60.48-430
56.24-me
59, 17.9 2075
19.19
ProtoNet
56.99
60.47
58.60
62.05
ProtoNet (20-way)
57.66
59.85
61.60
58.82
61.78
DKT
57.01
59.12.4m
56.36_no
58.09 28.09... 1m
SimpleShot
52.99
60.46:ca0
51.44
54.20
37.14
MAML
56.25=004
59.72
50,27
46.76 46.76
MMI
56.99
$5.90 15
41.45
47.17 17.17
40.85 .077
BMAML
56.58
58.20=b34
61.02
53.55
57.25
60,50-000
BMAMI (chaser)
52,23
35.18
32,33 uo:0
47.76
38.84_.14
23.68 23.68_n0m
21

---

Under review as a conference paper at ICLR 2021
C.4 PRECISION AND RECALL TABLES
Table 6: Precision and recall for linear 1-9shot 5-way tasks after Standard meta-training
Recalk95SCC) 0
Avt. F1(95%)D
K2
3
Ng
5
Kg
Rg
Kg
Rg
Ng
Baseline (I-NN)
0.32
0:d6
0.50 o.s
0.d8
0.45
0.13
0.29
.00 0.41 1od
0.50.a1
0.57 .ox
0.22 0.0
Baseline (fine-tune)
0.63
0.6d..m
a.50
0.00
0.19-sm
0.53
0.75.
0.90..
0.30-co
Baseline++
0.02
0.45.r44
0.66
0.52
0.39.r
0.00
0.13
0.48axm
0.76.x0
0.54..3
0.37.0x
Matching Net
2.5
0.67
0.62.m
0.55gam
0.15
0.61amm
0.73grxx
0.84.0
0.50
ProteNet
a.18
0.70.cis
0.62
i.54
0.5
0.03
0.37.cc
0.65
0.78
0.84.o
0,47.cx
ProteNet (20-way))
0.10
0.64-1
0.55.mm
o.4%..m
0.01.m0
0.8_.. 0.79..m
0.45-ci
Relation Net (CT)
0.4
0.62.0
0.56.au
0.54.rm
0.09.x>
0.48
DKT
0.57.am
0.63
066--
0.57.amm
a.
0.22.am
0.48
0.65.m
0.73
0,79->>
0.5
SimpleShoo
0.01 1Ms
0.59.cc
0.66.0m oz>
0.51
0.00.10
0.15 .03
0.52
0.77
0.9..ox oz
0,38
MAML
0.65
0.58..n
0.42...z
0.00
0.26 0
0.55 ...
0.75
015
0.4
ProtoMAMI.
0.18:
0.50
0.55.n
0.44gmm
0.38
0.03gare
0.23
0.68
0.76
0.36
BMAMI.
0:00
0.06
0,38_n
0.5
0.27
0.00
0.01
0.16.im
0.51
0.98
0.22gm
BMAML (chaser)
0.00
0.00.cot
0.15
0.46
0.24
0.00
0.00
0.04
0.37
0.99.cx
0.16.09
Table 7: Precision and recall for linear 1-9shot 5-way tasks after Random Shot meta-training.
Procision (KC))
Ave. F1(956CC
Ki 1
A2 3
Ky - 5
K4
Ks =
Kg -
A2
Kg 5
K4 7
Kg
Baseline (3-NN)
0.32
0.48.ca
a.59
0.46...3
0.13
0.30-cia
0.30
0.42
0.50.c3
0.58 ois
0.33 -333--00
Raveline (fine-time)
B.06.
06
0.63.mm
g.50
04
0:00
0.19
0.53gmm
0.76
0.50..1
0.30
Basclinet-
04
0.67
B.52gam
0.00
0.13:1
0.49-in
0.77
0.94
0.37
Matching Net
0,47 xx58
0.65.en
0.62
0.55
0.45cmm
0.24
0.36.015
o.55
0.6k:e1
0,84.0
0.49.002
ProtoNet
0.51-aa
0.49
0.48.cs
0.17
0.44-c
0.63-27
0,45-ca
ProteNet way?
0.5
0.16:
Relation Net (CE)
2.55
0.62
0/64-s
0.64..m
0.48:.11
0.58
0.6..m
0.66ssor
0.67.pa
0.59.cm
DKT
0.58
0.63.co
0.57
0.55.ca
0.21xx1
0.48
0.65.xi
0,74
0,79.0
0,53.00
SimpleSitot
0.06 c.o
0.64:ca
0.62
om
0.50
as
0.44.ca
0,01.10
0.22 c.
0.56 mm
0,76.
0.87-o3
oz)
0.1--ca cc
MAML
0.07 1x0
0.45
0.50
0.42 imo
0.36 co:
0.01:10
0,16 03
0.44
0.6d110)
0.8
0.33
ProteMAMML
0.15
0.59
0.55..o
6.02
0.23
0.49
0.68
0.79
0.37
BMAMI.
0,14
0:42
0.51
0.44
0.37
0.03
0.19.0
0.4
0.62
0.80.
0.33
BMAML (chaser)
0.06
0.44
0.40
0.35
0.01
0.13
0.36
0.60
0,78
0.28
Table 8: Precision and recall for linear 1-9shot 5-way after Random Shot (ROS) meta-training.
Recalk95%
Avr. F1(95&CC)
Ki 1
A2
3
Kg - 5
K1
7
Ks- -
Rg
Kg
3
Kg 5
K4 7
Kg- 9
Baseline (1- NN)
0:47:ces
0.50 ozo
0.49
0.16.co
0.13
0.29 .ci
0.42
0.50
0.57 .0o
0.32 on0
Baseline (iine tune)
0.16
0.09
0.02
0.52
0.44.ca
0.02
0.26-c14
0.57
0.76.si
0.88
0.42
Basclinee
0.40
0,71
0.64
2.57
0.52ara
0.12atms
0:45:n4
0.66..m
0.78.xm
0.86.mm
0.52gne
Matching Net
0,56
0.60.con
0.62
0.61ixs
0.6l:cs
0.46
0.57.cm
0.57
0.52
0.64.10
0.66
0,57
ProteNet
0.55
0.49.ca
0.15
0.5%
0447oxx
ProtoNet (20-way)
0.60
0.54.as
052.co
0:15
0.47-c12
0.62
0.69
0.75-om
0.50-co
Relation Not (CF)
2.56
0.62gmm
0.6
0.63.ass
0.64.rm
0.58_
0.64.m
0.66
0.60.mm
0.60
DKT
0.59 su
0.63ico
0.61 ozr
0.59 us
0.57.co
0,27
0,51.60
0.56 iu
0.72
0.77 .ox
0.55
SimpleShot
0.10
0.61:02
0.60.am
0,50
0.41:00
0.01:x00
0.22
0.50
0,71
036
0.39.0x
MAML
0.52
0.59
0.56
0,52
0.49
0.15
0.43 41
0.50
0,70
0.76-o2 oz
0.49
ProtoMAMM.
m.
0.58
0,53
B.46gam
1144gr
0.07
0.37
0. 0.54
0.63am
0.72gmm
0.40
BMAML
0.59
0.57
0.59
i.6
0.63.c1
0.54
0.55
0.58
0.55
0.54_ous
0.52
BMAML (chaser)
0.54.m
0.55
0.57
0.51
0.51.00
0.47.0
0.47
Table 9: Precision and recall for linear 1-9shot 5 5-way after Random Shot (ROS+) meta-training
Recal199SCD
Avr. F1(906CC
K: 1
N2 -3
Kg 5
K4
7
Ng
8g
Rg
5
7
Kg
Baseline (1 NNN
0.36 um
0.33
0.42
0.46
0.48
0.57 km
0.73 c0
0.30
0.13 cm0
0.15 om
0.26
Bascline (fine-tune)
0.60
0.16urm
0.63.m
0.58
0.55.r
m.28
0.52.mm
0.53
0.69
0.78-nm
0.53
Baselinct
0.61.m
0.64
0.60:
0.54.c
0.37
0:47:015
0.54
0.72:co
0.82
0.55.00
Matching Net
0.55
0.62.m
0.59..u
0.56.ra
0.43
0:47.00
0.58
0.72
0.55_re
ProteNet
0.59
0.55
cs
0.58 cu
0.68
ProtoNer (20-way)
0.02
0.56
0.57-1m
0.62.aua
0.6
0.25
0,72
0.69
0.63-.
0.56
Relation Ne: (CE)
1.55
0.57.con
0.60.as
0.60.
0.6..o
0.41
0.54.:12
0.63
0.66.100
0.66..
0.56.co
DRT
0.56
0.61
0.62.nm
0.58
0.57.ra
0.37
0.45
0.61axm
0.70.rm
0.76.
0.55
SimpleShoe
0,50
0,56
0.58
0,52 uu
0.52 css
0.11
0.11
011
0.51 ixd
0,76 uss
0.79 03
0.47
MAML
1.48
0.47
0.52-mm
0.53gam
0.56
0.38
0.47
0.54
0.59
0.60.on
0.50
ProtoMMM
0.43
0.43
0.47-ac
0.48
0.1--ca
0.32
0.53-o44
0.4-cz
RMAMI.
0.4k
0.57
0.60.m
0.56urm
0.54
0,68
0.40./m
BMAML (chaser)
0.54
0.5l.ci:
0.57
0.54.cx
0.48is15
0.41.c0
0.46
22

---

Under review as a conference paper at ICLR 2021
C.5 IMBALANCED REDUCED META-TRAINTE DATASET In this section, we present the results for the reduced meta-training MinilmageNet dataset (with 32 classes).
MinilmageNet
Acc. Diff. to balanced,
Accuracy,
50
16
16
24
# Minority
Figure 10: Combined model average performance with increasing minority classes. Left: Com- bined accuracy of all models and training scenarios. Right: Relative performance difference the balanced dataset.
Table 10: Table showing full results for the reduced meta-training MinilmageNet dataset (with 32 classes), evaluated on (meta-test split of) MinilmageNet. The last two columns show the average and maximum model's (absolute) performance difference to the balanced task. RelationNet suffers the most from the imbalanced meta-tran vecaatraining.
Imbalance Z
balanced
step
Avr. Diff.
Max. Ditt.
Max. # class samples
150
190
270
$10
Min. class samples
150
30
30
Minority
Baseline (I-NN)
40.4 40.46.205
41.86.nm
40.93
40.52
0.5
Baseline (fine-tune)
54.13=020
53.72.s0.
53.36
52.45
-0.7
Baselinett
54.40=364
54.15
53.55
52.89=0ss
-0.7
-1.5
Matching Net
53.56
53.34
52.69
51.622ns7
-0.8
-1.9
ProtoNet
54.14
53.54
53.38
52.81
-0.7
-1.3
ProtoNet (20-way)
55.05
54,98
53.40
51.98-ast
-1.2
-3.1
Relation Net (CF)
53.83
52.80
49.97.a.
49.55
-2.3
-4.3
DKT
$4.09=a.s
$3.57
53.01
52.27
SimpleShot
56.05 Laz
55.96
55.70.em
54.83_ov
-0.4
-1.2
MAMI.
50.40_aams
49.95
49.27
49.07
-0.7
-1.3
ProtoMMML
42.57
40.75
42.10
45.96=nss
0.3
-1.8
Table 11: Table showing full results for the reduced meta-training MinilmageNeet dataset (with 32 classes), evaluated on (meta-test split of) CUB. The last two columns show the average and maximum model's 's (absolute) performance difference the balanced task. RelationNet suffers the most from the imbalanced meta-training dataset.
Imbalance Z
balanced
Avr. Diff.
Max. Ditt.
Max. #l class samples
150
190
270
510
Min. class samples
150
30
30
30
Minority
Baseline (1-NN)
41.87
41.85
41.38
40.42=nst
-0.5
-1.4
Baseline (fine-tune)
53.16=80
52.63+eum
52.98
50.69
-0.8
Baselinett
51.65
52.40
52.31
48.67
-0.4
-3.0
Matching Net
49.90 =9m
19.12.amm
49.12
47. 47.82=nse
-0.9
-2.1
ProtoNet
50.01 =azi
49,78+am2
49. 49.34+071
48.97
-0.5
-1.0
ProtoNet (20-way)
50.09 aam
49.40
49.161671
47.74_nst
-1.0
-24
Relation Net (CE)
50.71 Lann
48. 6,.6 s0/9
46.78=nns6
-1.6
-3.9
DKT
63.84
$2.45
53.75
52.42=aus
-0.7
SimpleShot
53.01
52.60
52.03
50.44_mm0
-1.0
MAML
51.15
50.67
50.23
49.90=nss
-0.7
-1.2
ProtoMMM
45.20
43,22
43.81
48.69=n.s
0.0
-2.0
23

---

Under review as a conference paper at ICLR 2021
D ADDITIONAL EXPERIMENTS
D.1 ADDITIONAL IMBALANCE STRATEGIES
In Figure 11 we compare Standard meta-training and Kandom-Shor meta-training with focal loss. We observe significant advantage of using Focal Loss Random- Shot meta-training over the Stan- dard meta- cocta-trainin experiments.
Baseline (fine-tune)
Baseline++
MAML
ProtoMAML
70
60
40
&
20
BMAML
BMAMLL (chaser)
70
Standard IROS+ Inference)
K. Standard If ooocal Loss at inference)
Standard (Weighted Lass
40
Random Shot (ROS+ at inference)
Random Shot (Focal Loss inference)
(Weighted inference)
1
2
4
8
2
5
8 9
Imbalance, P Kear/Kemin
Figure 11: Standard episodic (meta-) )training VVnyals et al. 2017 and random-shot episodic meta- training (Triantafillou et al. 2020) with Weighted Loss (Buda al. 2018 and Focal Loss (Lin al.
2017 applied to the inner- loop of optimization based functions and fine-tune baselines.
D.2 BTAML
We implemented and trained Bayesian TAML (Lee et al. 2019); however, the training performance graphs suggest mistake in our implementation, which we did not manage to identify time for the submission. For this reason, we have left out these results from the main paper. The BTAML. (a, w, 7, 2) corresponds to full BTAML, while others indicate variants with the corresponding com- ponents turn off. We provide models' performances with the full performance in Appendix E
Table 12: Performance of our implementation of BTAML 5shot 5-way task. The BTAML
(a, w,y, 2) indicates the full version of the proposed model.
Ace (95%CI) Fl (95%CI)
Model
BTAML (at, W/ 7, 2)
52.94.076
52.30
BTAML 7;
54.89 20.75
54.52 1139
BTAML (a:
52.19 60.66
51.85 11.41
50
30
20
100
150
100
150
200
Validation performance of BTAML through epochs using Standard 5-shot 5- 5-way meta- training. and Random Shot (1-9shot random). The shaded areas show standard deviation over three repeats on different seeds.
Figure 12:
D.3 ANALYSIS SAMPLES PER CLASS IN
DATASET
24

---

Under review as a conference paper at ICLR 2021
Table 13: Meta-/Pre- Training with reduced number of samples in the meta-training dataset of Mini- ImageNet (all 64 classes are balanced). Setting with 600 samples uses 64 channels for each convolutional layers instead of channels as was done for the other *#Class Samples' settings. In addition, all three Baselines were trained using conventional split on Dtrain instead of D Piyain- The table suggests that the number of samples per class in the meta-training dataset is not very significant on the performance of FSL algorithms beyond certain point. All settings were trained on 50k tasks, apart from 600 " trained on 100k tasks.
Class Samples
50
100
300
600
Model
Baseline (1-NN)
38.75
39.82
42.69
39,72
Buseline (fine-tune)
44.34
46.10-aa
51.26
62.67
Buseline++
41.63 anw
49.51 Lave
48. 48.44_0o
66.43 286
Matching
55.90_om
58.30
58.26
62.27 Low
ProtoNet
58.43
60.199
60.65__mm
64.37
ProtoNet (20-way)
$7.87 1om
59.88 cm70
60.91 Lom
65.76 100
Relation Net (CE)
56.50=sm
60.93
62.78
64.76-tx
DKT
56.27
57.93
58.09
62.92
25

---

Under review as a conference paper at ICLR 2021
E FULL RESULTS FOR MAIN EXPERIMENTS
Figures 14 and 13 show accuracy and FI scores, respectively, on imbalanced tasks of Standard (5- shot) and Random Shot (1-9shot) meta-training. Their corresponding result tables are in Tables [4
to 21 Table 22 provides the full results for the main imbalanced meta-dataset experiments.
(Standard, accuracy)
(Random-Shie (ROS), accuracy)
(Random-Shot, accuracy)
(Random-Shot (ROS with aug.), accuracy)
5shot (balanced)
70
10
1-9shot random
3-7shot random
4-6shot linear
70
60
3-7shot linear
Accuracy,
1-9shot linear
1-9shot step 1minor
70
1-9shot step 4minor
60
10
1-6shot step 1minor
70
10
1-21shot step 4minor
Baseline
(1-NN)
Baseline
(ime-tune)
Baseline+++
Matching
ProtoNet
ProtoNet (20-way)
Relation Net (CE)
DKT
SimpleShot
MAML
ProtoMAML
BMAML
BMAML (chaser)
BTAML
G. V.
BTAMM
(z)
BTAML (at)
Net
Figure Accuracy
26

---

Under review as a conference paper at ICLR 2021
(Standard, f1)
(Random-Shot (ROS), f1)
(Random-Shot, f1)
(Random-Shot (ROS with aug.). f1)
5shot (balanced)
20
10
0
1-9shot random
50
3-7shot random
000000
4-6shot linear
accccoo
3-7shot linear
1-9shot linear
F1
1-9shot step 1minor
(000000
1-9shot step 4minor
10
1-6shot step 1mnnor
40
10
1-21shot step 4minor
Baseline
(1-NN)
Baseline
(eeetune)
Baseline++
Matching
ProtoNet
ProtoNet (20-way)
Relation
Net Net (CE)
DKT
SimpleShot
MAML
ProtoMAML
BMAML
BMAML (chaser)
BTAML
60, Y.
BTAML
(.)
BTAML (a)
Net
Figure 14: Scores.
27

---

Under review as a conference paper at ICLR 2021
Table 14: Standard (accuracy)
Sahot
1-9shor random
3-7shot random
4-6shot
1-9shot linear
1-9shot step Iminor
1-9shot step
s-or Aminor
Rank
4minor
step
Baseline (I-NN)
39.72
38. ...
39, 20.68
...
37 2
39.67
31 :1.68:
29. 81 cra
11.6
Baseline (lne tune)
62.67 0.00
49.72
57.17 .m.
59.78 .m
47 17.40
58.39
21 21.26
20.33 cm
7.8
Baseliness
66.43
49.
57.20
61.17
46
60.93..m
21 17
20.28
Matching
62.27
60.77 57.26...x
59,09...
40.35
ProtaNet
64.37 -001
55.54
61.68 unto
63.10
53 43.42-1..
59.28 1s5
27.1
25.64:00
4.1
ProtoNet (20-way)
65.76 6/00
$1.14 oun
61.66 ixt2
63.92 am
52.43
60. 60.18
23.42 1::8
22.50
4.0
Relation Net (CE)
64.76
58.86
63,50.007
64,14 1M0
56 19
60.28
37, 1177
35.77.o0T
2.0
62.922cmm
58
61.522gmm
57.30.as
60.54:30
47.27
45.18:am1
SimpksShot
63.74 c.a
49.05 ou
56.11 1x17
59.77 1171
16.75 1.:0
58.89 355
20:22 1Xx
20.13
8.6
MAML
61,83 1-71
50.83
57.70.
59.87-348
48.41:1M
36,77
23.09:134
20.32
7.4
PrOtOMMML
59.86
46 14:n
54.00aMt
56.76 26.6613
43.62:1
45.00:3M
21 19
20.79
10.1
BMAMI
59.89
37.04-031
48-27-11a 44.35
33.
54.93.11
20:00
20.00
BMAML (chaser)
-6.48..ar
$2.05
37.73
40.47-11a
27.90
52.25
20.00
20.00
ETAMI (nt,a) 5)
52.94-
16
31.39
22.06.374
20.00
20.00
14.6
RTAMI 2,.9
$4,89
37.32
mm.92
53.51
20.00
20.00
13.5
BTAML
52.19
46.15
50.16
51.04.509
44.61amM
49.41
12.6
Table 15: Kandom-SShot (accuracy)
Schot balanced
random
1-9shoc
random
3-Tshot
4-6chot
linear
1-9shot lincar
step T'mnne
1-9khat
1-9shot step 4minor
step 4minor
1-21shot
Rank
Baseline (I-NN)
40.44_.177
40.53
38 68
40.60
32. 18
20.08
11.8
Baseline (fine-tune)
62.83
49. .0.
57.31
59.94-.0m
58.60...
20
Baselinc++
66.74 eas
49.06 -ea
57.44
61.41-14
46.46
61.07
21.13
20.28
6.5
Matching Net
59.58 cup
54.79
58.21 uas
58,79 30
53.47 1L:
$8.19 aMs
39.20 :04
33.11 ios
5.1
ProtoNet
51.65
49,21
50.87
51.31
48.57
50,89
39 39.88111
39.43.ost
7.2
ProtoNer (20-way)
59.79
59.52-11
$4.66
57.32
42.24
41.29.00
Relation Not (CE)
63.50 0.00
61.33 ode
62,99 (071
63.30 k:
69.64 (.77
63,33 aM
51.20 6.6
51.69 6025
1.1
DKT
63.23 c.e
58.92 .m7
61.97 LM
62.72
57.42
60.75 1M4
47.37
45.20%.00
2.2
SimpleShot
61.99
50..4
56.77
59:35
48. 19
57.57
21 08
20.67:0:3
7.1
MAMI.
50.799.mM
43.14
47.75
48
23
20122
PrOuMAML
54,78
46.09 .cds
51.61 Exts
53. 12:
44.04
49.69.30
22 20.
21.10
9.0
BMAML
50.51
43.60).mm
48.12
20.3
41 15.55..
47.26.2314
26.6 26.61
21.33
10.2
BMAMI (chaser)
45.63
38.66-mm
44.03 42,70aave
37 63
43.35
23 23.48
20145.0
12.4
STAML (o.w), 3.2)
20.00-.a::
20.00
20.00
20.00..3e
20.00::.
20.00
16.0
STAML (o.9.9)
20.00
20.00
20.00
20.00
20.00...a
20.00
BTAML (a)
52.66 6m
46.85 ade
51.13 102s
52.10
45.08 1.:
50,12 1367
10.2
Table 16: Random-Shoe Shot (ROS) (accuracy)
Sahot
1-9shoe
3-7shot random
4-6shht
1-9chot lincar
1-9shot step Iminse
1-9chot step 4minor
1-21
Rank
random
step 4minor
Baseline (1 (I-NN)
30.90
38.29....
39.43...
39.63...
37.95..:
39.83
31.76..a
29.81
14.4
Baseline (finc tune)
63.01 cue
51.38 ous
58.61 1012
60.78 sm
49.75 11.0
$8,90
23.53 1332
21.64:o2:
7.8
Baseliness
66.25
59.53
64.36-ux
65.27
57 57.27
62.31
36 36633.1
29, 20
Matching
61.75
50.53
61.50
59.07
61.92
51 51.53
$1.86
ProtuNet
55.35 cas
51.32 ado
53.79 uus
$4.40 208
50.40 1::
$3.76 322
39.72 cs
39.08 oas
8.9
ProtuNet (20-way)
59.52 6m
$1.81 Loos
57.67 unt2
58.30
53.19 11:7
$6.89 au
11.50:1m7
40.62
6.5
Relation Net (CE)
64.12..m1
61.32
62.86.5m
63.31
60.93
64,29
52.16.47
52.60..0
1.4
63.21
59,80
1/44
61.52
46.92.00
Simpk-Shut
59.52 cm
47.63 .na0
$3.35 Lxts
55.39
16.04
$5.67 .363
22 20.12
21.60..20
11.0
MAML
59.10l.no
54.09
$7.61 2m7
58. 33430
$2.81:tM
56.71
39 24:1.
57.23
7.5
PrOOMAMI.
54.50
48.04_mm:
$3.96.21 12.59
46 16.99::
45.06.30
31 19
29.23gm
12.2
BMAML
58.98-1
58.99....
$7.70..m
58:24:10
$7.54
48 22
28.58
(shaser)
53.52
50.90...:
52.62
52.87-144
50.41..
52.20
44.48
44.61
BTAMI. (o.4): 5)
47.75
45.57
47.17
47.47
45 35.04.mm
47.44:111
15.1
BTAME (0.7,2)
53.34
50.84....
52.75 mm
52.84
49.97
$3.33
12.4
STAML
51.59
49.00
50.90..0r
51,23
48.62
51.68
13.4
Table 17: Random-Shor (ROS+) (accuracy)
Sahot balanced
random
1-9ihoe
random
3-Tshot
4-6shot linear
1-9khot linear
1-tkhot step Immnoe
1-9shot step 4minor
step Aminor
1-21shor
Rank
Baseline (I-NN)
29.00
28. 28.4
28.93 2319
33 48
34.74
33 49
24.75
15.1
Baseline (fine- tune)
60.46....
57.53
59.44-113
12:
61.81
39.00)...
Baselince+
63.72 eas
55.60 uuo
60.47 u03
62.31 kM
58.41
64.69 as4
45.79 16:4
43.05 10:5
3.5
Matching Net
60.05 cas
$7.26 ad0
59,33 ius
59.59 1m
57 11:0
60.36 1nc
49.00
48.93 625
1.8
ProteNet
60.05
57.35
59.94...
59.98
56 36.89...
60.57
46 46.23..0
45.8 45.83
5.4
ProtoNet (20 way)
61.21
59.89
137)
58.58
62.84
47.02
46.77
Relation Not (CE)
60/48 cm
58. .m
60.01 JXTI
60.50
$7.99
61.4 a3
18.61 11:7
48.35 Lo3
3.1
DKT
61.16.ne7
S8. 18.18.m0 u0::
60.08 2ns5
60.17
57.71
61.87
48.22 18.22:17
47.01.271
3.2
SimpleShot
49.95 59.44ggm
51.57 2m11
$7.89
$1.5 m
59.84.1m
34 70
32.64gm
988
MAMI
$4.60
51.04.mm
$3.57
54.41
51
56.35.m
41. 41.99
41 41,84
PrEuMAML
50.38 us
45.84_.0
48.41..1
49.29.377
46.01.17
50.38
30 88
39.24.00
13.1
BMAMI
56.52_m
53.112m
55.81am
56.07 114
53.23
56.92,214
46.02
26.0
7.8
BMAML (chaser)
$0.49
50.65_.m1
52.71 2m
$2.92
50.68::1
$2.29
43.59
43.72
10.2
STAML (o..).).2)
47.97
44,28
46.29
46.85...4
43.60
47.05
STAML (o.
52.66
51.54
47.79 11.2
$1.97
BTAML (c)
$1.80 cm5
47.69 .036
19.84 .m75
50.18 177
16.91
50.91 (375
13.1
28

---

Under review as a conference paper at ICLR 2021
Table 18: Standard (FI)
Sahot
1-9shor
3-7shot random
4-6shot
1-9shot linear
1-9shot step Iminor
1-9shot step
s-or step Aminor
Rank
random
4minor
Baseline (I-NN)
-4
32.37
34.47
34.64....
31 c..
33.90
23. 13
21
11.5
Baseline (lne tune)
61.77 120
43.18
54.10.139
57.82
39. 19.19...
52,
8.87 .1.
7.27
7.8
Baseliness
65.06-1
41.17
52.31
57.84-10
36
24.29.:m
2.59
7.16
Matching
60.922..1
53.76.
58.87-112
49 .4
24.20...a
34
26
ProtaNet
63.76 125
50.68 cu
60.13-1154
62.09
46.94
53.47 L1:
17.26
15.25 1::0
4.1
ProtoNet (20-way)
65.15 1:4
18.37
59.82 11:44
62.68 11:4
15.26 1.24
51,17 Lu0
12.10:101
10.74
4.1
Relation Net (CE)
64.000
55.27
62.32
63,19
51.39.144
15.12.1125
30.63
28.
2.0
DKT
61.51=1
15.65-15
59.71
60.73:::4
53,49
56.73.1.
42.8%:17
40.12:111
SimpksShot
63.19 132
41.90 12>
13.10
57.83 (1:4
38.28 (1:)
$2.79 11:5
7.07
6.91 1407
8.5
MAML
61.22 130
45:22 2a
$5.65 11.55
58.56.1:41
20.94:12
50.48 1105
11.66.191
7.26
7.1
PrOtOMMML
59.13-1
39.41
51,06a11
54.84:100
35 58
36,01
10.04
8.14
10.1
BMAMI.
56,87-1
26.26-m
35.54.-1
40.34:
47.76am
6,7
6.6721
BMAML (chaser)
13.26-1
20.50
27.48
30.174..
15.93
45.20
ETAMI (nt,0) 5)
$2.30-21
29.27-mm
50.06:::
$1.09::0
6.67
6.67.mm
14.5
RTAMI (0.7)4)
$4.52-1
30.19
51,30
52. 12.79=
6.67am
6.67.amm
13.2
BTAML
51.85-1
42.61
49.09
50.35
39.5
44.33
12.0
Table 19: Random-Shhee Shot (FI)
Schot balanced
1-9shot
3-7shot
4-6shot linear
1-9shot lincar
1-9shat
1-9shot step 4minoor
1-21shot
Rank
random
random
step Tmince
step 4minor
Baseline (I-NN)
36, 16
-3.45
35.71 35
32
25.07
23 M0
20.75
11.8
Baseline (fine-tune)
61.92-1m
43.29
54.26-112
27.99..4
52.29..um
7.27
Baselinc++
65.34 1.20
40.82
52.59 12m
58.07 ../
36.80..4
54.37 .US
8.51 -.5)
7.14
7.5
Matching Net
$7.51 1:0
50.70 1s0
$5.55 11:0
56.41
18.84
$3.99 L1:
32.62
24.83
5.0
ProtoNet
50.65
46.87
19.70.149
50,26
15:44:14
47.66
17717
31.35
6.9
ProtoNer (20-way)
53.45-130
58.12-112
58.75
50.95
53.57
37.07 :134
35.76-11
Relation Not (CE)
62.54 1.0
60.18 131
62.01 11:4
62.35 1.0
59.34 1.4
62.15 L14
49.25 1M4
49.67 154
1.1
DKT
61.80
55.88 174
60.15 11.9
61.11
53.45
$6.74
42.75:114
39.91
2.2
SimpleShot
61
44,37
54.13atm1
57.8221
40.59:131
$1.59
8.56
7.86
7.1
MAMI.
48.61-119
36.44_.17
46.17
33.49::1
41.91
13
7.201:
ProuMAML
54.12.:14
10.52.15
49.38 11:2
$1.70
37.19
43.34
10.60
8.72
9.1
BMAML
46.11=1m
35
42.55a1m0
44.29 1:
33.23:19
39:84:144
16.07.11
8.76
10.8
BMAMI (chaser)
41.04-1
30.50
36.51ar
38.572
28 44
35.99
12.01gmm
7.46
12.4
STAML (o.w),
6.67
6.67-mm
6.67
6.67
6.67
STAML (o.9..9
6.67
6.67-axa
6.67
6.67
BTAML (a)
52.37 1.0
43:46 122
50.09 104
51.49 (24
40.09 1:2
9.6
Table 20: Random-Shar (ROS) (F1)
Sahot
1-9shoe random
3-7shot random
4-6shot
1-9chot linear
1-9shot step Imince
1-9chot step
1-21
Rank
4minor
step 4minor
Baseline (1 (I-NN)
34,95 1100
32.32-1.x
34.26.1.
34.54....
31.20..az
34.08
:..s
23. 23.11
20.31
14.8
Baseline (finc tune)
62.15 128
45.48 2.1
56, 16.10
59.16 1.:
42.44
52.77
12.44
9.46
8.9
Baseliness
64.83-1.
15.35
62.44.1m
63.64-1a
51.73.:4
26.90
29.57=:0
20.73
Matching
57.85
57.28
60.21
49.51
ProtuNet
$4.56 141
48.61 120
52.55 (1:
53.38 L:
46.74
50.01
11:5
34.41 11:)
33.41 1125
9.1
ProtuNet (20-way)
58.87 1.0
$2.07 180
$6.51.115
$7.40
19.73
$3.03
36. 36.20:10
31.92
6.0
Relation Net (CE)
63.16.1
60.08
61.81
62.30.:.0
59.56:1.4
62.95
50.27
50.62:144
1.4
61.83
57.22
61.36
42.28
Simpk-Shut
58.93 133
41.58 1.0
50.55 11:22
$3,33
38.84113
49.96.111
10.36.177
9,401.c
11.9
MAML
5H.35
51.10
36.34:144
57.32
48 48:30:14
$2.56.1:3
34.05:17
51 51.26::1
7.5
PROOMAMI.
52.10-1m
42.91
29.85
$1.54:
40 70
36.75
23 18
20146
12.5
BMAML
55.76-1-1
52,63-1m
54.37
55.00..01
54.31
44 44.72
44.87s1
(sbaser)
49.72-171
47.17-13:
49.14
46.75
48.55
40.92
41.06
BTAMI (0..): 5)
45.92
43.88
45.4 45.45
45.71
43.35
45.48
14.5
BTAME (0.7,2)
$2.79
49.97
52.17
52.23
48.83
$2.06.:
10.8
STAML
51.27
48.29
50.51
50.86
47.62
50.53
Table 21: Soinddo-Shor (ROS+) (F1)
Schot halanced
random
1-9shoe
random
3-Tshot
4-6shot
linear
1-9chot
linear
step Iminoe
1-tkhot
1-9shot step 4minor
step 4minor
1-21shor
Rank
Baseline (I-NN)
20.25
13
19,70a
38
26 25.00.
27.86
26 26.61=
27.97a:m:
15.1
Baseline tune)
59.51
49.43-139
55.84.1.7
48.32...4
59.00.1.7
32.47
Baselince+
61.95 1.
51.37 180
58.12 1125
60.26 1.0
55 (1:))
61.95 (14
40.69 L25
36,90-17
4.1
Matching Net
$8.50 1.:
$1,58 un
57.50.152
57.84
54.93
58, 18 1147
15.52 1M0
41.99
5.1
ProteNet
59.30
55.62
59.10.142
59.22
54 54.66-14
58.59...4
43.40
42.31
ProtoNet (20 way)
60.27
58.69
60.56
60.73
42.93
Relation Not (CE)
59.41 -1
16.51.1.9
58.76
59,35 1::4
$6.15 (1:5
59.91 11:55
46.31 1::4
45.70.144
2.2
DKT
59.15
55.52
57.97.1.5
$7.88
55 12:15
59.44:144
44 44.27.1m
42.37
4.5
SimpleShot
58:09-1
44.9
44.91:mm
$3.63ar
56. 20
46 a
55.49.:1
27 mm
24.72a11
10.2
MAMI
53.90
49.73
$3.60 $2.64:14:
50.
55.21
40 12
39.78
PrEuMAML
48.51
47.94.171
45.99
43.11
47.51
37. 17.76
36.63
12.9
BMAMI
$2.78
28
$1.90
$1.97amm
$2. 78
49. 49.12
13.26,:11
41 41.32
41.21
8.8
BMAML (chaser)
46.35-1m
49.07 46.48
49.46
47.17 47.17
48.88.m
39.64:11
39.89
11.4
STAML (o..).).2)
46.32
42.63-134
44.64.152
45.14
41.97..u
45.32.155
STAML (o.9.9)
$2.06
48.10-1.
50.90
$1.12
BTAML (c)
51.41 1:0
47.16.1.z
19.11 11A1
50.05 L144
16.35
50.21 11.9
12.5
29

---

Under review as a conference paper at ICLR 2021
42.99 20.00
53,72 2071
49.42 20.00
49,95 5:074
51.34-67
52.61 LO72
52.76-04
L072
$2.57 50.2
1 19000
51.29
=0006
600
120
40
49.42
54.05
53.11
LUB-200-22 200-2011 1
Imbal Ioaalaneed Mini- ImageNet + CUB-200
step
570
30
32
42.19
10006
12.68
52.68
10
50.51
00.51400
52.42
51.33 1072
54.44 60
52.87 +071
53.96
1000
51.64
6067
48
49
51
42.03 20.05
52.47 60.1
48. 48.12- 12 20.00
53.32 20.5
51.15 1072
51.30 =0.8
53.95 6:71
52.90 60.70
53.65 3::00
$1.55=00
=0.4
462
30
24
50.82
51.55
random
43.39 20:00
10.52
20.57
52.31 1073
$2.13-0.8
70
60.72
50.75
570
53.52
50.48
50.77
52.06
$2.88
55.14
45.27
51.95 6:072
46.83 20:47
51.32 2077
52.60 .074
54.41 +021
54.19 L071
53.46 -0.71
53.26 10:70
51.25 20:00
linear
570
30
51.85
balanced
43.21 2::e
53. 53.192007
38.8 :0. 20.9
50.92
52. 26.66.. 2.
1.80 10.2
54.32 +0
54.62 LATE
53.16 33.66:03 807
53.4 53.46300
2007
48. 04.52mm
300
300
Table 22: Full results for Table I
600
40
41.77 20:00
(6.48:00 20.7
50.54.00
58.18 50.70
60.07=00
60.07
60.8 87 1.700
61.35 -.,00
57.64 L0:0
58.60
55.65
1902 3072
54.60=0.0
56.4
41.45 2005
10.70
20.9
56.53 20
58 10.70
57.93 +072
55.91 1000
10.71
54.30.070
20:70
53.56=071
step
570
30
32
52.47
51.88
58.69 28.69.200
58.89
54.30
Imbalanced Mini- ImageNct
2000
2071
2071
20.6
1071
58. 58.80-1 -0F1
56 99 2007
58.86 H072
2002 2902
20.1
462
30
24
39.02
51.54
49.01
57.06
59.34 34
59.02 .2.02..0
55.37
54.39
12.15.m 20:06
54.16 24
51.47 20:7
58.45 00455000 20:8
00.37 10.70
62.35 +8700
57.81 P0AT
58.74 00.72
54 11.97 2005
45 94 20.75
random
570
30
60. 10.16.20
linear 570
43.42 20
50.13
47.18
28.24.006
59.17 $9.17.00.5
61.39 HUU
57.59
59.78 60.3
55.14 2072 2997
54.57
30
60
ballanedd 300
42. 69 2m:8
21.26
48. 2844
58. 201s
20.35
62.78 12.78-10 78.78
58. 099 20/9
59.55 -0.72
24.43.n09
51.31
300
10.91
Max. # samples, Kmox Lpose
Min. # samples, Mmin
Baseline (fine- tunee)
Minority. Neam
troin
ProtoNet (0-
Relation Net (CE)
Baseline line (I-NN)
T-Distribution
Baselinct+
Matching Net
ProtoMAML
SimpleShot
ProtoNet
MAML
DKT
30


Reviews:
[{"review_id": "uQ6FQ8V3zpu", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper conducts extensive comparison experiments to study the effect of class-imbalance for many few-shot approaches. A detailed study of few-shot class-imbalance along three axes: dataset vs. support set imbalance, effect of different imbalance distributions (linear, step, random), and effect of rebalancing techniques, are presented. Also, this paper is clearly written and easy to understand. \n\n1. Though eleven few-shot approaches are considered, some strong baselines are missing, such as [1];\n2. In the contribution part, this paper declares \"compare over 10 state-of-the-art few-shot learning methods using backbones of different depths on multiple datasets\", however, \"backbones of different depths\" is commonly used in few-shot learning literature. Therefore it cannot reflect much contribution of this paper;\n3. Some related work is not discussed [2,3]. For instnace, prior work [2] discusses the effect of different value of $k$ in meta-training and meta-testing, which is pretty much similar to the concept \"imbalance\" studied in this paper;\n4. Overall, the contribution of this paper is somewhat limited. Apart from conducting extensive experiments, more informative observations and conclusions should be made.\n\n[1] A Baseline for Few-Shot Image Classification. ICLR 2020.\n[2] A Theoretical Analysis of the Number of Shots in Few-Shot Learning. ICLR 2020.\n[3] Learning to Stop While Learning to Predict. ICML 2020.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kzvrrP8ob61", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a detailed study of few-shot class-imbalance along three axes: dataset vs. support set imbalance, effect of different imbalance distributions (linear, step, random), and effect of rebalancing techniques. The authors extensively compare over 10 state-of-the-art few-shot learning methods using backbones of different depths on multiple datasets. The analysis reveals that 1) compared to the balanced task, the performances of their class-imbalance counterparts always drop, by up to 18.0% for optimization-based methods, although feature-transfer and metric-based methods generally suffer less, 2) strategies used to mitigate imbalance in supervised learning can be adapted to the few-shot case resulting in better performances, 3) the effects of imbalance at the dataset level are less significant than the effects at the support set level. \n\n\nPros: \n1) the paper covers the state-of-the-art few-shot learning methods, over 10 methods are compared in the paper;\n2) the work reveals some interesting insights in few-shot learning, such as the three analysis summarized in Abstract. \n3) the experiments are reasonable. There are a number of comparisons between different methods on different data sets. The codes to reproduce the experiments is released under an open-source license. \n\nCons: \n1) the paper does not provide a new model and the contribution is marginal. \n2) the experiments does not introduce new datasets as benchmark, all the datasets are heavily manipulated during testing. Is there any new data sets provides to test the assumptions of class-imbalance few-shot learning?\n3) the paper does not fully discuss new possible research directions in the field of class imbalance few learning. Although the authors discuss some insight into the previously unaddressed CI problem in the (meta-) training dataset and conclude that the effects of imbalance at the dataset level are less significant than the effects at the support set level, the future work along this direction seems still unclear.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QQ0Mc9qIIon", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper analyses the effect of class imbalance on few-shot learning problems. It draws a number of interesting (but kind of expected) conclusions e.g., the support set imbalance has a larger influence on the FSL performance compared to base class imbalance, a high impact of imbalance on gradient-based meta-learning methods compared to metric learning approaches. The paper is overall  \n\nPros: \n\n+ The paper is nicely written with a clear structure and exposition of ideas. \n+ An extensive number of FSL methods have been tested with three imbalance settings (linear, step, and random imbalance) on multiple datasets (Meta-Dataset and Mini-ImageNet) across various backbones. \n+ The paper considers class imbalance in both the base training and finetuning on the support set. \n+ Overall, the paper presents a thorough and detailed analysis of the class imbalance problem in FSL. \n\nCons: \n- An approach to deal with the imbalance in FSL settings could have made the paper even more stronger. \n- Specifically, two very simple rebalancing methods are studied in the paper i.e., Random over-sampling and Random shot meta-training. An algorithmic approach for appropriate rebalancing in the loss function (e.g., [a,b,c]) would be intreresting to analyze. \n\n[a] Ren et al., Learning to Reweight Examples for Robust Deep Learning\n\n[b] Khan et al., Cost-sensitive learning of deep feature representations from imbalanced data \n\n[c] Cui et al., Class-Balanced Loss Based on Effective Number of Samples\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5DBmxItPA2k", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary \n\nThis paper introduces a new benchmark for imbalanced few-shot learning where the number of samples per class is different. The authors extensively evaluate 10 SOTA few-shot methods on this benchmark and show consistent performance drop in this challenging setting. They also show that simple over-sampling techniques can alleviate the imbalanced issue in few-shot learning. \n\nPros\n\n-This paper introduces a new benchmark to evaluate the imbalance problem in few-shot learning.  \n-The evaluation is quite extensive and includes 10 SOTA method under different imbalanced settings\n\nCons\n\n-The authors ignore the long-tail recognition literature [1, 2, 3] which is highly relevant and in my opinion, more important than the proposed imbalanced few-shot learning problem. [1] introduced a long-tail recognition benchmark where the ImageNet classes are divided into many-shot, medium-shot and few-shot classes based on the number of training examples. This setting is more realistic because the statistics of real-world datasets also follow a long-tail. I do agree that the imbalanced problem is very important. But I am not convinced that the proposed imbalanced few-shot learning (or meta-learning) evaluation protocol is appealing to the imbalanced problem community. \n\n[1] Large-Scale Long-Tailed Recognition in an Open World. Liu et al., CVPR'19\n\n[2] Learning imbalanced datasets with label-distribution-aware margin loss. Cao et al., NeurIPS'19\n\n[3] Decoupling representation and classifier for long-tailed recognition. Kang et al., ICLR'20\n\n-The authors also ignore the generalized few-shot learning literature [4, 5, 5] which is closely related to imbalanced problems and few-shot learning. In particular, [4] introduced a benchmark that evaluates the performance on both base and novel classes where base classes have many samples and novel classes have only few shot examples.  \n\n[4] Low-shot visual recognition by shrinking and hallucinating features. Hariharan et al., ICCV'18\n\n[5] Low-shot learning with imprinted weights. Qi et al., CVPR'18\n\n[6] Low-shot learning from imaginary data. Wang et al., CVPR'18\n\n-There is no novelty except the proposed benchmark. The over-sampling techniques are standard and expected to improve the performance. \n\nJustification of the rating\n\nAs an evaluation paper, the authors ignore a large group of highly relevant works in long-tail recognition and generalized few-shot learning. I think the proposed benchmark is somewhat incremental to the existing long-tail recognition benchmark and recommand a rejection.  \n\n----------------------------------------------------\nPost-rebuttal\n\nI have read the rebuttal and other reviews. The first version of this submission fails to cite any highly relevant works in long-tailed recognition and generalized few-shot learning. I believe that addressing this issue would require a major revision. The authors argue that their proposed few-shot imbalanced setting is different from the long-tailed recognition problem and generalized few-shot learning setting.  But this is only partially true. It is unclear whether previous approaches for long-tailed and generalized few-shot learning can be already applied to address the few-shot imbalanced problem. Moreover, I am not convinced that the proposed setting is more appealing than those two existing imbalanced settings because the 5-way classification problem seems to be an artificial setting that rarely happens in practice. Finally, I realize that the proposed setting is actually not new. [Lee et al., ICLR 2020] have explored a very similar setting. Thus, I would keep my original review and recommend rejection.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the effectiveness of few-shot learning techniques in settings where the training labels are imbalanced. While addressing an interesting practical problem, reviewers raised concerns about the paper's technical depth, insufficient distinction to existing techniques for coping with label imbalance, and limited qualitative conclusions from the results. The authors incorporated some of these comments in their revision, but a more comprehensive update on the latter two points appears appropriate.
Decision: Reject