Towards Understanding the Cause of Error in Few-Shot Learning
Authors: Liang Song
Jinlu Liu
Yongqiang Qin
Venue:  (2021)
Keywords: upper bound of error
feature separability
classifier discrepancy
few-shot learning

Abstract: Few-Shot Learning (FSL) is a challenging task of recognizing novel classes from scarce labeled samples. Many existing researches focus on learning good representations that generalize well to new categories. However, given low-data regime, the restricting factors of performance on novel classes has not been well studied. In this paper, our objective is to understand the cause of error in few-shot classification, as well as exploring the upper limit of error rate. We first introduce and derive a theoretical upper bound of error rate which is constrained to 1) linear separability in the learned embedding space and 2) discrepancy of task-specific and task-independent classifier. Quantitative experiment is conducted and results show that the error in FSL is dominantly caused by classifier discrepancy. We further propose a simple method to confirm our theoretical analysis and observation. The method adds a constraint to reduce classifier discrepancy so as to lower the upper bound of error rate. Experiments on three benchmarks with different base learners verify the effectiveness of our method. It shows that decreasing classifier discrepancy can consistently achieve improvements in most cases.

Under review as a conference paper at ICLR 2021
TOWARDS UNDERSTANDING THE CAUSE OF ERROR IN FEW-SHOT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Few-Shot Learning (FSL) is a challenging task of recognizing novel classes from scarce labeled samples. Many existing researches focus on learning good repre- sentations that generalize well to new categories. However, given low-data regime, the restricting factors of performance on novel classes has not been well studied. In this paper, our objective is to understand the cause of error in few-shot classi- fication, as well as exploring the upper limit of error rate. We first introduce and derive a theoretical upper bound error rate which constrained to 1) linear separability in the learned embedding space and 2) discrepancy of tpsk-specific and indepeenedddn classifier. Quantitative experiment is conducted and results show that the error in FSL is dominantly caused by classifier discrepancy. We fur- ther propose a simple method to confirm our theoretical analysis and observation. The method adds constraint to reduce classifier discrepancy so as lower the upper bound of error rate. Experiments on three benchmarks with different base learners verify the effectiveness of our method. shows that decreasing classifier discrepancy can consistently achieve improvements in most cases.
INTRODUCTION
Learning novel concepts from few samples is one of the most important ability in human cogni- tion system (Chen et al. (2018) : Dhillon et al. (2019); Wang et al. (2020)). By contrast, massive achievements of modern artificial intelligent systems are dependent upon lots of data and annotation which are hard to acquire in many scenarios. Blocked by the difficulty in obtaining large labeled datasets, community shows more interests developing algorithms with high data-efficiency. is so-called few-shot learning that learns to generalize well to new categories with scarce labeled samples (Sung et al.] (2018): : Vinyals et al. (2016)). Existing methods deal with few-shot learning in the general framework of meta-learning where base learner is developed and optimized across dif- ferent episodes (or tasks). Episodes are formed a N-way K-shot fashion where K support samples per class available for training. The overall objective is enabling the base learner to exploit on base classes and to transfer learnt knowledge to recognize novel classes with few support data. Since training and evaluation performed different tasks, the base learner holds different task-specific classifiers that depend on data sampling.
In general, classification model has two components: feature extractor and classifier (Simonyan & Zisserman (2015); He et al. (2016) Zagoruyko & Komodakis 2) Most approaches of few-shot learning exploit from according perspectives: learning a good embedding and finding a right base learner. Rethinking-FSSC (Tian et al. (2020)) demonstrates that a good learned embedding space can be more effective than many sophisticated meta-learning algorithms. argues for the performance on meta set where embeddings are learnt in supervised or self-supervised way. Goldblum et al.) 2020)) reveal the importance feature clustering in few-shot learning. Since classifier performance sample-dependent especially in one-shot scenario, variance of feature is expected to be small so as to retain good performance. shows that classifier performance is not stable across different tasks. MetaOptNet (Lee et al. ()11)) and R2-D2 (Bertinetto et al. (2018) explore training and optimiza- tion routines for linear classifier, enabling good few-shot performance through simple base learner. These literatures develop specific algorithms from the aspects of learning good representation or op- timizing base learner. Most recent methods use linear classifier as base learner, so we also consider linear model in this paper. To our best knowledge, there has been little research focusing on how the

---

Under review as a conference paper at ICLR 2021
two components (aka feature representation and classifier) respectively influence the performance on novel classes in FSL.
In this paper, we introduce upper bound of eπror rate in few-shot learning, indicating that the error comes from two aspects: 1) linear separability in the embedding space and 2) classifier dis- crepancy between task-specific and task-independent classifiers. The ideal classifier is viewed to be task-independent since its performance is not sample-dependens (Goldblum et al. (2020)). To quantitively estimate each term, an experiment is performed where we use error rate of supervised classification tasks on novel classes to measure feature separability, and use disagreement of results obtained from different classifiers measure discrepancy. comes to interesting observation that features learned through simple methods are sufficiently discriminative and the eπror mainly comes from classifier discrepancy. Based on our observation and theoretical analysis, we propose simple method of reducing classifier discrepancy so as to boost few-shot performance. Experiments on three benchmarks are conducted to empirically verify our theory. Results on different datasets with various base learners show consistent improvements, supporting our finding and theory in few-shot learning.
The main contributions of this paper are:
The upper bound of error rate on novel classes is theoretically analyzed. From derived equa- tions figure out that the error in FSL is caused by linear separability in the feature space and discrepancy between -ase-ipe and task-independent classifiers.
2. Quantitative experiments conducted to verify the theoretical analysis. Results show that the error is dominantly caused by classifier discrepancy.
3. Based on the theoretical analysis and the experiment results, constraint is proposed to reduce classifier discrepancy so as to decrease the upper bound of error rate in FSL.
4. Further experiments on mini- ImageNet, of the proposed method. It shows that decreasing classifier discrepancy can consistently achieve improvements in most cases.
and CIFAR-FS confirm the effectiveness
2 RELATED WORK
Algorithms of Few-Shot Learning Prototypical Network (Snell et al. (2017)) is a classical al- gorithm for its simplicity and effectiveness, which performs few-shot classification by nearest- prototype matching. Since class prototype is the mean of features, the linear separability in feature space has direct impact on classification results. Performance of following series of prototype based methods (Allen et al. (2019) Liu et al. (2020) is also limited by feature separability. Different with these methods using nearest-neighbot classifier, |Bertinetto et al, (2018) adopt ridge regression and logistic regression as base learner. Similarly, Lee al. (2019) use classical linear classifier SVM in few-shot learning to learn representations. Simple linear classifier shows competitive performance and in this paper, we use linear classifier in measuring linear separability and classifier discrepancy.
Theoretical Analysis of Few-Shot Learning Cao et af. (2019) introduce a bound for accuracy of Prototypical Network (Snell al. (2017). demonstrating that the intrinsic dimension of the em- bedding function's 's output space varies with the number of shots. They further propose a method to overcome the negative impact of mismatched shots in meta-train and meta-test stages. As in (Liu et al. (2000), they give lower bound for accuracy cosine similarity based prototypical network. Two key factors: intra-class bias and cross-class bias are theoretically formulated. We also analyze theoretical bounds in few-shot learning. Theory in this paper does not focus specific algorithm like Prototypical Network but on general scenarios, from the perspective of feature separability and classifier discrepancy.
Theoretical Analysis Domain Adaptation: Methods of domain adaptation (Ben-David et al. 2007 2010); Ganin & Lempitsky (2015) solve the problem of how to train a classifier source domain and guarantee the classifier performs well on target domain. A classifier's ' target error is bound by its source error and the divergence between the two domains in (Ben-David et al. (2010)) They utilize H-divergence and -ive-diivergence to measure discrepancy between two domains. H -ii-eeiieeeee can be computed from finite unlabeled data, allowing to directly estimate the
2

---

Under review as a conference paper at ICLR 2021
error of nource-trained classifier the target domain. Inspired by their work, also use HAH- divergence to measure discrepancy between sets on novel classes and base classes.
BACKGROUND
3.1
PROBLEM SETUP
The common setup of few-shot learning used in this paper is described below. A space of class is divided into two parts: base classes Chass and novel classes C'noed where Cbase n C 0. Dataset Dlase of base classes is used for model training and the model is evaluated on dataset Dmovel whose samples belong to unseen classes during training. The model composed of a feature extractor F and a classifier h. In few-shot learning, we usually consider N-way K-shot Q-query tasks T. In task T = (D,, D.,, h), the support set D; includes K data label E {ei, .... CN) ). The goal is predict labels for query data in Dÿ given DJ. In this paper, use error rate on novel classes to evaluate the few-shot performance of a trained model. The error rate is formulated as: Goovll = F(er]
R per class and its true
M Q
1
Σ
1 (((F((r))) yo.))
(1)
M Q
where M the number of sampled tasks T₁ ~ Tnovel 1(-) indicator function.
3.2 DISTRIBUTION DIVERGENCE
We adopt following concepts to explore the cause of error in few-shot scenarios.
Definition Given a set = {(I1, ((c1,91). y1), .... (Im, 9m)] where r; E X and yı E y, for any mappings h2 E X, disagreement is defined in Eqn. t measure the difference of these two mappings.
dis(h1, h2) = PawDx (h1(x) + h2(x))
(2)
Definition Given domain with Dj and D22 probability distributions over X', let be a hypoth- esis class on X' and denote by I(h) the set for which h E H is the characteristic function; that is, x (h) <> h(x) 1. H. divergence between D, and D2 is
(D D2) 2supheH| PrD, [Th] Pro,[((h))
Definition 3 For hypotheses h, h E H. the symmetric difference hypothesis space H.H is the set of hypotheses g HAH < g(x) = h(x) e h'(x)) where the XOR function. H.H divergence over distributions is defined as following:
(HAH(Di: D2) = 2suphem Prz-D, [b(x) + n'(x)] ProND, (Mx)) + M(x))l
(4)
PROPOSED METHOD
4.1 MEASURING CLASSIFICATION PERFORMANCE ON NOVEL CLASSES
A classification model generally consists of two parts: feature extractor and classifier. Hence, the model holds two expectations: 1) Extracted features are expected be discriminative for classi- fication (Snell et al. [2017)] Lee et al. (2019)) 2) Classifier is supposed to be stable concerning different tasks (Cao et al. (2019); ; Liu et al. (2020)). Consider linear classifier this paper, we in- vestigate from linear separability and classifier stability to measure classification performance on novel classes. To quantitively estimate these two terms, we design experiments using a base model with ResNet-12 backbone and FC (fully-connected) layer classifier. The base model is first trained on base classes in supervised way. When testing on novel classes, we replace FC layer classifier with ProtoNet (Snell et al.) (2077) and ridge regression (RR) in paper Ye et al. (2020). The enovet(h" (i.e. error rate on mooel with classifier h²),, is used to approximate and quantify feature separability on novel classes (Eqn.
NXQ
Gnovel = N Q+
1
Σ
1(0: + 4v)
(5)
3

---

Under review as a conference paper at ICLR 2021
Table 1: Experiment of linear separability and classifier discrepancy of 5-way (N=5) classification
tasks on novel classes.
dis h²)
Dataset
Classifier
Gnovel (h*)
1-shot
5-shot
mini-ImageNet
11.72%
39.32%
16.23%
PN
tiered-Imee -mageeet
11.28%
32.65%
14.60%
mini-ImageNet
6.72%
38.34%
19.45%
RR
tiered-ImageNet
5,97%
31.10%
16.94%
In Eqn. 5 Q" is the number of all samples of each class fci...., Cy.. Us is predicted label and yi is true label. h* trained under supervised way from large set of samples and is used to approximate the expected ideal N-way classifier. On the other hand, we use disagreement defindd Eqn. to measure classifier discrepancy.
NXQ dis(h, h:) NXQ Σ 1(0: +
(6)
where Q is the number of query samples. h is task-specific classifier that differentiates among tasks, decided by sampled support data. ht is task-independent concerning these N classes. Thus, dis(h, h") indicates the discrepancy between the task-specific classifiers and the ideal classifier.
Table I shows results two benchmarks: mmgg-ett and ticred- Gerred-mmgeNee. From Table I can see that "noveth"" (h*) ) is generally lower than dis(h, h²) in a large margin. For example, 1-shot dis(h, h²) mini-ImageNet with RR is up 38.34% while Grovet is 6.72%, which is five times lower. Furthermore,the obvious drop of dis(h, h²) from I-shot to 5-shot indicates obvious raising of classifier discrepancy. An interesting conclusion can be drawn from this experiment that the error on novel classes is dominantly caused by classifier discrepancy in low-data regimes rather than linear separability. More details about this experiment are presented in the appendix.
4.2 UPPER BOUND OF ERROR RATE
In this section, we introduce an upper bound of error rate on novel classes in few-shot learning.
Proposition 1 Consider a feature extractor F and a hypothesis space H. Based on triangular inequality, for h, h² E H. it follows that:
<(;; F) S e(h";; + dis(h, h²; F))
(7)
ht is the ideal hypothesis H. holding that ht = arg min NE (le-(h; F)). Proof is in the ap- pendix.
In few-shot learning, error rate on novel classes is usually denoted by enovel = Eler (h;P))) F)]. Hence, the upper bound is:
Enovel S Eler (h";; F)] + Eldis, (h, F))
(8)
where 7 ~ Tnovel From Eqn. boosting few-shot performance can be achieved by minimizing the two terms right side of above inequality. However, h" is unavailable when testing on novel classes. In order to connect the performance on novel set and it on base set we consifer following setting.
Consider N-way K-shot tasks where N is assumed to be same in meta-train and meta-test stages. h, hí' are linear classifiers of novel classes and base classes respectively. For classification weights W,, W,, E Nxd of base and novel classes, there exists linear transformation matrix W e dxd that has W2 W,, W. We define the linear transformation between the ideal hypothesis on the novel set and that on the base set as A, h²* = A(h*), ), h² = - (h'*). For query samples X = {z, E Rd), predicted results are:
h (X; W.) XW. x(W.W-1)"
According to the above analysis we know that performing a transformation on the classifier equiv- alent to perform the transformation on the data, A(h)(X) h h({())).
4

---

Under review as a conference paper at ICLR 2021
Lemma Let H. be hypothesis space of VC dimension d. ht h² are ideal hypotheses D
and Douse There is ideal hypothesis h arg minkeR (novet + <base (A-1(h))- Then for h? and h!:
Enovel (h") I Ebase (h'*) + 2 ddnn (D DDbaso
(10)
where À Cnonel (h) Chase (A~ (h)) is the combined error of the ideal hypothesis h.
Lemma' 2 For linear hypotheses h, E H and ideal hypotheses he on Dnovel h on Dbase, there exists:
dis(h, h"; Dnovel)
1
ddis(h, <dis(h,A((")) Dhase) + HHHHH novel A(TDaase))
(11)
1
S &dis(h', Doase) + dis(h' A(h); Dbuse) + + (DHAH(Drooe (
Proofs of Lemma and Lemma are provided the appendix. We give the core theory in this paper by plugging Lemma] and Lemma into Eqn.
Theorem
The upper bound of error rate on novel classes in seo-shot learning is:
Gnovel <hase (h²* dis(h', N²; Doase)
1 2 jdnam DDoove A(Dbesa))
+ dAs(AA ( (h), D base) + +
2 dHaH (OMAH(Dnamn nowel Dhase) +
(12)
Based on theoretical analysis and experiments, we come to several conclusions:
1. In theory, the error of few-shot classification is influenced by linear separability of feature representation and classifier discrepancy between task-specific and task-independen classifiers. Ex- periment results indicate that the main cause of error in few-shot learning classifier discrepancy.
2. From Theorem I we can see that the upper bound of error rate on novel classes is positively related 1) linear separability on Dbase 2) classifier discrepancy on Dasse the combined error and 4) HLAV-divergence of Drovel and Diase measuring the discrepancy between two domains.
4.3 REDUCING CLASSIFIER DISCREPANCY FOR FEW-SHOT LEARNING
Based on our theoretical analysis. we propose simple method reduce the upper bound of error rate, boosting few-shot performance by reducing classifier discrepancy.
Measuring Classifier Discrepancy It proved in Sec. 4.2 that reduce error on novel classes can be achieved by improving linear separability and reducing classifier discrepancy. Furthermore, exper- iment reveals that the cause is the discrepancy between task-specific classifier and ideal classifier. For these reasons, we target to reduce the upper bound by decreasing classifier discrepancy. Dis- crepancy denoted in Eqn. Ílis non-din -ifferentiablle so that we propose two measurements of classifier discrepancy to ease gradient propagation in training stage. Since consider linear classifier in this paper, an intuitive way to reduce classifier discrepancy to reduce distance between classification weights. Squared euclidean distance can be used
measurement:
(13) where W is the weight of task-specific classifier h and W is the weight of task wask-independent clas- sifier h.. dis MSE measures the variance of classification weights. Since task-specific classifier is decided by data sampling, taking data distribution into consideration, we suggest to calculate the dif- ference of logit predicted by various classifiers. Consider commonly used KL divergence to measure the difference of logits:
disMsE(h, h"; W, W,; W*) = [[]]W-W"II
diskr(h, h²)= =
= E[KLD(L(DD DD) D*), (KK)
(14)
Training Policy Training process of proposed method Reducing Classifier Discrepancy (RCD) consists two phases. In the first phase, model is trained in conventional supervised way on base classes. Loss function in phase is:
Tesup Teae (IF())(()))) with (x. v) DDeeee Daase
(15)
5

---

Under review as a conference paper at ICLR 2021
Table Performance of RCD. 5-way classification accuracies (%) without/with disMSE constraint. Results in bold indicate performance is improved by reducing classifier discrepancy.
Backbone
Setting
Dataset
PN
LR
RR
disMsE
diaMsE
disuss
mini Mani-lmaeeNet
60.18
61.42
61.28
61.43
62.30
63.64
1-shot
ticred- Imageet
66.24
65.84
64.23
65.03
66.77
67.48
CIFAR-FS
67.90
71.02
66.86
70.77
67.92
70.27
ResNet-12
mini ImageNet
79.07
80.10
77.64
77.27
79.34
80,94
5-shot
tiered- ImageNet
82,98
82.35
81.62
82.64
82.63
83.12
CIFAR-FS 84.38
86.33
83.14
85.20
82.73
85.55
mini-
mini-TmageNed
52.61
54.39
51.43
51.99
53.40
54.05
I-shot
fiered-ImageNNtt
59.71
61.92
57.44
57.60
60.11
61.27
ConvNct-64
CIFAR-FS
68.59
68.20
68.21
69.10
mini-ImageNet
71.33
72.55
70.39
72.58
5-shot
tiered- Tenee-ee
74.18
75.06
74.39
75.01
75.35
76.22
CIFAR-FS
78.05
79.80
78.54
78.66
79.46
80.14
where Loe is standard cross-entropy loss. The classifier ht arg min, Laup obtained in this stage is treated as the ideal classifier, also the task-independent classifier, Dbase, With fixed lt, we train the model on meta tasks Tiuus with loss in Equ. 16
Limet = Lo (F((()))) v) + dis(h, h²), with (I,,) ~ Those
(16)
The second training procedure aims to reduce classifier discrepancy on base classes. Consequently, the upper bound of Gnovel be decreased as verified in Theorem
Our method shows great flexibility that training in the first phase is free of carefully designing base learners. Moreover, policy in the second phase can generalize different base learners. Algorithm is shown in the appendix.
5 EXPERIMENTS
5.1 IMPLEMENTATION DETAILS
Dataset and Backbone We conduct experiments on three benchmarks: mini-ImageNNt (Vinyals al. (2016)),
(Ren al. (2018)) and CIFAR-FS (Bertinetto et al. (2018)). ResNet- 12 (Lee et al. (2019) and ConvNet-64 (Snell et al. (2017) are employed as backbones in this paper. Details about data setting and architectures shown in the appendix.
Base Learner To illustrate the effectiveness of our proposed method, we use three base learners: PN, Ridge Regression (RR) and Logistic Regression (LR) (Bertinetto et al. (2018)). 1)PN is derived from (Snell et al. (207)) which finds the nearest prototype based on cosine similarity. Prototype is computed from support samples: / = norm? K Σ. F(æ,)). Predicted labels of query samples are given by y arg min, Cos(Pe (os(Pe,X)) 2) RR Classification weight is estimated by - Yy where Y is one-hot labels of support samples and I is identity matrix. Prediction of query samples are Y = X W. 3) LR Classification weight in logistic regression is W arg milly Lee LD*, W). Query samples predicted by Y X W. Descriptions detailed in the appendix.
(X" +
5.2 RESULTS OF RCD
We evaluate the proposed Reducing Classifier Discrepancy (RCD) on three benchmarks, with three base learners and two backbones. Table 2 and Table 3 summarize few-shot results with dis and disKL be auxiliary loss respectively. Overall, RCD achieves consistent improvements in most cases.
Auxiliary Loss As displayed in Eqn. 16 two measurements of classifier discrepancy can be added auxiliary loss in meta-train stage. We compare the results without and with auxiliary constraints in Table 2 and Table 3 Each second column of three base learners shows few-shot accuracies by reducing classifier diserepancy constrained by disMSE diskL. Generally, dis disMSE an useful

---

Under review as a conference paper at ICLR 2021
Table 3: Performance of RCD. 5-way classification accuracies (%) without/with diskl constraint. Results in bold indicate performance is improved by reducing classifier discrepancy.
Backbone
Setting
Dataset
PN
LR
RR
dis
diegi
diskz
miti- ImageNet
60.18
63.25
61.28
62.59
62.30
64,41
1-shot
tiered-Imagee Treee-et
66.24
67.37
64.23
65.11
66.77
67.97
CIFAR-FS
67.90
71.35
66.86
70.21
67.92
71.06
ResNet-12
mini-ImageNet
79.07
80.82
77.64
78.50
79.34
80.27
5-shot
tiered-ImageNet
82.98
83.32
81.62
82.03
82.63
83.40
CIFAR-FS
84.38
86.49
83.14
85.96
82.73
85.49
mini-ImageNet
52.61
55.52
51.43
52.02
53.40
54.25
I-shot
tiered-ImageNet
59.71
62.94
57.44
57.97
60.11
62.04
CIFAR-FS
68.59
68.62
67.94
68.10
69.10
70.73
ConvNet-64
mini- ImageNe
71.33
73.40
70.95
72.15
72.58
73.88
5-shot
Gered-ImageNet
74.18
76.23
74.39
75.09
75.35
77.63
CIFAR-FS
78.05
78.69
78.54
79.31
79.46
79.55
Table Changes classifier discrepancy on novel classes. Columns of Stage 1 show discrepancy on novel classes after the fist conventional training stage. Columns of Stage 2 show discrepancy on novel classes after the second training stage with auxiliary constraints dis disASSE or d. Differ- ences are highlighted by green and red respectively. Green/red means reduced/enlarged discrepancy.
Backbone this experiment: 12
PN
L.R
RR
RCD
Setting
Dataset
Stage
Stage
Stage
Stage
Stage
Stage 2
mini-Iman Imageeet
39.32%
37.96% (.36)
39.48%
36.9994(22449)
38.34%
36.24%
I-shot
tiered-ImageNNet
32.65%
$2.829((+0.1 17)
32.21%
32.01%(-0.20)
31.10%
30.28% (-0.82)
IIFAR-FS
30.73%
27.49%((33224
31.15%
29.64%((1.51)
30.64%
29.03%(-1
disuss
mini- ImageNet
16.23%
15.81%6(-0:42)
16.26%
16.40%((0.14)
19.45%
18.86%(-0.55)
5-shot
tiered-ImageNett
14.60%
14.769((+0.6)
15.92%
15.6456(-028)
16.94%
16.54%
CIFAR-FS
12.58%
11.25%%(-1.33)
13.78%
12.03%((11.55
14.17%
12.30%(-1.87
mini- Momi-ImageNel
39,32%
33.85%(($5:47)
39.48%
34.57%((4491)
38.34%
33.21%((3113)
1-shot
tiered-ImageNet
32.65%
31.84%(--0811
32.21%
31.7296(-0.9)
31.10%
30.11%(-0.99)
CTFAR FS
30.73%
26.03%
31.15%
27.5994-3.56)
30.64%
27.44%
disgt.
mini ImageeNet
16.23%
(5.70%
16.26%
16.0256(-024)
19.45%
(6.43%(333022
5-shot
ticred-ImageNet
14.60%
14.025 -0..8))
15.92%
15.399((-053)
16.94%
(-.99)(-
CIFAR-FS
12.58%
(0.80%(-11.7)
13.78%
12.74%6-1.04
14.17%
(2.62%(-1155))
constraint which results in improvements up 2.82% 1-shot CIFAR-FS with RR. By contrast, diskL shows superiority in reducing classifier discrepancy that by adding constraint diskL. L. accu- racy is raised in all cases, up 3.45% 1-shot CIFAR-FS with PN. Under the same conditions, reducing classifier discrepancy can bring in larger improvements in 1-shot scenarios. For exam- ple, mini-ImageNet with PÑ, reducing classifier discrepancy through diminishing KI. divergence achieves improvement by margins of 2.91% % in I-shot and 2.07% in 5-shot. Experiment results are consistent with our theory. In 1-shot settings, discrepancy is larger due data scarcity. Thus, RCD makes more obvious increase on 1-shot tasks.
Base Learner We argue for the flexibility and generalization of proposed RCD. For verification, we adopt three commonly used linear classifiers as base learner in meta-train stage. Results in Table 2 and Table demonstrate feasibility of our method in improving few-shot performance regardless of specific classifier. It indicates that reducing classifier discrepancy on base set is effective to lower the upper bound of <novel-
Backbone In this section, ResNet- 12 and ConvNet-64 are used for ablation study. Overall per- formance on ResNet-12 is prominently better than performance on ConvNet-64. Moreover, RCD makes relatively larger improvements when taking ResNet-12 as architecture. Note that dimension of features extracted by ConvNet-64 is 64 while dimension of ResNet-12 features is 640. Higher dimension indicates larger hypothesis space. That is, classifier in digeer-dimension space is less stable so classifier discrepancy is larger. ResNet- 12 holds a higher-dimension space where reducing classifier discrepancy can result in more obvious improvements.

---

Under review as a conference paper at ICLR 2021
16-
96
mini-ImageNet
Feasure space convertioral trahed.
Hogrure speces traines with diaer.
Feature space tiared
-22
-3
(b) CIFAR-FS
Figure 1: t-SNE visualization of features of novel classes. The first column: feature space is learnt in conventional training. The second and third column: feature space is learnt with dis disMSE and diskL. In each dataset, 5 novel classes are randomly sampled. Backbone: ResNet-12. Base learner: PN. Best viewed in color.
5.3 CLASSIFIER DISCREPANCY
We compare changes of discrepancy in Table clearly illustrate the proposed method is effective reducing classifier discrepancy. Changes are denoted in colors. It can be clearly see that after the second training stage with constraints disMSE and diskL. classifier discrepancy is reduced in nearly all settings, especially on mini-ImageNet and CIFAR-FS. Downward trend of classifier discrepancy is positively correlated to decreasing the upper bound of error rate. Results in Table are consistent with accuracy increment Table and Table 3 further proving the rationality of our proposed theory.
5.4
VISUALIZATION
T-SNE visualization (Matten & Hinton (2008) is provided in Fig. to give intuitive understand- ing of our method. Fig. figures in the first column display the distribution of features that trained in conventional way. Figures in latter columns visualize features trained with proposed con- straints dis and disKL- We can see that features within same classes cluster more tightly and the boundaries among different classes become more clear. Our method enables larger separability in feature space.
CONCLUSION
In this paper, we theoretically analyze the upper bound of enror rate on novel classes in few-shot learning. We derive that the upper bound is decided by feature separability and classifier discrep- ancy. Furthermore, the observation shows that classification error mainly caused by classifier dis- crepancy in few-shot scenarios. Based on our observation and theory, we propose simple method to lower the upper bound of classification error by reducing classifier discrepancy, Two differen- tiable discrepancy measurements are proposed auxiliary constraints in our method RCD which feasible on different base learners. To verify the feasibility and generalization of proposed RCD, comprehensive experiments conducted on three few-shot benchmarks with three base learners. Experiment results powerfully prove that RCD effective to reduce classifier discrepancy and con- sequently lower the upper bound of error rate in few-shot learning.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Kelsey Allen, Evan Shelhamer, Hanul Shin, and Josh Tenenbaum. Infinite mixture prototypes for few-shot learning. ICML, pp. 232-241, 2019.
Brandon Amos and Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In ICML, pp. 136-145, 2017.
Shai Ben-David John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In NIPS, pp. 137-144, 2007.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine Learning, 79(1-2):151-175 2010.
Luca Bertinetto, Joao F Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. In ICLR, 2018.
Tianshi Cao, Mare T' Law, and Sanja Fidler. A theoretical analysis of the number of shots in few-shot learning. In ICLR, 2019.
Wei-Yu Yu Chen, Yen-Cheng Liu. Zsolt Kira. Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. In ICLR, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, pp. 248-255. leee, 2009.
Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. In ICLR, 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, pp. 1 1180-1189, PMLR, 2015.
Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriia Cherepanova, and Tom Goldstein. Unraveling meta-lcarning: Understanding feature representations for few-shot tasks. ICML, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, pp. 448-456, 2015.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR, pp. 10657-10665, 2019.
Jinlu Liu, Liang Song, and Yongqiang Qin. Prototype rectification for few-shot learning. ECCV, 2020.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 97Nv))2559--265, 2008.
Tomasz Malisiewicz, Abhinav Gupta, and Alexei A Efros. Ensemble of exemplar-svms for object detection and beyond. ICCV, pp. 89-96. IEEE, 2011.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, pp. 807-814, 2010.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017.
Mengye Ren, Sachin Ravi, Fleni Triantafillou,, Jake Snell, Kevin Swersky, Josh B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification. In ICLR, 2018.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation. In CVPR, pp- 3723-3732, 2018.

---

Under review as a conference paper at ICLR 2021
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2013.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NIPS, pp. 4077-4087, 2017.
Flood Sung, Yongxin Yang. Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In CVPR, pp. 1999-1208, 2018.
Yonglong Tian, Yue Wang. Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? ECCV, 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NIPS, pp. 3630-3638, 2016.
Yaqing Wang, Quanming Yao, James Kwok, and Lionel Ni. Generalizing from few examples: A survey on few-shot learning. ACM Computing Surveys (CSUR), 53(3):1-34, 2020.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In CVPR, pp. 8808-8817, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. BMVC, 2016.
10


Reviews:
[{"review_id": "rzOaGvLwQFW", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have not had the time to write a detailed review but I have gone through the paper carefully.\nMotivation\nThe paper is motivated well and presents an interesting mathematical derivation that drives the subsequent approach based on reducing classifier discrepancy. The manuscript has a number of typos and grammatical errors for example \"consider\" is spelled as \"consifer\" and \"first\" as \"fist\" among others. Despite those problems, the logical flow is good so the paper is acceptably readable.\n\nMethod\nThe mathematical results is derived for linear classifiers which lend themselves to simple analysis. The authors then say that experimentally classifier discrepancy is the more significant factor, and propose a technique based on reducing such discrepancy. The authors test their approach on various datasets.\n\nResults\nThe authors' results are decent. The improvements are small but consistent which lends some credence to their approach. I find their results modest.\n\nQuality, clarity, originality and significance\nThe clarity of this paper is acceptable and will be better if the typos and grammatical errors are corrected. The originality of the paper as well as the quality and significance are modest. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "qAQu4AIDnty", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper seeks to understand theoretically the current bottleneck in few-shot learning and address it with a new way of training the embedding. The authors find that the key issue in few-shot learning is not the separability of the novel classes but the discrepancy between classifiers trained on large datasets and few-shot classifiers. They then propose a way to reducee this discrepancy.\n\nPros:\n+ The analysis seems to be along a worthwhile direction\n+ the proposed modfications look straightforward and simple.\n\nCons:\n- The theoretical parts of the paper are very poorly written and peppered with unexplained notations and incorrect claims. Some examples:\n-- Definitions (3) and (4) seem to implicitly assume that $h$ is boolean, which makes no sense in the multiclass few-shot learning setup.\n-- Equations (5) and (6) seem to change notations between them, with $h^*(x)$ being referrred to as $\\hat{y}$  in (5) but as $\\hat{y}^*$ in (6)\n-- The statement of Proposition 1 mentions triangle inequality without any indication of there being norms or metric spaces where a triangle inequality might apply.\n-- The paragraph before Equation (9) assumes that the classifiers for novel classes are a linear transformation of the classifiers for the novel classes, and that this linear transformation is invertible (9). Neither assumption is ever proved.\n-- The proof for Lemma 1 introduces terms such as $\\epsilon_N(h^*, h'^*)$ that are never defined. This term seems to relate to $dis(h, h')$ introduced in the main paper for measuring the discrepancy between two classifiers, but that makes no sense since $h^*$ and $h'^*$ operate on completely different label spaces.\n(There are probably more; I ran out of steam at this point).\n- Comparisons to state-of-the-art in few-shot learning is missing.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fMOaASHkBev", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper aims to understand the cause of error in few-shot classification. The authors are particularly interested in the upper-bound of the error rate, which they break down into linear separability in the feature space of the meta-train classes and classifier discrepancy on the meta-train classes, among other terms. Empirical results show that the latter is the dominant term. After identifying this, the authors propose a method, Reducing Classifier Discrepancy, to reduce the classifier discrepancy, lowering the upper-bound of the error rate. Empirical results show the benefits of the proposed method on three few-shot datasets.\n\nPros:\n1. This work investigates the cause of error in few-shot classification and finds an upper-bound for it.\n2. The proposed method, Reducing Classifier Discrepancy, is simple. Empirical results on three few-shot datasets show its effectiveness.\n\nCons:\n1. Through experimentation, the authors find that classifier discrepancy dominates the few-shot classification error. This could be due to the dataset being used for the experiments. This might not hold true in a cross-domain dataset, such as Meta-Dataset [1]. Can results be provided on such a dataset?\n2. The proposed method, Reducing Classifier Discrepancy, has two training phases. There have been works [2, 3] that show that after the first phase of conventional supervised training, the model does very well on few-shot tasks. How does the performance after the first phase compare to the that of the proposed method? Is the gain in the proposed method coming from the model being trained longer?\n3. Additionally, the classifier discrepancy loss forces better clustering of samples in the meta-train dataset. Can this not be achieved by training for longer or using better hyper-parameters?\n\nClarifications:\n1. In Equation 9 of the appendix, is the expansion of the discrepancy between \\Lambda(h) and h^{\\prime *} required for the result?\n\nNotes:\n1. It is hard to follow the equations. Additionally, the notation changes going from the main paper to the appendix. I would suggest cleaning that up for better readability.\n2. Equation 12 seems to have a typo - \\Lambda^{-1} instead of \\Lambda.\n\n[1] Mengye Ren et al. Meta-Learning for Semi-Supervised Few-Shot Classification.\n[2] Wei-Yu Chen at al. A Closer Look at Few-shot Classification.\n[3] Guneet S. Dhillon et al. A Baseline for Few-Shot Image Classification.", "extra_scores": {}, "extra_text": {}}, {"review_id": "J_fJukvQcgB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-----------------------------------\nSummary:\nThis paper analyzes the upper bound of error rate on novel classes in few-shot learning theoretically. It derives that the upper bound is decided by feature separability and classifier discrepancy and shows that classification error is mainly caused by classifier discrepancy in few-shot scenarios. In addition, this paper proposes a new method to lower the upper bound of classification error by reducing classifier discrepancy. \n\n-----------------------------------\nPros:\n1. The author proposes an upper bound of the error rate on few-shot learning.\n2. The classifier discrepancy loss is further introduced on top of the traditional methods to reduce the error rate on novel classes.\n3. Extensive experiments and a visualization of the feature space are provided to demonstrate the proposed method's effectiveness.\n\n-----------------------------------\nCons:\n1. In Eq.5, isn't it supposed to be (\\hat{y_i^{\\ast}}\\neq y_i)? Otherwise, it contradicts Eq.6.\n2. Table 1 should include \\epsilon_novel(h). Otherwise, it is hard to support the statement that the error on novel classes is dominantly caused by classifier discrepancy. For a fixed \\epsilon_novel(h^{\\ast}), a large dis(h, h^{\\ast}) does not necessarily imply a large \\epsilon_novel(h).\n3. Eq.10 is too loose. As the base dataset and the novel dataset have no intersection, it is not feasible to approximate the classifier on the novel dataset with a classifier trained on the base dataset by domain adaptation. The marginal distribution won't be matched, let alone the optimal joint error.\n4. The linear transformation \\tidle{W} does not appear in the final objective Eq.16; thus, the reviewer doesn't know whether it is used during the training. Besides, by comparing Eq.16 and the proposed upper bound Eq.12, it seems that only the first two terms of the upper bound are optimized, which makes the entire method not persuasive enough.\n5. Linearity assumption for classifiers makes no sense as a simple ReLU function will introduce non-linearity, and the reviewer doesn't think it is necessary according to the final objective.\n\n-----------------------------------\nReasons for Score:\n\nThe reviewer rates 4/10 because the third concern is critical. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a contribution aiming at understanding the cause of errors in few-shot learning. The motivation is interesting but the reviewers pointed out many aspects that require more precisions and polishing in addition to the fact that the upper bound provided it rather loose. The rebuttal provided addresses some concerns, but there are still some remarks that require some clarifications en work.
Hence, I propose rejection.
Decision: Reject