A Closer Look at Prototype Classifier for Few-shot Image Classification
Authors: Mingcheng Hou
Issei Sato
Venue: ICLR 2022 Submitted (2022)
Keywords: few-shot
meta-learning
prototypical network
fine-tuning
prototypical clasifier

Abstract: The prototypical network is a prototype classifier based on meta-learning and is widely used for few-shot learning because it classifies unseen examples by constructing class-specific prototypes without adjusting hyper-parameters during meta-testing.
Interestingly, recent research has attracted a lot of attention, showing that a linear classifier with fine-tuning, which does not use a meta-learning algorithm, performs comparably with the prototypical network.
However, fine-tuning requires additional hyper-parameters when adapting a model to a new environment. In addition, although the purpose of few-shot learning is to enable the model to quickly adapt to a new environment, fine-tuning needs to be applied every time a new class appears, making fast adaptation difficult.
In this paper, 
we analyze how a prototype classifier works equally well without fine-tuning and meta-learning.
We experimentally found that directly using the feature vector extracted using standard pre-trained models to construct a prototype classifier in meta-testing does not perform as well as the prototypical network and linear classifiers with fine-tuning and feature vectors of pre-trained models.
Thus, we derive a novel generalization bound for the prototypical network and show that focusing on the variance of the norm of a feature vector can improve performance.
We experimentally investigated several normalization methods for minimizing the variance of the norm and found that the same performance can be obtained by using the L2 normalization and embedding space transformation without fine-tuning or meta-learning.

dur- Under review as a conference paper at ICLR 2022
A CLOSER LOOK AT PROTOTYPE CLASSIFIER FOR FEW-SHOT IMAGE CLASSIFICATION
Anonymous authors Paper under double-hlind review
ABSTRACT
The prototypical network is prototype classifier based on meta-learning and is widely used for few-shot learning because it classifies unseen examples by constructing class-specific prototypes without adjusting meta-testing. Interestingly, recent research has attracted lot of attention, showing that training a new linear classifier, which does not use meta-leamiing algorithm, performs comparably with the prototypical network. However, the training of a new linear classifier requires the retraining of the classifier every time a new class appears. In this paper, we analyze how a prototype classifier works equally well without training new linear classifier and meta-learning. We experimentally find that directly using the feature vector extracted using standard pre-trained models construct a prototype classifier in meta-testing does not perform well the prototypical network and training a new linear classifiers and feature vectors of pre-trained models. Thus, we derive novel generalization bound for the proto- typical network and show that focusing on the variance of the norm of a feature vector can improve performance. We experimentally investigate several normal- ization methods for minimizing the variance of the norm and find that the same performance can be obtained by using the L2 normalization and embedding space transformation without training a new classifier or meta-learning.
during
INTRODUCTION
Few-shot learning is used to adapt quickly to new classes with low annotation cost. Meta-learning is standard training procedure to tackle the few-shot learning problem and Prototypical Network (Snell et al., 2017) a.k.a ProtoNet a widely used meta-learning algorithm for few-shot learning. In ProtoNet, we use prototype classifier based meta-learning predict the classes of unob- served objects by constructing class-specific prototypes without adjusting the ing meta-testing.
ProtoNet has the following advantages. (I) Since the nearest neighbor method is applied on query data and class prototypes during the meta-test phase, no hyper-parameters are required in meta-test phase. (2) Since the number of data for few-shot is small, the inference time is almost negligible. (3) The classifiers can quickly adapt to new environments because they do not have to be re-trained for the support set when new classes appear. The generalization bound of ProtoNet in relation to the number of shots in support set has been studied (Cao al. 2020). The bound suggests that the performance of ProtoNet depends on the ratio of the between- class variance to the within-class variance of features of a support set extracted using the treantt model.
There have been studies on training a new linear classifier on the features extracted using pre- trained model without meta-learning, which can perform comparably with the meta-learned models (Chen et al.] [019) Tian et al., 2020). We call this approach as ineearevalluation-baaeee approach. In these studies, the models are trained with the standard classification problem, i.e., models are trained with cross-entropy loss after linear projection from the embedding space to the class-probability space. The emear-evaluatton-baee based approach has the following advantages over meta-learning. (1) Training converges faster than meta-learning. (2) Implementation is simpler. (3) Meta-learning decreases in performance the number of shots does not match between meta- mnea-training and meta- testing (Cao al. 2020); however, the Aimear-ealluation-aaeee approach do not need to take this

---

Under review as a conference paper at ICLR 2022
Activation of 2nd neuron
Activation 2nd
60
Activation of 1st neuron
25
Activation of 1st neuron
-39
ProtoNet
(b) cross-entropy loss with linear projection layer
Figure 1: Distribution of features extracted using a neural network with two dimensional final layer trained on CIFAR-10 with (a): ProtoNet loss (b): cross-entropy loss with linear projection layer. The ProtoNet features distribute closer to its class center than the features extracted using the model trained cross-entropy loss with linear projection layer.
into account. However, the linear-evaluation-basse approach requires retraining a linear classifier every time new class appears.
In contrast, prototype classifier can be applied any trained feature extractor and does not require model learning in the testing phase. Therefore, prototype classifier can be a practical and useful first step for few-shot learning problems. In order to avoid meta-learning during the training phase and the linear evaluation during the testing phase, we focus on using a prototype classifier on the testing phase and training models in standard classification manner. As discuss in section 4 we found that when we directly constructed prototypes from the feature vectors extracted using pre- trained models and applied the nearest neighbor method in the testing phase in ProtoNet, this does not perform well as the enncar-vallattion- based approach.
We hypothesize that the reason is the difference between the loss function in ProtoNet and pre- trained models, As described in section 3 if we consider a prototype as pseudo sample average of the features in each class, the loss function of ProtoNet can be considered having regularizing effect that makes it closer to the sample average of the class. Since standard classification training computes cross-entropy loss with additional linear projection to make the features linearly separa- ble, the loss function does not have such an effect and can cause large within-elass variance. Figure T shows a scatter plot of the features extracted using a neural network with two dimension output trained on cifar- 10 with ProtoNet(la) and cross-entropy loss with linear projection layer This figure implies that the features extracted using a model trained a standard classification manner distributes away from the origin and causes large within-class variance along the direction of the norm of class mean vectors, while that of ProtoNet is more centered to its class means. This phc- nomenon also observed face recognition literatures (Wen al. 2016 Liu al. 2017 Wang et al. 2018 Deng et al. 2019).
We focus the theoretical analysis prototype classifier. A current study (Cao al., 2020) analyzed an upper bound of risk by using a prototype classifier. The bound depends on the number of shots of support set, between-class variance, and within-class variance. However, the bound requires the class-conditioned distribution of features to be Gaussian and to have the same covariance matrix among classes. In addition, since the bound does not depend on the norm of the feature vectors, it is not clear from the bound what performance improvement. Thus, we derive novel bound for a prototype classifier.
method can lead to
Our contributions are threefold.
1. We relax the assumption; specifically, the bound does not require that the features distribute any specific distribution, and each covariance matrix does not have be the same among classes.
2. We clarify the effect of the variance of the norm of the feature vectors on the performance of prototype classifier.
3. We investigate the effectiveness of reducing the variance of the norm empirically.
2

---

Under review as a conference paper at ICLR 2022
2 RELATED WORK
We summarize related work by describing a prototype classifier with meta-learning, linear- evaluation-based approach without meta-learning, and theoretical analysis related to the few-shot learning problem.
A prototype classifier with meta-learning On the basis of the hypothesis that features well distin- guished in the training phase are also useful for classifying new classes, constructing one or multiple prototypes for classifying unseen examples is widely used approach (Vinyals et al.) 2016)] [nnell
ct al.
2017 Pahde et al. 2021] Ji et al. 2021] Sung et al. 2018 Allen et al. 2019) Doersch et al, 2020
Qi al. 2018). Certain algorithms compute similarities between multiple prototypes and unseen examples by using their own modules, such as attention mechanism (Vinyals et al. 2016 Doersch et al. (020), relation network (Sung et al. 2018), reweighting mechanisms by taking into account between-class and within-class interaction (Ji et al. 2021), and latent clusters (Allen et al., also constructed in multi-modal way (Pahde et al.] (021). Another line of research is transforming the space of extracted features to a better distinguishable space (Simon) al. 2020 Yoon al. [2019] Das et al. 2020 Das & Lee 2020) taking variance of fetures into account (Bateni et al. 2020) Because of its convenience, the approach with a prototype classifier adapted other domain such as semantic segmentation (Dong & Xing 2018), text classification
2019). Prototypes
(Sun et al. 2019), and speech recognition (Wang et al. 2019a).
Update-based meta-learning In contrast to the approach with prototype classifier and meta- learning, update-based meta-learning approaches, model parameters are adjusted the test phase so that model can adapt to new classes. Model-agnostic meta-Learning (MAML) and its variants (Finn et al., 2017: 2018, Rajeswaran et al.] 2017) search for a good initialization parameters that adapt classes with few labeled data and few update steps of the parameters. Another ap- proach involves learning an effective update rule for the parameters of a base-learner model through sequence of training episodes (Bertinetto et al. 2019 Lee et al., 2019). Both approaches require additional learning of hyper-parameters and training time; thus, they prevent quick adaptation to new classes.
dinear-evaluaaion-base approach without meta-learning Interestingly, recent studies have shown that training a new linear classifier with features extracted using a model trained with cross- entropy loss on base-dataset performs comparably with meta-learning based methods (Chen et al., 2019) (Wang et al.) 2019b). More effective method for training new classifier in few-shot settings has been proposed (Yang et al. 2021 Phoo & Hariharan] 2021), such as calibrating distribution generated by support set (Yang al. 2021) self-supervised learning on query data (Phoo & Har-
iharan 202I). and distilling knowledge to obtain better embeddings (Tian et al., 2020). [iu et al.
(2020) focuses pretraining phase and found the negative margin cross- entsopy helps improving the performance in few-shot settings. However, similar to update-based meta-learning, these studies require additional hyper-paramette and training apply
classes; thus if we tackle these points, we would have an alternative method that is easier and more convenient to use.
Theoretical analysis of few-shot learning Even though much improvement has been empirically made few-shot learning, theoretical analysis is scarce. the context of meta-learning, Du et al. (2021) provided risk bound on the meta-testing phase that is related to the number of meta- meta-training data and meta meta-testing data. Lucas al. (202T) derived information -heorretc oower-bounds on mini- max rates of convergence for algorithms that are trained on data from multiple sources and tested on novel data. Cao et al.) (2020) derived new bound on prototype classifier and theoretically demon- strated that the mismatch regarding the number of shots in support set between meta-training and meta-testing degrades the performance of prototypical networks, which has been only experimen- tally observed. However, their bound depends several assumptions: the class-conditional distri- butions of features are Gaussian and have the same covariance matrix among classes. In contrast, derive novel bound that does not depend any specific distribution.
3

---

Under review as a conference paper at ICLR 2022
3 THEORETICAL ANALYSIS OF PROTOTYPE CLASSIHIER IN TERMS OF VARIANCE OF NORM OF FEATURE VECTORS
In this section, first formulate our problem settings and point out the drawbacks of the current theoretical analysis on prototype classifier. Next we provide our novel bound of a prototype classi- fier with the bound related the variance of the of the feature vectors. Finally, we list several methods that can improve the performance of prototype classifier based on our bound.
3.1 PROBLEM SETTING
Let y be a space of a class, т a probability distribution over V. X' space of input data, probability distribution over X, Dy a probability distribution over A' given class y. We define Disnk and D by D I Dy and D II == Diok respectively. We sample N classes from 7 to form the N --w classification problem. Denote by K a number of annotated data in each class and E X, y Y/ as input data and its class respectively. We define set of support data of class e sampled from 7 as S. = {a:; Ui) E X' x y Nyy = c) and set of support data in the N-way K-shot classification problem as S U Se. Suppose a feature extractor computes function Φ + R.D, where is the number of the embedding dimensions. ((S.) defined by ((Sz) = 1 ExEs, Φ(x). Let & be a space of the extractor function 6. Denote by K
M 4 x X' (X >)NK + RN a prototype classifier function that computes probabilities of
input x belonging to class as follows.
exp (-11)(x)- (66))) o(S.))l))
2( = PM = c/o,, 0)
(I)
Ei=1 exp (-l1o(x)) -
where ||u||/= - Ed=1 (n(d))² and vld) is the d-th dimension of vector The prediction of an input
r, denoted by 1 J. is computed by taking argmax for M(é, æ, S), i.e., 9 argmaxM(d. æ, S).
We denote by Exng(=) [=])] an operation to take the expectation of g(=) over distributed as f(2) and we simply denote Ex [[]]] as R. [=[]]] when is obviously distributed on g(=). We define
Varz=g(=) [[()] as an operation take variance of g(2) over distributed as q(z). With I denoting the indicator function, we define expected risk R. a prototype classifier as
(Ø) = Eg-Dens Bo,, E, [ MargmaxMMo. x, S) + c).
For simplicity, we now discuss the binary classification setting. We show case of multi-class classification in Appendix A.n due to lack of space.
Let ci and 02 denote any pair of classes sampled from 7. We consider that query data belongs to class C₁ and support sets S consist of class ej 's support set and co 's support set. Then, equation 2 is written as follows.
(M) = Es-Don Eej MargmaxMMod x, So, USO.)) # cill-
(3)
3.2 WHAT FEATURE-TRANSFORRE TRANSFORMATIONN METHOD EXPECTED TO BE EFFECTIVE?
The current theoretical analysis for a prototype classifier (Cao et al. 2020 has the following two drawbacks (see Appendix A.I for the details). The first is that the modeling assumption requires dlasscoonditioed distribution of the features follow Gaussian distribution with the same covari- ance matrix among classes. For example, when we use the ReLU activation function in last layer, it not normally distributed and the class-conditioned distribution does not have the same covariance matrix as shown in Figure The second drawback is that it is not clear what feature-transfrmmnn transformation method can reduce the upper bound. A class variance and minimize the within-class variance such linear discriminant analysis (LDA) (Fukunaga 1990) and Embedding Space Transformation (EST) (Cao et al. 2020) can be expected improve performance; however, it is not clear how the second term of the denominator changes.
method to maximize the between-
From Figure [ the distribution of each class feature stretches in the direction of the norm of its class-mean feature vector. This property is also observed in metric learning literatures (Wen al.
4

---

Under review as a conference paper at ICLR 2022
2016 Liu al. 2017). A model trained in the cross-entropy loss after linear projection from the embedding space to the class- cuass-rrbabbiltty space computes probability of input belonging to class given by
exp (Φ(x) We)
ply = c/ W)
(4)
Σド= exp (o(x))T]))
where W is a weight matrix that transforms features from the embedding space to the class- probability space.
Comparing equation 4 with equation 1 we found that equation does not have regularization term similar to the one appearing in equation I that forces the features to be close to its class-mean feature vector. This implies the features extracted using model trained with the cross-entropy loss on equation are less close to its class-mean feature vector than the ProtoNet loss equation I Through this observation and the property mentioned above, we hypothesis that normalizing the norm of the feature vectors push the features to each class-mean feature vector and can boost the performance of prototype classifier trained with cross-entropy loss on equation
3.3 RELATING VARIANCE OF NORM TO UPPER BOUND OF EXPECTED RISK
To understand what effect the variance of the norm of the feature vectors has the performance of a prototype classifier, we analyze how variance contributes to the expected risk when an embedding function ó is fixed. The following theorem provides a generalization bound for the expected risk of a prototype classifier in terms of the variance of the norm of the feature vectors computed using feature extractor.
Theorem 1.
Let M be an operation of a prototype classifier on binary classification defined by equation For pe E<>>,,(<z)) Σ. (c)()()) ((xx)
-pe) ", u= Eewr(pe) [pe].
Ben Seee (He - μ)) and [e-r[ie] = Er, if 6(x) has the variance of its norm, then the miss-classiffcation risk of the prototype classifier on binary classification R.M satisfies
4(Tr(E))
RM( $1 EV]hLa(((z)) + (I, + Vwit (E, E, 11) + dist (He, Mex)
(5)
where
[V[h12(0(x))]
4
Beut
Var [10(=))1
(6)
K
4
2
VIr(Ec::
Varoer [Tr (Eec))
K
K2
8
Vwit (2,, E, u)
Tr(2,) (Tr(E-)) + Tr(2)) + H 11) + 44Tr(T]))))))))) + TT)
(8)
K
((KE - Moz) (Me: Mcz))
E dist (Me, Mez) =Eerez
(9)
Remark. The term EV (V(h1a(6(Ss))) is the variance of the norm of the feature vectors. The term (2,(E,,) is the variance of the sunmation with diagonal element of the covariance matrix from each class embedding: it can be interpreted as the difference of the distribution shape constructed from each class embedding. The term V W_,, u) is correlated with the within-clat variance
because (F........1) is secondary expression for the ratio of between-class variance and within- Tr(2]
class variance. The term E dist?, Moz) is the expectation of the Euclidean distance between the class mean vectors.
This bound has the following properties.
1. Its derivation does not require the features to be distributed on any specific distributions and the caasscooddt covariance matrix does not have to be the same among classes
2. The bound can decrease when any of the following statistics decreases with fixed between- class variance (E): (i) the variance of the norm of the feature vectors, as what we discussed
in Section 3.2 (ii) the difference in the distribution shape constructed from each class
embedding, (ii) the within-class variance (E,), (iv) the Euclidean distance between the class-mean vectors.
5

---

Under review as a conference paper at ICLR 2022
As result, bound loosens the modeling assumption of Theoren in appendix A.1 and has its property in 2-ii).. We show the proof in Appendix A.2
3.4 FEATURE- TRANSFORMATION METHODS
We hypothesize from Theorem that in addition to a feature-transfomne transformaation method related to equa- tion 6 eliminating the affect from the difference in the distribution shape among classes and low- ering the ratio of between-class variance to within-class variance can improve the performance of prototype classifier. Regarding equation 9 the ratio of the Euclidean distance between the class mean vectors and the between-class variance is supposed to be constant. Thus, we focus on the transformation methods relating equation 6 equation 7 a equation We analyze the following LDA, EST,
ceature-transfommen methods: Ly-normalization (Lz-norm), and EST+Zy-norm. We show the detail of each method in Section A.3 due lack of space.
4 EXPERIMENTAL EVALUATION
In this section, we experimentally analyzed the effectiveness of the Ceature-transformmaioon methods mentioned in Sectior The center loss (Wen et al., 20T6) and affinity loss (Hayat al.) 2019) have been proposed to efficiently pull the features of the same class to their centers in the training phase; however, we focus on a widely used pre-trained model in this experiments following the line of studies Chen et al. (2019) and Tian et al. (2020).
4.1 DATASETS AND EVALUATION PROTOCOL
minilmageNet. The miniImageNet dataset (Vinyals et al. 2016) is standard bench-mark for few- shot learning algorithms for recent studies. contains 100 classes randomly sampled from Im- ageNet. Each class contains 600 images. We follow the widely-used splitting protocol (Ravi & Larochelle 2017) to split the dataset into 64/16/20 for training/validationnessti respectively.
fieredImageNet The fieredlmageNet dataset (Ren et al. 2018) is another subset of ImageNet but has 608 classes. These classes are grouped into 34 higher level categories in accordance with the ImageNet hierarchy and these categories are split into 20 training (351 classes), 6 validation (97 classes), 8 testing categories (160 classes). This splitting protocol ensures that the training set is distinctive enough from the testing set and makes the problem more realistic since generally cannot assume that test classes will be similar to those seen in training.
al. 2019) is recently proposed fewshot image classification benchmark, consisting of all 100 classes from CIFAR-100 (Krizhevsky et al.). The classes randomly split into 64, 16, and 20 for meta-training, meta-validation, and meta-testing respectively. Each class contains 600 images of size 32 x 32.
CIFAR-FS The CIFAR-FS dataset (Bertinetto
FC100 The FC100 dataset (Lee et al. 2019) is another dataset derived from CIFAR-100 (Krizhevsky eet al.), containing 100 classes which are grouped into 20 categories. These categories are split into 12 categories for training (60 classes), from 4 categories for validation (20 classes), 4 categories for testing (20 classes).
4.2 IMPLEMENTATION DETAILS
We compared the feature-rrasfoormation methods applied on against ProtoNet (Snell et al. 2017). Baseline and Baseline++ (Chen al. 2019). In Baseline, the linear projection layer is trained on support set and in Baseline++,, the norm of the linear projection layer and features are normalized be constant. We call Baseline and Baselinet+ as "linear evlauation" methods. We also com- pared with the feature-transfomman In that study they transformed the features so that the mean of all features to be origin before L normalization and then use prototype classifier without training new linear classifier. We call this operation as Centering+L--nnne normm. We re- re-implemented these methods following the training pro- cedure in previous study (Chen al., 2019). In the pre-training stage, where the cross-entropy loss was used and meta-learning was not used, we trained 400 epochs with a batch size of 16. In the meta-training stage for ProtoNet, trained 60, 000 episodes for 1-shot and 40, 000 episodes
method proposed in previous study (Wang et al.] 20196).

---

Under review as a conference paper at ICLR 2022
Table Classification accuracies on minilmageNet and rieredmageNeet of ProtoNet
linear evaluation methods (Chen et al. 2019). and ours. The best performing methods and any other runs within 95% confidence margin are bold.
Method
minilmageNet
ImageNl
ResNet12
ResNet18
ResNet12
ResNet18
1-shot
5-shot
I-shot
5-shot
1-shot
5-shol
1-shot
5-shol
ProtoNet
53.48%
73.56%
56.26%
74.02%
55.40%
77.67%
60.50%
81.40%
В@FT
54.51%
76.50%
55.41%
76.95%
61.67%
81.62%
63.38%
83.18%
B+@FT
56.33%
74.62%
55.07%
71.15%
63.02%
81.07%
64.20%
81.62%
B@CL2
58.66%
75.98%
57.67%
70.78%
64.88%
80.42%
65.26%
81.63%
B+@CL2
57.50%
74.00%
37.00%
74.06%
63.31%
79.19%
65.67%
81.41%
B
46.36%
73.97%
43.86%
72.36%
50.60%
78%10%
56. 16%
80.33%
B@L2-N
57.18%
77.12%
56.57%
76.44% %
64.04%
81.73%
65.19% %
82.93%
B@VN
63.55%
62.78%
66.08%
75.56%
B@LDA
73.75%
73.54%
77.44%
80.85%
B@EST
51.28%
72.80%
44.19%
72.99%
53.90%
78,02 0%%
57 12%
80.59%
B@EST+L2N
58.00%
76.90%
56.39%
76.24% %
64.54 %
81.40%
64.71 %
83.24%
B+
41.18%
73.97%
36.80%
63. 63.76%
46.52%
73. 85%
48 27%
75.87%
B+@LL-N
57.96%
75.38%
57.21%
71.89%
64.96%
81.08 %
65.40 %
82.49%
B+@ V-N
55.92%
54.31%
62. 62%19%
64.33%
B+@LDA
69.60%
74.38%
72.88%
75.55%
@+@EST B+@EST+L2-N
47.11%
69.01%
47.21% %
69.11%
52.01%
74.26%
53.49%
75.38%
58.32%
77.49%
57.00 %
77.13%
57.63%
80.99%
60.87%
81.80%
for 5-shot tasks. We used the validation set to select the training episodes with the best accuracy. In each episode, sampled N classes to form N-way classification (In meta-training N=20 and meta-testing N=5 following the original study of ProtoNet (Snell et al., 2017)). For each class, we selected K labeled instances
support set and 16 instances for the query set for K-shot task.
In the linear evaluation or meta-testing stage for all methods, we averaged the results over 600 trials. In each trial, we randomly sampled 5 classes from novel classes, and in each class, also selected K instances for the support set and 16 for the query set. For Baseline and Baseline++,, we used the entire support set to train new classifier for 100 iterations with batch size of 4. With ProtoNet, we used the models trained in the same shots as meta-testing stage since the mismatch of number of shots between meta-training and meta-testing degrades performance (Cao al.. 2020). All methods were trained from scratch, and the Adam optimizer with initial learning rate 10 was used. We applied standard data augmentation including random crop, horizontal-fie -lip. and color jitter in both the training stages. We used a ResNet12 network and ResNet18 network following the previous study (Chen et al. 2019 Yoon al. 2019).
Since LDA constructs a within-class covariance matrix from support set and variance- normalization computes sample variance, we did not investigate the LDA score and variance- normalization score of 1-shot settings. For LDA we set À 0.0001 for equation [8] in Appendix A.3 and for EST we set the dimensions of the projected space to 60 following the settings of the original study (Cao et al. 2020
4.3 RESULTS
We present the experimental results on Tan Table on the basis of backbones with ResNet and ResNet18 for comprehensive comparison. Table I shows the results on minilmageNet and fieredImageNet, and Table shows the results on CIFARFS and FC100. We show the detailed performance results with 95% confidence margin and other transformation methods in A.6
Method notations in experimental results We denote #TFT and B+@FI as the linear evalua- Baseline and Baseline++.. We also denote B@CL2, B+@ CL2 as centering+Lz-nomm WWang et al. (019bb on features trained with Baseline and Baseilne++. B. B@L2-N, B@ V-N, B@LDA, B@E and B@EST+L2-N stands for a prototype classifier with applying no trans- formation, L2 Ly-nomm, variance- nommaiiaaooiiiiionn...mmmm LDA, EST, and EST+L2-N on the features extracted using a model pre- pre-trained in the same manner as BQFT. B+. B+@L2-N, B+@ V-N, B+@LDA, B+ @ EST, and B+ @EST+L++-N stands for prototype classifier with applying no transformation,
tion methods:
7

---

Under review as a conference paper at ICLR 2022
Table 2: Classification accuracies on CIFARFS and FC100 of ProtoNet, linear evaluation methods (Chen et al. 2019), and ours. The best performing methods and any other runs within 95% confidence margin are in bold.
Method
CIFARES
FC100
ResNet12
ResNet18
ResNer12
ResNet18
1-shot
5-shot
I-shot
5-shot
1-shot
5-shot
1-shot
5-shol
ProtoNet
58.65%
75.33%
62.05%
78.25 %
35.56%
51.12%
36.02%
51.02%
B@FT
55.97%
76.50%
56.46%
77. 77.43%
39.72%
56.04%
40.06%
57.04%
B+@FT
61.08%
76.15%
61.34%
77.86%
36.01%
50.73%
36.93%
50.41%
B@CL2
58.61%
74.73%
58.09%
76.52%
41.58%
55.82%
41.51%
56.44%
B+@CL2
61.59%
76.10%
63.17%
77.49%
37.51%
50.38%
38.30%
51.06%
B
44.12%
72.47%
47.86%
75.66%
31 31.60%
52.50%
35.90%
55.20%
B@L2-N
59.43%
77.45%
58.51%
77.43%
40.34%
56.61 %
40.49%
57.71%
B@VN
44.66%
67.72%
41.99%
49.69%
B@LDA
75.73%
75.25%
52.90%
55.44%
B@EST
52.62%
72.80%
55.57%
75. 10%
44.45%
52.97%
47 49%
55.68%
B@EST+L2N
60.70%
77.04%
61.14%
77.48 %
47.57 %
56.96% %
50.13 %
59.94%
B+
43.73%
73.97%
36.80%
63.76 73.766%
72%
46.85%
30. 76%
47.62%
B+@LL-N
61.07%
76.71%
63.48 %
77.86
37.16%
51.10%
38.55%
50.15%
B+@ V-N
57.20%
57.16%
36.76%
40.66%
B+@LDA
68.58%
69.86%
47.10%
47.63%
@+@EST
50.18%
70.32%
49.8 19.82%
71.07%
37. 37.60%
46.81%
39.92%
47.65%
B+@EST+L2-N
59.83%
76.32%
63.00 %
77.99%
40.88%
50.45%
41.65%
50.74%
I2-norm, and variance-normalizatioe on the features extracted using the same pre-trained model as with B+ FT.
Comparison of the feature transformation methods with ProtoNet, linear evaluation meth- ods, and centering+L2-non classifier with L2-norm and EST+Ly-norm performs comparably with ProtoNet and the linear- evaluation-based approach in all settings and EST+L2-norm performs the best among the data trans- formation methods. Comparing the ceaturettransformation methods described in Section 3.4 with centering+ (Wang et al., (019b),, centering+L/z-nnne of the prototype classifier in several 1-shot settings. However, in 5-shot settings, the boost decreases and even performs worse than L2-norm, e.g. minilmageNet and tieredImageNet with ResNet12.
From Table [ and Table 2 we can observe that the prototype
slightly improve the performance
Comparison of the feature transformation methods with B and B+ In the 1-shot setting, al- though EST falls short of ProtoNet and evaluation- approach, it also improves the per- formance of prototype classifier. The performance gain of both Læ-norm, EST and EST+La-norm decrease when the number of shots increases. This phenomenon can be explained through Theorem The term relating to the variance of the norm and the ratio of the between-class variance the within-class variance depends on K. K.Since the term diminishes with increasing K, the performance gain of the tranorrrtrannoo methods decreases.
Discussion the feature transformation methods Surprisingly, variance normalization per- forms worse than applying no feature transformation. This can be explained by the following two reasons. The first is that the term in the bound related the difference among the class distribution shapes is smaller than other terms such as the variance of the norm and the ratio of the between-class variance the within-class variance. Figure 2 shows the term associated with the bound computed from each dataset and backbone. The larger the value is, the more it contributes to larger risk. We found the term associated with equation 7 is small compared with other values and thus variance- normalization did not worked well in our experiment. The second reason is that estimating variance with a support set is unstable and make prediction unreliable in few-shot learning. LDA and EST in 5-shot setting does not improve performance so much compared with L2-norm variants while in the 1-shot settings, EST helps improving performance. This because in the 5-shot settings, the values computed from equation 8 gets smaller with larger K and the effect of equation 8 on the risk of prototype classifier in the I-shot settings is larger. Especially, EST outperforms L norm FC100 with the s-shot setting. We found from Figure the features of FC100 shows the largest ratio of the
8

---

Under review as a conference paper at ICLR 2022
1.0
ResNet12
88
ResNet18
86
04-
04
02-
80
EV hh (xxx))
Vrr Ex
mini ImageNet
tiered ImageNet
FC100 CIFARFS
Tr(2)²
Tr(E)²
Tr(E)²
Figure 2: Left: We plotted the values shown in equations 6 8 divided by Tr(E)² The values computed from lest-split each dataset with ResNet12 and ResNet18. We scaled the values so each dataset's (wit(E- be for simplicity. Right: We showed the ratio of the within-class variance the between-class Tr]3jj
variance computed from the test-split's 's features extracted using each backbone trained each corresponding datasct. We scaled the values so that the maximium value to be 1 for simplicity.
62
0.3
06
80
0.1
02
03
04
05
0.6
07
08
cosine similarity
cosine similarity
Figure 3: Histogram of cosine similarities with class mean vectors and largest eigen vectors of Left:E, Right:E.
within-class to between-class variance among all dataset. Thus the method reducing the ratio works better with FC100 features than any other dataset's features.
4.4 EFFECT ON COVARIANCE MATRIX
We further analyzed how Tr(E.) and the trace of class covariance matrix Tr(Et.) fluctuate with L2-norm. Since L2-norm lowers the variance in the direction of the normalized vector, if the co- variance matrix stretches the direction of the class mean vector, L2-norm can reduce its trace. Thus we analyzed the cosine similarities between class-mean feature vectors and the eigenvectors with the largest eigen value of the covariance matrices. Figure iillutrates the distribution of cosine similarities of each class mean feature vector and the eigenvectors in test class of minilmageNet. This figure shows Σ, tends to stretches to the direction of class mean vector and Σ stretches in the orthogonal direction norm can reduce Tr(E.) and has less effect on Tr(E). L2-norm can reduce the variance of the norm and the ratio of the within-class variance to the between-class variance.
each class-mean feature vector. From this observation,
found that L---
CONCLUSION
We theoretically and experimentally analyzed how the variance of the norm of feature vectors affects the performance of a prototype classifier and found that using EST+Lg-norm makes the classifier comparable with ProtoNet and the linear-evaluation-besse approach. We also found that when the number of shots a support set increases. the performance gain from ceature-rransformation method decreases, which consistent with the results of the theoretical analysis. A prototypical classifier is expected to a practically useful first step in tackling few-shot learning problems because of its simplicity.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Kelsey R. Allen, Evan Shelhamer, Hanul Shin, and Joshua B. Tenenbaum. Infinite mixture proto- types for few-shot learning. PMLR, pp. 232-241, 2019.
Peyman Bateni, Raghav Goyal, Frank Wood Vaden Masrani, and Leonid Sigal. Improved few- shot visual classification. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 3-14, 2020.
Luca Bertinetto, João Henriques, Philip H.S. Torr, and Andrea Vedaldi. Meta-leaming with differ- entiable closed-form solvers. In International Conference Learning Representations (ICLR), 2019.
Tiansi Cao, Mare T.Law, and Sanja Fidler. A theoretical analysis of the number shots in few-shot learning. In International Conference Learning Representations (ICLR), 2020.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations (ICLR), 2019.
Debasmit Das and C.S. George Lee. A two-stage approach to few-shot learning for image recogni- tion. In IEEE Transactions on Image Processing, volume 29, 2020.
Debasmit Das, J. H. Moon, and C. S. George Lee. Few-shot image recognition with manifolds. In International Symposium on Visual Computing (ISVC), pp. 3-14, 2020.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2019.
Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformerr: spatially-aware few-shot transfer. Advances in Neural Information Processing Systems, pp. 4003 4014,4014, 2020.
Nanqing Dong and Eric Xing. Few-shot semantic segmentation with prototype learning. British Machine Vision Conference (BMVC), 2018.
Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qei Few-shot learning via learn- ing the representation, provably, International Conference on Learning Representutions((CLRR)) 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 1126-1135. PMLR, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Ad- vances in Neural Information Processing Systems, pp. 9516-9527, 2018.
Keinosuke Fukunaga. Introduction to Statistical Pattern Recognition (2Nd Ed.). Academic Press Professional, Inc., 1990.
Munawar Hayat, Salman Khan, Syed Waqas Zamir, Jianbing Shen, and Ling Shao. Max Mar-mn class imbalanced learning with gaussian affinity. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6469-6479, 2019.
Zhong Ji, Xingliang Chai, Yunlong Yu, and Zhongfei Zhang. Reweighting and information information-guidance networks for few-shot learning. In Neurocomputingg volume 423, pp. 13-23, 2021.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100.
K Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10657-10665, 2019.
Bin Liu, Yue Cao, Yutong Lin, Qi Li. Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in few-shot classification. In European Conference on Computer Vision (ECCV), 2020.
10

---

Under review as a conference paper at ICLR 2022
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2017.
James Lucas, Mengye Ren, Irene Raissa KAMENI KAMENI, Toniann Pitassi, and Richard Zemel. Theoretical bounds estimation error for meta-learning In International Conference on Learn- 2021.
ing Representations//
Frederik Pahde, Mihai Puscas, Tassilo Klein, and Moin Nabi. Multimodal prototypical networks for few-shot learning. In Winter Conference on Applications of Computer Vision WACV), pp-
2021.
Michael D. Perlman. Jensen's inequality for a convex vector-valued function on an infininite- dimensional space. In J Multivar Anal, pp. 52-65, 1974.
Cheng Perng Phoo and Bharath Hariharan. Self-training for few-shot transfer across extreme task difference. In International Conference on Learning Representationss//II..) 2021.
Hang Qi, Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp- 5822-5830, 2018.
Aravind Rajeswaran, Chelsea Finn, Sham M. Kakade, and Sergey Levine. Meta-learning with im- plicit gradient. PMLR, pp. 1126-1135, 2017.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In International Conference on Learning Representations (ICLR), 2017.
Mengye Ren, Sachin Ravi, Eleni Triantafillou, Jake Snell, Kevin Swersky, Josh B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-lcarning for semi-supervised fewshot classification. In International Conference on Learning Representations (ICLR), 2018.
Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi. Adaptive subspaces for few- shot learning. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2020.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
Shengli Sun, Qingfeng Sun, Kevin Zhou, and Tengchao Lv. Hierarchical attention prototypical networks for few-shot text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 476, 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In Computer Vision and Pattern Recognition (CVPR), 2018
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? European Conference on Computer Vision (ECCV), 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Advances in Neural Information Processing Systems, 2016.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2018.
Jixuan Wang. Kuan-Chieh Wang, Mare T. Law, Frank Rudzicz, and Michael Brudno. Centroid- based deep metric learning for speaker recognition. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3652-3656. IEFF, 2019a.

---

Under review as a conference paper at ICLR 2022
Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens der Maaten. Simpleshot: Revis- iting neighoo classification for few-shot learning. volume abs/: 2019b.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning ap- proach for deep face recognition. In European Conference Computer Vision (ECCV), 2016.
Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In International Conference on Learning Representations/(CLL. 2021.
Sung Whan Yoon, Jun Seo, and Jaekyun Moon. Tapnet: Neural network augmented with task- adaptive projection for few-shot learning. In Proceedings of the 36th International Conference Machine Learning (ICML.,, 2019.
A APPENDIX
A.1 EXISTING UPPER BOUND ON EXPECTED RISK FOR PROTOTYPE CLASSIFIER
To analyze the behavior prototype classifier, we start from the current study (Cao et al. 2020
The following theorem is the upper bound of the expected risk of prototypical networks with the next conditions.
The probability distribution of extracted feature ((x) given its class y = cis Gaussian i.e
Dy = [Nee, Me), where Ae = Ex-D, [o(x)] and De = ((o(x) (ó()) Me)
All class-conditioned distributions have the same covariance matrix, i.c., v(c, Ee Ege
Theorem 2.
Let M be an operation of a prototype classifier on binary classification defined by Then for μ Eeor [ue] and Eewr [Me =)( p)) ]. ], the miss- cissssllasification risk
equation
[
of the prototype classifier on binary classification RM satisfies
(T)
RM( S1 8(1 + 1/h32 Tr (E2) - 16(1 + 1/k) Tr (EE.) + TE dist ( Moz)
(10)
where E dist (Mes Mea) Ee
((Me - Mca) (Me, - 1²
We show the detail of the derivation in Appendix A.2
A.2 DERIVATION DETAILS OF THEOREM
2
we will explain the rough sketch of derivation of Theorem 2 In prototype classifier, from equation
and equation Ry written with sigmoid function as follows:
= (0)
(a(ll(()) S-
SDDKKK
o(S..) ||((x) - o (SS..)))
=Pr(a < 0),
(11)
where A [lδ(x)) お o(S.,)) [()(=)) ((S.))) Ill bound equation 11 with expectation and
variance a by following proposition.
Proposition From the one-sided Chebyshev's 's inequality, it immeddaately follows that:
EgnDaa egg Ea D., [a]2
RM( Pr(a < 0) 51 Vars, [a) + ES-Dov E, oerr EanDD [a]2
(12)
equation II can be further write down follows by Law of Total Expectation
Vars,e. (a) = ES Be, [a?] - (Es- Ber [a])
= Ee,,,, Ex c1 c2] -
BewN,, (al)²
= EE /Var (2) + E. (aler,
Bg Bgae EanDD E EswDon [a]2.
12

---

Under review as a conference paper at ICLR 2022
Therefore,
Pr(a < <0 S Bsw Dew Be,, Bu Be D.
(13)
Boy,e [ar Dej Pe..5 s(a] E Deg [|]]
We write down the expection and variance of with following Lemmas and Lemma 3. Under the same notation and assumptions as Theorem then,
ES-Deu D.w., [a] (11, M (H Moa) Bes carer EAND. Dr, ESNDaOK [a] = 2Tr(E).
Lemma 4. Under the same notation and assumptions Theorem then,
1 K
1 ) + 22) E1((1
Eay,r [ara, (aler,, c2]] S 8
Tr
(14)
The proofs of the above lemmas are in the current study CCo et al. 2020).
With Proposition
1 and Lemma Lemma obtain Theorem
A.3 DETAIL OF FEATURE- -AERRE--TAAA TRASNFORMING METHODS
Le normalization (L2-norm) From equation normalizing the norm of the feature vectors can improve the performance of prototype classifier. We denote a function normalizing the norm as L2 given by
<(x:
()2(0(x)) = ||o(=)]
(15)
naraaice-oommalzzatio From equation 7 normalizing the Euclidean distance in equation [ by the variance of each class embedding can improve the performance of a prototype classifier. This can be interpreted as a variation of the Mahalanobis distance with a diagonal covariance matrix. We denote a function that measures the distance of two vectors normalized by its variance at each dimension as follows:
( (ol=)))(-((a))) (xx)))
mmaha (0(x))).
(16)
)arlo())))(a))
dml
where Var[2] is sample variance of z. equation is rewritten with e'maha(@(z))) (#(21), ((x2)) as the following equation.
exp U-aaaa (olat), (bS))
M(o. x, 2),
(17)
Σ exp
--maha
(xx)) (SS)
This normalization similar to Bateni et al. (2020)'s 's work but our method differs in that we do not need to do meta-learning and we do not need to add other modules to the backbone network.
LDA From equation increasing the ratio of the between-class variance to the within-class vari- can improve the performance of a prototype classifier. LDA (Fukunagal 1990) is widely used search for a projection space that maximizes the between-class variance and minimizes the within- class variance. It computes the eigenvectors a matrix S ÉÉ, where & the covariance matrix of prototypes and Σ, is the class-conditioned covariance matrix. Since the number data is small in few-shot settings, Si cannot be estimated stably and add regularizer term to E and define it 2r
ETrg = Σ + AI,
(18)
where À E Rd. E RDxD is identity matrix.
13

---

Under review as a conference paper at ICLR 2022
EST Since computation of LDA is unstable, also analyze the effect of EST (Cao al. 2020).
EST computes eigenvectors of matrix Σ pΣA: the difference between the covariance matrix of the class mean vectors and the class mean covariance matrix with weight parameter Similar to LDA, EST also searches for the projection space that maximizes the E and minimizes the E, We compute the eigenvectors of the matrix with two small modification. The first is that we remove hyper-parameter rho and then simply computes the cigenvectors of Σ Σi, The second is that we constructs the matrix from a support in the multi-shot setting while we follows the original study in the 1-shot setting.
EST+/22norm We hypothesize that the combination of the transforming methods can improve the performance of prototype classifier independently from each other. Specifically, focus reducing 6o equation 8 by combining EST and Ly-norm. We first apply EST to reduce equation 8 and after that we apply L2-norm reduce equation [ At the end of the operation want the variance of the norm to be 0, thus we apply EST and after that we apply L2-norm.
variance-oormallaatioot//-zooo
We focus on reducing equation 6 and equation 7 by combin- ing variance-normalizztioo and I2-norm. Following the similar procedure of EST+ E we first apply variance-normalization and after that we apply Ln-norm. We show the experimental results of this transformation methods in A.6
LDA+Lg-norm We focus reducing equation g and equation by combining LDA and L2- norm. Following the similar procedure of EST++L-norm. we first apply LDA and after that we apply Io-norm. We show the experimental results of this transformation methods in A.6
A.4 DERIVATION DETAILS OF THEOREM I
We start the proof from equation 13
We first prove the following Seemmmarrrlattt to the expectation
statistics of a in equation 13
Lemma 5, Under the same notations and assumptions as Theorem then,
1
Es- EEwD, [a]
(Tr(Ec,) - Tr(Ee,)) + (Mez Meg T Mozs)
K
E E ESnDeax [a] 2Tr(E).
Proof. First, from the definition of a, we split Dea [a] in to two parts and examine them seperately.
Es- EEW [a] E ||(()) o(Scz)
E
((xx) ((SE))
(19)
(+4)
In regular conditions, for random vector X, the expectation of the norm E T x] Tr(Var(X)) + [
(20)
and the variance of the vector is
=ar(X)= -
(21)
E,, 2 VarmDD. (6(x))
(22)
Hence,
ii EDwDD Es (b(x)) ((Sogg) 27
=Tr (VaronDey (6(x)= ((Sor) ) EyLSS (0(x)) o(S.z)
Ex (0(x) o(S.z)
(23)
14

---

Under review as a conference paper at ICLR 2022
where the first term inside the trace can be expanded as: Var (6((z) - (S(2z))
((Sez) Var [0(x))] [x(x)) E [0(x))] Var (}(ccg)
(0(z) - o((cz))
(Me, - (Hg)) Heg)
E ( E (0(Soz)
Hog 4o
He, Moz (He, (He - Moz
Des
Dez (Last terms cancel out).
(24)
The second term in equation
is simply as follows.
Da, Es
p(xx) o(Sc.) = Mcs Hez
(25)
From equation 24 and equation 25 obtain
1 Tr(Eog)) + (Hei (oa)"(Ha Hez)
(i) Tr(Eo,) +
(26)
K
Similarly for ii,
(ii) = (Varz- Deyis (0(x)) -o(sa)))) E Es (d(x) o(S.a) -
=Tr(Ze))+ + K Ir(Ca).
(27)
From (i) and (ii), and equation
19
ES-Deu Ba.DEEEEDD... BawD., [a]
(Tr (Txz)
-Tr (Eor)) + (He, - Mo,)) (4 Hog)
(28)
K
Since ee x, s|a] Eer [LS
l BaoD, [a],
E E2 [a] Beggo
K (Tr(Engg) Tr(Ect)) (Her Moz)) (H Hoz)
(He, -Hom) T (He, Hoz)
Eeg.0ger
He Hta + H Hes - -H Hez - H Hey
=Er(E) +H + (E) + p H- 2u T μ
(2).
(29)
Next prove the following Lemma to the conditioned variance of Lemma 6. Under the same notation and assumptions as Theorem 7
E,,c, Var SwDS2N [a] s K
Eor Var [|o(z)]]] + Var [Tr(Ee] + Vwi E, E.,,. u),
K
(30)
where
8 Vwnt E,,,E, (Tr(E,)) (Tr(E) + 4 TT)) 4 (Tr(E) =T)²
(31)
K
Proof. We start with the inequality between the variance of 2 random variables. We define
Cov(A, B) as covariance of 2 random variables A, B.
Var(A A + B] = Var[A] + Var B] + 2Cov(A, B)
Var[A] + Var B + 2 Var Var[A]]ar
Var [A] Var
Var[A] + Var B +
2ar[[A] 2Var
(32)
15

---

Under review as a conference paper at ICLR 2022
For Var ,s (a],
Var aa s/a]
Var
((x) ((S..)
[(xx) x(Sc,)
Var (((
)(S
-2(x)) (0(Scz) ((Sc.))
|
2Vr
4Var (0(z0)
(o(S.z) (55 ) equation 32)
2Var
| )|
2
+
2ar
|
4Var x(xx) T (0(S.z) ((
(33)
From 3rd line to 4th line we deompose the variance of (Se..) of (SS,,) and Φb (S with their class given. Next we focus on Var (6(x)) (o(S.)) (SS..)
((Scz)
use the independence
Var ((x)
(o(Sc,)) (
-((Sa)) = (o(x) (0(Scz) - o(Sa,)
-BB(6())] T E (((cc)) (((Se.)
2
S (1Id(2)) p(Scz) ((8
((8 11) (H, T(Hoz H ))²
[10(=)]]] E [((s.n))) o(Sa))
(n (He, M ))²
(34)
From the 2nd line to the 3rd line use Schwarz inequality. For E [(Sc.) ((Se,)
with equation 20
1
E
((Szz) ((Sa)
(Tr (EEe) Tr (Eeg)) (Ho, Mo,) (Me, Hc,) (35)
16

---

Under review as a conference paper at ICLR 2022
Thus E2gT Be De [1((
BEnDey |0((5a) -((a)) [=]
|
is calculated as follows.
|1o[sw] - o(So,) 27
Be BEWDD [1(())11 BEWDoy.s
(Tr(Zes) ) + Mog HatHaa) ( K
(Tr(Ecs)) + Tr(Eep)) + (Hez Me, (Hes Hon))
=Be
1
Tr (Ee,) Tr((cr) Ir (Eer))
K
2 (Tr (E+)) Her Hei + Mej He, (Her Hea) (Heg Ham)
+ Bui
K
EEo,ean (Tr(F:o,))) cooer (Tr(Ecc))) )1²)
2 (Tr(E))) (Tr H 14) [e Ho, (Moz Hop (Hec M
K
Vareur [T [Tr (Ee)
K
+ Tr (T,))))
2
K (Tr (Tr(E) + 4 =THH) + Reyos Mo Ho, (H2r - Hoo) T (Hoz - Ho.))
K
Vareur [Tr(xc]]
2 Tr(E,) (Tr(E,)+ Tr(2) 4 ()) Eor,2 [Mo] Ho, (Mos Mo, ) (Hoz Hoa)
+ K
(36)
Now we take into account the term ((Mc, T(Her - Mey M2,))² equation 34
Ec,,0
| Me, (Hez Mei) (Moz
Me,) ) (Ho) (Mez Me,
E [ He, Mor
(n k )
s Em [o He, Hoy Ho.]
(Tr(E) + u)
(37)
Thus Ea Varld(x) (Ō(S.z)) ((((()))((((s)))) (1)
is calculated as follows.
Eeroz
Var[ō(x) (6((s)) o(S.))))
(38)
2 K Vareer [Tr(Ee)] + Tr(E,, (Tr(E.+ TT(E) + + =TT)) HT) (Tr(E) + TT nT)
(39)
K
Regarding ((S)) since the function computing square norm convex, next equation holds with
D-dimensional Jensen's 's inequlality (Perlman 1974).
((S.)
Σ
((x)
(wx)
(40)
Se
17

---

Under review as a conference paper at ICLR 2022
Combining equation
33
equation 38
and equation 40 obtain
ee aa SWDDEN (ax]
4
Bent VaraaD [16((2)11 + Var [Tr(Zn)]
Tr((),) (Ir(2i)) TT(2)) 4 u) 4 (T+(2) + 1 [111) 1))
(41)
K
To complete the proof of Theorem we calculate E,, st [a]?
S
Bel,eyBeen Do, De, Sw. [a]2
(42)
27
=Fa,ee
-T((E)))T(((E))) + (Me, -Ma) Mea)) K
Be
(Tr(Eer) -TT(Ee))))
+ Eee,, [(Hb) Meza) (He, Me
K
2.
(Tr(Eees) Tr(Eee))) (1Ha, -Ho)) -He)))
1 (Tr(Ees) Tr(Ena)) + Exs,
1 (Tr((ee))) (M(Eas))))
1
-aren
K
K
+ Berrez ((Hes
Moz)" ((He M2z))
=
K
Vareer [Tr(E.)] + Ber,e
[(He) - Hoe)) (Me, - m²
(43)
From 2nd line to 3rd line, the symmetry of the last term with respect and e2 and erase the term. Combining equation 13 Lemma Lemma 6 and equation 42 we obtain the bound.
A.5 THEOREM WITH N-WAY CLASSIHICATION
The upper bound on the risk of N-way prototype classifier is as follows.
Theorem 7. Let operation of binary class prototype classifier M as defined in equation 7 Then for ((S)) = K Emes, ((x), He = Em= E, = [()(x) ())()))) - Me) 1. =
1
Bor [He].E = Ee (H μ)) JJ, Ecwr [Cl] = Σ, if Φ(x) has its fourth moment, miss classification risk of binary class protorype classifier R.M satisfy
R(M(O, x, (S.)
N
{(Tr(E))
N [V]hradd<z)] (T, Vwit (2i,, E.H4, 11) E dist (Myy me))
(44)
where
EV[hr2(o(x))
4
Eysr
Varz. [10(2))1|]
K
4
2
Vrr(E.y) =
Vareur Tr (Er)),]
K
K²
Vwit(2i, 2.
Tr(Ei, (Tr(E;,) + Tr(2) + u u) + 4(T((2)) +uul uTm)²
Edistt2214y =Ry ((He, - mo) (nl - pc)) 2
18

---

Under review as a conference paper at ICLR 2022
Table 3: Classification accuracies on minilmageNet of ProtoNet linear evaluation meth- ods (Chen et al. 2019). and ours, The best performing methods and any other runs within 95% confidence margin are in bold.
Method
ResNet12
ResNet18
1-shot
5-shol
1-shot
5-shot
ProtoNet
53.48 + 0.89%
73.56 = 0.65%
56.26 + 0.85%
74.02 = 0.65%
B@FT
54.54 0.80%
76.50 = 0.62%
55.41 + 0.82%
76.95 + 0.61%
B+@FT
56.33 1 0.81%
74.62 + 0.65%
55.07 1: 0.81%
74.15 1 0.66%
B@CL2
58.66 = 0.83%
75.98 = 0.62%
57.67 = 0.83%
70.78 0.67%
B+@CL2
57.50 + 0.81%
74.00 = 0.61%
57.00 + 0.64%
74.06 + 0.61%
B
46.36 + 0.58%
73.97 + 0.62%
43.86 + 0.80%
72.36 + 0.92%
B@L2-N
57.18 + 0.80%
77.12 0.44%
56.57 + 0.80%
76.44 + 0.61%
B@V-N
63.55 1 0.49%
62.78 1 1.02%
B@ V-N
65.42 1: 0.64%
64.09 0.67%
B@LDA
73.75 1 0.64%
73.54 1: 0.85%
B@LDA+L2AN
74.80 + 0.64%
73.70 + 0.62%
B@EST
51.28 + 0.88%
72.80 = 0.64%
44.19 - 0.82%
72.99 1 0.63%
B@EST+L2-N
58.00 + 0.86% %
76.90 + 0.62%
56.39 0.79%
76.24 + 0.64%
B+
41.18 0.76%
73.97 + 0.66%
36.80 + 0.76%
63.76 + 0.71%
B+@L2-N
57.96 + 0.83%
75.38 + 0.61%
57.21 + 0.83%
74.89 + 0.65%
B+@V-N
55.96 + 0.63%
54.31 + 0.64%
B+@ V-N+L-N
56.20 + 0.64%
55.42 + 0.68%
@+@LDA
69.60 + 0.70%
74.38 0.65%
B+@LDA+L2-N
74.34 + 0.63%
74.38 + 0.63%
B+@EST
47.11 + 0.81%
69.01 + 0.63%
47.21 + 0.77%
69.11 = 0.64%
B+@EST+L2-N
58.32 + 0.81 %
77.49 = 0.63%
57.00 + 0.75%
77.13 + 0.64%
Proof.
Let y be the input and its class of query data. We define a, by a = -0(S.)
-
(((x)) - x(,) Then a prototype classifier miss-classify a class of input x. G. if E [1, N/.c
Hence: (Ø) Pr(U = ae < <)
N
y. ac
cF
By Frechet's inequality, next equation holds:
RM( & Σ Pr(a < 0).
Noting that Theorem [aa be applied to each term in the summation and then we obtain Theorem
i
A.6 DETAILED PERFORMANCE RESULTS
We show the detailed performance results in this section with 95% confidence margin. We add the two future normalization methods, variance normalization before Lg-normalization (V-N+L2- N) and LDA before Lz-normalization (LDA+L2-N) to the tables. Table 3 and Table g show the performance on minilmagenet and tieredlmagenet; table and table 6 show the performance on CIFARFS and FC100.
A.7 THE COMPARISON THE RATIO THE BETWEEN-CLASS VARIANCE THE WITHIN-CLASS VARIANCE BEFORE AND AFTER APPLYING L2-NORM
We show the ratio of the between-class variance the within-class variance before and after ap- plying norm in Figure 4 We calculated the ratio of the between-class variance to each class's variance and averaged over the test classes of each dataset. Although n.gnoraalizationn is expected to reduce Tr(2)) rather than D. according to 4.4 the figure shows L2-norm marginaly reduces the ratio CIFARES dataset and the ratio does not changed so much in the other dataset.
19

---

Under review as a conference paper at ICLR 2022
Table 4: Classification accuracies on magred of ProtoNet, linear evaluation meth- ods (Chen et al. 2019), and ours. The best performing methods and any other runs within 95% confidence margin are in bold.
Method
ResNet12
ResNer18
1-shot
5-shot
1-shot
5-shot
ProtoNet
55.40 + 0.98%
77.67 0.70%
60.50 1.01%
81.40 0.68%
B@FT
61.67 = 0.92%
81.62 0.64%
63.38 0.91%
83.18 - 0.64
B+@FT
63.02 + 0.91%
81.07 + 0.69% %
64.20 + 0.92%
81.62 + 0.69%
B@CL2
64.88 + 0.86 %
80.42 1 0.64%
65.26 + 0.88%
81.63 1 0.64%
B+@CL2
63.31 + 0.91%
79.19 0.68%
65.67 0.91%
81.41 = 0.68%
B
50.60 + 0.87%
78.10 1:00::000 0.67%
56.16 0.89%
80.33 1 0.66%
B@L2-N
64.04 0.89
81.73 0.66%
65.19 + 0.87% %
82.93
B@VN
66.08 + 0.69%
75.56 + 0.75%
B@V-N-L2-N
66.85 + 0.70%
76.66 + 0.67%
B@LDA
77.44 11:1:::00 0.74%
80.85 1 0.67%
B@LDA+L2N
81.01 0.69%
81.48 1 0.68%
B@EST
53.90 1 0.94%
78.09 1 0.67%
57.12 0.93%
80.59 1: 0.64%
B@EST+L2-N
%
81.40 + 0.64% %
64.71 + 0.91% %
83.24 -.667 %
B+
46.32 = 0.87%
73.85 = 0. 70%
48.27 77 - 0.91%
75.87 0.71%
B+@L2-N
64.96 + 0.90%
81.08 0.69%
65.40 + 0.94%
82.49 0.68%
B+@V-N
65.19 + 0.79%
64.33 0.77%
B+ V-NL-N
66.57 + 0.74%
67.15 + 0.74%
B+@LDA
72.88 + 0.74%
75.55 + 0.77%
B+ B+@LDA+L2-N
79.57 1 0.72%
81.98 + 0.67%
B+@EST
52.01 + 0.90%
0.70%
53.49 0.90%
75.38 1 0.71%
B+@EST+L2-N
57 67.63/7 1 0.88%
80.99 0.72%
60.87 1.01%
81.80 1 0.67%
Table 5: Classification accuracies on CIFARFS of ProtoNet, linear evaluation methods (Chen et al. (019). and ours. The best performing methods and any other runs within 95% confidence margin are in bold.
Method
ResNet12
ResNet18
1-shot
5-shot
1-shot
5-shot
ProtoNet
58,65 1.07%
75.33 0.73%
62.05 0.97%
78.25 1 0.71
B@FT
55.97 0.86%
76.50 0.74%
0.84% 56.46
77.43 + 0.66%
BT
61.08 + 0.95%
76.15 + 0. 7.7%%
61.34 + 0.93%
77.86 + 0.71%
B@CL2
58.61 0.86%
74.73 + 74%
58.09 + 0.84%
76.52 - 0.71%
B+@CI.
61.59 + 0.91 %
76.10 0.73%
63.17 + 0.94%
77.49 + 0.72%
44.12 0.89%
7
47.62
0.84%
75.66 0.71%
B@L2-N
59.43 0.90%
77.45 0.70
58.51 0.86%
77.43 0.69%
B@VN
44.66 + 1.44%
+ 0.72%
B@V-N+L2-N
55.91 + 0.70%
67.70 6..99%0.69% 0.69%
B@LDA
72.61 + 0.70%
75.25 1 7.669%
B@LDA+ L2-N
75.73 0.71%
77.12 0.71%
B@EST
52.62 1 0.91%
72.80 1 0.71%
55.57 0.90%
75.10 1.711%
B@EST+L2-N
60.70 - 0.95 %
77.04 + 0.70 %
61.14 + 0.96%
77.48 + 0.68%
B+
45.73 0.88%
73.97 + 0.76%
44.42 + 0.89%
70.85 + 0.76%
B+@L2-N
61.07 - 0.93 %
76.71 0.74%
0.94% 63.48
77.86 0.71%
B+@ V-N
57.20 0.70%
57.16 1. 0.72%
B+ V-N+L2-N
57.59 1 0.69%
57.24
B+@LDA
68.58 + 0.74%
69.86 1: 0.74%
B+@LDA4L2N
76.09 + 0.75%
69.86 1: 0.74%
B+@ EST
50.18 0.93%
70.32 = 0.79%
49.82 0.94%
71.07 0.70%
B+ EST+L2-N
59.83 = 0.98%
76.32 = 0.72%
63.00 + 0.94%
77.99 0.71%
20

---

Under review as a conference paper at ICLR 2022
Table 6: Classification accuracies on FC100 of ProtoNet, linear evaluation methods (Chen
et al. 2019), and ours. The best performing methods and any other within 95% confidence margin are in bold.
Method
ResNet12
ResNet18
1-shot
s-shot
1-shot
5-shot
ProtoNet
35.56 + 0.77%
51.12 0.71%
1 3 0.70%
51.02 1 0.72%
B@FT
39.72 0.68%
56.04 0.76%
40.06 = 0.68%
57.04 0.71%
B+@FT
36.01 + 0.64%
50.73 - 0.72%
36.93 = 7.70%
50.41 + 0.73%
B@CL2
41.58 + 0.74%
55.82 + 0.76%
41.51 + 0.72%
56.41 + 0.71%
B+@CL2
37.51 0.71%
50.38 = 0.72%
38.30 = 0.74%
51.06 + 0.70%
B
31.60 0.61%
52.50 -1: 0.74%
35.90 1 0.61%
55.20 0.77%
B@L2-N
40.34 0.71%
56.61 0.71 %
40.49 = 0.71%
57.71 0.76%
B@ V-N
41.99 - 0.63%
49.69 0.67%
B@ N-N-2N
42.03 + 0.65%
49.99 + 7.70%
B@LDA
52.90 + 0.72%
55.41 0.72%
B@LDA+L2-N
55.12 0.70%
56.32 + 0.75%
B@EST
44.55 0.88%
52.97 + 0.81%
17.49 + 0.91%
55.68 0.74%
B@EST+L2-N
47.57 + 0.83%
56,96 0.77%
50.13 + 0.91%
59.94 + 0.74%
B+
29.72 0.57%
46.85 + 0.67%
30.76 = 0.62%
47.62 :: 0.69%
B+@L2-N
37.16 0.67%
51.10 0.71%
38.55 + 0.72%
50.15 0.71%
B+@ V-N
36.76 0.59%
40.66 + 0.64%
B+@ V-N+L2-N
36.55 + 0.55%
40.16 0.61%
B+@ LDA
47.10 + 0.70%
47.63 + 0.69%
@+@ @LDLDA+L2-N
49.03 1 0.70%
50.60 1 0.72%
B+@EST
37.60 1 0.78%
46.81 1 0.67%
39.92 1 0.88%
47.65 1 0.69%
B+@EST+L2-N
40.88 0.83%
50.45 0.72%
41 1 0.85%
50.74 0.70%
before
10
after /2
ResNet12 ReaNer18
ReaNet12 Eieredimagenet tieredimagenct CIFARFS
ResMart1
ReaNet12
ResNet18 CIFARFS
RaaNat12 FC100
ResNet12 FC100
minilmagenet minilmagenet
Figure 4: We plotted the ratio of the between-class variance the within-class variance after applying Lz-norm averaged over the test classes of each dataset.
Tr(Ee)
before and Tr(E))
21


Reviews:
[{"review_id": "cMsLjaM22F", "reviewer": "Reviewer_8nVE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The interesting aspects of the paper are as follows:\n1. Theoretical analysis of protonets with relaxed constraints is interesting.\n2. Evaluation on benchmarks and experimental results seems to be consistent.\n\nHowever, some concerns are as follows:\n1. The theoretical analysis just relax the assumption of having particular class distribution (in both class distribution and class covariance matrix), but the overall approach is marginally novel compared to Cao et al.\n2. The theoretical results derived in this paper are quite intuitive though, e.g., L2 normalization of features is effective. These are quite well-known tricks. I was looking for more interesting insights which are non-trivial.\n3.  Variance normalization performs worse - can you connect this with the theoretical analysis performed here?\n4. How this paper addressed the second drawback, i.e., what is the optimal feature transformation, that is not clear to me. Several feature transformation methods are evaluated, however, there is no guarantee that the chosen feature transformation method is optimal. \n5. The flow of the paper might be improved, specially in the experimental results section.", "summary_of_the_paper": "In this paper, the authors have derived a generalization bound for the prototypical networks, which provides some insights towards improving protonets without finetuning using simply normalizing the feature vectors and reducing the variance of the norm. The derivation has relaxed the previous assumption of considering feature distribution and class covariance matrix to be a particular form. To justify the claim, the authors experimented on several feature transformation methods on standard few-shot benchmarks.\n\n", "summary_of_the_review": "Overall the theoretical analysis is interesting, however the novelty seems to be marginal. The insights provided in the paper are quite intuitive  and proper connection with theory and experiments need to be explained better. I would like to see how the authors response the raised issues.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Hb54ABzR1SD", "reviewer": "Reviewer_krD7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n- It is interesting to focus on the different loss functions in feature training (softmax loss) and testing (Prototype classifier). \n\n- The method which works well without meta-learning and fine-tuning is practical. \n\n- L2 normalization is well known to improve performance. The paper explains how the variance of the norm is related to the bound of a prototype classifier. \n\n[Weakness]\n\nThe comparison of normalization is incomplete. \n- The L2-N is applied only to EST. Why L2-N is not applied to V-N and LDA. \n- It is not clear if B@CL2 and B+@CL2 use a prototypical classifier, or fine-tuning is conducted. \n\nThe discussion of Sec.4.4 is not completely validated. It claims that  the L2-norm normalization less influences ${\\rm Tr}(\\Sigma)$ than ${\\rm Tr}(\\Sigma_c)$. These values before and after the normalization also should be compared. \n\nThere are several unclear or incorrect descriptions. \n\n- It is claimed that the proposed bound does not assume any specific distribution and the same covariance matrix among classes. However, it seems that the variance and mean of Gaussian distribution are used when expanding the results of one-sided Chebyshev's inequality, eg., Eq.(22). Also, p.15 Eq.(29), the first line to the second line, it seems to be assuming ${\\rm Tr}(\\Sigma_{c2}) = {\\rm Tr}(\\Sigma_{c1})$.\n\n- P.5  What means “a secondary expression for the ratio of between-class variance and within-class variance.”?\n\n- P.6  The authors wrote that “Regarding equation 9, the ratio of the Euclidean distance between the class mean vectors and the between-class variance is supported to be constant.” Why can it be supported to be constant? The feature transformation changes the distance of the mean vectors of each class. \n\n- Abstract; “the same performance can be obtained “. The performance is not completely the same as fine-tuning nor meta-learning.\n\n- Variance-normalization of Eqs.(12) and (13) are not clear. Is the different variance calculated for each class, or is the variance common for all samples? \n\n- For covariance matrix, $(\\phi(x)-\\mu_c)^T( \\phi(x) - \\mu_c )$ seems to be $(\\phi(x) - \\mu_c) (\\phi(x) - \\mu_c)^T$ and $(\\mu_c - \\mu)^T(\\mu_c - \\mu)$ seems to be $(\\mu_c - \\mu)(\\mu_c - \\mu)^T$. \n\n- p.12 the last row: $\\lambda \\in R^D$ seems to be $\\lambda \\in R^1$.\n\n", "summary_of_the_paper": "This paper analyzes how a prototype classifier works well without fine-tuning and meta-learning. The feature is trained with cross-entropy loss with a linear projection layer. In testing, the classifier is a Prototype classifier.\n\nThis paper derives a novel generalization bound for the prototypical network and shows that focusing on the variance of the norm of a feature vector can improve performance.\nThe proposed upper bound is a modification of (Cao et al., 2020), and it does not require the features distributed on any specific distribution. Each covariance matrix does not have to be the same among classes. \n\nThe authors experimentally investigated several normalization methods (L2-N, V-N, LDA, EST, EST-L2-N) for minimizing the variance of the norm.\n", "summary_of_the_review": "The theoretical analysis of the variance of the norm to impact the bound of the prototypical classifier is interesting. However, the descriptions seem to contain errors, and experimental validation is not satisfactory. \n\n==\nPost rebuttal\n==\n\nMy concerns have been solved. I have slightly increased the score accordingly.  \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1dBcJwpIAR", "reviewer": "Reviewer_M6Y5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topic studied in the paper is relevant to the few-shot learning research community, and the prospect of applying prototypical classifiers to a wider range of pre-trained feature extractors is appealing. The related work section is thorough and does a good job of contextualizing the proposed approach.\n\nMy expertise is limited in terms of evaluating the correctness of the derived bound; I will leave this to other reviewers and take it at face value in my review. The main issue I have with the submission is clarity. In several instances the notation tripped me up, or the writing was imprecise, or some experimental details were unclear:\n- The term \"linear classifier with fine-tunining\" doesn't make sense to me. I know this formulation has been used in the literature before, but calling it \"fine-tuning\" is incorrect when the backbone is frozen and a new classifier is learned from scratch.\n- Are there typos in the joint probability distributions defined in Section 3.1's third line? In both cases, the index \"i\" in the product is not used in the expression.\n- In Equation 2, should c ~ \\tau read as c_1, c_2 ~ \\tau?\n- Where is Tr(\\Sigma_c) defined? It's used in Section 4.4, but the closest definition I could find was the within-class variance \\Sigma_\\tau discussed in item 2 at the bottom of page 5.\n- Since the drawbacks of Cao et al.'s bound are discussed in this work, it would be good to show the bound in the submission. It would avoid having to pull up Cao et al.'s paper and put it side by side with the submission just to compare the bounds.\n- Comparing notations with Cao et al., I find their choice of notation and the way they introduce it much clearer, and I wish this submission adopted their notation (with attribution, of course).\n\nI also have some reservations about the experiments:\n- Can the authors expand on why the center loss and affinity loss are left out of the comparisons? Is it because they need to be applied during pre-training? If so, it should be made clearer.\n- Again, the following is common to a lot of few-shot classification papers, but given the tiered-ImageNet results, what additional evaluation signal is expected from the mini-ImageNet results? Isn't it redundant? The same can be said about FC100 vs CIFAR-FS.\n- How is Baseline++ trained in the re-implementation? Section 4.2 mentions the normalization of the projection layer and features as a difference between it and Baseline, but omits the cosine classifier.\n- The gap between B+@FT and B@FT is surprisingly small, given the improvements of Baseline++ over Baseline reported by Chen et al. Do the authors have an explanation? Is it possible that Baseline++ was not pre-trained properly in their re-implementation? In particular, Baseline++ scales the normalized weight vectors independently for each class, which is not clearly mentioned in Chen et al.'s paper, but is necessary for good performance. Is that detail also present in the re-implementation?\n- The submission goes beyond standard practice and uses a principled criterion for bolding table entries by comparing 95% confidence intervals, which is a good thing that I would like to see in more submissions. I would like to see those 95% confidence intervals in the Appendix, though, so that future papers can also compare to those confidence intervals.\n\nIn terms of significance, I would argue that \"comparable with ProtoNet and the fine-tuning approach\" is not that high a bar. I like the premise of being able to plug a prototypical classifier into any pre-trained model, and I feel that it could have been better exploited in the submission, for instance by using a larger-scale pre-trained classifier out of the box and achieving even better few-shot classification accuracies. I would say however that I am not entirely sold on a prototypical classifier having more desirable properties than a linear classifier or an SVM (like is done with MetaOptNet (Lee et al., 2019)). The paper justifies this through the lens of hyperparameter tuning, but work like Big Transfer (Kolesnikov et al., 2020) shows that heuristics such as BiT-HyperRule can be designed to avoid hyperparameter sweeps.\n\nAdditional questions/comments:\n- The abstract alternates between present and past verb tense. It would flow better if it was consistent in its verb tense use.\n- Since the variance-normalization transformation is described as a variation of the Mahalanobis distance with a diagonal covariance matrix, it would be relevant to cite Bateni et al. (2020)'s Simple CNAPs approach, which uses a Mahalanobis distance-based classification criterion.\n\nReferences:\n- Bateni, P., Goyal, R., Masrani, V., Wood, F., & Sigal, L. (2020). Improved few-shot visual classification. In CVPR.\n- Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., & Houlsby, N. (2020). Big Transfer (BiT): General visual representation learning. In ECCV.\n- Lee, K., Maji, S., Ravichandran, A., & Soatto, S. (2019). Meta-learning with differentiable convex optimization. In CVPR.", "summary_of_the_paper": "The submission introduces a generalization bound for Prototypical Networks that does not depend on assumptions on the class-conditional distributions. This bound decreases as the variance in norm of the feature vectors decreases, and the authors empirically investigate five feature transformation approaches that are meant to lower this variance: L2-normalization (L2-norm), variance-normalization, Linear Discriminant Analysis (LDA), Embedding Space Transformation (EST), and EST+L2-norm.\n\nExperimental results are reported on the mini-ImageNet, tiered-ImageNet, CIFAR-FS, and FC100 benchmarks in the 1-shot and 5-shot settings, using ResNet-12 or ResNet-18 as the backbone architecture. The five feature transformation approaches are applied on top of the Baseline and Baseline++ learners before using the extracted features to build a prototypical classifier. The submission compares against the following:\n- Prototypical Networks\n- Baseline and Baseline++ with a linear classifier on top of the extracted features\n- Baseline and Baseline++ with Wang et al.'s centering+L2-norm approach on top of the extracted features.\n- Baseline and Baseline++ with no additional feature transformation and a prototypical classifier on top of the extracted features.\n\nThe submission concludes that EST+L2-norm performs best, and that its performance-boosting effect decreases as the number of examples per class increases.", "summary_of_the_review": "The paper studies an interesting topic but lacks clarity. I have concerns with the experimental setup, and with how the paper justifies the significance of its proposed approach.\n\n\n---\n\n**POST-REBUTTAL**: Most of my concerns are addressed, but I remain concerned with the submission's significance (see discussion thread).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores prototype vs linear classifiers for few-shot learning. It has been found that pre-training a classifier network, followed by training a linear head can produce competitive results to meta-trained prototypical networks. A natural question therefore is whether one can directly derive prototypical classifiers from pre-trained classifiers. Naively applying this idea doesn’t work well in practice though, and this paper performs a theoretical investigation to determine why. The theory is a generalization of Cao et al., 2020, that doesn’t require assumptions on the class-conditional distributions. The theory suggests that the variance of the norm of the feature vectors plays a role, so the paper explores a few feature transformations to reduce this. It demonstrates on a few benchmark datasets that transforming the feature vectors can indeed allow us to create prototypical classifiers from pre-trained networks. As a minor quibble, the paper twice refers to “direction of the norm of class mean vectors” - This should just be the direction of the class mean vectors, right? Norm is not a direction, it’s a magnitude.

During the discussion phase, a number of questions arose, mainly around the clarity of the presentation and a request for a few additional baselines (e.g., L2 normalization combined with LDA/V-N). These points were resolved by the authors. The main outstanding issue is whether there is enough novelty/significance in the paper to merit acceptance, and on that point, the reviewers felt this is borderline. On the one hand, the theory is more general and does directly point to aspects of the feature space that can yield better generalization results. On the other hand, tricks like L2 normalization are already known, and the utility of prototype classifiers over linear classifiers like SVMs is unclear.

After careful consideration, further discussion with the reviewers, as well as the program committee, it was generally agreed that this paper does not quite meet the bar in terms of the novelty or significance of its contribution. The authors mentioned time and space benefits relative to fine-tuned classifiers in their response, and I think one way to improve the paper would be to demonstrate this advantage in a real-world application or challenging scenario.
Decision: Reject