Year: 2021

Abstract:
The prototypical network is a prototype classifier based on meta-learning and is widely used for few-shot learning because it classifies unseen examples by constructing class-specific prototypes without adjusting hyper-parameters during meta-testing. Interestingly, recent research has attracted a lot of attention, showing that training a new linear classifier, which does not use a meta-learning algorithm, performs comparably with the prototypical network. However, the training of a new linear classifier requires the retraining of the classifier every time a new class appears. In this paper, we analyze how a prototype classifier works equally well without training a new linear classifier and meta-learning. We experimentally find that directly using the feature vector extracted using standard pre-trained models to construct a prototype classifier in meta-testing does not perform as well as the prototypical network and training a new linear classifiers and feature vectors of pre-trained models. Thus, we derive a novel generalization bound for the prototypical network and show that focusing on the variance of the norm of a feature vector can improve performance. We experimentally investigate several normalization methods for minimizing the variance of the norm and find that the same performance can be obtained by using the L2 normalization and embedding space transformation without training a new classifier or meta-learning.

heading: 1 INTRODUCTION
text: Few-shot learning is used to adapt quickly to new classes with low annotation cost. Meta-learning is a standard training procedure to tackle the few-shot learning problem and Prototypical Network (Snell et al., 2017) a.k.a ProtoNet is a widely used meta-learning algorithm for few-shot learning. In ProtoNet, we use a prototype classifier based on meta-learning to predict the classes of unobserved objects by constructing class-specific prototypes without adjusting the hyper-parameters during meta-testing.
ProtoNet has the following advantages. (1) Since the nearest neighbor method is applied on query data and class prototypes during the meta-test phase, no hyper-parameters are required in meta-test phase. (2) Since the number of data for few-shot is small, the inference time is almost negligible. (3) The classifiers can quickly adapt to new environments because they do not have to be re-trained for the support set when new classes appear. The generalization bound of ProtoNet in relation to the number of shots in a support set has been studied (Cao et al., 2020). The bound suggests that the performance of ProtoNet depends on the ratio of the between-class variance to the within-class variance of features of a support set extracted using the meta-trained model.
There have been studies on training a new linear classifier on the features extracted using a pretrained model without meta-learning, which can perform comparably with the meta-learned models (Chen et al., 2019; Tian et al., 2020). We call this approach as linear-evaluation-based approach. In these studies, the models are trained with the standard classification problem, i.e., models are trained with cross-entropy loss after linear projection from the embedding space to the class-probability space. The linear-evaluation-based approach has the following advantages over meta-learning. (1) Training converges faster than meta-learning. (2) Implementation is simpler. (3) Meta-learning decreases in performance if the number of shots does not match between meta-training and metatesting (Cao et al., 2020); however, the linear-evaluation-based approach do not need to take this
into account. However, the linear-evaluation-based approach requires retraining a linear classifier every time a new class appears.
In contrast, a prototype classifier can be applied to any trained feature extractor and does not require model learning in the testing phase. Therefore, a prototype classifier can be a a practical and useful first step for few-shot learning problems. In order to avoid meta-learning during the training phase and the linear evaluation during the testing phase, we focus on using a prototype classifier on the testing phase and training models in a standard classification manner. As we discuss in section 4, we found that when we directly constructed prototypes from the feature vectors extracted using pretrained models and applied the nearest neighbor method as in the testing phase in ProtoNet , this does not perform as well as the linear-evaluation-based approach.
We hypothesize that the reason is the difference between the loss function in ProtoNet and pretrained models. As described in section 3, if we consider a prototype as a pseudo sample average of the features in each class, the loss function of ProtoNet can be considered having a regularizing effect that makes it closer to the sample average of the class. Since standard classification training computes cross-entropy loss with additional linear projection to make the features linearly separable, the loss function does not have such an effect and can cause large within-class variance. Figure 1 shows a scatter plot of the features extracted using a neural network with two dimension output trained on cifar-10 with ProtoNet(1a) and cross-entropy loss with a linear projection layer(1b). This figure implies that the features extracted using a model trained in a standard classification manner distributes away from the origin and causes large within-class variance along the direction of the norm of class mean vectors, while that of ProtoNet is more centered to its class means. This phenomenon is also observed in face recognition literatures (Wen et al., 2016; Liu et al., 2017; Wang et al., 2018; Deng et al., 2019).
We now focus on the theoretical analysis on a prototype classifier. A current study (Cao et al., 2020) analyzed an upper bound of risk by using a prototype classifier. The bound depends on the number of shots of a support set, between-class variance, and within-class variance. However, the bound requires the class-conditioned distribution of features to be Gaussian and to have the same covariance matrix among classes. In addition, since the bound does not depend on the norm of the feature vectors, it is not clear from the bound what feature-transformation method can lead to performance improvement. Thus, we derive a novel bound for a prototype classifier.
Our contributions are threefold.
1. We relax the assumption; specifically, the bound does not require that the features distribute on any specific distribution, and each covariance matrix does not have to be the same among classes.
2. We clarify the effect of the variance of the norm of the feature vectors on the performance of a prototype classifier.
3. We investigate the effectiveness of reducing the variance of the norm empirically.

heading: 2 RELATED WORK
text: We summarize related work by describing a prototype classifier with meta-learning, linearevaluation-based approach without meta-learning, and theoretical analysis related to the few-shot learning problem.
A prototype classifier with meta-learning On the basis of the hypothesis that features well distinguished in the training phase are also useful for classifying new classes, constructing one or multiple prototypes for classifying unseen examples is a widely used approach (Vinyals et al., 2016; Snell et al., 2017; Pahde et al., 2021; Ji et al., 2021; Sung et al., 2018; Allen et al., 2019; Doersch et al., 2020; Qi et al., 2018). Certain algorithms compute similarities between multiple prototypes and unseen examples by using their own modules, such as attention mechanism (Vinyals et al., 2016; Doersch et al., 2020), relation network (Sung et al., 2018), reweighting mechanisms by taking into account between-class and within-class interaction (Ji et al., 2021), and latent clusters (Allen et al., 2019). Prototypes are also constructed in a multi-modal way (Pahde et al., 2021). Another line of research is transforming the space of extracted features to a better distinguishable space (Simon et al., 2020; Yoon et al., 2019; Das et al., 2020; Das & Lee, 2020) or taking variance of fetures into account (Bateni et al., 2020) . Because of its convenience, the approach with a prototype classifier is adapted to other domain such as semantic segmentation (Dong & Xing, 2018), text classification (Sun et al., 2019), and speech recognition (Wang et al., 2019a).
Update-based meta-learning In contrast to the approach with a prototype classifier and metalearning , in update-based meta-learning approaches, model parameters are adjusted in the test phase so that a model can adapt to new classes. Model-agnostic meta-Learning (MAML) and its variants (Finn et al., 2017; 2018; Rajeswaran et al., 2017) search for a good initialization parameters that adapt to new classes with a few labeled data and a few update steps of the parameters. Another approach involves learning an effective update rule for the parameters of a base-learner model through a sequence of training episodes (Bertinetto et al., 2019; Lee et al., 2019). Both approaches require additional learning of hyper-parameters and training time; thus, they prevent quick adaptation to new classes.
linear-evaluation-based approach without meta-learning Interestingly, recent studies have shown that training a new linear classifier with features extracted using a model trained with crossentropy loss on base-dataset performs comparably with meta-learning based methods (Chen et al., 2019) (Wang et al., 2019b). More effective method for training a new classifier in few-shot settings has been proposed (Yang et al., 2021; Phoo & Hariharan, 2021), such as calibrating distribution generated by a support set (Yang et al., 2021) , self-supervised learning on query data (Phoo & Hariharan, 2021), and distilling knowledge to obtain better embeddings (Tian et al., 2020). Liu et al. (2020) focuses on pretraining phase and found the negative margin in cross-entropy helps improving the performance in few-shot settings. However, similar to update-based meta-learning, these studies require additional hyper-parameters and training to apply to new classes; thus if we tackle these points, we would have an alternative method that is easier and more convenient to use.
Theoretical analysis of few-shot learning Even though much improvement has been empirically made in few-shot learning, theoretical analysis is scarce. In the context of meta-learning, Du et al. (2021) provided a risk bound on the meta-testing phase that is related to the number of meta-training data and meta-testing data. Lucas et al. (2021) derived information-theoretic lower-bounds on minimax rates of convergence for algorithms that are trained on data from multiple sources and tested on novel data. Cao et al. (2020) derived a new bound on a prototype classifier and theoretically demonstrated that the mismatch regarding the number of shots in a support set between meta-training and meta-testing degrades the performance of prototypical networks, which has been only experimentally observed. However, their bound depends on several assumptions: the class-conditional distributions of features are Gaussian and have the same covariance matrix among classes. In contrast, we derive a novel bound that does not depend on any specific distribution.

heading: 3 THEORETICAL ANALYSIS OF PROTOTYPE CLASSIFIER IN TERMS OF VARIANCE OF NORM OF FEATURE VECTORS
text: In this section, we first formulate our problem settings and point out the drawbacks of the current theoretical analysis on a prototype classifier. Next we provide our novel bound of a prototype classifier with the bound related to the variance of the norm of the feature vectors. Finally, we list several methods that can improve the performance of a prototype classifier based on our bound.

heading: 3.1 PROBLEM SETTING
text: Let Y be a space of a class, τ a probability distribution over Y , X a space of input data, D a probability distribution over X , Dy a probability distribution over X given a class y. We define D⊗nk and D⊗ny by D⊗ny = Πni=1Dy and D⊗nk ≡ Πni=1D ⊗k i , respectively. We sample N classes from τ to form the N -way classification problem. Denote by K a number of annotated data in each class and x ∈ X , y ∈ Y as input data and its class respectively. We define a set of support data of class c sampled from τ as Sc = {xi | (xi, yi) ∈ X × Y ∩ yi = c}Ki=1 and a set of support data in the N -way K-shot classification problem as S = ⋃N c=1 Sc. Suppose a feature extractor computes a function ϕ : X → RD, where D is the number of the embedding dimensions. ϕ(Sc) is defined by ϕ(Sc) = 1K ∑ x∈Sc ϕ(x). Let Φ be a space of the extractor function ϕ. Denote by M : Φ × X × (X × Y)NK → RN a prototype classifier function that computes probabilities of input x belonging to class c as follows.
M(ϕ,x, S)c = pM(y = c|x, S, ϕ) = exp
( −∥ϕ(x)− ϕ(Sc)∥2 ) ∑N
l=1 exp ( −∥ϕ(x)− ϕ(Sl)∥2 ) , (1) where ∥v∥2 = ∑D d=1 ( v(d) )2 , and v(d) is the d-th dimension of vector v. The prediction of an input x, denoted by ŷ ∈ Y , is computed by taking argmax for M(ϕ,x, S), i.e., ŷ = argmaxM(ϕ,x, S). We denote by Ez∼q(z)[g(z)] an operation to take the expectation of g(z) over z distributed as f(z) and we simply denote Ez∼q(z)[g(z)] as Ez[g(z)] when z is obviously distributed on q(z). We define Varz∼q(z)[g(z)] as an operation to take variance of g(z) over z distributed as q(z). With I denoting the indicator function, we define expected risk RM of a prototype classifier as
RM(ϕ) = ES∼D⊗nkEc∼τEx∼Dc [I[argmaxM(ϕ,x, S) ̸= c]. (2)
For simplicity, we now discuss the binary classification setting. We show a case of multi-class classification in Appendix A.5 due to lack of space.
Let c1 and c2 denote any pair of classes sampled from τ . We consider that a query data x belongs to class c1 and support sets S consist of class c1’s support set and c2’s support set. Then, equation 2 is written as follows.
RM(ϕ) = ES∼D⊗2kEc1∼τEx∼Dc1 [I[argmaxM(ϕ,x, Sc1 ∪ Sc2) ̸= c1]]. (3)

heading: 3.2 WHAT FEATURE-TRANSFORMATION METHOD EXPECTED TO BE EFFECTIVE?
text: The current theoretical analysis for a prototype classifier (Cao et al., 2020) has the following two drawbacks (see Appendix A.1 for the details). The first is that the modeling assumption requires a class-conditioned distribution of the features to follow a Gaussian distribution with the same covariance matrix among classes. For example, when we use the ReLU activation function in last layer, it is not normally distributed and the class-conditioned distribution does not have the same covariance matrix as shown in Figure 1. The second drawback is that it is not clear what feature-transformation method can reduce the upper bound. A feature-transformation method to maximize the betweenclass variance and minimize the within-class variance such as linear discriminant analysis (LDA) (Fukunaga, 1990) and Embedding Space Transformation (EST) (Cao et al., 2020) can be expected to improve performance; however, it is not clear how the second term of the denominator changes.
From Figure 1 , the distribution of each class feature stretches in the direction of the norm of its class-mean feature vector. This property is also observed in metric learning literatures (Wen et al.,
2016; Liu et al., 2017). A model trained in the cross-entropy loss after linear projection from the embedding space to the class-probability space computes a probability of input x belonging to class c given by
p(y = c|x,W, ϕ) = exp
( ϕ(x)⊤Wc )∑N j=1 exp (ϕ(x) ⊤Wj) , (4)
where W is a weight matrix that transforms features from the embedding space to the classprobability space.
Comparing equation 4 with equation 1, we found that equation 4 does not have a regularization term similar to the one appearing in equation 1 that forces the features to be close to its class-mean feature vector. This implies the features extracted using a model trained with the cross-entropy loss on equation 4 are less close to its class-mean feature vector than the ProtoNet loss on equation 1. Through this observation and the property mentioned above, we hypothesis that normalizing the norm of the feature vectors can push the features to each class-mean feature vector and can boost the performance of a prototype classifier trained with cross-entropy loss on equation 4.

heading: 3.3 RELATING VARIANCE OF NORM TO UPPER BOUND OF EXPECTED RISK
text: To understand what effect the variance of the norm of the feature vectors has on the performance of a prototype classifier, we analyze how variance contributes to the expected risk when an embedding function ϕ is fixed. The following theorem provides a generalization bound for the expected risk of a prototype classifier in terms of the variance of the norm of the feature vectors computed using a feature extractor. Theorem 1. Let M be an operation of a prototype classifier on binary classification defined by equation 1. For µc = Ex∼Dc [ϕ(x)], Σc = Ex∼Dc [(ϕ(x)− µc)(ϕ(x)− µc)⊤], µ = Ec∼τ [µc],Σ = Ec∼τ [(µc − µ)(µc − µ)⊤], and Ec∼τ [Σc] = Στ , if ϕ(x) has the variance of its norm, then the miss-classification risk of the prototype classifier on binary classification RM satisfies
RM(ϕ) ≤1− 4(Tr(Σ))2
EV[hL2(ϕ(x))] + VTr(Σc1) + Vwit(Στ ,Σ,µ) + Edist2L2(µc1 ,µc2) , (5)
where
EV[hL2(ϕ(x))] = 4
K Ec∼τ
[ Varx∼Dc [ ∥ϕ(x)∥2 ]] , (6)
VTr(Σci) =
( 4
K +
2
K2
) Varc∼τ [Tr (Σci)] , (7)
Vwit(Στ ,Σ,µ) = 8
K Tr(Στ )
( Tr(Στ ) + Tr(Σ) + µ ⊤µ ) + 4(Tr(Σ) + µ⊤µ)2, (8)
Edist2L2(µc1 ,µc2) =Ec1,c2 [( (µc1 − µc2) ⊤ (µc1 − µc2) )2] . (9)
Remark. The term EV[hL2(ϕ(Sy))] is the variance of the norm of the feature vectors. The term VTr(Σc1) is the variance of the summation with diagonal element of the covariance matrix from each class embedding; it can be interpreted as the difference of the distribution shape constructed from each class embedding. The term Vwit(Στ ,Σ,µ) is correlated with the within-class variance because Vwit(Στ ,Σ,µ)Tr(Σ) is a secondary expression for the ratio of between-class variance and withinclass variance. The term Edist2L2(µc1 ,µc2) is the expectation of the Euclidean distance between the class mean vectors .
This bound has the following properties.
1. Its derivation does not require the features to be distributed on any specific distributions and the class-conditioned covariance matrix does not have to be the same among classes
2. The bound can decrease when any of the following statistics decreases with fixed betweenclass variance (Σ): (i) the variance of the norm of the feature vectors, as what we discussed in Section 3.2, (ii) the difference in the distribution shape constructed from each class embedding, (iii) the within-class variance (Στ ), (iv) the Euclidean distance between the class-mean vectors.
As a result, our bound loosens the modeling assumption of Theorem2 in appendix A.1 and has its property in 2-(iii). We show the proof in Appendix A.2.

heading: 3.4 FEATURE-TRANSFORMATION METHODS
text: We hypothesize from Theorem 1 that in addition to a feature-transformation method related to equation 6, eliminating the affect from the difference in the distribution shape among classes and lowering the ratio of between-class variance to within-class variance can improve the performance of a prototype classifier. Regarding equation 9, the ratio of the Euclidean distance between the class mean vectors and the between-class variance is supposed to be constant. Thus, we focus on the transformation methods relating to equation 6, equation 7 and equation 8. We analyze the following feature-transformation methods: L2-normalization (L2-norm), variance-normalization, LDA, EST, and EST+L2-norm. We show the detail of each method in Section A.3 due to lack of space.

heading: 4 EXPERIMENTAL EVALUATION
text: In this section, we experimentally analyzed the effectiveness of the feature-transformation methods mentioned in Section3.4. The center loss (Wen et al., 2016) and affinity loss (Hayat et al., 2019) have been proposed to efficiently pull the features of the same class to their centers in the training phase; however, we focus on a widely used pre-trained model in this experiments following the line of studies Chen et al. (2019) and Tian et al. (2020).

heading: 4.1 DATASETS AND EVALUATION PROTOCOL
text: miniImageNet. The miniImageNet dataset (Vinyals et al., 2016) is a standard bench-mark for fewshot learning algorithms for recent studies. It contains 100 classes randomly sampled from ImageNet. Each class contains 600 images. We follow the widely-used splitting protocol (Ravi & Larochelle, 2017) to split the dataset into 64/16/20 for training/validation/testing respectively.
tieredImageNet The tieredImageNet dataset (Ren et al., 2018) is another subset of ImageNet but has 608 classes. These classes are grouped into 34 higher level categories in accordance with the ImageNet hierarchy and these categories are split into 20 training (351 classes), 6 validation (97 classes), 8 testing categories (160 classes). This splitting protocol ensures that the training set is distinctive enough from the testing set and makes the problem more realistic since we generally cannot assume that test classes will be similar to those seen in training.
CIFAR-FS The CIFAR-FS dataset (Bertinetto et al., 2019) is a recently proposed fewshot image classification benchmark, consisting of all 100 classes from CIFAR-100 (Krizhevsky et al.). The classes are randomly split into 64, 16, and 20 for meta-training, meta-validation, and meta-testing respectively. Each class contains 600 images of size 32× 32. FC100 The FC100 dataset (Lee et al., 2019) is another dataset derived from CIFAR-100 (Krizhevsky et al.), containing 100 classes which are grouped into 20 categories. These categories are split into 12 categories for training (60 classes), from 4 categories for validation (20 classes), 4 categories for testing (20 classes).

heading: 4.2 IMPLEMENTATION DETAILS
text: We compared the feature-transformation methods applied on against ProtoNet (Snell et al., 2017), Baseline and Baseline++ (Chen et al., 2019). In Baseline, the linear projection layer is trained on a support set and in Baseline++, the norm of the linear projection layer and features are normalized to be constant. We call Baseline and Baseline++ as “linear evlauation” methods. We also compared with the feature-transformation method proposed in a previous study (Wang et al., 2019b). In that study they transformed the features so that the mean of all features to be origin before L2normalization and then use a prototype classifier without training a new linear classifier. We call this operation as centering+L2-norm. We re-implemented these methods following the training procedure in previous study (Chen et al., 2019). In the pre-training stage, where the cross-entropy loss was used and meta-learning was not used, we trained 400 epochs with a batch size of 16. In the meta-training stage for ProtoNet, we trained 60, 000 episodes for 1-shot and 40, 000 episodes
for 5-shot tasks. We used the validation set to select the training episodes with the best accuracy. In each episode, we sampled N classes to form N -way classification (In meta-training N=20 and meta-testing N=5 following the original study of ProtoNet (Snell et al., 2017)). For each class, we selected K labeled instances as our support set and 16 instances for the query set for a K-shot task.
In the linear evaluation or meta-testing stage for all methods, we averaged the results over 600 trials. In each trial, we randomly sampled 5 classes from novel classes, and in each class, we also selected K instances for the support set and 16 for the query set. For Baseline and Baseline++, we used the entire support set to train a new classifier for 100 iterations with a batch size of 4. With ProtoNet, we used the models trained in the same shots as meta-testing stage since the mismatch of number of shots between meta-training and meta-testing degrades performance (Cao et al., 2020). All methods were trained from scratch, and the Adam optimizer with initial learning rate 10−3 was used. We applied standard data augmentation including random crop, horizontal-flip, and color jitter in both the training stages. We used a ResNet12 network and ResNet18 network following the previous study (Chen et al., 2019; Yoon et al., 2019).
Since LDA constructs a within-class covariance matrix from a support set and variancenormalization computes sample variance, we did not investigate the LDA score and variancenormalization score of 1-shot settings. For LDA we set λ = 0.0001 for equation 18 in Appendix A.3 and for EST we set the dimensions of the projected space to 60 following the settings of the original study (Cao et al., 2020).

heading: 4.3 RESULTS
text: We present the experimental results on Table 1 and Table 2 on the basis of backbones with ResNet12 and ResNet18 for a comprehensive comparison. Table 1 shows the results on miniImageNet and tieredImageNet, and Table 2 shows the results on CIFARFS and FC100. We show the detailed performance results with 95% confidence margin and other transformation methods in A.6.
Method notations in experimental results We denote B@FT and B+@FT as the linear evaluation methods: Baseline and Baseline++. We also denote B@CL2, B+@CL2 as centering+L2-norm (Wang et al., 2019b) on features trained with Baseline and Baseilne++. B, B@L2-N, B@V-N, B@LDA, B@EST, and B@EST+L2-N stands for a prototype classifier with applying no transformation, L2-norm, variance-normalization, LDA, EST, and EST+L2-N on the features extracted using a model pre-trained in the same manner as B@FT. B+, B+@L2-N, B+@V-N, B+@LDA, B+@EST, and B+@EST+L2-N stands for a prototype classifier with applying no transformation,
L2-norm, and variance-normalization on the features extracted using the same pre-trained model as with B+@FT.
Comparison of the feature transformation methods with ProtoNet, linear evaluation methods, and centering+L2-norm From Table 1 and Table 2, we can observe that the prototype classifier with L2-norm and EST+L2-norm performs comparably with ProtoNet and the linearevaluation-based approach in all settings and EST+L2-norm performs the best among the data transformation methods. Comparing the feature-transformation methods described in Section 3.4 with centering+L2-norm (Wang et al., 2019b), centering+L2-norm can slightly improve the performance of the prototype classifier in several 1-shot settings . However, in 5-shot settings, the boost decreases and even performs worse than L2-norm, e.g. miniImageNet and tieredImageNet with ResNet12.
Comparison of the feature transformation methods with B and B+ In the 1-shot setting, although EST falls short of ProtoNet and linear-evaluation-based approach, it also improves the performance of a prototype classifier. The performance gain of both L2-norm, EST and EST+L2-norm decrease when the number of shots increases. This phenomenon can be explained through Theorem 1. The term relating to the variance of the norm and the ratio of the between-class variance to the within-class variance depends on K.Since the term diminishes with increasing K, the performance gain of the feature-transformation methods decreases.
Discussion on the feature transformation methods Surprisingly, variance normalization performs worse than applying no feature transformation. This can be explained by the following two reasons. The first is that the term in the bound related to the difference among the class distribution shapes is smaller than other terms such as the variance of the norm and the ratio of the between-class variance to the within-class variance. Figure 2 shows the term associated with the bound computed from each dataset and backbone. The larger the value is, the more it contributes to larger risk. We found the term associated with equation 7 is small compared with other values and thus variancenormalization did not worked well in our experiment. The second reason is that estimating variance with a support set is unstable and make prediction unreliable in few-shot learning. LDA and EST in 5-shot setting does not improve performance so much compared with L2-norm variants while in the 1-shot settings, EST helps improving performance. This is because in the 5-shot settings, the values computed from equation 8 gets smaller with larger K and the effect of equation 8 on the risk of a prototype classifier in the 1-shot settings is larger. Especially, EST outperforms L2-norm in FC100 with the 1-shot setting. We found from Figure 2 the features of FC100 shows the largest ratio of the
Tr(Σ)2
to be 1 for simplicity. Right: We showed the ratio of the within-class variance to the between-class
variance computed from the test-split’s features extracted using each backbone trained on each corresponding dataset. We scaled the values so that the maximium value to be 1 for simplicity.
within-class to between-class variance among all dataset. Thus the method reducing the ratio works better with FC100 features than any other dataset’s features.

heading: 4.4 EFFECT ON COVARIANCE MATRIX
text: We further analyzed how Tr(Σ) and the trace of class c’s covariance matrix Tr(Σc) fluctuate with L2-norm. Since L2-norm lowers the variance in the direction of the normalized vector, if the covariance matrix stretches in the direction of the class mean vector, L2-norm can reduce its trace. Thus we analyzed the cosine similarities between class-mean feature vectors and the eigenvectors with the largest eigen value of the covariance matrices. Figure 3 illustrates the distribution of cosine similarities of each class-mean feature vector and the eigenvectors in test class of miniImageNet. This figure shows Σc tends to stretches to the direction of class mean vector and Σ stretches in the orthogonal direction to each class-mean feature vector. From this observation, we found that L2norm can reduce Tr(Σc) and has less effect on Tr(Σ). L2-norm can reduce the variance of the norm and the ratio of the within-class variance to the between-class variance.

heading: 5 CONCLUSION
text: We theoretically and experimentally analyzed how the variance of the norm of feature vectors affects the performance of a prototype classifier and found that using EST+L2-norm makes the classifier comparable with ProtoNet and the linear-evaluation-based approach. We also found that when the number of shots in a support set increases, the performance gain from a feature-transformation method decreases, which is consistent with the results of the theoretical analysis. A prototypical classifier is expected to be a practically useful first step in tackling few-shot learning problems because of its simplicity.

heading: A APPENDIX
text: A.1 EXISTING UPPER BOUND ON EXPECTED RISK FOR PROTOTYPE CLASSIFIER
To analyze the behavior of a prototype classifier, we start from the current study (Cao et al., 2020). The following theorem is the upper bound of the expected risk of prototypical networks with the next conditions.
• The probability distribution of an extracted feature ϕ(x) given its class y = c is Gaussian i.e Dy = N (µc,Σc), where µc = Ex∼Dc [ϕ(x)] and Σc = Ex∼Dc [(ϕ(x)−µc)(ϕ(x)−µc)⊤]. • All class-conditioned distributions have the same covariance matrix, i.e., ∀(c, c′),Σc = Σc′ .
Theorem 2. Let M be an operation of a prototype classifier on binary classification defined by equation 1. Then for µ = Ec∼τ [µc] and Σ = Ec∼τ [(µc − µ)(µc − µ)⊤], the miss-classification risk of the prototype classifier on binary classification RM satisfies
RM(ϕ) ≤ 1− 4Tr(Σ)2
8(1 + 1/k)2 Tr (Σ2c) + 16(1 + 1/k) Tr (ΣΣc) + Edist2L2(µc1 ,µc2) , (10)
where Edist2L2(µc1 ,µc2) = Ec1,c2 [ ((µc1 − µc2)⊤ (µc1 − µc2)) 2 ] .
We show the detail of the derivation in Appendix A.2.
A.2 DERIVATION DETAILS OF THEOREM 2
we will explain the rough sketch of derivation of Theorem 2. In prototype classifier, from equation 1 and equation 3, RM is written with sigmoid function σ as follows:
RM(ϕ) =Prc1,c2∼τ,x∼Dc1 ,S∼D⊗2K ( σ(∥ϕ(x)− ϕ(Sc2)∥ − ∥ϕ(x)− ϕ(Sc1)∥) ≤ 1
2 ) =Pr(α < 0), (11)
where α ≜ ∥ϕ(x) − ϕ(Sc2)∥ − ∥ϕ(x) − ϕ(Sc1)∥. we bound equation 11 with expectation and variance of α by following proposition. Proposition 1. From the one-sided Chebyshev’s inequality, it immediately follows that:
RM(ϕ) = Pr(α < 0) ≤ 1− ES∼D⊗2kEc1,c2∼τEx∼Dc1 [α] 2
VarS,c1,c2,x[α] + ES∼D⊗2kEc1,c2∼τEx∼Dc1 [α] 2 . (12)
equation 11 can be further write down as follows by Law of Total Expectation VarS,c,x(α) = ES∼D⊗2kEc1,c2∼τEx∼Dc [α2]− ( ES∼D⊗2kEc1,c2∼τEx∼Dc1 [α] )2 = Ec1,c2Ex,S [α2|c1, c2]− ( ES∼D⊗2kEc1,c2∼τEx∼Dc1 [α]
)2 = Ec1,c2 [Varx,S(α|c1, c2) + Ex,S [α|c1, c2]2]− Ec1,c2∼τEx∼Dc1ES∼D⊗2K [α] 2.
Therefore,
Pr(α < 0) ≤ 1− ES∼D⊗2kEc1,c2∼τEx∼Dc [α] 2
Ec1,c2 [Varx∼Dc1 ,S [α] + Ex∼Dc1 ,S [α] 2] . (13)
We write down the expection and variance of α with following Lemmas 3 and 4. Lemma 3. Under the same notation and assumptions as Theorem 2, then,
ES∼D⊗2kEx∼Dc1 [α] = (µc1 − µc2)⊤(µc1 − µc2) Ec1,c2∼τEx∼Dc1ES∼D⊗2K [α] = 2Tr(Σ).
Lemma 4. Under the same notation and assumptions as Theorem 2, then, Ec1,c2 [Varx,S [α|c1, c2]] ≤ 8 ( 1 + 1
K
) Tr ( Στ ((1 + 1
K )Στ + 2Σ)
) . (14)
The proofs of the above lemmas are in the current study (Cao et al., 2020).
With Proposition 1 and Lemma 3, Lemma 4, we obtain Theorem 2.
A.3 DETAIL OF FEATURE-TRASNFORMING METHODS
L2 normalization (L2-norm) From equation 6, normalizing the norm of the feature vectors can improve the performance of a prototype classifier. We denote a function normalizing the norm as ψL2 given by
ψL2(ϕ(x)) = ϕ(x)
∥ϕ(x)∥ . (15)
variance-normalization From equation 7, normalizing the Euclidean distance in equation 1 by the variance of each class embedding can improve the performance of a prototype classifier. This can be interpreted as a variation of the Mahalanobis distance with a diagonal covariance matrix. We denote a function that measures the distance of two vectors normalized by its variance at each dimension as follows:
ψmaha(ϕ(x)1), ϕ(x)2)) = D∑ d=1
( ϕ(x)1) (d) − ϕ(x)2)(d) )2
Var[ϕ(x)1)(d)] , (16)
where Var[z] is sample variance of z. equation 1 is rewritten with ψmaha(ϕ(x1), ϕ(x2)) as the following equation.
M(ϕ,x, S)j = exp
( −ψmaha ( ϕ(x), ϕ(Sj) )) ∑N
l=1 exp ( −ψmaha ( ϕ(x), ϕ(Sl) )) . (17) This normalization is similar to Bateni et al. (2020)’s work but our method differs in that we do not need to do meta-learning and we do not need to add other modules to the backbone network.
LDA From equation 8, increasing the ratio of the between-class variance to the within-class variance can improve the performance of a prototype classifier. LDA (Fukunaga, 1990) is widely used to search for a projection space that maximizes the between-class variance and minimizes the withinclass variance. It computes the eigenvectors of a matrix Σ̂τ −1 Σ̂, where Σ̂ is the covariance matrix of prototypes and Σ̂τ is the class-conditioned covariance matrix. Since the number of data is small in few-shot settings, Σ̂τ −1 cannot be estimated stably and we add a regularizer term to Σ̂τ and define it as Σ̂τ reg.
Σ̂τ reg = Σ̂τ + λI, (18)
where λ ∈ R1, I ∈ RD×D is identity matrix.
EST Since computation of LDA is unstable, we also analyze the effect of EST (Cao et al., 2020). EST computes eigenvectors of a matrix Σ̂ − ρΣ̂τ : the difference between the covariance matrix of the class mean vectors and the class mean covariance matrix with weight parameter ρ. Similar to LDA, EST also searches for the projection space that maximizes the Σ and minimizes the Στ . We compute the eigenvectors of the matrix with two small modification. The first is that we remove hyper-parameter rho and then simply computes the eigenvectors of Σ̂− Σ̂τ . The second is that we constructs the matrix from a support set in the multi-shot setting while we follows the original study in the 1-shot setting.
EST+L2-norm We hypothesize that the combination of the transforming methods can improve the performance of a prototype classifier independently from each other. Specifically, we focus on reducing equation 6 and equation 8 by combining EST and L2-norm. We first apply EST to reduce equation 8 and after that we apply L2-norm to reduce equation 6. At the end of the operation we want the variance of the norm to be 0, thus we apply EST and after that we apply L2-norm.
variance-normalization+L2-norm We focus on reducing equation 6 and equation 7 by combining variance-normalization and L2-norm. Following the similar procedure of EST+L2-norm, we first apply variance-normalization and after that we apply L2-norm. We show the experimental results of this transformation methods in A.6.
LDA+L2-norm We focus on reducing equation 8 and equation 7 by combining LDA and L2norm. Following the similar procedure of EST+L2-norm, we first apply LDA and after that we apply L2-norm. We show the experimental results of this transformation methods in A.6.
A.4 DERIVATION DETAILS OF THEOREM 1
We start the proof from equation 13. We first prove the following Lemma 5 related to the expectation statistics of α in equation 13.
Lemma 5. Under the same notations and assumptions as Theorem 1, then,
ES∼D⊗2kEx∼Dc1 [α] = 1
K (Tr(Σc2)− Tr(Σc1)) + (µc1 − µc2)⊤(µc1 − µc2)
Ec1,c2∼τEx∼Dc1ES∼D⊗2K [α] = 2Tr(Σ).
Proof. First, from the definition of α, we split ES∼D⊗2kEx∼Dc1 [α] in to two parts and examine them seperately.
ES∼D⊗2kEx∼Dc1 [α] = E [∥∥∥ϕ(x)− ϕ(Sc2)∥∥∥2]︸ ︷︷ ︸
(i)
−E [∥∥∥ϕ(x)− ϕ(Sc1)∥∥∥2]︸ ︷︷ ︸
(ii)
. (19)
In regular conditions, for random vector X , the expectation of the norm is
E[X⊤X] = Tr(Var(X)) + E[X]⊤E[X], (20)
and the variance of the vector is
Var(X) = E[XX⊤]− E[X]E[X]⊤ (21)
Σci ∆ = Varx∼Dci (ϕ(x)). (22)
Hence, (i) = Ex∼Dc1ES [∥∥∥ϕ(x)− ϕ(Sc2)∥∥∥2]
= Tr ( Varx∼Dc1 ,S [ ϕ(x)− ϕ(Sc2) ]) + ExES [ ϕ(x)− ϕ(Sc2) ]⊤ ExES [ ϕ(x)− ϕ(Sc2) ] ,
(23)
where the first term inside the trace can be expanded as: Var [ ϕ(x)− ϕ(Sc2) ] = E [( ϕ(x)− ϕ(Sc2) )( ϕ(x)− ϕ(Sc2) )⊤] − (µc1 − µc2)(µc1 − µc2) ⊤
= Var [ϕ(x)] + E [ϕ(x)]E [ϕ(x)]⊤ +Var [ ϕ(Sc2) ] + E [ ϕ(Sc2) ] E [ ϕ(Sc2) ]⊤ − µc2µ ⊤ c1 − µc1µ ⊤ c2 − (µc1 − µc2)(µc1 − µc2) ⊤ = Σc1 + 1
K Σc2 (Last terms cancel out). (24)
The second term in equation 23 is simply as follows. Ex∼Dc1ES [ ϕ(x)− ϕ(Sc2) ] = µc1 − µc2 . (25)
From equation 24 and equation 25 we obtain
(i) = Tr(Σc1) + 1
K Tr(Σc2) + (µc1 − µc2) ⊤(µc1 − µc2). (26)
Similarly for ii, (ii) = Tr ( Varx∼Dc1 ,S [ ϕ(x)− ϕ(Sc1) ]) + ExES [ ϕ(x)− ϕ(Sc1) ]⊤ ExES [ϕ(x)− ϕ(Sc1)]
= Tr(Σc1) + 1
K Tr(Σc1). (27)
From (i) and (ii), and equation 19
ES∼D⊗2kEx∼Dc1 [α] = 1
K (Tr (Σc2)− Tr (Σc1)) +
( µc1 − µc2 )⊤ ( µc1 − µc2 ) . (28)
Since Ec1,c2,x,S [α] = Ec1,c2∼τ [ES∼D⊗2kEx∼Dc1 [α]], Ec1,c2,x,S [α] = Ec1,c2∼τ [ 1
K (Tr(Σc2)− Tr(Σc1)) +
( µc1 − µc2 )⊤ ( µc1 − µc2 )] = Ec1,c2∼τ [( µc1 − µc2 )⊤ ( µc1 − µc2
)] = Ec1,c2∼τ [ µ⊤c1µc1 + µ ⊤ c2µc2 − µ ⊤ c1µc2 − µ ⊤ c2µc1
] = Tr (Σ) + µ⊤µ+Tr (Σ) + µ⊤µ− 2µ⊤µ = 2Tr (Σ) . (29)
Next we prove the following Lemma 6 related to the conditioned variance of α. Lemma 6. Under the same notation and assumptions as Theorem 1,
Ec1,c2Varx∼Dc1 ,S∼D⊗2N [α] ≤ 4 K Ec∼τVar
[ ∥ϕ(x)∥2 ] + 4
K Varc∼τ [Tr(Σc)] + Vwit(Στ ,Σ,µ),
(30)
where
Vwit (Στ ,Σ,µ) = 8
K (Tr(Στ ))
( Tr(Σ) + µ⊤µ ) + 4 ( Tr(Σ) + µ⊤µ )2 . (31)
Proof. We start with the inequality between the variance of 2 random variables. We define Cov(A,B) as covariance of 2 random variables A,B.
Var[A+B] = Var[A] + Var[B] + 2Cov(A,B) ≤ Var[A] + Var[B] + 2 √ Var[A]Var[B]
≤ Var[A] + Var[B] + 2 · Var[A] + Var[B] 2
= 2Var[A] + 2Var[B]. (32)
For Varx∼Dc1 ,S [α],
Varx∼Dc1 ,S [α] = Var [∥∥∥ϕ(x)− ϕ(Sc2)∥∥∥2 − ∥∥∥ϕ(x)− ϕ(Sc1)∥∥∥2] = Var
[∥∥∥ϕ(Sc1)∥∥∥2 − ∥∥∥ϕ(Sc2)∥∥∥2 − 2ϕ(x)⊤ (ϕ(Sc2)− ϕ(Sc1))] ≤ 2Var
[∥∥∥ϕ(Sc1)∥∥∥2 − ∥∥∥ϕ(Sc2)∥∥∥2] + 4Var [ ϕ(x)⊤ ( ϕ(Sc2)− ϕ(Sc1) )] (∵ equation 32)
= 2Var [∥∥∥ϕ(Sc1)∥∥∥2]+ 2Var [∥∥∥ϕ(Sc2)∥∥∥2]+ 4Var [ϕ(x)⊤(ϕ(Sc2)− ϕ(Sc1))] . (33)
From 3rd line to 4th line we deompose the variance of ∥∥∥ϕ(Sc1)∥∥∥2−∥∥∥ϕ(Sc2)∥∥∥2 use the independence
of ϕ(Sc1) and ϕ(Sc2) with their class given. Next we focus on Var [ ϕ(x)⊤(ϕ(Sc2)− ϕ(Sc1)) ] .
Var [ ϕ(x)⊤(ϕ(Sc2)− ϕ(Sc1)) ] = E [( ϕ(x)⊤ ( ϕ(Sc2)− ϕ(Sc1) ))2] − ( E [ϕ(x)]⊤ E [( ϕ(Sc2)− ϕ(Sc1)
)])2 ≤ E [( ∥ϕ(x)∥2
∥∥∥ϕ(Sc2)− ϕ(Sc1)∥∥∥)2]− (µc1⊤(µc2 − µc1))2 = E [ ∥ϕ(x)∥2 ] E [∥∥∥ϕ(Sc2)− ϕ(Sc1))∥∥∥2]− (µ⊤c1 (µc2 − µc1))2 .
(34)
From the 2nd line to the 3rd line we use Cauchy–Schwarz inequality. For E [∥∥∥ϕ(Sc2)− ϕ(Sc1)∥∥∥2],
with equation 20
E [∥∥∥ϕ(Sc2)− ϕ(Sc1)∥∥∥2] = 1K (Tr (Σc1) + Tr (Σc2)) + (µc2 − µc1)⊤ (µc2 − µc1) . (35)
Thus Ec1,c2∼τ [ Ex∼Dc1 ,S [ ∥ϕ(x)∥2 ] Ex∼Dc1 ,S [∥∥∥ϕ(Sc2)− ϕ(Sc1)∥∥∥2]] is calculated as follows. Ec1,c2∼τ [ Ex∼Dc1 ,S [ ∥ϕ(x)∥2 ] Ex∼Dc1 ,S
[∥∥∥ϕ(Sc2)− ϕ(Sc1)∥∥∥2]] = Ec1,c2∼τ [ (Tr(Σc1) + µ ⊤ c1µc1) ( 1
K (Tr(Σc1) + Tr(Σc2)) + (µc2 − µc1)
⊤(µc2 − µc1) )]
= Ec1,c2∼τ [ 1
K
( Tr (Σc1) 2 +Tr(Σc1) Tr (Σc2) )] + Ec1,c2∼τ [ 2
K (Tr (Στ ))µ
⊤ c1µc1 + µ ⊤ c1µc1 ( µc2 − µc1 )⊤ ( µc2 − µc1 )] = 1
K
( Ec1,c2∼τ [ Tr(Σc1) 2 ] + Ec1,c2∼τ [Tr(Σc1)] 2 )
+ 2
K (Tr (Στ ))
( Tr (Σ) + µ⊤µ ) + E [ µ⊤c1µc1(µc2 − µc1) ⊤(µc2 − µc1) ]
= 1
K Varc∼τ [Tr (Σc)] + 2
K Tr (Στ )
2 +
2
K (Tr (Στ ))
( Tr (Σ) + µ⊤µ ) + Ec1,c2 [ µ⊤c1µc1 ( µc2 − µc1 )⊤ ( µc2 − µc1 )] = 1
K Varc∼τ [Tr(Σc)] + 2
K Tr(Στ )
( Tr(Στ ) + Tr(Σ) + µ ⊤µ ) + Ec1,c2 [ µ⊤c1µc1 ( µc2 − µc1 )⊤ ( µc2 − µc1 )] .
(36)
Now we take into account the term −(µc1⊤(µc2 − µc1)) 2 in equation 34,
Ec1,c2 [ µ⊤c1µc1(µc2 − µc1) ⊤(µc2 − µc1)− (µ ⊤ c1(µc2 − µc1)) 2 ]
= Ec1,c2 [ µ⊤c1µc1µ ⊤ c2µc2 − (µ ⊤ c1µc2) 2 ]
≤ Ec1,c2 [ µ⊤c1µc1µ ⊤ c2µc2 ] = ( Tr(Σ) + µ⊤µ )2 . (37)
Thus Ec1,c2 [ Var[ϕ(x)⊤(ϕ(Sc2)− ϕ(Sc1))] ] is calculated as follows.
Ec1,c2 [ Var[ϕ(x)⊤(ϕ(Sc2)− ϕ(Sc1))] ] (38)
= 1
K Varc∼τ [Tr(Σc)] +
2
K Tr(Στ )
( Tr(Στ ) + Tr(Σ) + µ ⊤µ ) + (Tr(Σ) + µ⊤µ)2. (39)
Regarding ∥∥∥ϕ(Sc)∥∥∥2, since the function computing square norm is convex, next equation holds with
D-dimensional Jensen’s inequlality (Perlman, 1974):
∥∥∥ϕ(Sc)∥∥∥2 = ∥∥∥∥∥∥∥ 1 K ∑ i=0
x∈Sc
ϕ(x) ∥∥∥∥∥∥∥ 2
≤ 1 K ∥∥∥∥∥∥∥ ∑ i=0
x∈Sc
ϕ(x) ∥∥∥∥∥∥∥ . (40)
Combining equation 33, equation 38, and equation 40 we obtain
Ec1,c2Varx∼Dc1 ,S∼D⊗2N [α]
= 4
K Ec∼τVarx∼Dc
[ ∥ϕ(x)∥2 ] + 4
K Varc∼τ [Tr(Σc)]
+ 8
K Tr(Στ )
( Tr(Στ ) + Tr(Σ) + µ ⊤µ ) + 4 ( Tr(Σ) + µ⊤µ )2 . (41)
To complete the proof of Theorem 1, we calculate Ec1,c2Ex∼Dc1 ,S∼D⊗2K [α] 2.
Ec1,c2Ex∼Dc1 ,S∼D⊗2K [α] 2 (42)
= Ec1,c2
[( 1
K (Tr(Σc1)− Tr(Σc2)) + (µc1 − µc2)
⊤(µc1 − µc2) )2]
= Ec1,c2
[( 1
K (Tr(Σc1)− Tr(Σc2))
)2] + Ec1,c2 [( (µc1 − µc2) ⊤(µc1 − µc2) )2]
+ 2Ec1,c2 [( 1
K (Tr(Σc1)− Tr(Σc2))
)( (µc1 − µc2) ⊤(µc1 − µc2) )]
= Varc1,c2∼τ
[ 1
K (Tr(Σc1)− Tr(Σc2))
] + ( Ec1,c2∼τ [ 1
K (Tr(Σc1)− Tr(Σc2)) ])2 + Ec1,c2 [( (µc1 − µc2) ⊤(µc1 − µc2) )2]
= 2
K2 Varc∼τ [Tr(Σc)] + Ec1,c2
[( (µc1 − µc2) ⊤(µc1 − µc2) )2] . (43)
From 2nd line to 3rd line, we use the symmetry of the last term with respect to c1 and c2 and erase the term. Combining equation 13, Lemma 5, Lemma 6, and equation 42, we obtain the bound.
A.5 THEOREM 1 WITH N-WAY CLASSIFICATION
The upper bound on the risk of N -way prototype classifier is as follows.
Theorem 7. Let operation of binary class prototype classifier M as defined in equation 1. Then for ϕ(Sc) = 1KΣx∈Scϕ(x), µc = Ex∼Dc [ϕ(x)], Σc = Ex∼Dc [(ϕ(x) − µc)(ϕ(x) − µc)
⊤], µ = Ec∼τ [µc],Σ = Ec∼τ [(µc − µ)(µc − µ)⊤], Ec∼τ [Σc] = Στ , if ϕ(x) has its fourth moment, miss classification risk of binary class prototype classifier RM satisfy
R(M(ϕ,x, {Si}Ni=1), y)
≤ N − 1− N∑ c=1 c ̸=y
4(Tr(Σ))2
EV[hL2(ϕ(x))] + VTr(Σy) + Vwit(Στ ,Σ,µ) + Edist2L2(µy,µc) , (44)
where
EV[hL2(ϕ(x))] = 4
K Ey∼τ
[ Varxc∼Dc [ ∥ϕ(x)∥2 ]] ,
VTr(Σy) =
( 4
K +
2
K2
) Varc∼τ [Tr (Σc)] ,
Vwit(Στ ,Σ,µ) = 8
K Tr(Στ )
( Tr(Στ ) + Tr(Σ) + µ ⊤µ ) + 4(Tr(Σ) + µ⊤µ)2,
Edist2L2(µy,µc) =Ey,c [(( µcy − µc )⊤ (µy − µc) )2] .
Proof. Let x, y be the input and its class of a query data. We define αc by αc = ∥∥∥ϕ(x)− ϕ(Sc)∥∥∥2−∥∥∥ϕ(x)− ϕ(Sy)∥∥∥2. Then a prototype classifier miss-classify a class of input x, ŷ, if ∃c ∈ [1, N ], c ̸=
y, αc < 0 . Hence: RM(ϕ) = Pr( ⋃N
c=1 c̸=y
αc < 0)
By Frechet’s inequality, next equation holds:
RM(ϕ) ≤ N∑ c=1 c̸=y Pr(αi < 0).
Noting that Theorem 1 can be applied to each term in the summation and then we obtain Theorem 7
A.6 DETAILED PERFORMANCE RESULTS
We show the detailed performance results in this section with 95% confidence margin. We add the two future normalization methods, variance normalization before L2-normalization (V-N+L2N) and LDA before L2-normalization (LDA+L2-N) to the tables. Table 3 and Table 4 show the performance on miniImagenet and tieredImagenet; table 5 and table 6 show the performance on CIFARFS and FC100.
A.7 THE COMPARISON OF THE RATIO OF THE BETWEEN-CLASS VARIANCE TO THE WITHIN-CLASS VARIANCE BEFORE AND AFTER APPLYING L2-NORM
We show the ratio of the between-class variance to the within-class variance before and after applying L2-norm in Figure 4. We calculated the ratio of the between-class variance to each class’s variance and averaged over the test classes of each dataset. Although L2-normalization is expected to reduce Tr(Σ) rather than Σc according to 4.4, the figure shows L2-norm marginaly reduces the ratio in CIFARFS dataset and the ratio does not changed so much in the other dataset.

Review summaries:
This paper analyzes how a prototype classifier works well without fine-tuning and meta-learning. The feature is trained with cross-entropy loss with a linear projection layer. In testing, the classifier is a Prototype classifier.  This paper derives a novel generalization bound for the prototypical network and shows that focusing on the variance of the norm of a feature vector can improve performance. The proposed upper bound is a modification of (Cao et al., 2020), and it does not require the features distributed on any specific distribution. Each covariance matrix does not have to be the same among classes.   The authors experimentally investigated several normalization methods (L2-N, V-N, LDA, EST, EST-L2-N) for minimizing the variance of the norm. 

The submission introduces a generalization bound for Prototypical Networks that does not depend on assumptions on the class-conditional distributions. This bound decreases as the variance in norm of the feature vectors decreases, and the authors empirically investigate five feature transformation approaches that are meant to lower this variance: L2-normalization (L2-norm), variance-normalization, Linear Discriminant Analysis (LDA), Embedding Space Transformation (EST), and EST+L2-norm.  Experimental results are reported on the mini-ImageNet, tiered-ImageNet, CIFAR-FS, and FC100 benchmarks in the 1-shot and 5-shot settings, using ResNet-12 or ResNet-18 as the backbone architecture. The five feature transformation approaches are applied on top of the Baseline and Baseline++ learners before using the extracted features to build a prototypical classifier. The submission compares against the following: - Prototypical Networks - Baseline and Baseline++ with a linear classifier on top of the extracted features - Baseline and Baseline++ with Wang et al.'s centering+L2-norm approach on top of the extracted features. - Baseline and Baseline++ with no additional feature transformation and a prototypical classifier on top of the extracted features.  The submission concludes that EST+L2-norm performs best, and that its performance-boosting effect decreases as the number of examples per class increases.

In this paper, the authors have derived a generalization bound for the prototypical networks, which provides some insights towards improving protonets without finetuning using simply normalizing the feature vectors and reducing the variance of the norm. The derivation has relaxed the previous assumption of considering feature distribution and class covariance matrix to be a particular form. To justify the claim, the authors experimented on several feature transformation methods on standard few-shot benchmarks.