Title: PROJECTIVE SUBSPACE NETWORKS FOR FEW-SHOT LEARNING

Abstract: Generalization from limited examples, usually studied under the umbrella of metalearning, equips learning techniques with the ability to adapt quickly in dynamical environments and proves to be an essential aspect of lifelong learning. In this paper, we introduce the Projective Subspace Networks (PSN), a deep learning paradigm that learns non-linear embeddings from limited supervision. In contrast to previous studies, the embedding in PSN deems samples of a given class to form an affine subspace. We will show that such modeling leads to robust solutions, yielding competitive results on supervised and semi-supervised few-shot classification. Moreover, our PSN approach has the ability of end-to-end learning. In contrast to previous works, our projective subspace can be thought of as a richer representation capturing higher-order information datapoints for modeling new concepts.

1. INTRODUCTION:

Supervised learning with deep architectures, though achieving remarkable results in many areas, requires large amount of annotated data. Various studies show that many deep learning techniques in computer vision, speech recognition and natural language understanding, to name a few methods stated in Hinton et al. (2012) ; Krizhevsky et al. (2012) , will fail to produce reliable models that generalize well if limited data annotations are available. Aside from the labor associated with annotating data, precise annotation can become ill-posed in some cases. One prime example of such a difficulty is object detection labeling which requires annotating bounding boxes of objects as explained in Alexe et al. (2012) .

In contrast to the current trend in deep learning, humans can learn new concepts from only a few examples. This in turn provides humans with lifelong learning abilities. Inspired by such learning abilities, several approaches are developed to study learning from limited examples Lake et al. (2015) ; Lazaridou et al. (2017) ; Vinyals et al. (2016) ; Triantafillou et al. (2017) ; Xu et al. (2017) ; Finn et al. (2017) ; Ravi & Larochelle (2017) ; Wang et al. (2018) ; Mishra et al. (2018) ; Qiao et al. (2018) ; Neill & Buitelaar (2018) . In machine learning, the diverse ideas in this context include embedding features through metric learning (Koch et al. (2015) , Vinyals et al. (2016) ), optimization technique (Finn et al. (2017) , Ravi & Larochelle (2017) ), and generative models (Fei-Fei et al. (2006) , Lake et al. (2015) ).

In this work, we propose a deep model that learns new concepts from limited data to address two challenging learning problems; namely: (i) few-shot classification and (ii) semi-supervised few-shot learning. The goal of few-shot classification is to learn a model that can discriminate a given query by comparing it to a few of samples (a.k.a. the support set). An example is to classify a motorcycle by viewing some different types of motorcycles (or other vehicles). The goal of semi-supervised few-shot learning is to additionally benefit from unlabeled data to boost the performance of the model.

Our method, coined Projective Subspace Networks or PSN for short, learns non-linear embeddings using subspaces, in a sense that samples of a class are modeled as a low-dimensional affine subspace. The use of subspaces to model images and sets has a long history in computer vision and machine learning. For example, it has been proved that the set of all reflectance functions (the mapping from surface normals to intensities) produced by Lambertian objects lie close to a low-dimensional linear subspace (Basri & Jacobs (2003)) .

q i q i x i q i (a) q i q i x i q i (b) q i q i x i q i (c)

Figure 1 : Feature embedding in (a) Matching Networks (Vinyals et al. (2016) ), (b) Prototypical Networks (Snell et al. (2017) ), and (c) our PSN method.

In spite of its intriguing properties, to the best of our knowledge, the subspace modeling has never been used previously to address few-shot learning problems. This makes our paper distinct and novel as compared to former studies (e.g., Vinyals et al. (2016) ; Snell et al. (2017) ). Fig. 1 provides a conceptual illustration of our approach.

We empirically observed that embeddings tailored towards capturing the structure of each class through low-dimensional affine subspaces could lead to discriminative models. Interestingly, such models can be built with minimum overheads and without opting for advanced methods in subspace creation (e.g., such as the notion of sparsity). Our conjecture here is that subspaces are less sensitive to perturbations such as outliers and noise compared to other embedding techniques for few-shot learning as illustratively shown in Fig. 2 .

Our contributions in this work are: i. Few-shot learning is formulated as an embedding problem through subspaces. We rely on a well-established concept stating that samples of a class (and hence variations such as pose and illumination) can be effectively captured by a low-dimensional affine space. ii. Adaptation from few-shot learning to semi-supervised learning is performed with a refinement through soft-assignment. The robustness of such a model is shown in our experiments. iii. We also introduce an evaluation mechanism to assess the generalization ability over unseen classes during test time.

2. RELATED WORK:

In this section, we briefly review the literature on few-shot learning and subspace clustering. Fewshot learning was originally introduced to imitate human learning capabilities in classification. Some of the early works made use of generative models and similarity learning to capture the variation within parts and geometric configurations of objects (Fei-Fei et al. (2006); Lake et al. (2015) ; Torralba et al. (2007) ). As of late, few-shot problems are mainly addressed through meta-learning techniques. For example, in Koch et al. (2015) ; Vinyals et al. (2016) ; Snell et al. (2017) ; Garcia & Bruna (2018) , the problem of few-shot learning is formulated as non-linear embedding or representation learning.

The closest approach to our proposed work is the Prototypical Networks (PN hereafter) model ( Snell et al. (2017) ). It uses the random choice of episode images which form the prototype center per class. In addition, recent metric learning approaches use complex architectures and pipelines in convolutional networks of Gidaris & Komodakis (2018) and Wang et al. (2018) . In contrast, our projective subspace concept is extremely simple by design as it constitutes just a single layer of the network.

Another common meta-learning approach to few-shot classification uses meta-learning and manipulates gradient updates to train the model parameters. Ravi & Larochelle (2017) The effect of outliers on prototypes and subspaces. The odd rows show the decision boundaries obtained by prototypes (with and without outliers) for two-and three-class problems.

The even rows depict how subspaces behave for the same problems. While affected, in general, subspaces show better resilience to perturbations and attain higher discriminatory power in comparison to prototypes. Best viewed in color.

task as well as a meta-gradient from combined tasks, and, as a result, it outperformed LSTM-based learner. Mishra et al. (2018) employed ResNets to model few-shot classification as a temporal solution with aggregation, thus, in meta-learning context, their model can refer to the past experience.

Our idea is based on the assumption that samples from any class in the support set form a lowdimensional subspace. As we will show, to train PSN, backpropagation through Singular Value Decomposition (SVD) is required. Backpropagation through matrix decomposition such as SVD is a well-studied problem Ionescu et al. (2015) with applications ranging from semantic segmentation to classification and visual recognition Ionescu et al. (2015) ; Li et al. (2017a) ; Gou et al. (2018) .

3. PROBLEM SET-UP:

We start by defining the terminology used in few-shot learning. The problem of N -way K-shot classification (e.g., 5-way 1-shot) is defined as classifying queries belonging to N classes by seeing only K samples from each class. For example, in 5-way 1-shot setting, the model needs to identify a query among five classes by seeing only one sample from each class. To obtain a reliable model, so-called episodes are used in training. An episode T i consists of two sets, the support set S and the query set Q. The system is then trained by minimizing a classification loss over episodes, simulating the scenarios it will encounter at test time. This episode setting is the same as proposed by Vinyals et al. (2016) .

A related problem is semi-supervised few-shot setting learning where unlabeled data is provided to the model. In the literature, various configurations are considered for semi-supervised few-shot learning (e.g., Garcia & Bruna (2018) In what follows, we introduce our PSN approach. The whole framework is trained end-to-end (see Fig. 3 ) which enables PSN to be used with various deep architectures. As an example, in § 5, we use PSN on top of the WideResNets (Zagoruyko & Komodakis (2016) ) for few-shot classification.

We start by introducing our notations for the (N -way, K-shot) few-shot learning. Each episode or task T i is composed of the support set

S = {(x 1,1 , c 1,1 ), (x 1,2 , c 1,2 ), • • • , (x N,K , c N,K )} and the query set Q = {q 1 , • • • , q N ×M }.

Here, x i,j denotes the j-th sample from class i and c i,j ∈ {1, • • • , N }. In the semi-supervised setting, there is an unlabeled set R = {r 1 , .., r U } within an episode. We propose to model points by subspaces {Z i } N i=1 . Each subspace Z i has a basis represented by

R D×n P i = [p 1 , • • • , p n ]; n ≤ D, with P i P i = I n . 4.1 PSN FOR FEW-SHOT CLASSIFICATION Let f Θ : X → R D be

a mapping from the input space X to some D-dimensional representation realized by a neural network. Our goal is to learn Θ, i.e., the embedding function in a way that the resulting space is suitable for subspace representation. For simplicity, we assume that every class in an episode can be described by just one subspace. Extension to multiple subspaces per class is straightforward though. Define µ k as the mean of class k in the embedded space. That is,

EQUATION

A basis for the subspace representing class k can be obtained by Singular Value Decomposition (SVD). To be specific, we define

X k = [x k,1 -µ k , • • • , x k,K -µ k ].

Applying truncated SVD on X k provides us with P k . We emphasize that more involved techniques to obtain robust subspaces from X k can potentially improve the PSN. Nevertheless, our goal is to assess whether the concept of subspace modeling for few-shot learning is justified or not and thus we opt for truncated SVD in our implementation. Now a query q j can be projected onto P k which yields:

EQUATION

The distance from the query to P k is:

EQUATION

We define the probability of the query assigned to class k using a softmax function as:

EQUATION

Now, we can minimize the negative log of Eqn. 4 to obtain Θ. To train the whole framework, backpropagation through SVD is required which is available in modern deep learning packages such as PyTorch (Paszke et al. (2017) ). Algorithm 1 explains the steps of training our PSN. The code will be released online on Github1 .

Algorithm 1 Train Projective Subspace Networks Input: Each episode

T i with S = {(x 1,1 , c 1,1 ), • • • , (x N,K , c N,K )} and Q = {q 1 , ..., q N ×M } 1: Θ 0 ← random initialization 2: for t in {T 1 , ..., T N T } do 3: L t ← 0 4: for k in {1, ..., N } do 5: X ← S k

Get examples in the support set from class k 6:

µ k ← 1 K x∈X f Θ (x)

Mean from the support set 7:

µ k ← Kµ k + i m i f Θ (r i ) K + i m i Refined mean(only semi-supervised learning) 8: X ← [x i -µ k , ..., x K -µ k ] 9: [U, Σ, V ] ← SVD( X)

Matrix factorization using SVD 10:

P k ← U 1,...,n

Truncate the matrix 11:

for q j in Q k do 12:

y j,k ← P k (f Θ (q j ) -µ k )

Query projection 13:

d j,k ← f Θ (q j ) -µ k -P k y j,k

Distance calculation 14:

p j,k ← exp(-d 2 j,k ) k exp(-d 2 j,k )

Softmax on distance scores 15:

end for 16:

end for 17:

L t ← 1 N 2 M k j -log (p j,k ) 18:

Update Θ using ∇ Θ L t 19: end for

4.2. PSN FOR SEMI-SUPERVISED FEW-SHOT LEARNING:

In what follows, we extend the model developed in § 4.1 to address semi-supervised few-shot learning. In doing so, we need to take advantage of the unlabeled data to fit better subspaces to our data. We achieve this by refining the center of each class according to

EQUATION

Here, m i is the soft-assignment score for unlabeled samples. To work at the presence of distractors, we use a fake class with zero mean. We empirically observed that such a simple modification to the means can improve the results without the need of refining the SVD step.

5. EXPERIMENTS:

Below we contrast and assess our method against state-of-the-art techniques on two challenging datasets, namely Mini-ImageNet (Ravi & Larochelle (2017) ) and Tiered-ImageNet (Ren et al. In contrast to the Mini-ImageNet, the training and test sets in Tiered-ImageNet represent distinct classes. Moreover, in designing the Tiered-ImageNet, the problem of few-shot learning with unlabeled data was taken into account and the labeled data is only within a small percentage when performing semi-supervised learning. In this large dataset, learning from the labeled data is still sufficient to produce reasonable representations even though the unlabeled data is set in huge portion e.g., 90%.

5.1. FEW-SHOT LEARNING:

We follow the general practice and evaluate our method on the Mini-ImageNet dataset when it comes to few-shot learning and classification. Various procedures such as pre-training using 64 classes (e.g., Qiao et al. (2018) ) and training with more classes/way have shown to increase the overall accuracy. Nevertheless, we avoid such procedures deliberately as we are mainly interested in contrasting the core idea, i.e., the role of subspaces in few-shot learning. So, we trained on 5way 5-shot and 20-way 5-shot, then applied the same classification task setup during testing. The CNN architecture is the same as the one used in Snell et al. (2017) with 4-convolutional stages. We also use WideResNets (Zagoruyko & Komodakis (2016) ) with 16 depth, 6 widening factor, and 0.3 dropout rate to compare with ResNets (He et al. ( 2016)) solution reported by Mishra et al. (2018) .

The feature dimensions from both architectures are 1600 and 384 respectively. We used ADAM ( Kingma & Ba ( 2015)) for optimizing our model and set the learning rate to 0.001 and cut it to half every 2.5K episodes for both architectures.

Results. By design, the method cannot accommodate learning with exactly one example, hence, the comparison here is provided only for 5-shot in 5-way and 20-way. In every episode, the query set contains 15 samples from each class. Our method outperforms the previous methods with 4convolutional stages shown in Table 1 . We also implemented WideResNets (Zagoruyko & Komodakis (2016)) using our method and obtained the performance for 5-way 5-shot: 69.92 ± 0.64% and 20-way 5-shot: 41.84 ± 0.24% that can outperform ResNets-based approach proposed by Mishra et al. (2018) with 68.88 ± 0.92% in 5-way and 5-shot classification task.

5.2. SEMI-SUPERVISED FEW-SHOT LEARNING:

In this experiment, the embedding architecture has 4-convolutional layers as PN (Snell et al. (2017) ).

We follow the experimental setup proposed by Ren et al. (2018) . The episode composition for labeled or support set and query set is similar to the few-shot learning classification task, but there is an additional unlabeled set provided in each episode. 2 : Semi-supervised few-shot classification results on the Mini-ImageNet and Tiered-ImageNet with 40% and 10% labeled data, respectively. We show the classification results with and without distractors. We compare our results to PN on semi-supervised learning (PN-SSL) with soft K-means (non-masked) and masked K-means (masked), as proposed by Ren et al. (2018) . Mini-ImageNet and Tiered-ImageNet with 40% and 10% of labeled data, respectively. We used the ADAM solver (Kingma & Ba (2015) ), the set the learning rate to 0.001 with the weight decay and cut the rate to half every 10K episodes. We trained in two settings: (i) supervised setting, where only labeled data is taken into account, and (ii) semi-supervised setting for which the unlabeled set is also used Results. In these experimental results, the performance is counted over 600 episodes. The results are averaged over 10 random splits of labeled and unlabeled sets. The supervised experiment shows that our method learns robust feature embedding from a small portion of labeled data. With the help of soft-assignment over unlabeled datapoints, the semi-supervised experiment detailed in Table 2 is demonstrated to outperform Prototypical Networks for Semi-Supervised Learning (PN-SSL) proposed by Ren et al. (2018) .

Mini-ImageNet:



5.3. GENERALIZATION BEYOND N -WAY K-SHOT:

As a measure of generalization ability, we propose to evaluate algorithms beyond the somehow inflexible testing protocol of few-shot learning. In particular, we assess whether a model trained on low number of classes can generalize well to classification tasks involving large number of classes.

To gain more insights, we further study how models trained with the K-shot assumption will perform if extra examples are available at the test time.

In the evaluation below, all models are trained with the 5-way 5-shot setting. At the test time, the models face the test protocol of N -way K-shot learning with N ∈ {5, 10, 15, 20} and K ∈ {3, 5, 10, 15, 20}. Table 3 contrasts the performance of PSN against PN. The table is selfexplanatory. In all experiments, PSN outperforms PN with the gap widened with more challenging settings (e.g., 20-way 20-shot).

Robustness to Perturbations. Our motivation to develop PSN is to devise a model that shows better resilience to perturbations. Intuitively, to have a noticeable change in the orientation of a subspace, one needs to induce drastic changes to the set. To empirically verify our claim, we assess how robust is PSN in comparison to PN (Snell et al. (2017) ) in the presence of perturbations. In particular, we considered the problems of 5-way 5-shot and 5-way 10-shot learning and introduced two types of perturbations at the test time to the trained models. Firstly, we randomly sampled examples from classes not presented in the support sets and included them in the support examples. Secondly, noisy examples are generated randomly using a multivariate Gaussian distribution with random mean and variance of σ = {0.15, 0.3, 0.4}. The results of this study are presented in Appendix A due to page limitations. To summarize, our experiments show that both PSN and PN are affected by outliers negatively. That said, the PSN exhibits a much better degree of resilience to outliers. When additive noise is considered, PSN behaves robustly for a wide-range of contamination. In contrast, the performance of PN drops rapidly and significantly in the presence of noise, reinforcing our idea that the use of subspaces indeed leads to a more robust model for the task in hand.

6. DISCUSSION:

Conceptually, the PN Snell et al. (2017) is the closest work to ours as both solutions obtain a refined representation for each class in the support sets, with the former using the class mean as the representation while in PSN affine subspaces model classes. That said, PSN also uses the mean of each class towards identifying their representative subspaces. Aside from the theoretical properties of affine subspaces2 , we empirically observed that for more challenging setups (e.g., 20-way 20-shot), utilizing the mean leads to a better performance. This is because prototypes lie on subspaces in the feature space. Additionally, prototypes in the PN can be easily utilized to design a hybrid with our approach.

Remark 1. Interestingly, in special cases, PSN simplifies to PN. For example, if samples of a class span a infinitesimal region, at the limit, collapsing to a point, then an affine subspace reduces to a point, recovering PN from PSN.

Remark 2. Since it is not possible to build an affine subspace from only one example, PSN cannot address 1-shot learning problems per se. However, simply applying augmentations to support images facilitates 1-shot learning. However, this issue is beyond the point we make in this work.

Subspace Dimension. In comparison to other models such as matching networks or PN, PSN comes with an extra hyper-parameter, the dimensionality of the subspaces (i.e., n). As a rule of thumb, we recommend to use n = K -1 to train the model, while n = 2 at the test time gives reasonable and robust results. That said, we thoroughly studied the effect of this parameter and summarized our findings in Appendix B. In short, PSN exhibits a great degree of robustness to n, which in turns, makes training of them painless.

Computational Complexity.:

The computational complexity of our PSN approach is O(min(N D 2 K, N DK 2 )), where K, N , and D are the number of shot, way, and feature dimensionality respectively. Compared to the complexity of the PN algorithm, i.e., O(N DK) , our algorithm is somehow slower due to the involvement of the SVD step. If the complexity is of a concern, fast approximate algorithms for SVD (e.g., Menon & Elkan (2011) ) can be considered.

7. CONCLUSIONS:

This paper presents the PSN, a novel approach for few-shot learning that employs nonlinear embeddings and modeling via affine subspaces. Empirically, we showed that the representations learned via PSN were expressive across a wide-range of supervised and semi-supervised few-shot problems.

In PSN, each class is represented by the subspace formed by all its examples, meaning that each class is modeled by the span of its samples. One possibility to extend the PSN modeling is to benefit -cleverly-from the null space associated with each class, in the hope of designing a more discriminative model.

Appendices:



A PERTURBATIONS EFFECT ON PERFORMANCE:

In this section, we study the effect of perturbation on the performance of PN and PSN. More specifically, we considered the problems of 5-way 5-shot and 5-way 10-shot learning and introduced two types of perturbations at the test time with the trained models. Note that, the model is obtained from 5-way 5-shot training without perturbations. Firstly, we randomly sampled examples from classes not presented in the support sets. Secondly, additive noise is generated randomly using a multivariate Gaussian distribution with random mean and variance of σ = {0.15, 0.3, 0.4}. Both examples in these two types are included in the support examples for prototypes and subspaces creation.

The results of this study are depicted in Fig. 4 . To summarize, our experiments show that both PSN and PN are affected by outliers negatively. That said, the PSN exhibits a much better degree of resilience to outliers. For example, modelling with PN leads to a drop of 19% and 12% percentage points when each support set has 20 outliers for the problem of 5-way 5-shot and 5-way 10-shot respectively. For the same experiment, the PSN modelling only suffers 11% and 9% percentage points of performance drop. When additive noise is considered, PSN behaves robustly for a widerange of contamination. In contrast, the performance of PN drops rapidly and significantly in the presence of noise, reinforcing our idea that subspaces form indeed a more robust model for the task in hand. 

B SUBSPACE DIMENSION EXPERIMENT:

In section, we report the results of experiments conducted to assess the effect of the subspace dimensionality (i.e., n) on the overall accuracy of the algorithm. In particular, we considered two few-shot problems, namely 5-way 5-shot and 5-way 20-shot. For the former, we varied the subspace dimensionality n ∈ {2, 3, 4, 5} and considered all combinations during training and testing (for example, n = 4 during training and n = 3 at the test time). For the experiment on 5-way 20shot problem, we varied the subspace dimensionality n ∈ {5, 10, 15, 20} and again considered all combinations.

Our experiments demonstrate that PSN behaves robustly over a wide-range of subspace dimensionality. For example, the lowest and highest performances for the 5-way 5-shot learning are 66.08 ± 0.67% and 66.62 ± 0.69%, respectively. For the problem of 5-way 20-shot learning, the lowest and highest performances are 75.03 ± 0.57% and 75.58 ± 0.57%, respectively.

:

https://github.com/anonymousauthor/PSN

One can argue why not modeling classes with linear subspaces, i.e., to drop the class mean for modeling. We note that linear subspaces are indeed special cases of more general affine subspaces where all subspaces intersect at the origin.

Figures and Tables:

Figure 3: Projective Subspace Networks Architecture

. The unlabeled set is composed of the examples from the classes in the support set and distractor classes. The number of supporting classes and distractor classes is set to five for training and testing. In the training stage, the number of examples in the unlabeled set is 50 consisting of five examples from each class. In the testing stage, the unlabeled set consists of 20 examples from each class. We also define the query set to have 20 examples per class for testing purpose.

Figure 4: The charts show 5-way 5-shot and 5-way 10-shot results of the PSN and the PN algorithms in the presence of outliers and additive noise. In the first column, sub-plots show the effect of introducing outliers among support samples (the classes of outliers are disjoint with the support classes of samples). The second to fourth columns show the effect of introducing noisy examples generated randomly from Gaussian distributions with random means and variance of σ = {0.15, 0.3, 0.4}, respectively. The performance is measured with increasing number of outliers and noisy examples (X-axes).

Few-shot classification results on Mini-ImageNet using 4-convolutional stages with 95% confidence intervals.(2018)). As the rule of thumb, the parameters of subspace dimension (n) that we used in the experiments are K-1 for training and two for testing stage. Mini-ImageNet. The Mini-ImageNet(Ravi & Larochelle (2017)) contains 60,000 images of the ImageNet(Russakovsky et al. (2015)) datasets. Images in the Mini-ImageNet are of size 84 × 84 and represent 100 classes with 64, 16, and 20 classes used for training, validation, and testing, respectively. Tiered-ImageNet. This dataset is also derived from ImageNet but contains a broader set of classes compared to the Mini-ImageNet. There are 351 classes from 20 different categories for training, 97 classes from 6 different categories for validation, and 160 classes from 8 different categories for testing.

Table content: <table><tr><td>Models</td><td>5-way 5-shot</td><td>20-way 5-shot</td></tr><tr><td>Matching Nets (Vinyals et al. (2016))</td><td>55.31 ± 0.73%</td><td>22.69 ± 0.20%</td></tr><tr><td>MAML (Finn et al. (2017))</td><td>63.11 ± 0.92%</td><td>19.29 ± 0.29%</td></tr><tr><td>Meta-Learner LSTM(Ravi &amp; Larochelle (2017))</td><td>60.60 ± 0.71%</td><td>26.06 ± 0.25%</td></tr><tr><td>Meta-SGD (Li et al. (2017b))</td><td>64.03 ± 0.94%</td><td>28.92 ± 0.35%</td></tr><tr><td>PN(Snell et al. (2017))</td><td>65.49 ± 0.25%</td><td>37.23 ± 0.21%</td></tr><tr><td>PSN</td><td colspan="2">66.62 ± 0.69% 38.26 ± 0.23%</td></tr></table>

Our model is trained on 100K episodes for

Table content: <table><tr><td/><td colspan="2">Mini-ImageNet</td><td colspan="2">Tiered-ImageNet</td></tr><tr><td>Models</td><td colspan="2">5-way 5-shot</td><td colspan="2">5-way 5-shot</td></tr><tr><td>PN, Supervised</td><td colspan="2">59.08 ± 0.22%</td><td colspan="2">66.15 ± 0.22%</td></tr><tr><td>PSN, Supervised</td><td colspan="2">63.43 ± 0.61%</td><td colspan="2">68.72 ± 0.49%</td></tr><tr><td/><td>w/o</td><td>w/</td><td>w/o</td><td>w/</td></tr><tr><td/><td>Distractors</td><td>Distractors</td><td>Distractors</td><td>Distractors</td></tr><tr><td>PN-SSL, Non-Masked</td><td>64.59 ± 0.28%</td><td>63.55 ± 0.28%</td><td>70.25 ± 0.31%</td><td>68.32 ± 0.22%</td></tr><tr><td>PN-SSL, Masked</td><td>64.39 ± 0.24%</td><td>62.96 ± 0.14%</td><td>69.88 ± 0.20%</td><td>69.08 ± 0.25%</td></tr><tr><td colspan="5">PSN, Semi-Supervised 68.12 ± 0.67% 66.10 ± 0.66% 71.15 ± 0.67% 69.15 ± 0.51%</td></tr><tr><td>Table</td><td/><td/><td/><td/></tr></table>

Generalization performance of PSN and PN with varying number of way (N ) and shot (K) at the test time.

Table content: <table/>

Peer review (iclr2019):
 This paper presents a new method for fully- and semi-supervised few-shot classification that is based on learning a general embedding as usual, and then learning a sub-space of it for each class. A query point is then classified as the class whose sub-space is closest to it. Pros: This is a neat idea and achieves competitive results. Learning a sub-space per class makes intuitive sense to me since it’s plausible that there is a lower-dimensional subspace of the overall embedding space that captures the properties that are common to only examples of a certain class. If this is indeed the case, it seems that indeed classifying query examples into classes based on their distances from the corresponding sub-spaces would lead to good discrimination. Cons: First, an inherent limitation is that this approach is not applicable to one-shot learning, and I have doubts in its merit for very low shot learning (explained below). Second, I’m missing the justification behind a key point used to motivate the approach, which requires clarification (explained below). Third, I feel that certain aspects of the approach were unclear (details to follow). Finally, I feel more analysis is needed to better understand the differences of this method from previous work (concrete suggestions follow). For semi-supervised learning, the novelty regarding how the unlabeled examples are incorporated is limited, as the approach used is previously-introduced in Ren et al, 2018. Overall, even though I like the idea and the results are good, there are a few points, mentioned in the above section that I feel require additional work before I can strongly recommend acceptance. Most importantly, relating to getting more intuition about why and when this works best, and tying it in better with previous approaches. A key point requiring clarification. There is a key fact that the authors used to motivate this approach which remains unclear to me: why is it the case that this approach is less sensitive to outliers than previous approaches? In Figure 1, an outlier is pictured in each of subfigures (a) and (b) corresponding to Matching and Prototypical Networks, but not in subfigure (c) which corresponds to PSN. No explanation is provided to justify this conjecture, other than empirical evaluation that is based on the overall accuracy only. In particular, since SVD is used to obtain the sub-spaces, instead of an end-to-end learned projector that directly optimizes the query set accuracy, it’s not clear why if a support point is an outlier it would not affect the sub-space creation. If I’m missing something, please clarify! (A) Comments on the approach. (1) Why define X_k as the support set examples minus the class prototype instead of just the support examples themselves? The latter seems simpler, and should have all the required information for shaping the class’ subspace. (2) Note that if X_k is defined as [x_{k,1}, .dots, x_{k,K}] as proposed in the above point (ie. without subtracting the class mean from each support point) then this method would have been applicable to 1-shot too. How would it then compare to a 1-shot Prototypical Network? Notice that in this case the mean of the class is equal to this one example. (3) In general, the truncated SVD decomposition for a class can be written using the matrices U, .Sigma and V^T with dimensions [D, n], [n, n] and [n, K] respectively, where D is the embedding dimensionality and K is the number of support points belonging to the given class. The middle matrix .Sigma in the non-truncated version would have dimensions [D, K]. Does this mean that when truncating, n is enforced to be smaller than each of D and K? This would mean that the dimensionality n of the sub-space is limited by the number of the support examples, which in some cases may be very small in few-shot learning. Can you comment on this? (4) How to set n (the dimensionality of each subspace) is not obvious. What values were explored? Is there a sweet spot in the trade-off between the observed complexity and the final accuracy? (B) Comparison with Prototypical Networks. (1) In what situations do we expect learning a sub-space per class to do better than learning a prototype per class? For example, Figure 4 shows the test-time performance as a function of the test ‘way’. A perhaps more interesting analysis would be to compare the models’ performance as a function of the test *shot*: if more examples are available it may be less appropriate to create a prototype and more beneficial to create a sub-space? (2) Can we recover Prototypical Networks as a special case of PSN? If so, how? It would be neat to show under which conditions these are equivalent. (C) Clarifications regarding the semi-supervised setup. (1) Are distractor classes sampled from a disjoint pool of classes, or is it that, for example, a class which is a distractor in an episode is a non-distractor in another episode. (2) Similarly for labeled / unlabaled at training time. Can the same example appear as labeled in one episode but unlabaled in another? In Ren et al, 2018, this was prevented by creating an additional labeled/unlabeled split even for the training examples. Therefore they use strictly less overall information at meta-training time than if that split weren’t used. To be comparable with them, it’s important to apply this same setup. (D) Additional minor comments. (1) “To work at the presence of distractors, we propose to use a fake class with zero mean”. Note that this was already proposed in Ren et al, 2018. They used a zero-mean, high-variance additional cluster whose aim was to ‘soak up’ the distractor examples to prevent them for polluting legitimate clusters (this was the second model they proposed). (2) In the introduction, regarding contribution iii. A more appropriate way to describe this is as exploring generalization to different numbers of classes, or ‘ways’ at test time than what was used at training time. (3) Gidaris and Komodakis (2018) is described in the related work as using a more complicated pipeline. Note however that their pipeline is in place for solving a more challenging problem than standard few-shot classification: they study how a model can maintain the ability to remember training classes while rapidly learning about new ‘test’ classes. (4) In the last line of section 5.3, use N-way instead of K-way since in the rest of the paper K was used to refer to the shot, not the way.