Projective Subspace Networks For Few-Shot Learning

1 INTRODUCTION . Supervised learning with deep architectures , though achieving remarkable results in many areas , requires large amount of annotated data . Various studies show that many deep learning techniques in computer vision , speech recognition and natural language understanding , to name a few methods stated in Hinton et al . ( 2012 ) ; Krizhevsky et al . ( 2012 ) , will fail to produce reliable models that generalize well if limited data annotations are available . Aside from the labor associated with annotating data , precise annotation can become ill-posed in some cases . One prime example of such a difficulty is object detection labeling which requires annotating bounding boxes of objects as explained in Alexe et al . ( 2012 ) . In contrast to the current trend in deep learning , humans can learn new concepts from only a few examples . This in turn provides humans with lifelong learning abilities . Inspired by such learning abilities , several approaches are developed to study learning from limited examples Lake et al . ( 2015 ) ; Lazaridou et al . ( 2017 ) ; Vinyals et al . ( 2016 ) ; Triantafillou et al . ( 2017 ) ; Xu et al . ( 2017 ) ; Finn et al . ( 2017 ) ; Ravi & Larochelle ( 2017 ) ; Wang et al . ( 2018 ) ; Mishra et al . ( 2018 ) ; Qiao et al . ( 2018 ) ; Neill & Buitelaar ( 2018 ) . In machine learning , the diverse ideas in this context include embedding features through metric learning ( Koch et al . ( 2015 ) , Vinyals et al . ( 2016 ) ) , optimization technique ( Finn et al . ( 2017 ) , Ravi & Larochelle ( 2017 ) ) , and generative models ( Fei-Fei et al . ( 2006 ) , Lake et al . ( 2015 ) ) . In this work , we propose a deep model that learns new concepts from limited data to address two challenging learning problems ; namely : ( i ) few-shot classification and ( ii ) semi-supervised few-shot learning . The goal of few-shot classification is to learn a model that can discriminate a given query by comparing it to a few of samples ( a.k.a . the support set ) . An example is to classify a motorcycle by viewing some different types of motorcycles ( or other vehicles ) . The goal of semi-supervised few-shot learning is to additionally benefit from unlabeled data to boost the performance of the model . Our method , coined Projective Subspace Networks or PSN for short , learns non-linear embeddings using subspaces , in a sense that samples of a class are modeled as a low-dimensional affine subspace . The use of subspaces to model images and sets has a long history in computer vision and machine learning . For example , it has been proved that the set of all reflectance functions ( the mapping from surface normals to intensities ) produced by Lambertian objects lie close to a low-dimensional linear subspace ( Basri & Jacobs ( 2003 ) ) . In spite of its intriguing properties , to the best of our knowledge , the subspace modeling has never been used previously to address few-shot learning problems . This makes our paper distinct and novel as compared to former studies ( e.g. , Vinyals et al . ( 2016 ) ; Snell et al . ( 2017 ) ) . Fig . 1 provides a conceptual illustration of our approach . We empirically observed that embeddings tailored towards capturing the structure of each class through low-dimensional affine subspaces could lead to discriminative models . Interestingly , such models can be built with minimum overheads and without opting for advanced methods in subspace creation ( e.g. , such as the notion of sparsity ) . Our conjecture here is that subspaces are less sensitive to perturbations such as outliers and noise compared to other embedding techniques for few-shot learning as illustratively shown in Fig . 2 . Our contributions in this work are : i. Few-shot learning is formulated as an embedding problem through subspaces . We rely on a well-established concept stating that samples of a class ( and hence variations such as pose and illumination ) can be effectively captured by a low-dimensional affine space . ii . Adaptation from few-shot learning to semi-supervised learning is performed with a refinement through soft-assignment . The robustness of such a model is shown in our experiments . iii . We also introduce an evaluation mechanism to assess the generalization ability over unseen classes during test time . 2 RELATED WORK . In this section , we briefly review the literature on few-shot learning and subspace clustering . Fewshot learning was originally introduced to imitate human learning capabilities in classification . Some of the early works made use of generative models and similarity learning to capture the variation within parts and geometric configurations of objects ( Fei-Fei et al . ( 2006 ) ; Lake et al . ( 2015 ) ; Torralba et al . ( 2007 ) ) . As of late , few-shot problems are mainly addressed through meta-learning techniques . For example , in Koch et al . ( 2015 ) ; Vinyals et al . ( 2016 ) ; Snell et al . ( 2017 ) ; Garcia & Bruna ( 2018 ) , the problem of few-shot learning is formulated as non-linear embedding or representation learning . The closest approach to our proposed work is the Prototypical Networks ( PN hereafter ) model ( Snell et al . ( 2017 ) ) . It uses the random choice of episode images which form the prototype center per class . In addition , recent metric learning approaches use complex architectures and pipelines in convolutional networks of Gidaris & Komodakis ( 2018 ) and Wang et al . ( 2018 ) . In contrast , our projective subspace concept is extremely simple by design as it constitutes just a single layer of the network . Another common meta-learning approach to few-shot classification uses meta-learning and manipulates gradient updates to train the model parameters . Ravi & Larochelle ( 2017 ) utilized Long-Short Term Memory ( LSTM ) -based learner to optimize the model parameters and their gradients . Moreover , the Model Agnostic Meta-Learner ( MAML ) proposed by Finn et al . ( 2017 ) used gradients per task as well as a meta-gradient from combined tasks , and , as a result , it outperformed LSTM-based learner . Mishra et al . ( 2018 ) employed ResNets to model few-shot classification as a temporal solution with aggregation , thus , in meta-learning context , their model can refer to the past experience . Our idea is based on the assumption that samples from any class in the support set form a lowdimensional subspace . As we will show , to train PSN , backpropagation through Singular Value Decomposition ( SVD ) is required . Backpropagation through matrix decomposition such as SVD is a well-studied problem Ionescu et al . ( 2015 ) with applications ranging from semantic segmentation to classification and visual recognition Ionescu et al . ( 2015 ) ; Li et al . ( 2017a ) ; Gou et al . ( 2018 ) . 3 PROBLEM SET-UP . We start by defining the terminology used in few-shot learning . The problem of N -way K-shot classification ( e.g. , 5-way 1-shot ) is defined as classifying queries belonging to N classes by seeing only K samples from each class . For example , in 5-way 1-shot setting , the model needs to identify a query among five classes by seeing only one sample from each class . To obtain a reliable model , so-called episodes are used in training . An episode Ti consists of two sets , the support set S and the query setQ . The system is then trained by minimizing a classification loss over episodes , simulating the scenarios it will encounter at test time . This episode setting is the same as proposed by Vinyals et al . ( 2016 ) . A related problem is semi-supervised few-shot setting learning where unlabeled data is provided to the model . In the literature , various configurations are considered for semi-supervised few-shot learning ( e.g. , Garcia & Bruna ( 2018 ) ; Boney & Ilin ( 2017 ) ; Ren et al . ( 2018 ) ) . In this work , we follow the challenging protocol in Ren et al . ( 2018 ) where so-called distractors are introduced . Here , an episode includes the support set S , query setQ , and unlabeled setR . The support ( labeled ) S and query Q sets are configured as in few-shot learning . Additionally , an unlabeled set R is provided to assist the classification task within an episode . In the unlabeled set , there are examples from two different sources : the support classes and the distractor classes . As the name implies , examples from distractor classes are irrelevant to the classification task and represent classes outside the support set . 4 PROJECTIVE SUBSPACE NETWORKS In what follows , we introduce our PSN approach . The whole framework is trained end-to-end ( see Fig . 3 ) which enables PSN to be used with various deep architectures . As an example , in § 5 , we use PSN on top of the WideResNets ( Zagoruyko & Komodakis ( 2016 ) ) for few-shot classification . We start by introducing our notations for the ( N -way , K-shot ) few-shot learning . Each episode or task Ti is composed of the support set S = { ( x1,1 , c1,1 ) , ( x1,2 , c1,2 ) , · · · , ( xN , K , cN , K ) } and the query set Q = { q1 , · · · , qN×M } . Here , xi , j denotes the j-th sample from class i and ci , j ∈ { 1 , · · · , N } . In the semi-supervised setting , there is an unlabeled set R = { r1 , .. , rU } within an episode . We propose to model points by subspaces { Zi } Ni=1 . Each subspace Zi has a basis represented by RD×n 3 Pi = [ p1 , · · · , pn ] ; n ≤ D , with P > i Pi = In . 4.1 PSN FOR FEW-SHOT CLASSIFICATION . Let fΘ : X → RD be a mapping from the input space X to some D-dimensional representation realized by a neural network . Our goal is to learn Θ , i.e. , the embedding function in a way that the resulting space is suitable for subspace representation . For simplicity , we assume that every class in an episode can be described by just one subspace . Extension to multiple subspaces per class is straightforward though . Define µk as the mean of class k in the embedded space . That is , µk = 1 K ∑ i , ci=k fΘ ( xi ) . ( 1 ) A basis for the subspace representing class k can be obtained by Singular Value Decomposition ( SVD ) . To be specific , we define Xk = [ xk,1 − µk , · · · , xk , K − µk ] . Applying truncated SVD on Xk provides us with Pk . We emphasize that more involved techniques to obtain robust subspaces from Xk can potentially improve the PSN . Nevertheless , our goal is to assess whether the concept of subspace modeling for few-shot learning is justified or not and thus we opt for truncated SVD in our implementation . Now a query qj can be projected onto Pk which yields : yj , k = P > k ( fΘ ( qj ) − µk ) . ( 2 ) The distance from the query to Pk is : dj , k = ‖fΘ ( qj ) − µk − Pkyj , k‖ . ( 3 ) We define the probability of the query assigned to class k using a softmax function as : pΘ ( c = k|qj ) = exp ( − d2j , k ) ∑ k′ exp ( − d2j , k′ ) . ( 4 ) Now , we can minimize the negative log of Eqn . 4 to obtain Θ . To train the whole framework , backpropagation through SVD is required which is available in modern deep learning packages such as PyTorch ( Paszke et al . ( 2017 ) ) . Algorithm 1 explains the steps of training our PSN . The code will be released online on Github1 . Algorithm 1 Train Projective Subspace Networks Input : Each episode Ti with S = { ( x1,1 , c1,1 ) , · · · , ( xN , K , cN , K ) } and Q = { q1 , ... , qN×M } 1 : Θ0 ← random initialization 2 : for t in { T1 , ... , TNT } do 3 : Lt ← 0 4 : for k in { 1 , ... , N } do 5 : X ← Sk . Get examples in the support set from class k 6 : µk ← 1K ∑ x∈X fΘ ( x ) . Mean from the support set 7 : µk ← Kµk + ∑ imifΘ ( ri ) K + ∑ imi . Refined mean ( only semi-supervised learning ) 8 : X̃ ← [ xi − µk , ... , xK − µk ] 9 : [ U , Σ , V > ] ← SVD ( X̃ ) . Matrix factorization using SVD 10 : Pk ← U1 , ... , n . Truncate the matrix 11 : for qj in Qk do 12 : yj , k ← P > k ( fΘ ( qj ) − µk ) . Query projection 13 : dj , k ← ‖fΘ ( qj ) − µk − Pkyj , k‖ . Distance calculation 14 : pj , k ← exp ( −d2j , k ) ∑ k′ exp ( −d2j , k′ ) . Softmax on distance scores 15 : end for 16 : end for 17 : Lt ← 1N2M ∑ k ∑ j − log ( pj , k ) 18 : Update Θ using ∇ΘLt 19 : end for

Summary: This paper considers the problem of few-shot learning and proposes a new embedding-based approach. In contrast to previous work (such as Matching Networks and Prototypical Networks) where distance is computed in pure embedding space, this work proposes computing a low-dimensional subspace to represent a class and using the distance from an embedded query point to this subspace. The low-dimensional subspace for a class is computed by running truncated Singular Value Decomposition on the normalized embeddings of all points in the support set for that class and using the top n left singular vectors as the basis for the class's subspace. The authors also propose an extension to their model to the semi-supervised few-shot learning setting by incorporating masked-mean computation and zero-mean cluster for distractor items (both ideas borrowed from Renn 2017 for prototypical networks). Experiments are conducted on Mini-Imagenet in the few-shot learning setting and on Mini-Imagenet and Tiered-ImageNet in the semi-supervised few-shot learning setting.