Projective Subspace Networks For Few-Shot Learning

1 INTRODUCTION . Supervised learning with deep architectures , though achieving remarkable results in many areas , requires large amount of annotated data . Various studies show that many deep learning techniques in computer vision , speech recognition and natural language understanding , to name a few methods stated in Hinton et al . ( 2012 ) ; Krizhevsky et al . ( 2012 ) , will fail to produce reliable models that generalize well if limited data annotations are available . Aside from the labor associated with annotating data , precise annotation can become ill-posed in some cases . One prime example of such a difficulty is object detection labeling which requires annotating bounding boxes of objects as explained in Alexe et al . ( 2012 ) . In contrast to the current trend in deep learning , humans can learn new concepts from only a few examples . This in turn provides humans with lifelong learning abilities . Inspired by such learning abilities , several approaches are developed to study learning from limited examples Lake et al . ( 2015 ) ; Lazaridou et al . ( 2017 ) ; Vinyals et al . ( 2016 ) ; Triantafillou et al . ( 2017 ) ; Xu et al . ( 2017 ) ; Finn et al . ( 2017 ) ; Ravi & Larochelle ( 2017 ) ; Wang et al . ( 2018 ) ; Mishra et al . ( 2018 ) ; Qiao et al . ( 2018 ) ; Neill & Buitelaar ( 2018 ) . In machine learning , the diverse ideas in this context include embedding features through metric learning ( Koch et al . ( 2015 ) , Vinyals et al . ( 2016 ) ) , optimization technique ( Finn et al . ( 2017 ) , Ravi & Larochelle ( 2017 ) ) , and generative models ( Fei-Fei et al . ( 2006 ) , Lake et al . ( 2015 ) ) . In this work , we propose a deep model that learns new concepts from limited data to address two challenging learning problems ; namely : ( i ) few-shot classification and ( ii ) semi-supervised few-shot learning . The goal of few-shot classification is to learn a model that can discriminate a given query by comparing it to a few of samples ( a.k.a . the support set ) . An example is to classify a motorcycle by viewing some different types of motorcycles ( or other vehicles ) . The goal of semi-supervised few-shot learning is to additionally benefit from unlabeled data to boost the performance of the model . Our method , coined Projective Subspace Networks or PSN for short , learns non-linear embeddings using subspaces , in a sense that samples of a class are modeled as a low-dimensional affine subspace . The use of subspaces to model images and sets has a long history in computer vision and machine learning . For example , it has been proved that the set of all reflectance functions ( the mapping from surface normals to intensities ) produced by Lambertian objects lie close to a low-dimensional linear subspace ( Basri & Jacobs ( 2003 ) ) . In spite of its intriguing properties , to the best of our knowledge , the subspace modeling has never been used previously to address few-shot learning problems . This makes our paper distinct and novel as compared to former studies ( e.g. , Vinyals et al . ( 2016 ) ; Snell et al . ( 2017 ) ) . Fig . 1 provides a conceptual illustration of our approach . We empirically observed that embeddings tailored towards capturing the structure of each class through low-dimensional affine subspaces could lead to discriminative models . Interestingly , such models can be built with minimum overheads and without opting for advanced methods in subspace creation ( e.g. , such as the notion of sparsity ) . Our conjecture here is that subspaces are less sensitive to perturbations such as outliers and noise compared to other embedding techniques for few-shot learning as illustratively shown in Fig . 2 . Our contributions in this work are : i. Few-shot learning is formulated as an embedding problem through subspaces . We rely on a well-established concept stating that samples of a class ( and hence variations such as pose and illumination ) can be effectively captured by a low-dimensional affine space . ii . Adaptation from few-shot learning to semi-supervised learning is performed with a refinement through soft-assignment . The robustness of such a model is shown in our experiments . iii . We also introduce an evaluation mechanism to assess the generalization ability over unseen classes during test time . 2 RELATED WORK . In this section , we briefly review the literature on few-shot learning and subspace clustering . Fewshot learning was originally introduced to imitate human learning capabilities in classification . Some of the early works made use of generative models and similarity learning to capture the variation within parts and geometric configurations of objects ( Fei-Fei et al . ( 2006 ) ; Lake et al . ( 2015 ) ; Torralba et al . ( 2007 ) ) . As of late , few-shot problems are mainly addressed through meta-learning techniques . For example , in Koch et al . ( 2015 ) ; Vinyals et al . ( 2016 ) ; Snell et al . ( 2017 ) ; Garcia & Bruna ( 2018 ) , the problem of few-shot learning is formulated as non-linear embedding or representation learning . The closest approach to our proposed work is the Prototypical Networks ( PN hereafter ) model ( Snell et al . ( 2017 ) ) . It uses the random choice of episode images which form the prototype center per class . In addition , recent metric learning approaches use complex architectures and pipelines in convolutional networks of Gidaris & Komodakis ( 2018 ) and Wang et al . ( 2018 ) . In contrast , our projective subspace concept is extremely simple by design as it constitutes just a single layer of the network . Another common meta-learning approach to few-shot classification uses meta-learning and manipulates gradient updates to train the model parameters . Ravi & Larochelle ( 2017 ) utilized Long-Short Term Memory ( LSTM ) -based learner to optimize the model parameters and their gradients . Moreover , the Model Agnostic Meta-Learner ( MAML ) proposed by Finn et al . ( 2017 ) used gradients per task as well as a meta-gradient from combined tasks , and , as a result , it outperformed LSTM-based learner . Mishra et al . ( 2018 ) employed ResNets to model few-shot classification as a temporal solution with aggregation , thus , in meta-learning context , their model can refer to the past experience . Our idea is based on the assumption that samples from any class in the support set form a lowdimensional subspace . As we will show , to train PSN , backpropagation through Singular Value Decomposition ( SVD ) is required . Backpropagation through matrix decomposition such as SVD is a well-studied problem Ionescu et al . ( 2015 ) with applications ranging from semantic segmentation to classification and visual recognition Ionescu et al . ( 2015 ) ; Li et al . ( 2017a ) ; Gou et al . ( 2018 ) . 3 PROBLEM SET-UP . We start by defining the terminology used in few-shot learning . The problem of N -way K-shot classification ( e.g. , 5-way 1-shot ) is defined as classifying queries belonging to N classes by seeing only K samples from each class . For example , in 5-way 1-shot setting , the model needs to identify a query among five classes by seeing only one sample from each class . To obtain a reliable model , so-called episodes are used in training . An episode Ti consists of two sets , the support set S and the query setQ . The system is then trained by minimizing a classification loss over episodes , simulating the scenarios it will encounter at test time . This episode setting is the same as proposed by Vinyals et al . ( 2016 ) . A related problem is semi-supervised few-shot setting learning where unlabeled data is provided to the model . In the literature , various configurations are considered for semi-supervised few-shot learning ( e.g. , Garcia & Bruna ( 2018 ) ; Boney & Ilin ( 2017 ) ; Ren et al . ( 2018 ) ) . In this work , we follow the challenging protocol in Ren et al . ( 2018 ) where so-called distractors are introduced . Here , an episode includes the support set S , query setQ , and unlabeled setR . The support ( labeled ) S and query Q sets are configured as in few-shot learning . Additionally , an unlabeled set R is provided to assist the classification task within an episode . In the unlabeled set , there are examples from two different sources : the support classes and the distractor classes . As the name implies , examples from distractor classes are irrelevant to the classification task and represent classes outside the support set . 4 PROJECTIVE SUBSPACE NETWORKS In what follows , we introduce our PSN approach . The whole framework is trained end-to-end ( see Fig . 3 ) which enables PSN to be used with various deep architectures . As an example , in § 5 , we use PSN on top of the WideResNets ( Zagoruyko & Komodakis ( 2016 ) ) for few-shot classification . We start by introducing our notations for the ( N -way , K-shot ) few-shot learning . Each episode or task Ti is composed of the support set S = { ( x1,1 , c1,1 ) , ( x1,2 , c1,2 ) , · · · , ( xN , K , cN , K ) } and the query set Q = { q1 , · · · , qN×M } . Here , xi , j denotes the j-th sample from class i and ci , j ∈ { 1 , · · · , N } . In the semi-supervised setting , there is an unlabeled set R = { r1 , .. , rU } within an episode . We propose to model points by subspaces { Zi } Ni=1 . Each subspace Zi has a basis represented by RD×n 3 Pi = [ p1 , · · · , pn ] ; n ≤ D , with P > i Pi = In . 4.1 PSN FOR FEW-SHOT CLASSIFICATION . Let fΘ : X → RD be a mapping from the input space X to some D-dimensional representation realized by a neural network . Our goal is to learn Θ , i.e. , the embedding function in a way that the resulting space is suitable for subspace representation . For simplicity , we assume that every class in an episode can be described by just one subspace . Extension to multiple subspaces per class is straightforward though . Define µk as the mean of class k in the embedded space . That is , µk = 1 K ∑ i , ci=k fΘ ( xi ) . ( 1 ) A basis for the subspace representing class k can be obtained by Singular Value Decomposition ( SVD ) . To be specific , we define Xk = [ xk,1 − µk , · · · , xk , K − µk ] . Applying truncated SVD on Xk provides us with Pk . We emphasize that more involved techniques to obtain robust subspaces from Xk can potentially improve the PSN . Nevertheless , our goal is to assess whether the concept of subspace modeling for few-shot learning is justified or not and thus we opt for truncated SVD in our implementation . Now a query qj can be projected onto Pk which yields : yj , k = P > k ( fΘ ( qj ) − µk ) . ( 2 ) The distance from the query to Pk is : dj , k = ‖fΘ ( qj ) − µk − Pkyj , k‖ . ( 3 ) We define the probability of the query assigned to class k using a softmax function as : pΘ ( c = k|qj ) = exp ( − d2j , k ) ∑ k′ exp ( − d2j , k′ ) . ( 4 ) Now , we can minimize the negative log of Eqn . 4 to obtain Θ . To train the whole framework , backpropagation through SVD is required which is available in modern deep learning packages such as PyTorch ( Paszke et al . ( 2017 ) ) . Algorithm 1 explains the steps of training our PSN . The code will be released online on Github1 . Algorithm 1 Train Projective Subspace Networks Input : Each episode Ti with S = { ( x1,1 , c1,1 ) , · · · , ( xN , K , cN , K ) } and Q = { q1 , ... , qN×M } 1 : Θ0 ← random initialization 2 : for t in { T1 , ... , TNT } do 3 : Lt ← 0 4 : for k in { 1 , ... , N } do 5 : X ← Sk . Get examples in the support set from class k 6 : µk ← 1K ∑ x∈X fΘ ( x ) . Mean from the support set 7 : µk ← Kµk + ∑ imifΘ ( ri ) K + ∑ imi . Refined mean ( only semi-supervised learning ) 8 : X̃ ← [ xi − µk , ... , xK − µk ] 9 : [ U , Σ , V > ] ← SVD ( X̃ ) . Matrix factorization using SVD 10 : Pk ← U1 , ... , n . Truncate the matrix 11 : for qj in Qk do 12 : yj , k ← P > k ( fΘ ( qj ) − µk ) . Query projection 13 : dj , k ← ‖fΘ ( qj ) − µk − Pkyj , k‖ . Distance calculation 14 : pj , k ← exp ( −d2j , k ) ∑ k′ exp ( −d2j , k′ ) . Softmax on distance scores 15 : end for 16 : end for 17 : Lt ← 1N2M ∑ k ∑ j − log ( pj , k ) 18 : Update Θ using ∇ΘLt 19 : end for

Summary: This paper proposes a Projective Subspace Network (PSN) for few-shot learning. The PSN represents each support set of classes as a subspace obtained by SVD. Then the method calculates distances between a query and classes by the projection error to the subspace. Instead of using the prototype of the class center, the subspace representation is more robust to outliers. Though the contribution seems to be incremental, it is a reasonable improvement upon Matching Networks and Prototypical Networks.