Meta-OLE: Meta-learned Orthogonal Low-Rank Embedding

We introduce Meta-OLE , a new geometry-regularized method for fast adaptation to novel tasks in few-shot image classification . The proposed method learns to adapt for each few-shot classification task a feature space with simultaneous inter-class orthogonality and intra-class low-rankness . Specifically , a deep feature extractor is trained by explicitly imposing orthogonal low-rank subspace structures among features corresponding to different classes within a given task . To adapt to novel tasks with unseen categories , we further meta-learn a light-weight transformation to enhance the inter-class margins . As an additional benefit , this light-weight transformation lets us exploit the query data for label propagation from labeled to unlabeled data without any auxiliary network components . The explicitly geometry-regularized feature subspaces allow the classifiers on novel tasks to be inferred in a closed form , with an adaptive subspace truncation that selectively discards non-discriminative dimensions . We perform experiments on standard few-shot image classification tasks , and observe performance superior to state-of-the-art meta-learning methods . 1 INTRODUCTION . Meta learning , also referred to as learning to learn , aims at acquiring knowledge from a distribution of tasks , and learn to quickly solve novel tasks sampled from the same or similar underlying task distribution . Meta learning is extensively studied under the context of few-shot learning ( FSL ) , and realized by models that adapt efficiently when given only a few labeled samples of novel tasks . The research is mainly driven by how to design the adaptation for acquiring task-specified models efficiently and robustly . Prototypical networks ( ProtoNets ) ( Snell et al. , 2017 ) adapt to new tasks by computing the prototype of each class simply as the average of feature vectors , with all the network parameters shared across tasks . MAML ( Finn et al. , 2017 ) adapts to new tasks by a few iterations of gradient descent , and this approach has inspired many subsequent methods ( Antoniou et al. , 2018 ; Finn et al. , 2018 ; Nichol et al. , 2018 ; Yoon et al. , 2018 ) . The adaptation of the entire network makes it hard to be scaled to large networks , and many recent efforts focus on adapting the last classification layer only ( Gordon et al. , 2019 ; Bertinetto et al. , 2019 ) , while assuming a universal feature extractor that is shared across all tasks . In this paper , we attempt to attack few-shot image classification from a new perspective of geometry regularization of the feature space . As observed in ( Lezama et al. , 2018 ) , training deep networks with softmax and cross-entropy loss does not simultaneously enforce intra-class similarity and interclass margins . On the other hand , encouraging features to be in a low-rank subspace in each class as well as orthogonal across classes can significantly improve the robustness of deep classification networks . While such explicit orthogonal low-rank geometry regularization has been proved successful in classical classification tasks Lezama et al . ( 2018 ) ; Qiu et al . ( 2018 ) ; Lezama et al . ( 2017 ) , it remains highly non-trivial to extend this geometry regularization approach to tasks with novel classes involved at the testing stage . In few-shot image classification , further challenges arise from the demand of robust generalization to novel unseen classes . As we will show in this work , large class margins resulting from explicit geometric regularization can potentially allow novel knowledge to be represented by a composition of existing knowledge , and can reduce interference across classes . An illustration is presented in Figure 1 . Under review as a conference paper at ICLR 2022 Novel classes Seen classes Motivated by the maximal-margin feature space geometry , we introduce meta-learned orthogonal low-rank embedding ( Meta-OLE ) to combine the simplicity of the prototype based methods and the adaptivity of the parameter adaptation based methods . Specifically , we encourage an orthogonal low-rank structure to the feature space across classes . Thus feature vectors of the same class reside in a subspace with imposed low-rankness , while subspaces across classes are encouraged to be as orthogonal as possible . While imposing geometry-regularization to the feature space over seen classes has been investigated Lezama et al . ( 2018 ; 2017 ) ; Wen et al . ( 2016 ) , the induced feature extractor does not guarantee to generalize well to novel object classes that are unseen during training . To extend an orthogonal low-rank embedding to a few-shot learning scenario , we introduce a metalearning framework with a light-weight adaptive orthogonal low-rank transformation that is able to adapt efficiently to novel classes with very few examples . We then show that , given the imposed lowrank orthogonal geometry , the final classification of query samples can be performed by subspace projections , where the projection matrices are directly inferred from the few labeled examples in a closed form . And we show that , to adaptively adjust the dimension of the projections based on the compactness of the feature subspace , the robustness of the classifier to outlier examples can be further improved . The closed-form inference of class labels allows unlabeled samples to be easily involved in the learning of the adaptive orthogonal low-rank transformation for label propagation , and improved performance is observed without any auxiliary parametric components to infer the pseudo labels . Despite being simple and geometry-motivated , the proposed method achieves on public FSL datasets superior performance to state-of-the-art methods that often involve more sophisticated components . In summary , our contributions are as follows : • We propose to impose low-rank orthogonal geometry in feature space for few-shot learning . • We introduce meta-learned adaptive orthogonal low-rank transformations for efficient adaptations to novel tasks with unseen classes . • Geometry-motivated classifier based on subspace projections with adaptive dimension selection is introduced for fast and robust class inference . • The effectiveness of the proposed Meta-OLE is validated with extensive experiments on few-shot image classification . 2 METHOD . In this section , we start with the basic formulation of FSL , and then we introduce each of the components of the proposed Meta-OLE framework in detail . 2.1 PRELIMINARY . Scalars , vectors and tensors are denoted as lower-case , bold lower-case , and bold upper-case letters , e.g. , n , x , X , respectively . For example , we denote an image as a vector x , and use X = [ x1 , x2 , . . . ] to denote a collection of images . Xc denotes the collection of images within X with label c. Few-shot image classification with episodic training . A few-shot learning ( FSL ) task is usually defined as a K-way N -shot learning problem , where N is usually a small number , e.g. , N = 5 . FSL with meta-learning is usually formulated as a series of episodic training . Typically , in each episode , one FSL task is generated by first sampling K categories from the training data , each of which contains N samples to form the support set St = { x1 , . . . , xK×N } . An adaptation to the FSL model is then performed on St , by , e.g. , computing prototypes ( Snell et al. , 2017 ) , or updating the network parameters ( Finn et al. , 2017 ) . After adaptation , samples from the same categories of each episodic , referred as the query set Qt = { x′1 , . . . , x′K×M } , are sampled to evaluate the updated model , and the error is propagated back to update the parameters . Orthogonal low-rank embedding . The idea of learning a linear transformation to recover the orthogonal low-dimensional intrinsic structures in data is originally proposed in ( Qiu & Sapiro , 2015 ) . In ( Qiu & Sapiro , 2015 ) , a linear transformation is learned to restore a low-rank structure for data from the same subspace , and , at the same time , force a maximally separated structure for data from different subspaces . This idea is further generalized to deep learning in ( Lezama et al. , 2018 ) , where orthogonal low-rank embedding ( OLE ) is introduced as a regularization term to the training of deep classification networks for improved performance . Given a collection of N samples in Rd for a K-way classification task , the transformation in ( Qiu & Sapiro , 2015 ) is computed as min T : Rd→Rd K∑ c=1 ||TXc||∗ − ||TX||∗ , ( 1 ) where || · ||∗ denotes the nuclear norm and serves as a convex lower bound of the rank function on the unit ball in the operator norm , and T is a linear transformation to be learned . Specifically , the first term in ( 1 ) encourages the transformed representations within each of the K classes to reside in a low-rank subspace . The second term in ( 1 ) promotes orthogonality of the subspaces across classes . Theorem 1 ( Qiu & Sapiro , 2015 ) ||A , B||∗ ≤ ||A||∗ + ||B||∗ , with equality satisfied if and only if the column spaces of A and B are orthogonal . According to Theorem 1 , the cost value of ( 1 ) is always nonnegative . Moreover , it achieves the minimum at zero if and only if different classes become orthogonal after the transform . See ( Qiu & Sapiro , 2015 ) for more details of this formulation . Based on ( 1 ) , ( Lezama et al. , 2018 ) introduces a generalization to the training of deep neural networks , where an orthogonal low-rank embedding ( OLE ) loss is proposed as a regularization term to facilitate the learning of typical image classification networks with the cross-entropy loss . Specifically , the OLE loss is defined as LOLE = K∑ c=1 ||Zc||∗ − ||Z||∗ = K∑ c=1 ||Φ ( Xc ) ||∗ − ||Φ ( X ) ||∗ , ( 2 ) where Φ denotes the nonlinear transformation associated with a deep network . When Φ is a network that ends with a ReLU activation , training with the OLE loss leads to orthogonal inter-class subspaces , which is equivalent to explicitly pushing feature from different classes to the maximum cosine distance ( Lezama et al. , 2018 ) . 2.2 ADAPTIVE ORTHOGONAL LOW-RANK SUBSPACE PROJECTIONS . In the proposed method , samples within a FSL task is first mapped by a universal feature extractor , parametrized by a deep CNN in our setting , to a feature space , where we explicitly encourage lowrank subspace for each class as well as orthogonality among different classes . To adaptively promote higher intra-class compactness and inter-class orthogonality in each task , we then meta-learn a lightweight orthogonal low-rank transformation , that learns on the samples of each task and adapts the parameters for task-specific feature transformations . After learning the task-specific transformations for better low-rank orthogonal embeddings , the classification of unlabeled samples in query set is accomplished by adaptive subspace projections , with the projection matrices inferred directly from feature vectors of the support set in a closed form . We further introduce adaptive subspace projections by selecting principal dimensions for projections , and truncating non-discriminative dimensions for improved robustness . All components are detailed next . Universal feature extractor . Following standard practice , we train a universal feature extractor Φ , which is typically a CNN that is shared across tasks . Specifically , given an image x , the feature extractor maps it to a d-dimensional feature vector z ∈ Rd = Φ ( x ) . Different from the common practice that the feature extractor Φ is solely learned using the gradient propagated back from the error at tasks , we explicitly encourage orthogonal low-rank feature geometry across classes in each training episode . This is achieved by supervising the parameters in Φ with an OLE loss as : LOLE = K∑ c=1 ||Φ ( Xc ) ||∗ − ||Φ ( X ) ||∗ . ( 3 ) The same feature extraction is applied to both support set and query set to obtain the respective features Z = { zi } K×Ni=1 , Z′ = { z′j } K×M j=1 . Meta-learned adaptive transformation . Training a universal feature extractor Φ on limited training classes can hardly guarantee that the orthogonal low-rank feature geometry to be perfectly generalized to novel tasks in practice . To fully exploit the support set samples , we therefore propose to meta-learn a light weight adaptive transformation Ψ , parametrized by a tiny network with parameters θ , to adaptively transform the feature vectors of novel tasks for more compact intra-class and orthogonal inter-space subspaces . The parameters θ in Ψ is adapted to the new task given features from the support set iteratively , and an initialization is learned across the episodic training . Specifically , given a collection of the features from the support set Zt , we perform P iterations of parameter updating to θ , in order to project the features to a space that better presents the lowrank orthogonal geometry . At each iteration p , the transformed features are computed as Zp = Ψ ( Z , θp−1 ) , and the parameters θ are then updated by θp = θp−1 − β∇θ ( LOLE ( Zp ) ) , p = 1 , . . . , P , ( 4 ) where θ0 = θ , which serves as the universal parameter initialization for all tasks and to be optimized by tasks in episodic training . The updated parameters at the last iteration θP are returned as the final parameter of Ψ for a specific task , and transform both the support set and query set features , Z̃ = { z̃i = Ψ ( zi ; θP ) } K×Ni=1 , Z̃ ′ = { z̃′i = Ψ ( z′i ; θP ) } K×Mi=1 . ( 5 ) Adaptive subspace projections as classifiers . The imposed geometry-regularization of the orthogonal low-rank subspaces naturally leads to a subspace projection-based classifier that fully utilizes the obtained feature geometry . Since now the desirable features of each class reside in a low-rank subspace , the inference of class labels of query samples can then be effectively computed by projecting the feature vectors to each subspace of classes , and observing the norm of the projected vectors . Specifically , in K-way N -shot FSL , given a collection of support set feature vectors of a class Zc ∈ RK×d , the subspace , i.e . span ( Zc ) , can be directly inferred in a closed-form . Let Zc = UcΣcVc be the singular value decomposition of the Zc , the rows of Uc = [ b1 , . . . , bN ] form an orthogonal bases of Zc . Then any feature vector z in the query set can be projected onto the subspace span ( Zc ) by projc ( z ) = UcU > c z . In practice , based on the rank of the features , there can be non-discriminative dimensions contained in U ( in the most optimal case , only a single dimension is sufficient to represent the subspace of a class ) . Given the singular values Σc = [ s1 , . . . , sN ] , the non-discriminative dimensions are bases in Uc that correspond to low singular values . When the singular values Σc are sorted in a descending order , we can easily truncate non-discriminative dimensions by discarding bases in Uc whose associated singular values are lower than a threshold . In practice , we introduce a non-negative hyperparameter τ < 1.0 , and truncate bases in Uc with singular values lower than τ × s1 . Formally , we obtain the projections projc ( z ) = Uc [ 1 : r ] Uc [ 1 : r ] > z , ( 6 ) where sr ≥ τ × s1 , and sr+1 < τ × s1 . Ideally , each query sample will lie in the subspace of its class , thus the projection will mostly preserve the norm of the feature vector . For each zj , we then define the norm of the projected vector of zj to the subspace of class c as the unnormalized probability of sample xj belonging to class c , i.e. , ŷc ( z̃j ) = P ( z̃j ∈ c ) = exp ( ||projc ( z̃j ) ||2 ) ∑K c′=1 exp ( ||projc′ ( z̃j ) ||2 ) , ( 7 ) and standard cross-entropy loss is then used for computing and back-propagating errors . The advantages are further validated with real-world experiments in Section 3 . Leveraging query samples . While meta-learning the adaptive orthogonal low-rank transformation Ψ , we can further leverage the query samples with no labels . In each iteration of updating θ in Ψ , we can augment the data by assigning a pseudo label to each query sample . This can be efficiently implemented by projecting each query sample to each subspace inferred from the support set as in ( 6 ) , and finding the label maximum probability as in ( 7 ) . This achieves transductive learning in FSL without introducing any auxiliary components to the network . And we introduce a non-negative hyperparameter α < 1.0 as the weight of the contribution from the query set with pseudo labels . In this transductive setting , the updating to θ in Ψ becomes : θp = θp−1 − β∇θ ( LOLE ( Zp ) +αLOLE ( [ Zp , Z′p ] ) ) , ( 8 ) for p = 1 , . . . , P , and [ Zp , Z′p ] here denotes concatenation of the transformed support and query features . In summary , the proposed method consists of a universal feature extractor Φ for projecting highdimensional image inputs to feature vectors . Iterative updating to the adaptive orthogonal low-rank transformation Ψ adapts the model to the task at hand . The final classification is performed by subspace projections obtained in a closed form . x Φ−→ z Ψ ( · ; θ ) −−−−→ z̃ projc−−−→ yc . All the parameters are jointly updated by the loss L = Lsoftmax ( ŷ , y ) + λLOLE ( [ Z , Z′ ] ) , ( 9 ) with ŷ being the inferred label from ( 7 ) and y the true label . We summarize the overall training of our method in Appendix Algorithm 1 .

Summary: This paper proposes Meta-OLE, which imposes a low-rank orthogonal geometry in feature space for few-shot learning. An adaptive orthogonal low-rank transformation is introduced for efficient adaptations to novel tasks with unseen classes. Also, a geometry-motivated classifier based on subspace projections with adaptive dimension selection is presented for fast and robust class inference  The proposed method is composed as follows:   1.　Samples within a FSL task is first mapped by a universal feature extractor trained with the OLE loss.  (Actually, with softmax loss as shown in Eq.(9))  2.　To exploit the support set samples, the proposed method meta-learns a lightweight adaptive transformation $\Psi$ with OLE loss as shown in Eq.(4).    3.　The proposed method projects each data to subspace, and its lengths are used for classification as shown in Eq.(7). (But it is effective only using one leading direction for each class.)  4.　The proposed method uses pseudo labels for using query samples for training.