Hyperspherical embedding for novel class classification

Deep neural networks proved to be useful to learn representations and perform classification on many different modalities of data . Traditional approaches work well on the closed set problem . For learning tasks involving novel classes , known as the open set problem , the metric learning approach has been proposed . However , while promising , common metric learning approaches require pairwise learning , which significantly increases training cost while adding additional challenges . In this paper we present a method in which the similarity of samples projected onto a feature space is enforced by a metric learning approach without requiring pairwise evaluation . We compare our approach against known methods in different datasets , achieving results up to 81 % more accurate . 1 INTRODUCTION . Humans have the ability to identify many different types of objects , Fields ( 2016 ) . Even when we are not able to name a certain object , we can tell it ’ s differences from a second object , which contributes to the identification of objects we have never seen before and group them into classes based on prior knowledge . Metric learning KAYA & BİLGE ( 2019 ) is a well adopted approach that identifies novel classes without fine tuning a model on these classes . The approach applies an optimization strategy , which guarantees that the classes a model has seen during optimization form disjoint clusters on the latent space according to a certain metric distance . Some common approaches that use this strategy are : the triplet loss Schroff et al . ( 2015 ) ; constrative loss Hadsell et al . ( 2006 ) ; prototypical networks Snell et al . ( 2017 ) ; constellation loss Medela & Picon ( 2020 ) ; and matching networks Vinyals et al . ( 2016 ) , here referred to as distance based learners . Another approach in metric learning is called similarity learning , where the model receives pairs of inputs and learns that they are similar if they belong to the same class or dissimilar otherwise , as discussed in Sung et al . ( 2018 ) . During inference on novel classes , distance based learners use the distance between labeled points of the novel class the model was not optimized upon to obtain a representation in the latent space for the novel class and then calculate the distance between new points and each class representation . When considering similarity based learners , a similarity score is calculated between every ( class , query ) point pair in order to find the most similar pair . However while enforcing metric properties on the latent space leverages the model knowledge to novel classes , it requires pairwise learning , which limits the scalability of such approaches given the amount of possible pairs . In this paper we take into account the normalized softmax loss function ( NSL ) , proposed by Wang et al . ( 2018 ) , and present how it enforces a latent space that obeys the cosine similarity . Based on this , we then present a methodology to apply the NSL to the novel classes classification problem . Considering a trained artificial neural network , we add a new neuron to it ’ s last layer and infer the weights that connect the penultimate layer of the network to this neuron . The connection and the new neuron are used to classify a novel class by using few labeled samples of it . Our approach for the open set problem allows us to classify new classes without fine-tuning the model . Instead , we use the same network parameters the model was optimized upon to classify its seen classes and only add a new neuron along with its inferred connection for new classes . We evaluate state-of-the-art approaches to solve the open set problem against our proposed approach , both in the disjoint and joint scenarios , for different datasets . The experimental results show that our approach outperforms other metric learning strategies and additionally , induces a more scalable training process , as it does not require pairwise learning , leveraging the open set problem technique to deal with large datasets . The remainder of this paper is structured as follows . First , it presents some theoretical background at section 2 . Our methodology and how to classify new classes is described in section 3 . Next , we present the results on the joint and disjoint open set problem in section 4 . Moreover , we present the use of the NSL approach in a more complex dataset in the field of botany , in section 4.4 . We compare our methods to incremental learning in section 4.5 . We present related work and lastly , we conclude in section 6 . 2 PRELIMINARIES . We are given a training dataset ( xi , yi ) i∈ { 1 , ... , n } where , for all i , the input xi belongs to an input space X ⊂ Rd , e.g . the space of images , and the output yi to an output space Y = { 1 , 2 , . . . , K } , the set of class labels , whereK is the number of classes . Based on this training set , the aim is to find a classifier h : X → Y which produces a single prediction for each input and generalizes well on unseen samples x ∈ X . When this classifier is a deep neural network , h can typically be expressed as : h ( x ) = max k η̂k ( x ) where η̂ ( x ) = ( η̂1 ( x ) , . . . , η̂K ( x ) ) is the vector of the estimated class probabilities computed as : η̂ ( x ) = ψ ( φ ( x ) ) with φ : X → RM being a succession of layers allowing to compute anM -dimensional feature vector representation φ ( x ) for any input image x ∈ X , and ψ : RM → RK being the final classification function , typically composed of a fully connected layer followed by a softmax activation function : ψk ( z ) = ewkz+bk∑K j=1 e wjz+bj ( 1 ) 2.1 THE OPEN SET PROBLEM . A classification problem can be formulated as a closed set or open set problem . In the closed set problem context , the optimization process trains a model to learn features that can classify the samples into classes present in the training set . The approach does not require the identification of classes not present in the training set . This is commonly tackled using the Softmax-cross-entropy loss He et al . ( 2016 ) , Simonyan & Zisserman ( 2015 ) , Szegedy et al . ( 2015 ) . In contrast , in the open set problem we are interested in not only identifying the classes present in the training set , but also to be able to use the model to classify new classes by exploiting properties in the latent space yielded during optimization . 2.2 CLASSIFYING NEW CLASSES . When tackling the open set problem , we are interested in optimizing models in which the full knowledge the network obtains during optimization can be exploited for classes outside of the training set . The usual softmax cross-entropy approach lacks the ability to extract features that obey this property , as the weights between the penultimate layer and the classification layer w are as important as the representation in the latent space of the penultimate layer z as seen in Eq . 1 , and the former is undefined for novel classes . Usual approaches for classifying novel classes are explored in metric learning . Metric learning strategies are interesting as novel classes can be identified . However , current strategies based on pairwise learning can be costly to optimize . We discuss in this paper a strategy to remove pairwise learning and still be able to define novel classes for a model . 2.3 NORMALIZED SOFTMAX LOSS . Proposed in Wang et al . ( 2018 ) , the NSL ( Normalized Softmax Loss ) is a modification of the softmax loss that enforces a cosine similarity metric between classes on the latent space . It enforces the features z that are projected into the latent space to be contained in a M dimensional hypersphere ( M > 3 ) where each region of the sphere contains features belonging to a certain class . If we look again at the classical softmax equation ( Eq . 1 ) , the constraints induced by NSL are : { bk = 0 , ∀k ‖wk‖ = 1 , ∀k ‖z‖ = ‖φ ( x ) ‖ = S , ∀x ( 2 ) and finally η̂k ( x ) = ψk ( φ ( x ) ) = ewkφ ( x ) ∑K j=1 e wjφ ( x ) = eS.cos ( wk , φ ( x ) ) ∑K j=1 e S.cos ( wj , φ ( x ) ) where cos ( u , v ) = u.v/ ( ‖u‖.‖v‖ ) is the cosine similarity , i.e . the cosinus of the angle between two vectors u and v. Note that the hyper-parameter S acts as a temperature of the normalized softmax allowing to control the degree of concentration of the output probabilities η̂k ( x ) . A geometrical representation that shows the relationship between the weights and the feature vectors obtained with NSL is shown in Figure 2 . One can see that the barycenter of the feature vector is aligned with it ’ s corresponding class weights . 3 PROPOSED METHODOLOGY . In this paper we aim to compare pairwise strategies , commonly used in metric learning , against the normalized softmax loss approach for the open set problem . In this manner we consider both the problem where during inference seen and unseen classes are disjoint , as well as the scenario where the model must identify both the seen and unseen classes together . More formally , once the network has been trained , we would like to extend the output space to a new set of classes Y∗ = { K + 1 , . . . , K +K∗ } for which we have only one or very few samples ( x∗i , y ∗ i ) i∈ { 1 , ... , n∗ } . In particular , we would like to obtain a new classifier h ∗ : X → Y∗ ( disjoint scenario ) or a new classifier h′ : X → Y ⋃ Y∗ ( joint scenario ) . Note that , whatever the scenario , we consider that the function φ is fixed as well as the pre-trained weights of the seen classes wk , ∀k ∈ { 1 , . . . , K } . 3.1 CLASSIFYING NEW CLASSES VIA NSL . Given that the function φ and the weights wk of the seen classes are fixed , our objective is reduced to optimizing the weights w∗k , ∀k ∈ { 1 , . . . , K∗ } of the unseen classes . Using the cross-entropy as the objective function , this can be expressed as : argmin w∗1 , ... , w ∗ K∗ n∗∑ i=1 −log ( η̂y∗i ( x ∗ i ) ) argmin w∗1 , ... , w ∗ K∗ n∗∑ i=1 −log e w∗y∗ i φ ( x∗i ) ∑K j=1 e wjφ ( x∗i ) + ∑K∗ j=1 e w∗jφ ( x ∗ i ) In the particular case where we have only one new class ( i.e . K∗ = 1 ) , this simplifies to : argmax w∗1 n∗∑ i=1 w∗1φ ( x ∗ i ) = argmax w∗1 w∗1 n∗∑ i=1 φ ( x∗i ) which leads , with the constraints of Eq . 2 , to : w∗1 = 1 n∗ n∗∑ i=1 φ ( x∗i ) ‖φ ( x∗i ) ‖ = 1 S.n∗ n∗∑ i=1 φ ( x∗i ) ( 3 ) The weight w∗1 of a new class can thus simply be computed by averaging the feature vectors of the images x∗i of the new class . This simple theoretical result does not hold anymore when there is more than one novel classes ( i.e . when K∗ > 1 ) . However , as we we will see in our experiments , using this estimation procedure for other new classes provides a good approximation of the exact optimal weights and is quite effective in practice . More formally , we propose to estimate the weights w∗k of each of K∗ new classes as : w∗k = 1 S ∑n∗ i=1 φ ( x ∗ i ) 1 ( y ∗ i = K + k ) ∑n∗ i=1 1 ( y ∗ i = K + k ) ( 4 ) In the joint scenario , we are interested in a classifier on both the seen classes and the new classes . This can be expressed as : hjoint ( x ) = n∗∑ i=1 max k ew ∗ kφ ( x ∗ i ) ∑K j=1 e wjφ ( x∗i ) + ∑K∗ j=1 e w∗jφ ( x ∗ i ) ( 5 ) where the wj and φ ( ) are pre-trained on the seen classes and the new weights w∗k are computed with Eq . 4 . In the disjoint scenario , we are interested in a classifier on the new classes only ( in a transfer learning way ) : hdisjoint ( x ) = n∗∑ i=1 max k ew ∗ kφ ( x ∗ i ) ∑K∗ j=1 e w∗jφ ( x ∗ i ) ( 6 ) where φ ( ) is pre-trained on the seen classes and the new weights w∗j are computed with Equ . 4 . A dataflow depicting our approach to infer the weights for novel classes is presented in Figure 1 .

Summary: This paper is about classification of images in an open set setting. Data coming from new classes are introduced to the network after training on data from a fixed set of known classes. The goal is to be able to correctly classify the old and new classes either jointly or not. This paper proposes a method for handling new classes by increasing the size of the classifier weights for each new classes. The network is trained using the Normalized softmax loss and new classifier elements are added by finding the center of mass of the data coming from the new set of class. The method performs on FASHION MNIST, CIFAR and Plantnet datasets.