One-way prototypical networks

Few-shot models have become a popular topic of research in the past years . They offer the possibility to determine class belongings for unseen examples using just a handful of examples for each class . Such models are trained on a wide range of classes and their respective examples , learning a decision metric in the process . Types of few-shot models include matching networks and prototypical networks . We show a new way of training prototypical few-shot models for just a single class . These models have the ability to predict the likelihood of an unseen query belonging to a group of examples without any given counterexamples . The difficulty here lies in the fact that no relative distance to other classes can be calculated via softmax . We solve this problem by introducing a “ null class ” centered around zero , and enforcing centering with batch normalization . Trained on the commonly used Omniglot data set , we obtain a classification accuracy of .98 on the matched test set , and of .8 on unmatched MNIST data . On the more complex MiniImageNet data set , test accuracy is .8 . In addition , we propose a novel Gaussian layer for distance calculation in a prototypical network , which takes the support examples ’ distribution rather than just their centroid into account . This extension shows promising results when a higher number of support examples is available . 1 MOTIVATION . One- or few-shot learning is an active area of research that focuses on modeling semantic concepts with very little training data . Such models learn information about the general task of comparing queries to one or several support examples of the requested classes , which can then be applied to unseen classes . A large amount of data is still necessary for training , but this data does not need to belong exactly to the domains that may become interesting in the future , but rather to a variety of similar problems . This notion is inspired by human learning ; humans can usually generalize to a new concept from just one or very few examples , using their prior knowledge of the world and related concepts . A special case occurs when we do not need to distinguish between multiple concepts , but rather want to enable models to recognize a single new concept . In many practical scenarios , we will often be interested in just one novel class , rather than multiple at once . Application examples include detecting news or social media texts about a certain events ; detecting one type of unseen object ( e.g . a face ) in a dataset ; verification of unseen fingerprints ; or outlier detection within an unseen class . In existing few-shot methods , this is commonly translated into a two-class problem with the classes “ belonging to the concept ” and “ not belonging to the concept ” . As these two classes have very different types of feature distributions , this is not the most elegant solution and poses problems , especially with regards to training data selection . It also requires effort for modeling a class ( the negative one ) that is not of interest . In this paper , we present an addition to few-shot learning approaches that enables generalization to a single class in order to determine whether queries belong to this ( previously unseen ) concept or not . This is implemented by introducing a “ null ” ( or “ garbage ” ) class which does not need to trained to the prototypical network approach . Application of this novel model type is demonstrated on the commonly used Omniglot and MiniImageNet data sets . 2 RELATED WORK . 2.1 ONE-CLASS CLASSIFICATION . The problem of one-class classification has been a topic of research for decades . It has also been interpreted as outlier detection , anomaly detection , novelty detection , and concept learning . Comprehensive overviews of “ traditional ” methods for solving this task are given in Khan & Madden ( 2014 ) and Chandola et al . ( 2009 ) . Chandola et al . ( 2009 ) identify several different fundamental techniques , e.g . statistics , nearest-neighbour methods , clustering , and classification . Strategies can be derived from research areas such as data mining , information theory , and machine learning . Within the machine learning domain , a common solution relies on One-class Support Vector Machines ( OSVMs ) [ Schölkopf et al . ( 1999 ) ; Tax & Duin ( 1999 ) ] , although earlier-stage neural networks were also utilized [ Hawkins et al . ( 2002 ) ] . In recent years , various deep learning approaches have been suggested as well . Auto-encoders are frequently used for dimensionality reduction ( corresponding to a feature extraction ) of the input data in this context . Some methods then directly calculate the reconstruction error to determine whether new queries belong to the same class or are outliers ( e.g . Sakurada & Yairi ( 2014 ) ; Chen et al . ( 2015 ) ) . Others add a subsequent classification network ( e.g . Morton & Griffin ( 2016 ) or Erfani et al . ( 2016 ) , where the auto-encoder is replaced with a Deep Belief Network ) . Ruff et al . ( 2018 ) take inspiration from OSVM approaches and train CNNs jointly with a classification SVM . Lately , adversarial approaches have seen attention in this domain as well , e.g . in Schlegl et al . ( 2017 ) and Sabokrou et al . ( 2018 ) , where they are used in a similar fashion as the previous auto-encoders . Direct training on a single annotated class is not implemented as frequently as the combination of automatic feature extraction and distance ( or reconstruction error ) calculation . Oza & Patel ( 2019 ) demonstrates a strategy for training a CNN on positive examples and artificial negative examples generated from Gaussian distributions , while Perera & Patel ( 2018 ) shows a method based on automatic feature extraction in combination with specialized loss functions and template matching . A broad overview of deep learning techniques from the anomaly detection perspective is provided in Chalapathy & Chawla ( 2019 ) . All of these approaches rely on the availability of data from the expected class for training . What is missing is a type of model that can determine whether a query belongs to a class from a small number of examples that are provided at evaluation time without the need to re-train . Such a classification is possible with few-shot learning . 2.2 FEW-SHOT LEARNING . The concept of one- or few-shot learning is motivated by human learning : If humans are presented with a new class of objects , they do not require more than a few examples to be able to match new instances to this class ( often , just one supporting example is sufficient ) . This ability is , among other factors , informed by general world experience , which teaches humans to set boundaries between object classes . Matching networks , which implement this idea , were first introduced by Vinyals et al . ( 2016 ) . Training is performed on sets of support examples for a subset of possible classes plus a query example belonging to one of the classes in so-called episodes . These episodes are generated for a wide range of class permutations . In the one-shot case , only one support example is given for each class , while there are multiple ones in few-shot models . Such a network has two input branches : One for the support examples , and one for the query example . In both branches , the inputs are run through a sub-network which performs an embedding . The embedding networks may or may not share their weights . The query ’ s resulting embedding is then compared against the supports ’ resulting embeddings using a pre-defined distance metric . This results in likelihood measures for each input class , commonly implemented with a softmax . The over-all comparison metric for the task is learned implicitly as the network learns an appropriate embedding . In the few-shot case , multiple support examples per class are treated independently of each other throughout the network . Example-related likelihoods are then summed up to obtain class likelihoods at the output . In this way , matching networks essentially implement a weighted nearest-neigbor classification . As a further development , Snell et al . ( 2017 ) introduced prototypical networks . Instead of treating all of a class ’ support examples independently , they compute a so-called “ prototype ” of the class after the embedding network . This prototype is commonly the mean vector ( or centroid ) of the class ’ s embedded support examples.When the squared Euclidean distance is used as the metric , this implements a linear classifier rather than the nearest neighbor classifier produced by matching networks . Finn et al . ( 2017 ) goes one step further and introduces model-agnostic meta learning ( MAML ) : An algorithm to train models that can be easily finetuned to new tasks with very few examples . This can also be applied to other few-shot methods such , as matching networks . When applying few-shot learning to a binary problem , support examples for two classes are required : Positive examples of the class of interest , and random other examples . This amounts to a workaround , as there are no two comparable classes . Instead , one of them merely consists of counterexamples ( a sort of “ garbage ” class ) . Examples in this class will have a much wider feature distribution . This makes selecting random examples difficult , as they should still be representative of this distribution , particularly when only a small number of support examples is used . It also means that computational power and time must be dedicated to a class that is not actually of interest , and as such is not very elegant . Siamese networks [ Koch et al . ( 2015 ) ] can be seen as an interesting way to solve this problem , as they implement a decision about whether two unseen examples belong to the same class or different ones . However , they are not able to exploit more than one support example per class like prototypical networks can . 3 PROPOSED APPROACH . 3.1 ONE-WAY PROTOTYPICAL MODELS . Motivated by the counterexample issue , we take the described approach one step further : We construct a prototypical network that only requires positive support examples , from which it generates an internal embedding of the whole class . The query embedding is then compared to this , and judged to belong to the same class or not . We call this type of network a “ one-way prototypical network ” . The sub-network for generating the internal embeddings is implemented just as in regular prototypical networks with shared weights between the support and query branches . An illustration is shown in figure 1a . This approach presents a challenge for the distance metric : Because Cosine or Euclidean distances are not bounded , we can not set a threshold for class belonging . In order to do this , we need a basis for comparison . We implement this by introducing a “ null class ” that models the whole space of possible examples without explicitly being given those examples ( as in the two-class implementation mentioned above ) . Queries are judged on whether they are more probable to belong to the ( positive ) class we are interested in rather than the null class . The question then becomes how to construct this class . As explained above , prototypical networks model classes by calculating the centroid of the class ’ s support examples . As the model includes batch normalization layers , we can assume that the centroid of the whole latent space will be close to the zero point in all embedding dimensions . A query ’ s distance to the centroid of the positive class is therefore compared to its norm ( i.e . its distance to the zero centroid ) to determine whether it is more likely to belong to the positive class or to the null class encompassing everything . Figure 1b shows a visualization where the null centroid is denoted as c0 . A similar idea was presented in Ren et al . ( 2018 ) , which suggests the use of unlabeled data to improve prototypes . The null ( or “ distractor ” ) class here is only used to identify examples not belonging to any of the trained classes , rather than training the model to explicitly distinguish between a single given class and the null class . As in regular prototypical models , Euclidean distance is used . One issue that arises is that in the state-of-the-art prototypical models , batch normalization is not performed at the end of the convolution blocks of the embedding network , but rather after the convolution layer and before the ( ReLU ) non-linearity and pooling layers . The final block is therefore not quite batch-normalized . As there is some debate about the ordering of these layers anyway [ Ioffe & Szegedy ( 2015 ) ; Mishkin ] , we propose reordering the block layout to a structure of : Convolution - ( Leaky ) ReLU - Max Pooling - Batch Normalization . An alternative to the null class solution would be a simple thresholding of the class likelihood . However , preliminary experiments showed that this threshold is very problem- and concept-dependent , and thus does not generalize well to unseen episodes .

Summary: Authors consider the 1-way few shot classification task. Argue that modeling it as a 2-way with a random negative sample is not efficient. Propose a novel technique applicable for prototypical networks. Their proposal is to use 0 as the prototype of null class. So the distance to the prototype is compared against the norm of the query embedding. They also propose modeling the distribution and not just the centroid using a multivariate gaussian. But in practice there is no benefit in doing so. Therefore, the main contribution is proposing to compare against norm of the embedding rather than a prototype for random negative samples. The benefit of this proposal decreases by more shots. Probably because the prototype of 20 random images is 0 anyway.