One-way prototypical networks

Few-shot models have become a popular topic of research in the past years . They offer the possibility to determine class belongings for unseen examples using just a handful of examples for each class . Such models are trained on a wide range of classes and their respective examples , learning a decision metric in the process . Types of few-shot models include matching networks and prototypical networks . We show a new way of training prototypical few-shot models for just a single class . These models have the ability to predict the likelihood of an unseen query belonging to a group of examples without any given counterexamples . The difficulty here lies in the fact that no relative distance to other classes can be calculated via softmax . We solve this problem by introducing a “ null class ” centered around zero , and enforcing centering with batch normalization . Trained on the commonly used Omniglot data set , we obtain a classification accuracy of .98 on the matched test set , and of .8 on unmatched MNIST data . On the more complex MiniImageNet data set , test accuracy is .8 . In addition , we propose a novel Gaussian layer for distance calculation in a prototypical network , which takes the support examples ’ distribution rather than just their centroid into account . This extension shows promising results when a higher number of support examples is available . 1 MOTIVATION . One- or few-shot learning is an active area of research that focuses on modeling semantic concepts with very little training data . Such models learn information about the general task of comparing queries to one or several support examples of the requested classes , which can then be applied to unseen classes . A large amount of data is still necessary for training , but this data does not need to belong exactly to the domains that may become interesting in the future , but rather to a variety of similar problems . This notion is inspired by human learning ; humans can usually generalize to a new concept from just one or very few examples , using their prior knowledge of the world and related concepts . A special case occurs when we do not need to distinguish between multiple concepts , but rather want to enable models to recognize a single new concept . In many practical scenarios , we will often be interested in just one novel class , rather than multiple at once . Application examples include detecting news or social media texts about a certain events ; detecting one type of unseen object ( e.g . a face ) in a dataset ; verification of unseen fingerprints ; or outlier detection within an unseen class . In existing few-shot methods , this is commonly translated into a two-class problem with the classes “ belonging to the concept ” and “ not belonging to the concept ” . As these two classes have very different types of feature distributions , this is not the most elegant solution and poses problems , especially with regards to training data selection . It also requires effort for modeling a class ( the negative one ) that is not of interest . In this paper , we present an addition to few-shot learning approaches that enables generalization to a single class in order to determine whether queries belong to this ( previously unseen ) concept or not . This is implemented by introducing a “ null ” ( or “ garbage ” ) class which does not need to trained to the prototypical network approach . Application of this novel model type is demonstrated on the commonly used Omniglot and MiniImageNet data sets . 2 RELATED WORK . 2.1 ONE-CLASS CLASSIFICATION . The problem of one-class classification has been a topic of research for decades . It has also been interpreted as outlier detection , anomaly detection , novelty detection , and concept learning . Comprehensive overviews of “ traditional ” methods for solving this task are given in Khan & Madden ( 2014 ) and Chandola et al . ( 2009 ) . Chandola et al . ( 2009 ) identify several different fundamental techniques , e.g . statistics , nearest-neighbour methods , clustering , and classification . Strategies can be derived from research areas such as data mining , information theory , and machine learning . Within the machine learning domain , a common solution relies on One-class Support Vector Machines ( OSVMs ) [ Schölkopf et al . ( 1999 ) ; Tax & Duin ( 1999 ) ] , although earlier-stage neural networks were also utilized [ Hawkins et al . ( 2002 ) ] . In recent years , various deep learning approaches have been suggested as well . Auto-encoders are frequently used for dimensionality reduction ( corresponding to a feature extraction ) of the input data in this context . Some methods then directly calculate the reconstruction error to determine whether new queries belong to the same class or are outliers ( e.g . Sakurada & Yairi ( 2014 ) ; Chen et al . ( 2015 ) ) . Others add a subsequent classification network ( e.g . Morton & Griffin ( 2016 ) or Erfani et al . ( 2016 ) , where the auto-encoder is replaced with a Deep Belief Network ) . Ruff et al . ( 2018 ) take inspiration from OSVM approaches and train CNNs jointly with a classification SVM . Lately , adversarial approaches have seen attention in this domain as well , e.g . in Schlegl et al . ( 2017 ) and Sabokrou et al . ( 2018 ) , where they are used in a similar fashion as the previous auto-encoders . Direct training on a single annotated class is not implemented as frequently as the combination of automatic feature extraction and distance ( or reconstruction error ) calculation . Oza & Patel ( 2019 ) demonstrates a strategy for training a CNN on positive examples and artificial negative examples generated from Gaussian distributions , while Perera & Patel ( 2018 ) shows a method based on automatic feature extraction in combination with specialized loss functions and template matching . A broad overview of deep learning techniques from the anomaly detection perspective is provided in Chalapathy & Chawla ( 2019 ) . All of these approaches rely on the availability of data from the expected class for training . What is missing is a type of model that can determine whether a query belongs to a class from a small number of examples that are provided at evaluation time without the need to re-train . Such a classification is possible with few-shot learning . 2.2 FEW-SHOT LEARNING . The concept of one- or few-shot learning is motivated by human learning : If humans are presented with a new class of objects , they do not require more than a few examples to be able to match new instances to this class ( often , just one supporting example is sufficient ) . This ability is , among other factors , informed by general world experience , which teaches humans to set boundaries between object classes . Matching networks , which implement this idea , were first introduced by Vinyals et al . ( 2016 ) . Training is performed on sets of support examples for a subset of possible classes plus a query example belonging to one of the classes in so-called episodes . These episodes are generated for a wide range of class permutations . In the one-shot case , only one support example is given for each class , while there are multiple ones in few-shot models . Such a network has two input branches : One for the support examples , and one for the query example . In both branches , the inputs are run through a sub-network which performs an embedding . The embedding networks may or may not share their weights . The query ’ s resulting embedding is then compared against the supports ’ resulting embeddings using a pre-defined distance metric . This results in likelihood measures for each input class , commonly implemented with a softmax . The over-all comparison metric for the task is learned implicitly as the network learns an appropriate embedding . In the few-shot case , multiple support examples per class are treated independently of each other throughout the network . Example-related likelihoods are then summed up to obtain class likelihoods at the output . In this way , matching networks essentially implement a weighted nearest-neigbor classification . As a further development , Snell et al . ( 2017 ) introduced prototypical networks . Instead of treating all of a class ’ support examples independently , they compute a so-called “ prototype ” of the class after the embedding network . This prototype is commonly the mean vector ( or centroid ) of the class ’ s embedded support examples.When the squared Euclidean distance is used as the metric , this implements a linear classifier rather than the nearest neighbor classifier produced by matching networks . Finn et al . ( 2017 ) goes one step further and introduces model-agnostic meta learning ( MAML ) : An algorithm to train models that can be easily finetuned to new tasks with very few examples . This can also be applied to other few-shot methods such , as matching networks . When applying few-shot learning to a binary problem , support examples for two classes are required : Positive examples of the class of interest , and random other examples . This amounts to a workaround , as there are no two comparable classes . Instead , one of them merely consists of counterexamples ( a sort of “ garbage ” class ) . Examples in this class will have a much wider feature distribution . This makes selecting random examples difficult , as they should still be representative of this distribution , particularly when only a small number of support examples is used . It also means that computational power and time must be dedicated to a class that is not actually of interest , and as such is not very elegant . Siamese networks [ Koch et al . ( 2015 ) ] can be seen as an interesting way to solve this problem , as they implement a decision about whether two unseen examples belong to the same class or different ones . However , they are not able to exploit more than one support example per class like prototypical networks can . 3 PROPOSED APPROACH . 3.1 ONE-WAY PROTOTYPICAL MODELS . Motivated by the counterexample issue , we take the described approach one step further : We construct a prototypical network that only requires positive support examples , from which it generates an internal embedding of the whole class . The query embedding is then compared to this , and judged to belong to the same class or not . We call this type of network a “ one-way prototypical network ” . The sub-network for generating the internal embeddings is implemented just as in regular prototypical networks with shared weights between the support and query branches . An illustration is shown in figure 1a . This approach presents a challenge for the distance metric : Because Cosine or Euclidean distances are not bounded , we can not set a threshold for class belonging . In order to do this , we need a basis for comparison . We implement this by introducing a “ null class ” that models the whole space of possible examples without explicitly being given those examples ( as in the two-class implementation mentioned above ) . Queries are judged on whether they are more probable to belong to the ( positive ) class we are interested in rather than the null class . The question then becomes how to construct this class . As explained above , prototypical networks model classes by calculating the centroid of the class ’ s support examples . As the model includes batch normalization layers , we can assume that the centroid of the whole latent space will be close to the zero point in all embedding dimensions . A query ’ s distance to the centroid of the positive class is therefore compared to its norm ( i.e . its distance to the zero centroid ) to determine whether it is more likely to belong to the positive class or to the null class encompassing everything . Figure 1b shows a visualization where the null centroid is denoted as c0 . A similar idea was presented in Ren et al . ( 2018 ) , which suggests the use of unlabeled data to improve prototypes . The null ( or “ distractor ” ) class here is only used to identify examples not belonging to any of the trained classes , rather than training the model to explicitly distinguish between a single given class and the null class . As in regular prototypical models , Euclidean distance is used . One issue that arises is that in the state-of-the-art prototypical models , batch normalization is not performed at the end of the convolution blocks of the embedding network , but rather after the convolution layer and before the ( ReLU ) non-linearity and pooling layers . The final block is therefore not quite batch-normalized . As there is some debate about the ordering of these layers anyway [ Ioffe & Szegedy ( 2015 ) ; Mishkin ] , we propose reordering the block layout to a structure of : Convolution - ( Leaky ) ReLU - Max Pooling - Batch Normalization . An alternative to the null class solution would be a simple thresholding of the class likelihood . However , preliminary experiments showed that this threshold is very problem- and concept-dependent , and thus does not generalize well to unseen episodes .

Summary: This paper addresses a method of applying prototypical networks (which are popular for few-shot learning problems) to few-shot one-classification problems where only one group of examples are available without any counter-examples. The main idea of prototypical networks is to learn an embedding function such that in the embedding space a distance metric well reflects the class structure. When such models are applied to one-class problems, a basis for comparison is required.