One-way prototypical networks
Authors: Anna Kruspe
Venue:  (2020)
Keywords: few-shot learning
one-shot learning
prototypical networks
one-class classification
anomaly detection
outlier detection
matching networks

Abstract: Few-shot models have become a popular topic of research in the past years. They offer the possibility to determine class belongings for unseen examples using just a handful of examples for each class. Such models are trained on a wide range of classes and their respective examples, learning a decision metric in the process. Types of few-shot models include matching networks and prototypical networks. We show a new way of training prototypical few-shot models for just a single
class. These models have the ability to predict the likelihood of an unseen query belonging to a group of examples without any given counterexamples. The difficulty here lies in the fact that no relative distance to other classes can be calculated
via softmax. We solve this problem by introducing a “null class” centered around zero, and enforcing centering with batch normalization. Trained on the commonly used Omniglot data set, we obtain a classification accuracy of .98 on the matched
test set, and of .8 on unmatched MNIST data. On the more complex MiniImageNet data set, test accuracy is .8. In addition, we propose a novel Gaussian layer for distance calculation in a prototypical network, which takes the support examples’ distribution rather than just their centroid into account. This extension shows promising results when a higher number of support examples is available.

Under review as a conference paper at ICLR 2020
ONE-WAY PROTOTYPICAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Few-shot models have become a popular topic of research in the past years. They offer the possibility to determine class belongings for unseen examples using just handful of examples for each class. Such models are trained on wide range of classes and their respective examples, learning a decision metric in the process. Types of few-shot models include matching networks and prototypical networks. We show class. These models have the ability to predict the likelihood an unseen query belonging to a group of examples without any given counterexamples. The diffi- culty here lies in the fact that no relative distance to other classes can be calculated via softmax. We solve this problem by introducing a "null class" centered around zero, and enforcing centering with batch normalization. Trained on the commonly used Omniglot data set, we obtain classification accuracy of .98 on the matched test set, and of . on unmatched MNIST data. On the more complex Minilma- geNet data set, test accuracy 88. In addition, we propose novel Gaussian layer for distance calculation in a prototypical network, which takes the support exam- ples' distribution rather than just their centroid into account. This extension shows promising results when higher number of support examples is available.
way of training prototypical few-shot models for just single
MOTIVATION
One- or few-shot learning is an active area of research that focuses on modeling semantic concepts with very little training data. Such models learn information about the general task of comparing queries to one or several support examples of the requested classes, which can then be applied to unseen classes. A large amount of data is still necessary for training. but this data does not need to belong exactly to the domains that may become interesting the future, but rather to a variety of similar problems. This notion is inspired by human learning; humans can usually generalize to a new concept from just one or very few examples, using their prior knowledge of the world and related concepts.
A special case occurs when we do not need to distinguish between multiple concepts, but rather want to enable models to recognize single new concept. In many practical scenarios, will often be interested in just one novel class, rather than multiple at once. Application examples include detecting news social media texts about a certain events; detecting one type of unseen object (e.g. face) in a dataset; verification of unseen fingerprints; or outlier detection within an unseen class. In existing few-shot methods, this is commonly translated into two-class problem with the classes "belonging to the concept" and "not belonging to the concept". As these two classes have very different types feature distributions, this is not the most elegant solution and poses problems, especially with regards to training data selection. It also requires effort for modeling class (the negative one) that is not of interest.
In this paper, we present addition to few-shot learning approaches that enables generalization to a single class in order to determine whether queries belong to this (previously unseen) concept or not. This is implemented by introducing "null" (or "garbage") class which does not need trained to the prototypical network approach. Application of this novel model type is demonstrated on the commonly used Omniglot and MinilmageNet data sets.

---

Under review as a conference paper at ICLR 2020
2
RELATED WORK
2.1 ONE-CLASS CLASSIHCATION
The problem of one-class classification has been a topic of research for decades. It has also been interpreted as outlier detection, anomaly detection, novelty detection, and concept learning. Com- prehensive overviews of "traditional" methods for solving this task are given in Khan Madden (2144) and Chandola al. 22009). Chandola et al. (2009) identify several different fundamental techniques, e.g. statisties, nearest- neighbour methods, clustering, and classification. Strategies can be derived from research areas such as data mining, information theory, and machine learning.
Within the machine learning domain, a common solution relies on One-class Support Vector Ma- chines (OSVMs) (Schölkopf et al. (1999); Tax & Duin (1999)], although earlier-stage neural net- works were also utilized [Hawkins et al. (.022]. J. In recent years, various deep learning approaches have been suggested as well. Auto-encoders are frequently used for dimensionality reduction (cor- responding to a feature extraction) of the input data in this context. Some methods then directly calculate the reconstruction error to determine whether new queries belong to the same class or are outliers (e.g. [Sakurada & Yairi (2014); Chen et al. (2015)) Others add subsequent classification network (e.g. |Mottnn& & Griffin (2016) or Erfani et al. (216)) where the auto-encoder replaced with Deep Belief Network). Ruff et al.
(018) take inspiration from OSVM approaches and train CNNs jointly with a classification SVM Lately, adversarial approaches have seen attention this domain well, c.g. in [Schleg] et al. (2017) and [Sabokrou et al. (2018). where they are used in
similar fashion as the previous auto-encoders.
Direct training on single annotated class is not implemented frequently the combination of automatic feature extraction and distance (or reconstruction error) calculation. |Oza & Patel (2019) demonstrates strategy for training CNN on positive examples and artificial negative examples generated from Gaussian distributions, while while|Perera & Patel (2018) shows a method based on auto- matic feature extraction in combination with specialized loss functions and template matching. A broad overview of deep learning techniques from the anomaly detection perspective is provided in Chalapathy & Chawla (019)).
All of these approaches rely on the availability of data from the expected class for training. What is missing is a type of model that can determine whether a query belongs to a class from a small number of examples that are provided at evaluation time without the need to re-train. Such classification is possible with few-shot learning.
2.2 FEW-SHOT LEARNING
The concept of one- or few-shot learning motivated by human learning: humans are presented with a new class of objects, they do not require more than a few examples to be able to match new instances to this class (often, just one supporting example is sufficient). This ability is, among other factors, informed by general world experience, which teaches humans to set boundaries between object classes.
Matching networks, which implement this idea, were first introduced by Vinyals et al. (2016). Train- ing is performed on sets of support examples for a subset of possible classes plus a query example belonging to one of the classes in so-called episodes. These episodes are generated for wide range of class permutations. In the one-shot case, only one support example is given for each class, while there multiple ones in few-shot models.
Such network has two input branches: One for the support examples, and one for the query ex- ample. In both branches, the inputs are run through sub-network which performs an embedding. The embedding networks may or may not share their weights. The query's resulting embedding is then compared against the supports' resulting embeddings using pre-defined distance metric. This results in likelihood measures for each input class, commonly implemented with a softmax. The over-all comparison metric for the task is learned implicitly as the network learns an appropriate embedding.
In the few-shot case, multiple support examples per class are treated independently of each other throughout the network. Example-related likelihoods are then summed up obtain class likelihoods at the output. In this way, matching networks essentially implement a weighted nearest-neigbor clas- sification. As further development, (Snell et al. (2017) introduced prototypical networks. Instead of treating all of a class' support examples independently. they compute so-called "prototype" of the class after the embedding network. This prototype is commonly the mean vector (or centroid) of
2

---

Under review as a conference paper at ICLR 2020
the class's embedded support examples. When the squared Euclidean distance is used the metric, this implements a linear classifier rather than the nearest neighbor classifier produced by matching networks.
Finn et al. (2017) goes one step further and introduces model-agnostic meta learning (MAML): An algorithm to train models that can be easily finetuned to new tasks with very few examples. This can also be applied to other few-shot methods such, as matching networks.
When applying few-shot learning to binary problem, support examples for two classes are re- quired: Positive examples of the class of interest, and random other examples. This amounts to a workaround, as there are no two comparable classes. Instead, one of them merely consists of counterexamples (a sort of "garbage" class). Examples in this class will have much wider feature distribution. This makes selecting random examples difficult, as they should still be representative of this distribution, particularly when only small number of support examples is used. It also means that computational power and time must be dedicated to class that is not actually of interest, and as such is not very elegant. Siamese networks (Koch et al. [ can be seen an interesting way to solve this problem, as they implement a decision about whether two unseen examples belong to the same class different ones. However, they are not able exploit more than one support example per class like prototypical networks can.
PROPOSED APPROACH
3.1 ONE-WAY PROTOTYPICAL MODELS
Motivated by the counterexample issue, we take the described approach one step further: We construct prototypical network that only requires positive support examples, from which it generates an internal embedding of the whole class. The query embedding is then compared to this, and judged to belong the same class or not. We call this type of network a "one-way prototypical network". The sub-network for generating the internal embeddings is implemented just as in regular prototypical networks with shared weights between the support and query branches. An illustration is shown in figure Ia]
This approach presents challenge for the distance metric: Because Cosine Euclidean distances are not bounded, we cannot set a threshold for class belonging. In order to do this, we need a basis for comparison. We implement this by introducing "null class" that models the whole space of possible examples without explicitly being given those examples (as in the two-class implementation mentioned above). Queries are judged on whether they are more probable belong to the (positive) class we are interested in rather than the null class.
The question then becomes how to construct this class. As explained above, prototypical networks model classes by calculating the centroid of the class's support examples. As the model includes batch normalization layers, we can assume that the centroid of the whole latent space will be close to the zero point in all embedding dimensions. A query's 's distance to the centroid of the positive class is therefore compared to its norm (i.c. its distance to the zero centroid) to determine whether it is more likely to belong to the positive class or the null class encompassing everything. Figure shows a visualization where the null centroid denoted as cp- A similar idea was presented in Ren et al.] (2018). which suggests the of unlabeled data to improve prototypes. The null (or "distractor") class here is only used to identify examples not belonging to any of the trained classes, rather than training the model explicitly distinguish between a single given class and the null class. As in regular prototypical models, Euclidean distance is used.
One issue that arises is that in the state-of-the-art prototypical models, batch normalization is not performed at the end of the convolution blocks of the embedding network, but rather after the convolution layer and before the (ReL.U) lineliiyyy and pooling layers. The final block is therefore not quite batch-normalized. As there is some debate about the ordering of these layers anyway [loffe & Szegedy (2015): Mishkin), we propose reordering the block layout to structure of: Convolution - (Leaky) ReLU Max Pooling Batch Normalization. An alternative to the null class solution would be simple thresholding of the class likelihood. However, preliminary experiments showed that this threshold is very problem- and deeendtt, and thus does not generalize well to unseen episodes.
3

---

Under review as a conference paper at ICLR 2020
Query example
Support examples
Class Likelhood
Class
m
Co
Pre processing Emoedding CNN
Diatance
(a) Network structure
(b) Prototype distance metric
Figure 1: One-way few-shot network structure and prototype calculation
3.2 ONE-WAY NORMAL PROTOTYPICAL MODELS
As described, prototypical networks only take into account the centroid of class's support exam- ples. This is sufficient when all classes can be assumed to be roughly on the same semantic level and therefore have similar feature distribution widths. However, this may not always be true (e.g. we may encounter cases where the model attempts to distinguish between queries of the class "Bobtail cat" and queries of the class "Insect"). This problem becomes even more prevalent for the one-class case, as unseen classes may have wider narrower distributions than unseen examples, and we do not have a basis for comparison defined by other classes.
Allen et al. (019) suggest solving this problem by dynamically adding prototypes per class neces- sary- - so-called Infinite Mixture Prototypes (IMP). We propose a different extension to prototypical networks: Modeling classes by their distribution instead of their centroid only. These distributions are fitted to the class's ' support examples. Due to the effect of batch normalization, we assume mul- tivariate normal distributions for this extension. A query's ss class assignment is then determined by its likelihood under each class's distribution. The previous embedding network is implemented just before, except that Leaky ReLU activations become necessary instead of ReLUs guarantee training stability. (To clarify: If the value of any of the embedding dimensions becomes exactly zero after the non linearity, the all-over probability becomes zero as well, leading to failure of gradient calculation during training. Leaky ReLU prevents this from happening).
This extension of prototypical networks also integrates with the one-way framework described above. Instead of assuming centroid at zero for the null class, we now assume multivariate normal distribution with a mean at zero and standard deviation of 1 (this is not guaranteed ex- actly, though: Batch normalization enforces centering zero and standard deviation of I, but does not enforce Gaussian-ness: A visualization is provided in figure[] (In the one-way case, this an advantage
IMP: Preliminary experiments show that adding the way concept IMP models does not lead to improvements over standard prototypical networks as the multiple positive prototypes tend overpower the one negative prototype.)
Of course, this idea has one obvious drawback: The model requires a sufficient number of support examples be able to determine salient standard deviation over them, particularly as the embed- ding dimensions are treated independently. We hypothesize that beyond a certain number of support examples, implementing class's 's distributions instead of their centroids only will enable the model to distinguish between them better.
(As side note, Fort (201)) also showed how to integrate a distribution concept into prototypical net- works, calling them "Gaussian Prototypical Networks". However, in this publication, the covariance matrix is calculated for each embedded example rather than over the various support embeddings. It is then taken into account explictly in the distance metric. This allows a better handling of uncer- tainties, particularly for noisy data. Our focus here is somewhat different: We are more interested in obtaining additional information from the relationships between support examples).

---

Under review as a conference paper at ICLR 2020
Gim.
den
Figure 2: Normal prototype example- Support examples and estimated class distribution in yellow, null class distribution in grey; positive query in orange, negative query in red
4 EXPERIMENTS AND RESULTS
4.1 DATA AND EXPERIMENTAL DESIGN
We first test our approach on a well-known toy data set for one- or few-shot learning: The Omniglor image data set [Lake al. (2015)). This data contains 1623 characters hand-drawn 20 times by Amazon Mechanical Turk" annotators. These characters come from 50 alphabets; a 30-20 split of the alphabets is suggested for training and validation. Per episode, we use 19 instances of character as support examples, and either one of the remaining examples or random other character as the query. As commonly done in literature, we also test the trained models on an unmatched data set: The MNIST' handwritten digit database [LeCCnn et al. (1998)).
Second, we run experiments on the much more complex MinilmageNer data set [Vinyals et al. [2016]] MinilmageNeet is a subset of the larger ImageNet data set [Russakovsky et al. (2015)) It contains 60,000 color images of size 84x84, split across 100 semantic classes with 600 examples each. Out of these, 80 classes
used for training and 20 for test. We use the splits provided by Ravi Larochelle (2017). Our experiments are performed with to 20 support examples per class. In both cases, pre-processing simply consists of normalization of the input image data to mean of 0 and a standard deviation of 1. All trainings are performed six times with random initializations and training episodes. The results are reported in terms of accuracy and averaged across runs; standard deviation between runs is generally very low (<.01). Each training is run for 20 (for Omniglor) or 10 (MinilmageNeet) episodes, with 32,000 randomly generated episodes each, resulting effectively in 640,000 and 320,000 training episodes respectively. Adam is used as the optimizer.
4.2 EXPERIMENT A: SIAMESE AND TWO-WAY MODELING
In our first experiment, we test the performance of previous architectures on the one-way problem: With Siamese, matching, and prototypical networks. The former can only integrate a single support example and compare this to the query. For the latter two, we implement the one-way problem as two-way classification, with the relevant "positive" class on the one hand, and the "negative" class on the other. Support examples are thus selected from the positive class, and randomly from the whole remaining data set. Results the Ommiglot and MinilmageNer data sets are displayed in figure 3, Supports range between and 19 (Omniglot) or 20 (MinilmageNett)
The results are generally acceptable, with prototypical models always delivering higher accuracies than matching networks. This difference becomes larger the more support examples are available, as this allows the prototypical networks to model the negative class better, whereas the nearest- neighbor classification of matching networks is not strongly improved with more support examples. Accuracy of the prototypical networks ranges between .75 for one support example, and .977 for 19
https:/ htt::://wwww.e.. www.w.turk... com/
5

---

Under review as a conference paper at ICLR 2020
Matching
Meiching
Protolyp call
Protooypical
Normal prototppill
Nor max prototypeal
0.8
0.6
0..0
Accut
0.0
0.2
0.2
19
spports
Supporrt
Omniglot
(b) MinilmageNel
Figure 3: Accuracies of models implementing the one-way problem with two classes
may prototypical
procoopiccal model
way prototypical model
I-way procoopical model
Tvnay mode with mordered leyers
1-way pr. model with reordered
08
0.8
0.6
Aucuanay
0.6
C23
02
0.2
10
19
10
20
Supppots
Supports
Omniglot
(b) MinilmageNer
Figure 4: Accuracies of one-way prototypical models
on the Omniglot data set, and between .59 (one support) and .74 (20 supports) on MinilmageNet. Siamese networks perform well (even best on Omniglot) for one support example, but are quickly surpassed by prototypical networks when more are available.
The novel normal prototypical networks work well for this problem, but do not surpass standard pro- totypical networks. For very few support examples, results do not rise much above the .5 baseline, or training may even fail completely. This is to be expected, as the distribution cannot be determined from such low number of supports. 5 examples appear to be necessary to make these models work at all, and 10 to make them generate competitive results.
We also test the models trained Omniglor the unmatched MNIST data set. Results not shown here, but follow the same trends, just somewhat lower. The accuracy of prototypical net- works is .686 for support example, and .883 for 19.
4.3 EXPERIMENT B: ONE- WAY PROTOTYPICAL MODELS
We test the novel one-way configuration for prototypical models which distinguishes between a "positive" class that we are interested in, and a "null" class that is fixed at a zero centroid as de- scribed in section 3.1 The results are shown in figure The first bar in each group the two-way result from the previous experiment for comparison. The second one displays the accuracy of the same type of model, but with the null class instead of an explicit "negative" class. As mentioned above, we also test model with reordered layers that batch normalization comes last before the prototype calculation; the third bar shows the results for this model.
We discover that the one-way way realization works better in almost all cases. Results improve partic- ularly in cases with few support examples. This makes sense, as few examples do not allow the two-way model to find the centroid of the negative class easily, whereas this is pre-defined in the one-way network. We also notice that the reordered model provides better results than the regular one; batch-normalizing latent representations right before the prototype calculation therefore

---

Under review as a conference paper at ICLR 2020
2-w model
2wwy model
Towey mosel
Towey model
huaay model with tramsill siddde
T-way model with or aanablle soadle
way model with reordered layeer
I-way medel with reordered layers
model with tnamable sitile and readered layers
T-ony medel with tratnablle walte and rececieeed
199
10/23
901
0.8
Acorscy
0.0
Accuray
0.6
590
99
60
5%
V2-1
1.2%
0.4
0.4
0.
15
19
10
Supports
Supports
(a) Omniglot
(b) MinilmageNet
Figure 5: Accuracies of one-way normal prototypical models
appears to be important (we observe the same effect when simply adding another batch normal- ization layer to the standard model). With this model, accuracy on Omniglot rises to .969 for one support example, and to .98 for 19. On the more complex MinilmageNNet data, accuracy becomes .64 for one support example and .8 for 20. For comparison, this re-ordering was also tested on the two-way prototypical model; significant differences in the results were found when compared to the original layer ordering.
Once again, we also test the Omniglot models on MNIST. For 19 support examples, the accuracy is slightly worse compared to the two-class model at .796. This makes sense as the batch normal- ization is optimized on Omniglot, and large number of supports allows for better estimation of the negative class' distribution. However, results are better or equal for a small number of support examples (e.g. .743 for one).
4.4 EXPERIMENT C: NORMAL PROTOTYPICAL MODELS
Finally, test the various configurations of our novel normal prototypical models described in section 3.2 The results are shown in figure 5 Out of these, the first bar in each group displays the two-way model accuracy from experiment in which the prototypes are modeled with normal distribution. The second bar shows the results for the same model, but with null class modeled with a unit Gaussian (mean O, standard deviation 1) instead of the explicit negative class. Third, we allow the model to train the standard deviation of the null class instead of fixing it at 1. The fourth and fifth bars show results for similar models, but once again with the embedding layers reordered so that the last step before the prototype calculation is batch normalization.
As described in experiment A, this type of model does not produce usable results with few support vectors, and may even fail to train. For this reason, only show results for 5 more support vectors. With three or more for Omniglot or 5 or more for MinilmageNet, we start to notice an improvement when using the one-way implementation over the two-way version just in the previous experiment. Once again, reordering the embedding layers to put batch normalization last increases accuracy, but the effect is not strong as in standard prototypical models. Allowing the model to adapt the standard deviation of the null distribution during training also helps. This is somewhat surprising because batch normalization enforces a standard deviation of over all data. We suspect that this happens because the actual distribution is not perfectly Gaussian, and therefore not modeled well by our assumptions. We tested adding a loss term to enforce Gaussian- but did not see any improvemtents. This effect will require closer investigation in the future.
Over-all, the results follow the same trends as those of the standard prototypical models, but mostly remain slightly below them. However, for a larger number of support examples (215), they surpass them slightly in accuracy on the Omniglot data set. Future experiments with even more supports necessary to determine where they become useful. Over-all, we obtain accuracies of .985 for Omniglot with 19 support examples, and .78 for MinilmageNeer with 20. When testing the Omniglor models on MNIST, they yield accuracy of .788.

---

Under review as a conference paper at ICLR 2020
2-aay maadhing
2-0ay, prosotppicel
Tonay, protutypccal
4m0
356
422
2-way, procotypecel
Tomay, protutypacal
2-way normal procotppical
2/wayy normal pronetypical
1 may, normal procoopicl
200
11 mwly, normal procotppcal
25
200
244
100
(a) Omniglot
(b) MinilmageNet
Figure 6: Average training time per epoch
4.5 EXPERIMENT D: TRAINING TIME
As an additional experiment, we analyze the training times of our various models on an NVIDIA Volta V100 with 16GB RAM. This is calculated by averaging the training times of the last three epochs of each model. Figure [6) displays the results on Omniglot and MinilmageNet.. As mentioned above, each epoch consists of 32,000 episodes, and choose the configuration with 5 support examples for comparison. The first two bars in each group correspond to the two-way matching and prototypical models from experiment A: their training times are roughly in the same range. Interestingly, the two-way normal prototypical model (fourth bar, also from experiment A) trains slightly faster on Omniglot and much faster for MinilmageNNer while achieving almost the same results as the prototypical model. We are currently investigating the reasons for this.
More importantly, bars three and five show the training times for the one-way implementations. For both prototypical and normal prototypical models, training time is almost halved. This is explained by the fact that only half the support examples need to be processed, and the centroids or distributions only need to be calculated for one class instead of two in each episode. As seen above, these models also usually deliver better results than the two-way implementations, thus providing another argument for their usage.
CONCLUSION AND FUTURE WORK
In this paper, presented approaches for few-shot learning in cases where only interested in a single class in each episode. Naively, this can be modeled with a two-way matching or pro- totypica model with support examples for the relevant, "positive" class, and random other support examples for the "negative" class. This is not very elegant as selecting negative examples is difficult, and those may not reflect the negative class very well, especially when few supports are used. We suggest introducing a "null" class in prototypical models instead which does not require support examples. As batch normalization enforces zero centering in the latent space, we simply assume a zero centroid for this null class. We show that such models work better than the two-way implemen- tation. Additionally, training time is reduced almost by a factor of two as only half of the support examples are required.
As another novel idea, we suggest modeling class prototypes not just by their mean (or centroid), but by their distribution. We assume normal distributions here, and model them via their mean and standard deviation. In the one-way case, we model the null class with a multivariate unit Gaussian (with mean 0 and standard deviation 1). This has the disadvantage of requiring sufficient number of support examples to estimate these parameters. If those are available, the approach generally works, but usually does not surpass the standard prototypical model below certain number of sup- port examples (around 15).
We will further investigate whether these models can be useful when a very large number of support examples are available. There also is some evidence that such models train faster than standard prototypical models, which we will analyze. We suspect that one problem the assumption of Gaussian-ness, which is not guaranteed in the latent space. It will be helpful to examine whether we can either enforce this, or otherwise find better ways to model the class distribution.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Kelsey R. Allen, Evan Shelhamer, Hanul Shin, and Joshua B. Tenenbaum. Infinite mixture pro- totypes for few-shot learning. In International Conference on Machine Learning (ICML), Long Beach, CA, USA, June 2019.
Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: A survey. arXiv, abs/1901.00 0034077 January 2019.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM Com- puling Surveys, 41 41 July 2009.
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. Event extraction via dynamic multi-pooling convolutional neural networks. In Annual Meeting of the Association for Computa- tional Linguistics (ACL), July 2015. doi: 10.31155w1/11--1017, URL
Anthology/P15-11 1017
Sarah M. Erfani, Sutharshan Rajasegarar, Shanika Karunasekera, and Christopher Leckie. High- dimensional and large-scale anomaly detection using a linear one-class svm with deep learning. Paitern Recognition, 18(C):121-134 October 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta meta-learning for fast adaptation of deep networks. International Conference Machine Learning (ICML), 2017.
Stanislav Fort. Gaussian Prototypical Networks for Few-Shot Learning on Omniglot. arXiv, haagg/abs/110 02735
abs/1708.027al 02735, 2017.
Simon Hawkins, Hongxing He, Graham J. Williams, and Rohan A. Baxter. Outlier detection using replicator neural networks. In International Conference on Data Warehousing and Knowledge Discovery (DaWaK), Six-en-Provence, France, 2002.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32Nd International Conference on Inter- national Conference on Machine Learning- Volume 37, ICML'15, pp. 448 448-444446666 456. JMLR.org. 2015.
URL http: acm. org/ http:/iaatiiiin....o cf 3045118.. 3045167
Sheroz Khan and Michael G. Madden. One-class classification: Taxonomy of study and review of techniques. The Knowledge Engineering Review, 29:345-374, January 2014.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In International Conference on Machine Learning (ICML), Lille, France, 2015.
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 150066266:1332-113333 December 2015.
Yann LeCun, Corinna Cortes, and Christopher J.C. Burges. MNIST handwritten digit database. 1998. UtL/htt:://yaat lerrannn.ecc... www.cooccceeoeeee httmm//xbm/mnii/////mmm
Dmytro Mishkin. Evaluation of the CNN design choices performance ImageNet-2012. http:///tibbbooooooocoeeeeritttt
Jerone T. A. Andrews Edward Morton and Lewis D. Griffin. Detecting anomalous data using auto-encoders, International Journal of Machine Learning and Computing, 6(1):21-26, February 2016.
Poojan Oza and Vishal M. Patel. One-class convolutional neural network. IEEE Signal Processing Letters, 26(2):277-28 January 2019,
Pramuditha Perera and Vishal M. Patel. Learning deep features for one-class classification. CoRR, abs/1801 803.05565, January 2018.

---

Under review as a conference paper at ICLR 2020
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In 5th In- ternational Conference on Learning Representations. ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL https: /ttpr////ppeeeeeeeeeeeeemee net. forum?
id=rJy0-Kcll.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-leaming for semi-supervised few-shot classifi- cation. In 6th International Conference on Learning Representations, ICLR, Vancouver, BC, Canada, 2018
Lukas Ruff, Nico Görnitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Robert A. Vandermeulen, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In In- ternational Conference on Machine Learning (ICML), Stockholm, Sweden, July 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211 1-2222, 2015. doi: 10.1007/s1 1263-015-0816-y
Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, and Ehsan Adeli, Adversarially learned one-class classifier for novelty detection. In Conference Computer Vision and Patlern Recognition (CVPR), Salt Lake City, UT, USA, June 2018
Mayu Sakurada and Takehisa Yairi. Anomaly detection using autoencoders with nonlinear dimen- sionality reduction. In Workshop Machine Learning for Sensory Data Analysis (MLSDA), Gold Coast, Australia, December 2014.
Thomas Schlegl, Philipp Seeböck, Sebastian M. Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International Conference Information Processing in Medical Imaging (IPMI), Boone, NC, USA, June 2017.
Bernhard Schölkopf, Robert Williamson, Alex Smola, John Shawe-Taylor, and John Platt. Sup- port vector method for novelty detection. In International Conference Neural Information Processing Systems (NIPS), November 1999.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In International Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA, December 2017.
David M. J. Tax and Robert P. W. Duin. Support vector domain description. Pattern Recognition Letters, 20111-13:1191111999 November 1999.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for shot learning. In International Conference Neural Information Processing Systems (NIPS), Barcelona, Spain, December 2016.
A
APPENDIX
You may include other additional sections here.
10


Reviews:
[{"review_id": "rJeP0M7TYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper looks at the problem of few-shot classification in the regime when only a single class is present. The task at hand is as follows: given a number of support images of a previously unseen class (not present during training) and a single unlabeled image, we need to decide if this image belongs to the class or not. While previous approaches would explicitly construct negative examples to contrast the positive ones with during training, the authors bypass this by using batch norm in the last layer, which, on average, centers embedding feature vectors at 0, defining effectively the embedding for the negative class. In addition, the authors look at modelling the distribution of support image embeddings to improve the performance of their model.\n\nOverview:\nI like the general idea and find the paper easy to follow. Using the centering effect of batch norm to effectively bypass the need to define the universal negative class embedding is an interesting approach. I think it is especially compelling due to the special role of the \\vec{0} point in the embedding space, where all embedding vectors of length 0, regardless of their direction, meet. I also appreciate that the authors used multiple training runs and their averages as their results, rather than the best result.\n\nI have a few points that I believe should be clarified:\n\n-- Point 1 --\nAre you assuming that the trainable offset in the batch norm will be close to \\vec{0}, or are you identically setting it to \\vec{0}? Since your definition of the “garbage” class relies on this, it might be worth checking what the actual learned parameters in the batch norm end up being, and if they are small, possibly fixing them to 0?\n\n-- Point 2 --\nI didn’t get what distance metric in particular you were using to determine if the unlabeled example X is closer to the centroid of the class c or the origin 0. If I understood it correctly, you effectively look at length(c-X) > length(X-0) as your condition determining if X belongs to c. What metric did you use as your length measure?\n\n-- Point 3 --\nWhen using the Gaussian approximation estimating the distribution of the support images in the embedding space, how does it affect the condition  length(c-X) > length(X-0)? In “Gaussian Prototypical Networks for Few-Shot Learning on Omniglot” by Stanislav Fort you cite, they incorporate the covariance of the Gaussian S in the length function, effectively using length(c,X,S). Their Euclidean metric uses the covariance matrix is a metric tensor to weight different axes differently. How exactly do you use yours?\n\n-- Point 4 --\nWhen comparing to the previous algorithms in the single-class (one-way) regime, you use a random image from one (at random) of the C_train - 1 remaining training classes as your negative example / example of the other class. Is that a fair comparison? How many images of those classes do you use? If it were the case that for a problem with the support size of 1 (or a few), sampling from these C-1 other classes might provide very unrepresentative embedding vector for the negative class. Might it be fairer to look at the average embedding vector over more images from the negative classes?\n\n-- Typos --\n= 2 typos on page 4 at the bottom: “*Is* is then taken into account *explictly* in the distance metric.”\n= 1 typo on page 5 at the bottom: “standard deviation between runs is generally very low (¡.01). Each train”\n\n-- Conclusion --\nOverall, I think the problem and solution proposed are interesting. The authors provide a large enough set of comparisons to existing algorithms, and explain their approach well.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HklxYwDCKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors consider the 1-way few shot classification task. Argue that modeling it as a 2-way with a random negative sample is not efficient. Propose a novel technique applicable for prototypical networks. Their proposal is to use 0 as the prototype of null class. So the distance to the prototype is compared against the norm of the query embedding. They also propose modeling the distribution and not just the centroid using a multivariate gaussian. But in practice there is no benefit in doing so. Therefore, the main contribution is proposing to compare against norm of the embedding rather than a prototype for random negative samples. The benefit of this proposal decreases by more shots. Probably because the prototype of 20 random images is 0 anyway.\n\nFig. 4b is wrong. The 2-way results match the matching result in Fig. 3b rather than the prototypical result. \n\nConsidering Fig. 4 seems the most gain comes from the reordering rather than one-way proposal. So a natural question is what is the effect of reordering on the 2-way prototypical. It may boost the 2-way one too.\n\nThe contribution of this paper is intuitive and interesting. But given the experiments it seems insignificant. Specially with the missing reordering of the 2-way model results.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1xX8Syx5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses a method of applying prototypical networks (which are popular for few-shot learning problems) to few-shot one-classification problems where only one group of examples are available without any counter-examples. The main idea of prototypical networks is to learn an embedding function such that in the embedding space a distance metric well reflects the class structure. When such models are applied to one-class problems, a basis for comparison is required. \nThe idea proposed in this paper is to introduce a null class which models the entire space of possible examples and queries are judged, compared to positive class as well as the null class. Another extension was made to model classes by their distribution instead of prototypes only, leading to one-way normal prototypical models.\n\n---Strength---\n- One-class extension of the prototypical network is a new idea, while prototypical networks are already popular for few-shot learning problems.\n- The proposed model can solve few-shot anomaly detection problems.\n\n---Weakness---\n- Regarding the null class, it was claimed that the centroid of the whole latent space can be assumed to be close to the origin in the embedding space, since batch normalizations are included in the model.  The batch normalization learns the scale and shift parameters to minimize the error function, so the center does not have to be close to zero in all dimensions. More clear justification should be added to make the paper more sense.\n- Regarding one-way normal prototypical models, calculating the distributions require sufficient number of examples. Thus, when applied to few-shot anomaly detection, the performance might be be satisfactory.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends prototypical networks to few shot 1-way classification. The idea is to introduce a null class to compare against with a null prototype. The reviewers found the idea sound and interesting. However, the response was mixed because the reviewers were not convinced of the significance of the improvements. Furthermore, there were questions raised about the motivation that were not sufficiently addressed in the rebuttal. Batch normalization layers will not necessarily lead to zero mean if the trainable offset is not disabled. The authors did not clarify whether they disable this offset. I encourage the authors to resubmit after addressing the issues raised by the reviewers.
Decision: Reject