Hyperspherical embedding for novel class classification
Authors: Rafael S. Pereira
alexis joly
Patrick Valduriez
Fábio Porto
Venue: ICLR 2022 Submitted (2022)
Keywords: Metric Learning
open set
deep learning

Abstract:   Deep neural networks proved to be useful to learn representations and perform classification on many different modalities of data. Traditional approaches work well on the closed set problem. For learning tasks involving novel classes, known as the open set problem, the metric learning approach has been proposed. However, while promising, common metric learning approaches require pairwise learning, which significantly increases training cost while adding additional challenges. In this paper we present a method in which the similarity of samples projected onto a feature space is enforced by a metric learning approach without requiring
pairwise evaluation. We compare our approach against known methods in different datasets, achieving results up to $81\%$ more accurate.

Under review as a conference paper at ICLR 2022
HYPERSPHERICAL EMBEDDING FOR NOVEL CLASS CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks proved to be useful to learn representations and perform classification many different modalities of data. Traditional approaches work well on the closed set problem. For learning tasks involving novel classes, known the open set problem, the metric learning approach has been proposed. How- ever, while promising, common metric learning approaches require pairwise learn- ing, which significantly increases training cost while adding additional challenges. In this paper we present method in which the similarity of samples projected onto feature space enforced by metric learning approach without requiring pair- wise evaluation. We compare our approach against known methods in different datasets, achieving results up 81 % more accurate.
INTRODUCTION
Humans have the ability to identify many different types of objects, Fields 2016). Even when we not able to name a certain object, we can tell it's differences from a second object, which contributes to the identification of objects have never seen before and group them into classes based prior knowledge. Metric learning KAYA & BILGE 2019) is well adopted approach that identifies novel classes without fine tuning a model on these classes. The approach applies an optimization strategy, which guarantees that the classes model has seen during optimization form disjoint clusters on the latent space according to a certain metric distance. Some common approaches that use this strategy are: the triplet loss (Schroff et al. (2015); constrative loss Hadsell et al. (2006); prototypical networks Snell et al. 2017); constellation loss Medela & Picon (2020); and matching networks |Vnyaals et al. (2016), here referred to as distance based learners. Another approach in metric learning is called similarity learning, where the model receives pairs of inputs and learns that they are similar if they belong to the same class or dissimilar otherwise, as discussed in (Sng al. (2018). During inference on novel classes, distance based learners use the distance between labeled points of the novel class the model was not optimized upon obtain representation in the latent space for the novel class and then calculate the distance between new points and each class representation. When considering similarity based learners, a similarity score is calculated between every (class.query) point pair in order to find the most similar pair.
Fields
However while enforcing metric properties on the latent space leverages the model knowledge to novel classes, it requires pairwise learning, which limits the scalability of such approaches given the amount of possible pairs.
In this paper we take into account the normalized softmax loss function(NSL), proposed by Wang al.] (2018), and present how enforees a latent space that obeys the cosine similarity. Based on this, we then present methodology to apply the NSL to the novel classes classification problem. Considering a trained artificial neural network, we add a new neuron to it's last layer and infer the weights that connect the penultimate layer of the network to this neuron. The connection and the new neuron are used to classify a novel class by using few labeled samples of Our approach for the open set problem allows to classify new classes without fine-tuning the model. Instead, we use the same network parameters the model was optimized upon to classify its seen classes and only add new neuron along with its inferred connection for classes. We evaluate state-of- the-art art approaches to solve the open set problem against our proposed approach, both in the disjoint and joint scenarios, for different datasets. The experimental results show that our approach outperforms

---

Under review as a conference paper at ICLR 2022
other metric learning strategies and additionally, induces more scalable training process, as it does not require pairwise learning, leveraging the open set problem technique to deal with large datasets.
The remainder of this paper is structured follows. First, it presents at section 2 Our methodology and how to classify new classes is described in section 3 Next, we
theoretical background
present the results the joint and disjoint open set problem in section 4 Moreover, [44 We the use of the NSL approach in more complex dataset in the field of botany, in section
present
compare our methods incremental learning in section 4.5 We present related work and lastly, conclude in section 6
2
PRELIMINARIES
We given training dataset (ara, (I).. where, for all the input xy belongs input space X F.d e.g. the space of images, and the output Y4 to an output space y {1, {1,2..... ..... K), the set of class labels, where K is the number of classes. Based this training set, the aim is to find a classifier X + y which produces a single prediction for each input and generalizes well on unseen samples X. When this classifier deep neural network, h can typically be expressed as:
h(x) max nu (z) &
where f)()) (f)()),,....... ...,, 1x (x)) is the vector of the estimated class probabilities computed as:
i(x) = ((((z))
with Φ X' M being a succession of layers allowing to compute an M-dimensional feature vec- tor representation d(x:) for any input image E X, and d R - RRK being the final classification function, typically composed of a fully connected layer followed by a softmax activation function:
e t
(/x(2)
Σ ΣØ=1 ew
2.1 THE OPEN SET PROBLEM
A classification problem can be formulated as closed set or open set problem. In the closed set problem context, the optimization process trains model to learn features that can classify the samples into classes present in the training set. The approach does not require the identification of classes not present in the training set. This is commonly tackled using the Softmax-cross crossserroosy loss He et al. Simonyan & Zisserman ( et al. (2015). In contrast, in the open set problem to be able to use the model to classify new classes by exploiting properties the latent space yielded during optimization.
are interested in not only identifying the classes present in the training set, but also
2.2 CLASSIPYING NEW CLASSES
When tackling the open problem, interested in optimizing models in which the full knowl- edge the network obtains during optimization can be exploited for classes outside the training set. The usual softmax cross-entropy approach lacks the ability extract features that obey this prop- erty, as the weights between the penultimate layer and the classification layer w are as important as the representation in the latent space of the penultimate layer & as seen in Eq. [ and the former is undefined for novel classes. Usual approaches for classifying novel classes are explored in met- ric learning. Metric learning strategies are interesting as novel classes can be identified. However, current strategies based pairwise learning can be costly to optimize. We discuss in this paper strategy to remove pairwise learning and still be able define novel classes for a model.
2.3 NORMALIZED SOFTMAX LOSS
Proposed in Wang et al. 2018), the NSL (Normalized Softmax Loss) is a modification of the softmax loss that enforces cosine similarity metric between classes the latent space. It enforces the
2

---

Under review as a conference paper at ICLR 2022
features that are projected into the latent space to be contained in M dimensional hypersphere (M > 3) where each region the sphere contains features belonging to a certain class.
If we look again at the classical softmax equation (Eq. D. the constraints induced by NSL are:
bg = 0, vk
1I0 || = 1, vk
(2)
|=||| = |1(())) S, Vx
and finally
(.cos(wa)(=)))
(x((x) = (x(((x))
= Σj= e",d(()
2j=1 (Scoo(ws
where cos(u, (L.l/[[[/]ll/ll)]] is the cosine similarity, i.c. the cosinus of the angle between two vectors u and v. Note that the syper-parameter S acts as a temperature the normalized softmax allowing control the degree of concentration of the output probabilities O (2:).
geometrical representation that shows the relationship between the weights and the feature vectors obtained with NSL shown in Figure 2 One can see that the barycenter of the feature vector is aligned with it's corresponding class weights.
3 PROPOSED METHODOLOGY
In this paper we aim to compare pairwise strategies, commonly used in metric learning, against the normalized softmax loss approach for the open set problem. In this manner we consider both the problem where during inference seen and unseen classes are disjoint, as well as the scenario where the model must identify both the seen and unseen classes together.
More formally, once the network has been trained, we would like to extend the output space to new of classes y {K + 1,..., K + K*) for which we have only one or very few samples (+t ME ii (x)........ In particular, we would like to obtain new classifier ht X + y (disjoint scenario) or a new classifier h' X + U'* (joint scenario). Note that, whatever the scenario, we consider that the function @ is fixed as well as the pre-trained weights of the seen classes Wk. Vk K}.
{1,
3.1 CLASSIFYING NEW CLASSES VIA NSL
Given that the function & and the weights We of the seen classes are fixed, our objective is reduced optimizing the weights w vk (1,..., K*} of the unseen classes. Using the cross-entropy as the objective function, this can be expressed as:
arg min
log (xf))
K
1=1
/ (xx))
arg mii
log
Ex em)((=))) + E}=
K (((()
wet 2. k 1=1
In the particular case where we have only one new class (i.e. K* = 1), this simplifies to:
wip(x)) arg max wi Σ [(())
arg max
wi
which leads, with the constraints of Eq.
2 to:
( [x(z))|
w
S.n
The weight wi of a new class can thus simply be computed by averaging the feature vectors of the images z of the new class. This simple theoretical result does not hold anymore when there is more will in our experiments, using
than one novel classes (i.e. when K* 1). However,
3

---

Under review as a conference paper at ICLR 2022
this estimation procedure for other classes provides good approximation of the exact optimal weights and is quite effective in practice. More formally, we propose to estimate the weights wi of each of K new classes
1 1 [( 1 = K
wi D 1 +)
In the joint scenario, we are interested in a classifier on both the seen classes and the new classes. This can be expressed as:
n2.
Rjoint (x)
max &
(5)
R
where the wy and @() are pre-trained on the seen classes and the new weights wo are computed with Eq.
In the disjoint scenario, we are interested in a classifier on the new classes only (in a transfer learning
way):
ddiajoint (z)
max
(6)
Dj= e ((x))
where @() pre-trained on the seen classes and the new weights wj are computed with Equ.
A dataflow depicting approach to infer the weights for novel classes is presented in Figure
Hitor
zigo
Hillo
Lewe
art CASE
dmarciry
Figure Diagram presenting the approach infer weights for the decision layer for new classes
4 RESULTS
4.1
EXPERIMENTAL SETUP
In this section we present the experimental setup. All experiments presented for the FASHION MNIST and CIFAR datasets
performed using google collaboratory. Experiments using the @ ntNet dataset were performed using a Dell PowerEdge R730 server, with 2 CPUs Intel (R) Xeon GB of RAM; and running on a Linux CentOS 7.7.1908 kernel version 3.10.0- 1.006.2..3.e1....... 64.4. The machine is equipped with a single NVIDIA Pascal P100 GPU, with 16GB RAM. Implementations were performed using Python 3.7 along with the Keras deep learning library.
(R) CPU E5- E5-2690 v3 @ 2.60GHz; 768
4.2 EVALUATING THE DISJOINT SCENARIO
In this section we show the results of evaluating the model accuracy on the test set from seen and unseen classes by employing VGG based model with two convolutional blocks.
We optimize the model on K = 10 K* seen classes, and we use the trained network to classify
the K" unseen classes, without considering the seen ones possible answers. The approach to do
4

---

Under review as a conference paper at ICLR 2022
Figure 2: Embedding obtained on the cifarl0 dataset when using latent space with two dimensions using NSL, each color represents different class. Inner points are the class weights while outer points come from the training set, notice how classes from the outer circle are aligned with the inner circle
K
NSL
Triplet
Constrative
2
0.852
0.663
0.700
3
0.725
0.570
0.551
4
0.629
0.406
0.422
0.545
0.296
0.328
Table 1: Model results for the cifar 10 dataset. K refers to the amount of unseen classes, while other columns refer to the method and accuracy obtained on the test set.
NSL
Triplet
Constrative
2
0.897
0.62
0.703
3
0.876
0.39
0.469
4
0.841
0.27
0.312
5
0.807
0.22
0.2
Table 2: Model results for the Fashion Mnist dataset. K* refers to the amount of unseen classes while other columns refer to the method and accuracy obtained on the test set.
so was presented in Eq. 6 in section Proposed Methodology. Results are presented in Tables and 2 where compare the adoption of the NSL approach against metric learning strategies in a disjoint settings. In the first line of the table, we present scenario in which we train with the other two unseen classes. In the second line, the experiment trained the model with classes and tested with 3 unseen ones, and so forth. Our results show that in both datasets the NSL outperformed those metric learning strategies for evaluating novel classes in disjoint scenario. To evaluate both the triplet loss well as the constrative loss methods, we first built the embedding representation, and then fed this representation into a k-nearest neighbours model trained on the average embedding of the class, using the same number of samples as NSL.
random classes and evaluate
4.3 EVALUATING THE JOINT SCENARIO
In this section, present the results when the novel classes must be integrated into the classification process along with the classes used for optimization. To this end, the function that we want to optimize is described in Eq. 5 The model is optimized with 10 K classes and we evaluate the accuracy on these and on the K unseen classes considering 10 total classes, Results are presented in Tables andd
Tables and depict the results comparing our approach using NSL with metric learning strate- gies: Triplet loss and Constrative loss, both Cifar and Fashion Mnist datasets, considering a joint

---

Under review as a conference paper at ICLR 2022
Seen
Unseen
K
NSL
Triplet
Constrative
NSL
Triplet
Constrative
2
0.559
0.195
0.205
0.501
0.226
0.189
3
0.578
0.128
0.182
0.433
0.156
0.176
4
0.620
0.082
0.166
0.391
0.127
0.160
0.641
0.05
0.215
0.357
0.146
0.176
Table 3: Model results for the Cifar10 dataset. K refers to the amount of unseen classes while other columns refer to the method and accuracy obtained on the test set. Seen refers to the accuracy the 10 K" classes while unseen on the K* classes
Seen
Unseen
K
NSL
Triplet
Constrative
NSL
Triplet
Constrative
2
0.854
0.708
0.657
0.773
0.724
0.613
3
0.842
0.705
0.440
0.771
0.716
0.390
4
0.860
0.688
0.284
0.739
0.680
0.286
5
0.878
0.690
0.190
0.719
0.691
0.181
Table 4: Model results for the fashion mnist dataset. K refers the amount of unseen classes while other columns refer to the method and accuracy obtained on the test set. Seen refers to the accuracy in the 10 K* classes while Unseen on the K* classes
scenario. The NSL approach outperformed both metric learning strategies on the two evaluated datasets, including seen and unseen classes predictions.
4.4 CASE STUDY: THE PL PI@NTNET DATASET
In order to assess the NSL approach on real world data, we evaluate it under the closed and open set problems on a dataset built from Pl@ntner database. Pl@ ntnet is one of the largest citizen science observatory in the world relying on mobile application Affouard et al.] (2017) that allows contributors to identify plants using their smartphone (an application that uses convolutional neural networks). The task is challenging as available pictures have different levels of quality, as well as multiple species from many different parts of the world as shown in Figure 3
The problem becomes relevant to the evaluation the proposed approach given that the scenario usually represented by long tail distribution, in which some classes are very common, while others are rare and lack significant available training data.
The subset of the @I@ ntnet data used was obtained from Garcin (2021) and has a total of 182 classes.
Figure 3: Distribution of the training dataset. Note the long tail distribution presenting how there many classes with small amounts of data and few with large amount.
4.4.1 EXPERIMENTAL DESIGN
As it clear from Figure 3 there is high imbalance among classes in the Plantnet dataset. Thus, there are many classes in the training set which have very small amounts of data. Since many plant species have few samples, we are interested in exploring the performance of NSL, where a model optimized only more common species and weights to classify uncommon species inferred, as discussed section 3.I] To this end, we perform experiments by optimizing the model only on classes where the number of samples larger equal to N {200, 100, 50, 25), which results

---

Under review as a conference paper at ICLR 2022
Number classes
10
28
43 0.3974
NSL accuracy
0.7349
0.6617
0.5382
Table 5: Model accuracy the test set optimized for 100 epochs weighted cross-entropy seen classes.
in K = {10, 16, 28, 43), where K is the number of classes, and present the results for joint and disjoint settings. Unseen classes will be selected randomly among those with number of samples M. M < N, and results will be presented with the average of 30 runs. All models are optimized on 100 epochs and weights that minimize validation loss are used for inference.
To describe the model architecture, first we define a convolutional block as two convolu- tional layers with same kernel size followed by maxpooling layer. The model is structured with four convolutional blocks as previously defined in sequence, with 3x3 kernels of sizes (25,6,25),
Following the four convolutional blocks we add flatten- ing layer and feed its output to fully connected layer with 1024 neurons and no activation function. The output is then normalized to norm and fed to the classifier. The Prepprocesiing step prepares the data by normalizing it to [0. 1] range and reshaping it to < 96, 96,3 > shape. Models are optimized with weighted cross-entropy by passing the class weights arguments to keras fit function deal with unbalanced classes. For the open tasks, we report on balanced accuracy better take into account class imbalance.
4.4.2 RESULTS
In this section, we present results using balanced accuracy for set of models built with varying number of classes in the last layer. As already discussed, we selected seen classes based filter on the number samples M Z N. Under this constraint, the relation between the number of samples and the number of classes is as follows: [25, 43), [80, [100, 16), [200, 10).
In Table 5 we present the model accuracy for the four different models to be used on the Plantnet experiment. All models were optimized by receiving the same amount of samples per epoch, as well as the number of epochs. This set of models then used to evaluate both the disjoint and joint scenarios.
4.4.3 DISJOINT SCENARIO
In this subsection, we present the disjoint analysis for the Plantnet dataset. As discussed in section 3.1 in our approach, we instantiate the model trained with seen data, without the last layer. We pass the samples from the unseen training set through the network and obtain their enbeddings from the penultimate network layer. Next, we compute the weights, using the enbeddings, according to Eq.
3 Observe that adopt random selection of unseen classes to compute weights and to perform inference. We present the results of 30 runs, for each value of K*, for novel classes inference using balanced accuracy. Results are presented in Figure 4
Seen Classes
Runber of unseen casses
Figure 4: Comparing models sharing the same architecture and optimized upon the same data and number of epochs. but with different amount of seen classes. We evaluate their ability to classify novel classes.
In Figure 4 show comparison among models trained different number of classes with the same amount of data, by presenting their ability to identify novel classes in a disjoint scenario. Our results show that the diversity of classes during training allowed the model to show robustness

---

Under review as a conference paper at ICLR 2022
for novel classes. This is observed by the worst results in Figure 4 obtained for the inference of novel classes, when training with only 10 seen classes. However it is also important to note that optimizing
higher number of classes is seen by the model, more updates or a more complex model to learn robust features for all seen classes. This is shown the curve obtained from the model optimized upon 43 classes, which ranks the second worst. Our best result was seen on the model with 28 classes, which had a high class diversity while also learning robust features during training.
complex problem, requiring more data to be
4.4.4 JOINT SCENARIO
In this subsection, we present the results of the joint scenario for the @ dataset. We instantiate the model trained on classes. Then, we add other K* unseen classes, so that the model must classify between K + K classes. Weights for the K classes are inferred described in section 3.1 We report the model accuracy for all classes, seen and unseen, as well as the model behavior for only unseen classes. Results presented in Figures and 6
Edgercal Accuncy
Seen Classes
classes
Figure 5: Analysis for the joint scenario showing the results for different models on seen and unseen classes.
Edungg Recuney
Figure 6: Analysis for the joint scenario showing the results for different models the unseen classes.
In Figures and present results for the joint scenario evaluating the overall model quality as new classes are added as well as showing the balanced accuracy calculated only on the unseen classes. Our conclusions can see that for both of these cases the model trained on 28 classes has the higher balanced accuracy, given the same number of total classes. Moreover, the model optimized on 43 classes shows the worse balanced accuracy. The lack of diversity of the model optimized on 10 classes influences it's quality the number novel classes increases in both scenarios.
the disjoint scenario, seen in Figure 44 also hold in these results. We
4.5
FEW SHOT SCENARIO FOR INCREMENTAL. LEARNING
In Eq. we show how our proposed methodology for inferring weights actually finds the set of weights that minimizes cross-entropy, whenever single novel class is included. However, when including multiple classes, our proposal may not yield the optimum set of weights for each new neuron. this section we present a set of experiments comparing the performances obtained by our inferred weights with the ones obtained through incremental learning, i.e. by minimizing the cross- entropy loss on samples of the new classes while freezing all the other network weights. Experiments performed using the Cifarl0 dataset. The initial model is trained on the training samples of K seen classes and the incremental learning phase is computed on the training samples of K" unseen classes (while freezing all the other network weights).
8

---

Under review as a conference paper at ICLR 2022
To evaluate the proposed strategy in few shot scenario, train the Resnet50 architecture using the NSL constraint, as provided by keras, on subset of classes of the cifar dataset. Once the network has been trained for 100 epochs examples of the classes that were unseen during training in a few shot scenario (one, five and twenty five shots). We infer the weights for the novel classes using the methodology described in section 3.1 and measure the model quality on the test set compared to the incremental learning approach (using the same few shots for each class). We report the results for one, five and twenty five shot scenarios on the CIFARIO dataset. Figure [ depicts similar results for all scenarios, considering the accuracy of predictions. Note that when the number of novel classes is smaller or equal to the number of seen classes our approach tends to achieve higher accuracy.
subset of the dataset,
sample small number of
Figure 7: Accuracy obtained in different few shot scenarios for the cifar10 dataset when comparing using inferred weights versus further optimizing them.
RELATED WORK
Creating models that are able to classify novel classes is task that is explored in different fields of artificial intelligence. There are two fields our work falls upon. The first is called metric learning discussed in section 2.2 while the second is called incremental learning briefly presented in section
4.5
In metric learning, which is sub-field of few shot learning, one aims to train model to identify classes via some property metric space, enforced during training. The enforced property can be for example that examples of the same class form a cluster according to some predefined metric and each class has its own cluster. These properties can then be explored to identify novel classes given that it is possible to determine the cluster of novel class with some labeled examples without retraining the model. There are many works that fall this category SScrroff et al. (2015) Hadsell eet al. (2006) Medela & Picon (2020) Vinyals et al. (2016). While all these approaches enforce metric properties on the latent space, they also require pairwise training, which our approach does not require.
6 CONCLUSION
In this paper, we presented how the normalized softmax loss can be employed on the open set prob- lem. We presented results on different datasets for both the disjoint and joint open set problems and compared them to metric learning strategies. We show that the NSL based approach demonstrates superior results producing more robust features and implementing less costly optimization proce- dure, as it does not require pairwise training. Results on real world use case evaluating on a subset of the @ ntnet data shows how our approach can be employed to identify classes unseen during optimization, with weights associated to the classification of new data inferred by the approach.
7 ACKNOWLEDDMENTT
The authors would like to thank Petrobras for supporting this work through the project "Develop- ment of an Intelligent software platform". We would also like to thank the INRIA INRIA-Brazil Associated Team cooperation project HPDaSe.

---

Under review as a conference paper at ICLR 2022
8 REPRODUCIBILITY STATEMENT
In this section we detail the steps we ensured to ensure that our work is reproducible. To ensure data availability we mostly use public datasets that are available in the keras. datasees interface. The subset of the @l@ ntnet dataset we used in this paper available as numpy arrays in the plantnet folder that is available via google drive link that presented in the appendix section.
Regarding data pre prepprssssing, all pre-processing steps as well as the structure of the models are presented in the main paper.
Concerning the mathematical formulation of the problem, the main mathematical formulation is presented in the main paper while additional information about the area of the problem is presented in the appendix metric learning and the latent space.
Lastly regarding experiment reproducibility, all experiments were organized into jupyter notebooks and these are organized into an folder that is available via a google drive link on the appendix.
REFERENCES
Antoine Affouard, Hervé Goëau, Pierre Bonnet, Jean-Christophe Lombardo, and Alexis Joly. Pl@ntnet app in the era of deep learning. In ICLR: International Conference on Learning Repre- sentations, 2017.
Chris Fields. Editorial: How humans recognize objects: Segmentation, categorization and individual identification. Frontiers in Psychology, 7:400, 2016. ISSN 1064 doi: 10.3389/fpsyg.200 2016. 00400. URL htt.ap/...........ooo.c...ooooop
00400
Camille Garcin, Projects garcin camille plantnet dataset, 2021. URL https:
inria. ft//cgarcin/pllatneet disete__atttse
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant
mapping. In Proceedings of the 2006 IEEE Computer Society Conference Computer Vision and Pattern Recognition Volume 2, CVPR '06, pp. 1735-1742, USA, 2006, IEEE Computer Society. ISBN 0769525970.
doi: 10.1 10.1109/CVP.22006 100. URL https: /ttp::://ddiioooo orq/10.
IICS/CVPR. 2006.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10. D0.100//VPP.2206.990
Mahmut KAYA and Hasan Şakir BILGE. Deep metric learning: survey. Symmetry, 11(9),
2019. ISSN 8973-8994. doi: 10.3390/syml 1091066. URL https: /www//wwwwkk mdpi. com/
2073 3994/11/9/1066
Alfonso. Medela and Artzai. Picon. Constellation loss: Improving the efficiency of deep metric learning loss functions for the optimal embedding of histopathological im-
ages. Journal of Pathology Informatics, 11(1):38, 2020. doi: 10.4103/jpi.pi..11222
URL https::/wwwt http://wwwpppptaooooooo.eeeeeeaa
2oar=2020; www.ooouume111m 1 http:///o..a.aaaaoooo/////ee/aea
Schroff, D. Kalenichenko, and Philbin. Facenet: A unified embedding for face recognition and clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 815-823, 2015. doi:
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceed- ings, 2015. URL http:
wbwpaa///////eeeeeee org abs 11199.10091111
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett
10

---

Under review as a conference paper at ICLR 2022
(eds.), Advances in Neural Information Processing Systems, volume 30, pp. 4077-4087 Cur- ran Associates, Inc., 2017. URL httppp/s/ppprcceeeie..aaa......p
F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. S. Torr, and T. M. Hospedales. Learning to compare: Relation network for few-shot learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1199-1208, 2018. doi: 10.1109/CVPR.201 2.0000011..
C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015. doi: 10. N0.1109/CVR..2055.225555
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, pp. (1630-3333 Cur- Associates, Inc., 2016. URL https:: fi elee/ 990 135 73355783355555 9836 2.0
.ew.reeeeeppppp CC/ wwwpppppppppppeceeccc.ec..pe..ee //16/
H. Wang. Y. Wang. Z. Zhou, X. Ji, D. Gong, Zhou, Z. Li, and W. Liu. Cosface: Large margin cosine loss for deep face recognition. In 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5265-5274, 2018.
doi: 10.1109/CVPR.20 2018.00052
A APPENDIX
The following sections contain additional analysis and theoretical discussion that is not integral to the understanding of the paper but the authors would like to share about the research performed.
A.1 METRIC LEARNING AND THE LATENT SPACE
Deep neural networks learn a set of transformations and relations among the inputs in order obtain the desired output during optimization. The network can be broken into two parts: the projection module (ø(x)). which takes the data and transforms into representation in the latent space, and the data representation processing, applied by the layer implementing the desired task (1((2). The latter enforces the latent space to have some properties that are defined by the task. When the network is trained with cross-entropy loss, the objective is the following:
arg min
-log(fvv. (x:))
where 0 is the set of all parameters of the network (for both (b(x) and Φ(x)). Thus, after optimization, we generally have that ny (x:) >> fly (ax) for j + yi which can only be achieved if: wy, (xx;) + by: >> ();;) + by for # yı- In other words, the softmax cross-entropy approach enforces the inequality W;Z; + b, >> w + by in the latent space, where i, represent different classes Wang et al. (2018). A proposed alternative that optimizes the latent space directly and can enforce metric properties that allows the model to be used for novel classes know as metric learning.
The metric learning approach learns set of features that obey a metric distance on the latent space. The model can be optimized to learn similarity metric between pairs, as proposed in [Sung et al. (2018), or can enforce the latent space to obey predefined metric distance like euclidean distance or cosine similarity. Some strategies, such as the constrative loss Hadsell et al. (2066), learn on pairs of data, while others learn using triplets like triplet loss Schroff et al. (2015).
Optimization on these approaches aim to obtain disjoint clusters for each class of interest in the latent space, according to predefined metric distance. As desired consequence of the approach, classification can be performed for novel classes by using the representation of an anchor example and calculating the metric distance between query point and the anchor.
A.2 HOW DOES THE AMOUNT OF SAMPLES AFFECT THE CLASS PROTOTYPE ?
common scenario in which the identification of unseen classes appears is the one where the amount of available data samples for classes of interest is small the cost of optimizing another
11

---

Under review as a conference paper at ICLR 2022
model include the new classes becomes too costly. Therefore, all strategies discussed in this paper classify new classes based on labeled examples without retraining. The influence of the number of samples needed to perform classification tasks using the NSL approach is shown in three different datasets in Figure for a model optimized for 30 epochs. We use keras base learning rate with adam optimizer. The experiment consider that the model was trained on all ten classes. Models for mnist and fashion mnist consider only dense layers, while cifar considers two convolutional blocks with 32 and 64 filters each. We create the class prototype using the inferred weights obtained via Eq an strategy similar to Snell et al. (2017).
675
24
0:0
Weight
Figure 8: Fl score on the test set for three different datasets where we show results when weights are: (a) Trained: the ones found during optimization; (b) Single anchor: weights inferred using a single random anchor example; and (c) weights are inferred using the training set to build a class prototype.
As it can be seen Figure weights inferred using Eq. on the training set maintain the same model accuracy as by using the weights obtained during optimization. We also can observe that model quality, when inferring via a single example, decays in relation the task complexity. On the x axis we present how the class weights were obtained where class prototype uses the whole training set infer the weights according to our methodology, while single anchor uses a single random example from the training set for weight inference. Y axis presents the F1-Score on the test set.
A.3 COMPARISON TO INCREMENTAL. LEARNING USING MANY SAMPLES
In Eq. show how proposed methodology of inferring weights actually finds the set of weights that minimizes cross-entropy, whenever a single novel class is included. However, when including multiple classes, proposal may not yield the optimum set of weights for each new neuron. In this section we present a set of experiments comparing the performances obtained by our inferred weights with the ones obtained through incremental learning, i.e. by minimizing the cross- entropy loss on samples of the new classes while freezing all the other network weights. Experiments were performed using the fashion mnist dataset. The initial model is trained on the training samples of K seen classes and the incremental learning phase is computed on the training samples of K* unseen classes (while freezing all the other network weights).
Number unseen classes
Figure 9: Cosine similarity between inferred weights, obtained as in Eq. S and optimized weights, obtained through cross-entropy optimization using the incremental learning constraint, for different numbers of novel classes on the fashion MNIST dataset
Figure II first shows the cosine similarity between the inferred and the optimized weights for differ- numbers of unseen class. As observe, when the number of novel classes is small, the two
12

---

Under review as a conference paper at ICLR 2022
sets of weights almost identical, which means that the inferred weights good the ones optimized through incremental learning (while being much faster and simpler to compute). With larger numbers of novel classes, This suggests that the gain of incremental learning might not be very high this case as well.
observe that the mean cosine similarity is still very high.
To quantify this gain, Figure TO presents the ratio of the accuracy achieved through incremental learning over the one obtained with the inferred weights (using MNIST test set). We see that when including small number of novel classes the ratio stays close improvement due to optimization. When including more classes, the gain of incremental learning can be higher (up to 1.55 for 8 unseen classes) but this requires 4-5 epochs on the training set. This suggests that the inferred weights may be used to initialize the incremental phase and get faster convergence when having lot of data for the novel class. In the next section we perform the same experiment considering small data.
showing no strong accuracy
Unseen Classes
Aecuevcy
33
6pochs
Figure 10: Ratio of accuracy after optimization versus inferred weights unseen = 10 866n classes when optimization occurs on seen classes for the fashion mnist dataset.
Opatne
Number
Figure 11: Cosine similarity between inferred weights, obtained as in Eq. 3 and optimized weights, obtained through cross-entropy optimization using the incremental learning constraint, for different numbers of novel classes on the fashion MNIST dataset when using only two samples to generate novel class
B EXPERIMENTS
All performed experiments are available via the following google drive folder: httpp::/tttt::::/eee
goog le. com/ wtt.ccoo//vi/e http:///....eeeeeee/22/ooo222110
shar ing
The subset the plantnet data is already available numpy array format inside the plantnet folder in this link
B. NSL ARCHITECTURAL SEARCH ON THE CLOSED SET
In this section, we present in Table an ablation study comparing NSL against softmax loss applied to the same network architecture. We define the following deep neural networks: Inception block, as a single Inception module as proposed in Szegedy et al. (2015); ; a VGG block is composition of two Convolutional layers with 3x3 kernel and 2x2 MaxPooling layer; while Resnet block is built with 3x3 convolution with a residual connection. For the number of filters we increase them the values.
order < 32, 64, 128
13

---

Under review as a conference paper at ICLR 2022
Architecture
Number of blocks
SL
NSL
Inception
1
0.473
0.633
Inception
2
0.47
0.746
Inception
3
0.42
0.803
ResNet
0.1
0.610
ResNet
2
0.1
0.75
ResNet
3
0.1
0.81
VGG
1
0.175
0.638
VGG
2
0.187
0.738
VGG
3
0.643
0.823
Table 6: Analysis on the closed set for different architectural setups optimized softmax nor- malized softmax for the cifarl0 dataset. SL and NSL present model accuracy on the test set.
In table 6 we compared the normalized softmax loss softmax loss different architectures. We note that for all architecture variations as well as variations in the network depth, the normalized softmax loss outperformed the softmax loss. We also denote that on softmax loss the choice of base architecture (Inception, Resnet, VGG) significantly influenced the model accuracy as much as depth. In contrast, the normalized softmax loss was more influenced by model depth in this dataset.
14


Reviews:
[{"review_id": "ji5DgSt128h", "reviewer": "Reviewer_t3Uk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) The paper completely missed the related works. Many papers addressing this problem are ignored. Basically only two metric learning approaches are introduced. The paper should improve the related work following this recent survey:\nGeng, Chuanxing, Sheng-jun Huang, and Songcan Chen. \"Recent advances in open set recognition: A survey.\" IEEE transactions on pattern analysis and machine intelligence (2020).\n\n2) In the reviewer's opinion the paper in its current shape is not ready to be published. The proposed method is loosely motivated and the comparisons are somewhat limited.\n\n3) Paper contributions are not highlighted, difficult for a reviewer to understand the novelty of the paper if any.\n", "summary_of_the_paper": "The paper presents how the normalized softmax loss can be used on the open set problem. Superior results seem to be achieved without the pairwise pairwise training", "summary_of_the_review": "Paper difficult to review in its current shape.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Y65EanNT7z", "reviewer": "Reviewer_AHvJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strengths:\n+ The presentation and writing is clear and easy to follow.\n+ The proposed approach is simple and efficient.\n\n#### Weaknesses:\n- The paper states that (deep) metric learning approaches (DML) suffer from high computitational complexity due to reliance on pairwise learning constraints. However, many approaches in DML effectively circumvent the computational burden using sampling strategies [e.g. A] or class proxies [e.g. B], etc. In particular the latter has shown to be both very computationally efficient and competitive with the state-of-the-art in DML.\n\n- In general lots of references to established works in Deep Metric Learning (DML) and Few-Shot Learning (FSL) are missing. For extensive discussions of both fields, e.g. see [C, D].\n\n- The core idea of the presented approach is adding additional neurons, respectively class weights, for new classes to the entropy-loss based classification. The class weights are set to the average over inferred weights based on few representatives of the new classes and subsequently fixed for classifiying unseen samples. However, it is questionable if the proposed approach would work on larger, more complex and finegrained datasets, as inferring class weights by averaging over few samples may be too unstable for representing fine-grained class features. More experiments on larger, standard benchmark sets would be very helpful.\n\n- The experimental section lacks details about the trained model, i.e. architecture details and optimization paramters. No standard architectures commonly used in DML or FSL ([C,D]) seem to be considered for fair comparison to the standard/State-of-the-art approachess. Moreover, comparison to state-of-the-art approaches in DML,FSL are lacking in the quantative evaluation. Only Tripet and Contrastive Learning seems to be considered – however, reference about the exact implementation details of these approaches are missing as well. Finally, the evaluation also does neither consider standard benchmark sets in DML (e.g. CUB200, CARS196, Standford Online Products) or FSL (Mini-ImageNet, CIFAR100), nor follows the established evaluation protocols of these areas. In total, the evaluation protocol significantly lacks comparison to established results in DML, FSL and, thus, does not allow for proper evaluation of the effectiveness of the proposed approach.\n\n- Evaluation on the PlantNet dataset lacks comparison to other approaches/baselines.\n\n---------------\nReferences:\n\n[A] Wu et al. 2017; Sampling Matters in Deep Embedding Learning\n\n[B] Kim et al 2020; Proxy Anchor Loss for Deep Metric Learning\n\n[C] Roth et al. 2020; Revisiting Training Strategies and Generalization Performance in Deep Metric Learning\n\n[D] Tian et al. 2020; Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?", "summary_of_the_paper": "The paper proposes an approach to Few-shot Learning based on the the CosFace (Normalized Softmax) Loss. After pretraining, class weights are added to the cross entropy loss for each new class in the test set, which are computed by averaging over inferred weights from a support set while fixing the remaining network weights (feature extractor). Experiments conducted on CIFAR10 and Fashion-MNIST indicated gains over baseline approaches.", "summary_of_the_review": "The paper lacks novelty and significance both for the presented approach and the empirical results. Further, the quantitative evaluation is based on non-standard datasets, evaluation protocols, architectures and baselines, hence does not provide a proper basis to evaluate the effectiveness of the proposed approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "5ATj7dtAfn8", "reviewer": "Reviewer_Tp5p", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a good empirical study on dataset showing the effect of the number of class seen and unseen. However, there is a concern regarding some statement made by the author and regarding the novelty presented in this work.\n\nStrength:\n- Case study in sec 4.4, this case study gives valuable insight on the effect of the data distribution on learning. Fig 4,5 and 6 shows that the choice of seen classes is important for learning on unseen classes and that inter-class diversity is more important than having a high number of classes.\n\nWeakness:\n - In the abstract, the author states that the results are 81 % more accurate but do not specify compare to what. I do not see in the table a jump in performance of more than 81% compare to the proposed baselines.\n- The authors chose to have metric learning methods as a baseline and compare their classifier with triplet/contrastive networks. While deep metric learning advanced, recent methods proposed frameworks that do not necessary need to form pairs.\nSee \"Proxy anchor loss for deep metric learning from S Kim et al\" or \"No fuss distance metric learning using proxies by Y Movshovitz-Attias et al\". The method proposed by the authors seems to be an easier setting than the method proposed in the stated paper since they are classifying in a zero-shot manner (the new classes are not seen in training). \n- The main issue that could bring concern is the design of the weight for the new class in Eq 4. By not fine-tuning the network on the new accessible classes, there is a chance that the features used to compute the new weights belong to a similar distribution than features from another existing classes. It could then happen that a new weights is equal to an already existing weight. How to make sure that it does not happen ?\n- Another issue is the lack of comparison with existing work. It would have been interesting to compare with state-of-the art incremental learning method or the newest metric learning method. It would have provided a strong support for claiming that the authors' work is new and performant.\n- There are too few related work reviewed in Sec 5.", "summary_of_the_paper": "This paper is about classification of images in an open set setting. Data coming from new classes are introduced to the network after training on data from a fixed set of known classes. The goal is to be able to correctly classify the old and new classes either jointly or not. This paper proposes a method for handling new classes by increasing the size of the classifier weights for each new classes. The network is trained using the Normalized softmax loss and new classifier elements are added by finding the center of mass of the data coming from the new set of class. The method performs on FASHION MNIST, CIFAR and Plantnet datasets.", "summary_of_the_review": "My recommendation for this paper is a strong reject. I am not convinced of the novelty proposed in this paper and there are too few comparison with existing work to strengthen the validity of the performance on the proposed datasets.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3nlmViLUY05", "reviewer": "Reviewer_qMdp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis work follows on from the Normalized sotmax loss function work by Wang et al 2018.\nIn that work, the last layer of a network learns feature vectors that are constrained/projected to a unit sphere.\nLearned simultaneously are weight vectors (w) for each class.\n\nThis work considers the arrival of a new set of examples for a set of new (previously unseen) classes.\nThese examples come with labels, but there are assumed to be few examples and  we would like to lever the features from the \"pretraining\" of the network on the original training set.\n\nIdeally, one would use the cross entropy loss to optimize a new set of classifying vectors (w) in the unit sphere for the new classes. The authors point out that for one class, this is equivalent to making the new class vector (w) to be the normalized average of all the feature vectors for the new class.\n(Note: the weight vectors for the classes and the features vectors share the same space in the unit sphere.)\nThen the authors consider what might happen if they apply the same method for more than one new class. This has less theoretical justification. In effect for each new class, a weight vector is generated by computing the normalized average of the feature vectors for examples from that class.\n\nThey consider different settings - joint and disjoint, and evaluate on a few different datasets.\nThe main results (tables 1,2,3,4) appear impressive, but I have some reservations..\n\n1) It is not clear what the feature dimensions are for the datasets.\n2) There is little detail about the triplet/contrastive methods used in the comparison.\n3) I would expect to compare against many more competing methods for this proposal.\n\nThe first three evaluation datasets were MINST, Cifar10 and Fashion MNIST.\nA lot of space in the paper is given to The Pl@ntnet dataset. (4.4.*)\nIt is a careful, appropriate and novel application of the method.\nThere is a discussion about how accuracy is affected for varied numbers of unseen classes under disjoint and joint scenarios with different training / unseen class splits. It is revealed that it is helpful to pre-train with a large number of seen classes, but that too many \"seen\" classes can be detrimental. So there is a trade of for this parameter, for this dataset at least.\nThis work (pl@ntnet) does not have any baselines or comparisons with other methods, it shows that the method can be applied but doesn't tell us much else - there are no strong conclusions or detailed analysis of the results.\n\nIn section 4.5 the authors compare their method to incremental learning. (In this case by training a new layer to the new classes while\nfreezing all other weights). This would be a section of great interest to anyone reading this paper but it is short, and the results are\n presented in one figure which is hard to read and no strong conclusions can be drawn from it.\n", "summary_of_the_paper": "This paper proposes a method to repurpose classifiers trained using the Normalized Softmax Loss function to novel classes by averaging the examples from the same previously unseen class to gain a new weight vector for that class.\nThe method is simple and effective, even though it is not theoretically justified for more than one class.\n", "summary_of_the_review": "I like the method - it is simple and intuitive, and the initial results appear impressive.\nBut for such a simple model the burden of proof is high, and there should be many more detailed comparisons with SOA zero-shot methods allowing the reader to compare and contrast benefits.\nSo I would like to see more detailed analysis on datasets and methods, for example, I would also like to see study on:\n1) how does the dimension of the feature vector affects performance?\n2) what happens when a set of new classes are very similar to a known class?\nPerhaps some synthetic data would allow to demonstrate the strengths and weaknesses of the method.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles an open-set setting where new classes (with few labeled examples) are introduced after the initial pre-training on different categories. A simple approach is proposed based on a normalized softmax classifier and feature averaging to generate a classifier for the new categories. Results are shown on a few standard datasets as well as the Pl@ntnet dataset. 

While reviewers found the topic and setting (as well as Pl@ntnet dataset) interesting, they had significant concerns on the novelty (t3Uk, Tp5p, AHvJ), contribution, and rigor of the empirical evaluation. Since the method is simple and largely leverages from prior works, the latter is especially important; reviewers pointed out that some of the latest in metric learning is ignored (e.g. Proxy Anchor, Tp5p and AHvJ), and no comparison is made to other classes of methods that (by the authors' admission) are very close to the setting such as open-set recognition (especially those that seek to classify new categories) and incremental learning.

Unfortunately, no rebuttal was provided by the authors, so these significant concerns remain and the paper cannot be accepted as-is. Since the reviewers did appreciate the setting and dataset, I recommend refining the paper and significantly beefing up the empirical evaluation for future resubmissions.
Decision: Reject