Venue: ICLR 2018 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: - The paper proposes to use RBF kernel based neurons with each training data point as a center of
  one of the RBF kernel neuron. (i) Kernel based neural networks have been explored before [A] and
  (ii) ideas similar to the nearest neighbour based efficient but approximate learning for mixture
  of Gaussians like settings have also been around, e.g. in traning GMMs [B]. Hence I would consider
  the novelty to be very low 
- The paper says that the method can be applied to embedding learning and classification, which were
  previously separate problems. This is largely incorrect as many methods for classification,
  especially in zero- and few-shots settings (on some of the datasets used in the paper) are using
  embedding learning [C], one of the cited and compared with paper (Sohn 2016) also does both
  (mostly these methods use k-NN classifier with Euclidean distance between learned embeddings)
- It seems that the method thus is adding a kernel neuron layer, with the number equal to the number
  of training samples, centers initialized with the training samples, followed by a normalized
  voting based on the distance of the test example with training examples of different classes
  (approximately a weighted k-NN classifier)
- The number of neurons in the last layer thus scales with the number of training examples, which
  can be prohibitively large 
- It is difficult to understand what exactly is the embedding; if the number of neurons in the
  RBF layer is equal to the number of training examples then it seems the embedding is the activation
  of the layer before that (Fig1 also seems to suggest this). But the evaluation is done with
  different embedding sizes, which suggests that another layer was inserted between the last FC
  layer of the base network and the RBF layer. In that case the empirical validation is not fair as
  the network was made deeper.
- Also, it is a bit confusing that as training proceeds the centers change (Sec3.3 first few lines),
  so the individual RBF neurons, eventually, do not necessarily correspond to the training examples
  they were initialized with, but the final decision (Eq4) seems to be taken assuming that the
  neurons do correspond to the training examples (and their classes). While the training might
  ensure that the centers do not move so much, this should be explicitly discussed and clarified.
  
Overall, the novelty of the paper seems to be low and it is difficult to understand what exactly is
being done.  

[A] Xu et al., Kernel neuron and its training algorithm, ICONIP 2001
[B] Verbeek et al., Efficient greedy learning of gaussian mixture models, Neural Computation 2003
[C] Xian et al., Latent Embeddings for Zero-shot Classification, CVPR 2016
review_title: The paper proposes a non-linear RBF kernel based layer on top of standard CNNs for improving classification. The presentation quality is not good, and the novelty is low. 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a loss that is based on a RBF loss for metric learning and incorporates additional per exemplar weights in the index for classification. Significant improvements over softmax are shown on several datasets.

IMHO, this could be a worthwhile paper, but the framing of the paper into existing literature is lacking and thus it appears as if the authors are re-inventing the wheel (NCA loss) under a different name (RBF solver).

The specific problems are:
- The authors completely miss the connection to NCA loss (https://papers.nips.cc/paper/2566-neighbourhood-components-analysis.pdf) and thus appear to be re-inventing the wheel.
  - The proposed metric learning scenario is exactly as proposed in the NCA loss works, while the classification approach adds an interesting twist by learning per exemplar weights. I haven't encountered this before and it could make an interesting proposal. Of course the benefit of this should be evaluated in ablation studies( Tab 3 shows one experiment with marginal improvements).
- The authors' use of 'solver' seems uncommon and confusing. What is proposed is a loss in addition to building a weighted index in the case of classification.
- In the metric learning comparison with softmax (end of page 9) the authors mentions that a Gaussian standard deviation for softmax is learned. It appears as if the authors use the softmax logits as embedding whereas the more common approach is to use the bottleneck layer. This is also indicated by the discussion at the end of page 10 where the authors mention that softmax is restricted to axis aligned embeddings. All softmax metric learning experiments should be carried out on appropriately sized bottleneck layers.
- Some of the motivations of what the various methods learn seem flawed, e.g. triplet loss CAN learn multiple modes per class and there is nothing in the Softmax loss that encourages the classes to fill a large region of the space.
- Why don't the authors compare on ImageNet?

Some positive points:
- The authors mention in Sec 3.3 that updating the RBF centres is not required. This is a crucial point that should be made a centerpiece of this work, as there are many metric learning works that struggle with this. Additional experiments that can investigate this point would greatly contribute to a well rounded paper.
- The numbers reported in Tab 1 show very significant improvements

If the paper was re-framed and builds on top of the already existing NCA loss, there could be valuable contributions in this paper. The experimental comparisons are lacking in some respect, as the comparison with Softmax as a metric learning method seems uncommon, i.e. using the logits instead of the bottleneck layer. I encourage the authors to extend the paper and flesh out some of the experiments and then submit it again.
review_title: Rejection of paper with potential if reframed as revisiting the NCA loss.

final_score: rating: 5: Marginally below acceptance threshold
review_content: (Summary)
This paper proposes weighted RBF distance based loss function where embeddings for cluster centroids and data are learned and used for class probabilities (eqn 3). The authors experiment on CUB200-2011, Cars106, Oxford 102 Flowers datasets.

(Pros)
The citations and related works cover fairly comprehensive and up-to-date literatures on deep metric learning.

(Cons)
The proposed method is unlikely to scale with respect to the number of classes. "..our approach is also free to create multiple clusters for each class.." This makes it unfair to deep metric learning baselines in figures 2 and 3 because DMP baselines has memory footprint constant in the number of classes. In contrast, the proposed method have linear memory footprint in the number of classes. Furthermore, the authors ommit how many centroids are used in each experiments.

(Assessment)
Marginally below acceptance threshold. The method is unlikely to scale and the important details on how many centroids the authors used in each experiments is omitted.
review_title: Review

Meta-review: This paper proposes a non-parametric method for metric learning and classification. One of the reviewers points out that it can be viewed as an extension of NCA. There is in fact a non-linear version of NCA that was subsequently published, see [1]. In this sense, the approach here appears to be a version of nonlinear NCA with learnable per-example weights, approximate nearest neighbour search, and the allowance of stale exemplars. In this view, there is concern from the reviewers that there may not be sufficient novelty for acceptance.

The reviewers have concerns with scalability. It would be helpful to include clarification or even some empirical results on how this scales compared to softmax. It is particularly relevant for larger datasets like Imagenet, where it may be impossible to store all exemplars in memory.

It is also recommended to relate this approach to metric-learning approaches in few-shot learning. Particularly to address the claim that this is the first approach to combine metric learning and classification.

[1]: Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure. Ruslan Salakhutdinov and Geoffrey Hinton.  AISTATS 2007 
Decision: Reject