Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method to learn neural nets that preserves the subclass
similarity in the embedding space. The proposed method adds a prototype layer,
which stores the representative prototypes of some subclasses, then a fully
connected MLP is used to output the class label based on examples' distance to
the prototypes.
main_review: I am not expert in deep learning, but I know that prototype method was very
popular in statistical learning. Many commonly used algorithms, such as KNN and
K-means, are exploiting this idea.

The proposed method adapts the idea of prototype-based classification to deep
neural networks, and uses the automatically extracted prototypes to increase the
explainability of neural nets. The proposed algorithm seems work well on MNIST
and ImageNet benchmarks.

My main concern is the novelty of this work, and its lack of comparison to
previous work in similar direction. For example, the DkNN method mentioned in
related works persues a very similar idea like this work, so I think the authors
should compare this work to it. Moreover, the main contribution of this work is
it extends MsDNN with a prototype layer. Again, I'm not familiar with this area,
so I'm not sure whether this improvement is significant or just incremental.

Other than the above problem, I wonder how difficult it is to apply the proposed
network on real problems. The authors have state that it should be applied on
classification problems that each class consists of many subclasses. In order
to show this idea, the authors manually create superclasses in MNIST and
ImageNet dataset. I understand that using this method can verify if the neural
net preserves the subclass structure in its embedding layer. Nevertheless, in
real problem we usually do not have such type of information, so I wonder how
will the proposed method perform on the original MNIST and ImageNet tasks
comparing to more related works.
summary_of_the_review: The novelty of this work could be limited, and the experiments seem not enough
for validating the effectiveness of the proposed approach, so I recommend to reject.
=======================
After reading other reviews and the authors' clarification, I have increased my rating.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a novel an interpretable neural network for classification tasks that works by assigning input cases to prototypical cases based on their euclidean distance in an embedding space learned by the network. This is achieved by minimizing a loss function consisting of three components: cross entropy loss; distance between each case and the prototype with the same class, distance between each prototype and the closest case to it. The proposed method is specially useful when classification at fine-grained level is performed where in addition to the class to which an instance belongs, predicting the subclass is desirable as well. Experimental results are provided in the paper that validates the merits of the method compared to a set of variations of the proposed method and a state of the art approach as well.
main_review: The paper addresses an interesting problems which is interpretability of the classification methods that can have various use cases and can demystify the neural net black box for technical and non-technical individuals. It is written well and is easy to follow. The method is positioned well in the existing body of works and the contributions of the work is clear. Experimental settings and results are throughly and very well discussed which makes the work repeatable. Authors have also done a good job discussing the experimental results and sharing the interesting observations.

A couple of recommendations:
- One of the main contributions of the paper that should get credit for it's superior fine-grained performance is the loss function introduced in equation 1. However, it's not clear to what extent r1 and r2 are contributing to the overall performance of the method and which one is more important. From the explanations at page 6, it looks like that the weight of the cross entropy component of the loss function is significantly higher than r1 and r2 weights which alludes to the fact that r1/2 play a less significant role in the method. It would have been good if the authors could have added one or more ablated versions to the experiments to check this.
- Based on the experimental results, C-NN and F-NN mostly outperform Coarse and Fine which makes the reader wonder if instead of classification nearest neighbor search in the embedding space should have been suggested. If the concern is computational costs, it would have been interesting to see the performance of C-NN and F-NN by using approximate nearest neighbor search (e.g. using LSH or k-d trees).

- Minor comments:
-- P3: "Training stops when accuracy using nearest-neighbor classification on a validation set is minimized"
If I understand correctly, training stops when classification error is minimized not the accuracy. Please check.
-- Some hyper parameters seem to be set arbitrarily (e.g. # prototypes = 30) and it's not clear how assigning a different value could have affected the accuracy of the proposed method or how authors have come up with the used values.
summary_of_the_review: Overall an interesting read and a well written paper. My recommendation is to accept the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel combination of prototype learning and deep nearest neighbor learning in order to achieve an embedding of the input data that is more friendly for prototype learning while maintaining the computational efficiency and intepretability of a prototype approach. In more detail, the approach first trains a Multi-scale deep nearest neighbor network to embed the input data. Then, it jointly learns a prototype layer and a fully connected layer which outputs class logits based on distances to prototypes. Finally, the learned prototypes are replaced by the closest sample from the actual training data to enhance interpretability. In experiments on MNIST, FashionMNIST, CIFAR-10, and ImageNet, the paper shows that the proposed approach outperforms state-of-the-art prototype learning.
main_review: The paper's main strengths are:
* The combination of nearest-neighbor learning and prototype learning is very natural and appears to work well, empirically. The combination is also well-motivated from a pretty comprehensive literature review. I particularly appreciate the acknowledgment of early prototype work (e.g. Kohonen) which is typically omitted in the recent literature.
* A particularly nice result is that a learned model can also be used as a one-nearest prototype classifier for finer-grained subclasses of the actual classes it was trained on. In other words: the proposed approach learns hierarchical structure that was not explicitly provided in the training labels. As far as I know, this result is a novelty in the literature and motivates the use of prototype/nearest neighbor approaches.
* The paper provides reasonable ablation studies and close baselines, which enables the reader to tease apart the effect of most architectural choices. Similarly, the discussion of results is detailed and illuminating.
* I found the paper easy to follow and clear, with each claim being justified both conceptually as well as empirically.

The paper's main weaknesses are:
* It seems to me that the architecture of the last 2 layers is needlessly complicated. While this architecture is in Line with Li et al. (2018), a more common option (starting from Kohonen, 1995) is to assign a fixed label to each prototype and classifying data by assigning the label of the closest prototype (as the paper does for the fine-grained classification). Typical loss functions include the GLVQ loss from [Sato and Yamada (1995)](https://proceedings.neurips.cc/paper/1113-generalized-learning-vector-quantization.pdf) or variants of the [LMNN loss (2009)](https://www.jmlr.org/papers/volume10/weinberger09a/weinberger09a.pdf). Such an architecture would also be more in line with MsDNN.
Instead, the present paper selects a pseudo-label for each prototype based on the closest data point and proposes new loss components. Why is that? Does it perform better? If so, I would appreciate another ablation baseline that drops the final, fully connected layer and, instead, uses only the distances to the prototypes for classification. 
* While the combination of MsDNN and prototype learning is very natural and makes a lot of sense, intuitively, it would have been nice to see a more formal, theoretical justification why this combination improves performance.
* The experimental evaluation includes very reasonable baselines. However, more baselines would have been possible, taking more recent work into account, such as Chen et al. (2019) or [Hase et al. (2019)](https://ojs.aaai.org/index.php/HCOMP/article/view/5265).
* The experimental hyper-parameter settings on page 6 appear ad hoc and are not justified.

Overall, my impression is that the paper is not quite ready, yet. I believe that more experiments and theoretical analysis are needed to fully develop the proposed idea. I do appreciate the core idea, though, and believe that a future iteration of the paper may well be a good contribution to the field.

Beyond the major points mentioned above, there are a few minor recommendations:

* on page 4, the meaning of the hyperparameter $\sigma$ should be quickly explained.
* Please provide standard deviation (or another measure of variance) in Tables 1-3.
summary_of_the_review: Overall, my impression is that the paper is not quite ready, yet. I believe that more experiments and theoretical analysis are needed to fully develop the proposed idea. I do appreciate the core idea, though, and believe that a future iteration of the paper may well be a good contribution to the field.

/edit The authors have resolved the most crucial points in their response. Provided that these crucial arguments are amended to the paper, I believe that it can be published.
review_title: null

Meta-review: The authors propose a neural network model to preserve the sub-class similarity. The key of the model is to add a prototype layer to a multi-scale deep nearest neighbor network. The prototype layer stores the representative prototypes of some fine-grained sub-classes. The use of the prototype layer preserves intepretability and computational efficiency. Experimental results demonstrate that the proposed approach reaches state-of-the-art prototype learning performance.

The reviewers generally find the paper clear and with sufficient contributions. The empirical validation is sufficiently thorough to back the claims in the paper. The main concern prior to the rebuttal among some of the reviewers was about the novelty of the paper (e.g. with respect to DkNN), but the authors convinced most of the reviewers in the rebuttal about the key differences. The authors are encouraged to highlight the novelty aspect more clearly in the revision. Another suggestion was to add an ablation study to justify the importance of the r1/r2 parameters, and the authors have done a successful job addressing the suggestion. Several other comments, such as explanations of the hyperparameters, have been taken into account in the revision. The reviewers thus reach the consensus to recommend acceptance.
Decision: Accept (Poster)