Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a method to integrate the cost of errors into a classification algorithm. The proposed loss function has two terms: one for pushing samples towards their corresponding class prototype (L_{data}), and another for forcing/guiding the pairwise distance between class prototypes to follow a predefined values (L_{distortion}).  

Marginally below acceptance. 

I like the idea of using a priory knowledge about similarities between classes to regularize the learning process.  But,

1) Out of 4 datasets, the proposed method works better than others on only two datasets for which in one case the addition of proposed term L_{distortion} has helped, but it actually degraded the results in the other dataset. So, the results are inconclusive.

2) pp.3 the assumption of having a symmetric misclassification cost seems very limiting. Effectively, this assumption implies the type 1 and type 2 errors must have equal cost. The proposed method is about treating different errors differently and yet it assumes type 1 and type 2 errors are equal. Referring to the example, provided in the introduction (paragraph 1), cost of misclassifying a crossing pedestrian as a street light should be way more than the cost of classifying a street light as a crossing pedestrian. 
Another assumption of the proposed method is triangle inequality i.e. D[k, l] + D[l,m] >= D[k,m] but this assumption is not justified except saying the cases where the assumptions do not hold, are out of scope of this paper. 

pp.1 says, "this error discrepancy is not taken into account ... in the evaluation metrics". This statement is incorrect. At least in a simple binary classification task, we have type 1 and type 2 errors. We have recall and precision. They are all about putting different weights to different errors. 

pp.2 Caption of Figure 1: tree is created based on authors perceived visual similarity. The tree should reflect the cost of class confusions. From paragraph 1 in page 1, it is inferred that the cost of a class confusion is determined by the actual consequences e.g. cost of confusing a streetlamp with a crossing pedestrian. Such cost is different than how similar the two classes are.  The notion of "cost" and "similarity" are not the same but used interchangeably in the paper. 

pp.7 the soft-label method is the top runner in cifar-100. Its performance on the other 3 datasets may be improved if the temperature parameter is adjusted to compensate the various number of classes. Of course, same can be said about tuning hyperparameters of the proposed method.  

review_title: assumptions are not adequately discussed and results are inconclusive

final_score: rating: 4: Ok but not good enough - rejection
review_content: In order to organize my review, I use the NeurIPS2020 template with slight adaptions to fit the ICLR requirements. 

### Summary and contributions: Briefly summarize the paper and its contributions.

The paper proposes a method to jointly learn a feature-extractor (a neural network) and a set of prototypes. Learning is performed in an end-to-end setting and the prototype distribution is regularized to follow a given distribution. For this purpose, the authors propose a cost matrix that assesses the severity of errors between classes. This matrix is organized in such a way that it yields a finite metric. The distribution of prototypes is regularized such that the distances between prototypes reflect this finite metric. In several experiments, the authors evaluate the advantages of such a setting with respect to an average cost measure.

### Strengths: Describe the strengths of the work.

The work is well motivated and how the entire network setup is realized is easy to understand and straightforward.


### Weaknesses: Explain the limitations of this work along the same axes as above.

A major weakness is that the authors have not cited all the relevant previous work about prototypical networks (see my comments below). Taking all the relevant literature into account, the contribution of the paper is small. Moreover, the proposed method and observed effects raise several questions that are not sufficiently addressed. For example, the authors observe that the learning with the squared Euclidean distance does not perform as good as with the Euclidean distance, but they have not investigated why this is the case (see my further comments below).


### Correctness: Are the claims and method correct? Is the empirical methodology correct?

The current version of the paper has some minor mathematical mistakes. However, they can be fixed and I consider them to be of little relevance for the correctness of the method. I will list them below.
One serious correctness point is that the authors claim (see page 4 top and page 2 contributions first bullet point) that they are inventing a principle to learn the prototypes jointly in an end-to-end setting. Considering the state of the art, this claim is incorrect.


### Clarity: Is the paper well written?

The paper is well written and the mathematical formulations have a good quality.


### Relation to prior work: Is it clearly discussed how this work differs from previous contributions?

The paper has not discussed all the relevant prior work. The idea to learn the prototypes jointly is not new:
* Chen, Chaofan and Li, Oscar and Tao, Daniel and Barnett, Alina and Su, Jonathan and Rudin, Cynthia. “This Looks Like That: Deep Learning for Interpretable Image Recognition.” NeurIPS (2019).
* Hong-Ming Yang and Xu-Yao Zhang and Fei Yin and Cheng-Lin Liu. “Robust Classification with Convolutional Prototype Learning.” CVPR (2018)
* Villmann, Thomas and Biehl, Michael and Villmann, Andrea and Saralajew, Sascha. “Fusion of deep learning architectures, multilayer feedforward networks and Learning Vector Quantizers for deep classification learning.” WSOM+ (2017)
* to name just a few…

Note that prototype-based classifiers and especially nearest prototype classifiers have been invented long before Tibshirani et al. (2002). For example, see Kohonen’s Learning Vector Quantization (LVQ) algorithms, including the original heuristic versions and the end-to-end trainable versions like Generalized LVQ (by Sato and Yamada).
Moreover, the incorporation of a cost matrix into the learning processes for prototypes was also done before. For example, Kaden, Marika, Wieland Hermann, and Thomas Villmann. "Attention Based Classification Learning in GLVQ and Asymmetric Misclassification Assessment." Advances in Self-Organizing Maps and Learning Vector Quantization. Springer, Cham, 2014. 77-87. Of course, in this work, the authors have not used the cost matrix to penalize the prototype positions, but it was included to account for different severity between class confusions.

### Reproducibility: Are there enough details to reproduce the major results of this work?

The paper provides a lot of details about the experimental setup, especially in the appendix. However, some information is not provided, for example, the initialization strategy of the architecture.


### Additional feedback, comments, suggestions for improvement, questions for the authors, and a **recommendation (accept or reject)**.

The authors should be precise about the assumptions of $D$. In the current version, the definition of $D$ is not correct to talk about a finite metric space (see page 3 Method and page 1). The missing property is the identity of indiscernibles.

The authors provide a good motivation for why the incorporation of a cost matrix is useful. However, could the authors explain why they then assume that the matrix $D$ is symmetric? Because, according to the motivation, the severity of misclassifying a *crossing pedestrian* as a *street lamp* is different from misclassifying a *street lamp* as a *crossing pedestrian*.

The first sentence in Section 3.1 is difficult to read due to “sample $n$”, consider writing $x_n$.

What is the difference of Equation (3) to a cross-entropy loss with a one-hot encoding for the correct class?

The authors observed that the squared Euclidean distance does not perform as well as the Euclidean distance. I assume that this happens because the squared Euclidean distance does not fulfill the triangle inequality that is, on the other hand, assumed for $D$. Could the authors comment on this? 

I don’t see that the non-differentiability of the square root at zero causes any issues in the training. If the argument becomes zero, the distance is zero and, therefore, the prototype is equal to the feature vector of the sample. Given the given interpretation of the attraction and repulsion forces on the data and the prototypes (see page 3 below), in which direction should they be pushed or repelled if they have a perfect match? There exists no update direction that could improve the model. Therefore, it is feasible to just stop the gradient or to ignore such samples.

Have the authors observed training instabilities due to large values of the distance $d$?

If I consider Figure 1, I wonder why the learned prototypes tend to be outside the class centers (e.g., see figure c). Can the authors elaborate on that?

The statement on page 1 bottom about the cross-entropy is incorrect. Whether the cross entropy “singles out the prediction” depends on the true probability vector.

The experimental results are not convincing as they don’t show a significant benefit compared with state-of-the-art methods. 

I have problems with the statement following Equation (4). Could the authors explain in more detail why a prototype arrangement regarding $D$ and low distortion is advantageous in terms of AC?

The google colab link provided in the supplementary was not accessible. 

Considering all the above points, I vote for rejection, since the paper does not have the right maturity level to be published at ICLR and the scientific contribution is small.

review_title: Reinvention of several known concepts; small scientific contribution

final_score: rating: 6: Marginally above acceptance threshold
review_content: Strengths and weaknesses:
+
Nice idea
Consistent improvements over cross entropy for hierarchical class structures
Improvements w.r.t other competitors (though not consistent) 
Good ablation study
-
The improvements are small
The novelty is not very significant
 

More comments:


Figure 1:
-	It is not clear what distortion is at this stage
-	It is not clear what perturbed MNist is, and respectively: why is the error of a 3-layer CNN so high (12-16% error are reported)? CNNs with 2-3 layers can solve MNist with accuracy higher than 99.5%?
-	This figure cannot be presented on page 2 without proper definitions. It should be either presented on page 5, where the experiment is defined, or better explained  
Page 4: It is said that s can be computed efficiently and this is shown in the appendix, but the version I have do not have an appendix
Page 6: the XE+EMD method is not present in a comprehensible manner. 1) p_k symbols are used without definition (tough I think I these are the network predictions p(\hat{y}=k|I)  2) the relation of the formula presented to the known EMD is not clear. The latter is a problem solved as linear programming or similar, and not a closed form formula  3) it is not clear what the role of \mu is and why can be set to 3 irrespective of the scale of metric D
page 7: 
The experiments show small, but consistent improvements of the suggested method over standard cross entropy, and improvements versus most competitors in most cases

I have read the reviews of others and the author's response. My main impression of the work remains as it was: that it is  nice idea with small but significant empirical success. However, my acquaintance with the previous literature in this subject is partial compared to the acquaintance of other reviewers, so It may well be possible that they are in a better position than me to see the incremental nature of the proposed work. I therefore reduce the rating a bit, to become closer to the consensus.
review_title: Nice idea, providing consistent (small) accuracy improvements in well conducted experiments

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

This paper proposes a prototype-based approach to learning with prior hierarchical knowledge of categories. The main idea behind their approach is to perform metric-guided penalization on top of a prototype-based cross-entropy loss. The penalization is scale-free, yet the loss can be computed in closed form, allowing for a hierarchical guidance to prototype locations.

Strengths:

The starting motivation and direction of the paper are clear. If hierarchical knowledge is available, it is intuitive to exploit such information. Care has been taken to make the research open and reproducible. Along with the submission, an anonymized repo and notebook with details are provided. Both help with understanding the approach and its workings in practice.

The scale-free component of the proposed regularization is interesting, most notably the fact that the corresponding loss can be computed in closed form. By making the hierarchical loss scale free, a clash with the cross-entropy loss - which tries to push prototypes as far away as possible from each other - is avoided.

Weaknesses:

The main weakness of the paper is that its innovation is limited, especially in light of missed recent works on prototype-based approaches using hierarchies in non-Euclidean spaces. The innovation of the paper is in Section 3.2, which means that the technical novelty of the paper is limited to a loss with hierarchical loss. More pressingly, relevant recent literature is missed on using hierarchical knowledge in deep networks. What the authors consider future work ("Among other promising avenues for further research, we plan to investigate the use of non-Euclidean embedding spaces, which are known to be well-suited for embedding hierarchies [...]", page 8), has readily been proposed, see e.g.:

[1] Liu, Shaoteng, et al. "Hyperbolic Visual Embedding Learning for Zero-Shot Recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[2] Long, Teng, et al. "Searching for Actions on the Hyperbole." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[3] Khrulkov, Valentin, et al. "Hyperbolic image embeddings." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. (note: this paper does not explicitly utilize hierarchical knowledge)

All three above papers, from the same conference as the often cited work by Bertinetto et al. (2020) in the submission, investigate prototype-based approaches in non-Euclidean spaces. [1,2] explicitly use hierarchical knowledge to steer the prototypes. Hyperbolic spaces are known to be a better fit for hierarchies, see e.g. [4].

[4] Nickel, Maximillian, and Douwe Kiela. "Poincaré embeddings for learning hierarchical representations." Advances in neural information processing systems. 2017.

Compared to recent works on prototypes in hyperbolic spaces with hierarchical knowledge, the paper provides limited novelty. As the authors state that moving the prototypes to non-Euclidean spaces is future work for this submission, the other recent works make the paper come across as somewhat outdated.

Empirically, the paper provides a comparison to multiple hierarchical alternatives on four datasets. The use of four datasets is appreciated, as it provides a clear picture of the strengths and limitations of the approaches. Besides missing comparisons to e.g. the supervised setup of [2], Figures 2 and Table 2 show that the difference with alternatives is minimal. E.g. the guided prototypes seem to only outperform soft labels on iNaturalist-19 in terms of Average Cost. Table 2 paints a picture that fixing the scale or using a coarser ranking-based approach hardly affect the Average Cost on all datasets and only provide some improvements on one of the four datasets.

Conclusion:

The paper addresses an open problem in deep learning literature in a clear manner. The open science setup of the submission is appreciated and the scale-free regularizer is interesting. In light of recent advances in hyperbolic prototype-based approaches with hierarchical knowledge, the innovation of this work is minimal. The empirical benefits of the proposed approach are also not always convincing. For the rebuttal, it would be interesting to discuss and compare the submission to hyperbolic hierarchical prototype approaches.

Opinion post rebuttals:

After reading the rebuttal and the other reviews, I remain of the opinion that more work is needed before warranting acceptance. This paper indeed learns prototypes on the fly in contrast to e.g. Mettes et al. 2019. That method was however not designed for hierarchical knowledge, while several recent CVPR papers were. Since the novelty over these papers is limited and direct comparisons are lacking, more research is needed.
review_title: Review

Meta-review: Thank you for your submission to ICLR.

This paper had somewhat dissenting reviews, but three of four reviewers felt negatively about the paper.  On the positive side, the reviewers noted good motivation for the problem, a good ablation study and, in some cases, good performance over standard cross entropy.  On the negative side, the reviewers noted limited novelty, missing discussion or comparison to prior work, and in some cases only marginal improvement over existing methods.

The positive reviewer still remained positive after discussion, but noted that their confidence was low on the paper and that they would defer to others' opinions.  The other reviewers still had several concerns after the discussion phase.  Ultimately, it seems that the paper could use some additional work before it is ready for publication.  I would strongly encourage the authors to keep the reviewer comments in mind when preparing a future version of the manuscript.
Decision: Reject