Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper addresses hierarchical classification, where the classes live in a hierarchy, and the cost of a mistake is the tree distance between the nodes.

The paper tests the latest cool algorithms for hierarchical-aware loss functions, versus a very old idea: CRM. In CRM, you make your best estimate of the posterior probability of a class y given input x P(y|x), and then you make a final decision based on minimizing the expected loss.

There seems to be a belief that modifying the loss function to be hierarchy-aware is clearly better than doing boring old CRM. But there is not much evidence in favor of that hypothesis. This paper offers negative evidence for that hypothesis, with two experiments:

1.  By comparing hierarchical loss to top-1 loss with modified loss functions, there is a tradeoff, and there does not seem to be an advantage in using the modified loss function.
2.  For the top-k case, using CRM clearly dominates the proposals for modifying the loss function.

These support the use of CRM.

I find this paper to be really nice -- I'd far rather have a paper with good experiments with known algorithms, where I can learn something useful; than a paper with a new algorithm with somewhat useless experiments. So I would argue for acceptance.

One thing for the authors to think about:

When they test the calibration of the modified loss functions, they find them to be poorly calibrated. This is not surprising, since the modified loss functions are not proper scoring rules. They attempt to calibrate by using a softmax  with variable T. Wouldn't it make more sense to train exp(alpha_i x + beta_i) / \sum_{i=1}^N exp(alpha_i x + beta_i) ? that is, a gain and offset for all classes after the first one?

review_title: A nice paper that thoroughly tests CRM for hierarchical classification using latest NNs

final_score: rating: 7: Good paper, accept
review_content: This paper proposes to use conditional risk minimization (CRM) for hierarchy aware classification. The proposed method simply amends mistakes using a cost matrix with the lowest common ancestor information. The method outperforms SOTA deep hierarchy-aware classifiers by large margins at ranking classes with little loss in classification accuracy.

As the authors mentioned, CRM was already proposed several decades ago, so the novelty of this paper is limited. However, the paper does demonstrate the power of this old technique when equipped with modern deep learning tools. Also, the simplicity and intuition are much appreciated.

I myself am not an expert in image classification. However, for text classification, recent studies (e.g., [1]) shows that using binary cross-entropy (i.e., viewing a multi-label classification problem as L binary classification task, where L is the number of classes) can achieve higher performance than multi-label cross-entropy. In this case, it is possible that more than one label can have p(y|x) > 0.5. I wonder whether your approach is still effective for models using binary cross-entropy. It would also be better if the authors can show some experimental results on hierarchical text classification.

[1] Liu et al. Deep Learning for Extreme Multi-label Text Classification. SIGIR'17.
review_title: A straightforward technique with satisfying performance

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 
The authors propose a model to improve the output distribution of neural nets in image classification problems. Their model is a post hoc procedure and is based on the tree structure of WordNet. The model revises the classifier output based on the distance of the labels in the tree. Intuitively, their solution is to pick the candidate label that is located in the region of the tree with a higher accumulated probability mass value. They also experimentally show that the previous evaluation metrics are inconclusive. 

Pros:
- The authors provide a different perspective on the evaluation procedure of the previous studies and experimentally show that it was incomplete. This is an important finding.
- Their experiments are thorough.

Cons:
- The article lacks enough novelty: The problem has been investigated before. The solution is not novel. The WordNet tree structure has been extensively used in the information retrieval community before.
- The article is not written well: There are informal vocabulary in the paper (e.g., “something similar” or “grossly miscalibrated”). There are also typos (e.g., see the paragraph before Theorem 1). In Section 3 it is not formally stated that the tree structure is derived from WordNet (the authors mention this in Abstract section). In Section 4 the baselines are not cited!
review_title: The authors show that previous related studies have used an incomplete metric for evaluation, which is an important finding. But their solution lacks enough novelty.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper addresses the problem of hierarchy-aware classification, which utilizes a hierarchy to specify certain mistakes as being worse than others. They make two main contributions. 

First, they claim that the metric used in prior work, average mistake severity, is flawed because it rewards methods that make many "easy" mistakes, as opposed to fewer, "harder" mistakes. Based on this analysis, they claim that no prior methods actually improve over the simplest baseline of cross-entropy (i.e. doing nothing).

Second, they introduce their own method, an adaptation of the classical CRM framework. They argue that CRM:
1) Is the only method to improve over cross-entropy under average mistake severity (the metric from prior work they argue is flawed)
2) Beats all other metrics under their new, improved metric
3) Improves on the calibration of the predictions

Strengths:
The method is very simple and easy to understand, builds upon existing work, and I could probably implement it from scratch in 30 minutes on top of my existing models. While ML reviewers very frequently about a lack of complexity, I think this is a great strength.

The methods section is very clearly written, and it was quite easy to quickly understand what the method is doing.

Weaknesses:
I am very concerned about the experiment sections. 

1)
To my understanding, Figure 2/Section 4.1 are factually incorrect. In particular, it appears that the soft-labels technique does essentially the same, or better than, CRM, across all fronts. In detail,
a) 
In Figure 2(a), the leftmost softlabel point is equal to or better than CRM (and cross-entropy)

b) 
Figure 2(b) really concerns me, as it appears that the hyperparameters have been chosen to make a fairly narrow point - that it is possible to have low average hierarchical distance, and high top-1 error. I agree that that indicates a problem with the metric. 

However, the authors make the fair broader claim that CRM is the only method which beats cross-entropy, which I do not think is justified. Looking at Figure 4 in A.1, it is readily apparent that choosing different hyperparameters for existing methods would yield similar error distributions to CRM. Having these plots in the appendix, combined with claims that the authors chose the best hyperparameters, feels a bit misleading.

c)
As in 1), soft labels is essentially on top of CRM and Cross entropy (for iNaturalist19, it looks like a higher beta value would be directly on top, it's unclear why the authors did not extend the curve further)

2)
These results, at first blush, seem fairly impressive. For the leftmost plots, I am concerned that the authors are using subpar hyperparameters, similarly to 1)(b) above. Strangely, in this instance the results for other hyperparameters are not included in the appendix. The remaining experiments are fairly convincing, though.

Reccomendation
I do not think this paper can be accepted in its current form. While I suspect that CRM is a good method that I would like to use, some of the core arguments (Figure 2/Section 4.1) in the paper appear to be fatally flawed.

Smaller notes:
- The paper could use an additional proofread, as there are often odd phrasings. I found the experiment section particularly hard to follow
- The acronym HXE is never defined, or linked to a citation
review_title: Promising, simple method but flawed experiments/arguments

Meta-review: The approach explore the use of Conditional Risk Minimization (CRM) as a post-hoc operation to amend a classifier decision by averaging a prior class hierarchy. The authors show that it is beneficial for ranking predictions without sacrifying top-1 accuracy.

The rebuttal period clarified some reviewers' concern on paper presentation and experiments, and all reviewers recommend acceptance after the discussion period.

Although the approach is simple and directly revisits the use of CRM for deep models, the AC considers that the contribution is meaningful, and that the proposed method provides predictions with good ranking and calibration properties. The paper also sheds light into interesting issues in state-of-the-art methods integrating class hierarchies during training.
The AC therefore recommends paper acceptance.

Decision: Accept (Poster)