Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presented a new approach for evaluating hierarchical classifiers for predicting non-leaf nodes at multiple operating points. With the new evaluation metric, the paper shows that the widely used softmax classifier performs reasonably well in many cases, although a structured classifier might have an advantage in classifying unseen classes. The paper also proposes two new loss functions. One of them outperforms the softmax classifier in all examinations. The experiments are on ImangeNet and iNat classification tasks.
strengths_and_weaknesses: The paper is well written and easy to follow. A large amount of literature is discussed and adequately summarized. The comparison between the paper's and previous methods is succinct but informative.

Regarding significance, comparing hierarchical classifiers at multiple operating points sounds a must-have for the evaluation. Therefore I am slightly surprised that the paper mentioned that "almost no prior work has compared methods at multiple operating points" (in lines 5 and 6). If the above claim is valid, I believe this work provides a valuable evaluation of the problem and is a significant contribution. I like to admit that I am not an expert on this topic and will cross-check with other reviewers for the claim. To strengthen the significance, the author may explain why evaluation at multiple operating points is not popular or feasible without the method in section 5.

Technically, I do not find obvious mistakes, but the clarity of the key methods may have room to improve. For example, (1) elaborate more on why Eq-17 gives a result better than flat in Figure 1; (2) use an algorithmic block to summarize the process of section 5.

Regarding the experiments, the paper uses large-scale datasets, has proper baselines and prior methods, and examines interesting aspects such as flat classifiers at different levels and unseen classes. The results provide conclusive insights and are easy to interpret. 
questions: Does Eq-17 have any limitations? For example, how was its accuracy in the most popular classification setting that only predicts leaf classes?
limitations: Lines 340-344 have the discussion and make sense to me.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors consider the tasks of hierarchical classification where predictions can be done at different levels of granularity, which can be seen in datasets like ImageNet, where a super class can be spiders and its fine-grained classes are the different types of spiders.

The authors describe and compare different existing loss functions on a new metric class based on "Operating characteristic curve" on the  iNat21-Mini dataset. They also propose two loss functions: soft-max-descendant that scores internal nodes in the classification hierarchy, and soft-max-margin which is a soft version of the structured hinge loss.

Discussions have been made about how different loss functions behave based on the structure of the problem such as out-of-distribution and in-distribution.
strengths_and_weaknesses: 
Strengths:

- Hierarchical classification is very interesting to the research community.


Weaknesses:

- There is no theoretical motivation or practical motivation for using "Operating characteristic curve" as a metric for Hierarchical classification, why is it better than the other metrics in terms of reliability, correctness, etc.? For example,  [26] proposed an improved version of the mistake-severity metric that  "could give a wrong impression of improvement while the model might just be making additional mistakes to fool the metric" which is a good motivation for their work. That type of motivation is missing in this work.

- The document is poorly written as it is difficult to find out what the authors actually did. The authors do not clearly present their contributions. I suggest the authors summarize their contributions at the end of the introduction as a paragraph or bullet points.

- The related work is poorly written, there is a large list of existing methods described but the authors do not show how relevant they are to the proposed metric/loss functions. It is important that by the end of related work paragraphs to describe what the paper proposes in relation to existing methods.

- The authors spend so much text explaining previous methods and metrics without explaining why they should be there as they do not help in understanding the proposed loss functions and metrics. A simple citation and few sentence explanations of these methods would be sufficient. This amount of detail just adds confusion about finding out what the authors are actually trying to propose.

- The proposed soft-max-descendant performs poorly in so many of the settings so it is not clear what the motivation is for including it in this work.
questions: Can the authors explain the main contributions and motivation of this work? It is not clear why having "Operating characteristic curve" is important and why having the two loss functions "soft-max-descendant" and "soft-max-margin" are important as their performance is poor in many settings.

A lot of loss functions have been compared against a new metric, what is the takeaway? What loss function should we use when there is a hierarchical classification problem, and why is the proposed metric a more reliable approach than others?
limitations: The authors state that "One limitation of our general approach is that it may not easily scale to millions of classes, where embeddings can be more suitable." It is not clear why that is the case, why embeddings are more suitable, and what "general approach" really means. Please clarify. 

The societal impact is poorly written,  can you clarify what this quote means? "While hierarchical classifiers may mitigate some of the risks of misclassification due to their more logical modes of failure, this may enable more widespread use of automatic classification or lead human users to place excessive trust in the system." It is not clear to me what "logical modes of failure" mean, and I don't see why humans would place excessive trust in the system.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Topic: the paper reviews the performance of hierarchical classification for deep learning classifiers.

They propose as scoring for internal nodes to sum up the softmax components of the leaves.
They propose for the above scoring two loss functions. 
They propose in section 5 a way to compute curves over varying thresholds \tau for node probabilities p(y), where the graphs show correct versus exact and precision vs recall, with the note that these measures are defined in the sense of hierarchical measures and not as conventional precision vs recall for flat classifiers.

They evaluate these curves for a number of methods.
They run an experiment with simulated unseen classes which are part of the hierarchy.

The topic is relevant.
For a better score, section 5 needs to be verifiable and better understandable.

strengths_and_weaknesses: 
Strengths:
A certain novelty, with combinations of loss functions and section 5, if correct.
It is an overview evaluation for a number of methods.
The topic is interesting and relevant.

Weaknesses:

It has a not so great readability.
Some parts appear to be too brief, and it appears as if it would profit from an extended journal version.

-the meaning of the choice of C(y,\hat{y}) in line 181 is a bit unclear. One seems to sum I(y) values from ancestor nodes starting at the prediction \hat{y}.
-eq (7) needs a discussion
-the changes made to deepRTC need a discussion.
-eq (17) needs a discussion with respect to the concrete C term used.

-Section 5 is hard to understand. 

The ordering is a double ordering, with two criteria. How to sort with two criteria?

p(\hat{y}_{k}) \ge p(\hat{y}_{k+1}) and I(\hat{y}_{k}) \le I(\hat{y}_{k+1}). What to do if p(\hat{y}_{k}) \ge p(\hat{y}_{k+1}) and I(\hat{y}_{k}) \ge I(\hat{y}_{k+1}) for example ?

There is a \Delta seemingly out of nowhere. How to merge the sequence of pairs ? Why does it work? 
There is an unclear y in the definition of z_k which might be related to the \tau.

Section 5 reads as if one has to trust that it works as described. 

-a smaller issue: it could be worth to cite some pre-deep learning hierarchical papers (but that causes no reduction in scoring) like:

Blaschko, Zaremba, Gretton Taxonomic Prediction with Tree-Structured Covariances
Zweig, Weinshall, Exploiting Object Hierarchy: Combining Models from Different Category Levels
Marszalek, Schmid, Semantic Hierarchies for Visual Object Recognition
and the like
questions: Questions: 

Are the p(y) for all methods probabilities? if yes, what is the set over which they sum up? That can be put in a table.


How does computing the curves in Section 5 work?
The ordering is a double ordering, with two criteria. How to sort with two criteria?
What is varied in line 233 with z_k and the \tau when considering the z_k? There is an unclear y in the definition of z_k which might be related to the \tau.
How to merge the sequence of pairs \delta into \Delta?

What is the meaning of eq(7) given the specific cost function choice C?
What is the meaning of eq(17) given the specific cost function choice C? It seems to give a higher weight to nodes y' such that the prediction y is on the path from y' ? 



limitations: Some discussion has been made. No issue regarding potential negative societal impact.
review_title: null

Meta-review: metareview: The submission benchmarks several hierarchical classification techniques showing that flat softmax on the leaf nodes dominates most methods.  This is quite a negative result for previous works, indicating that efforts on hierarchical classification losses are often not leading to better results.  The authors introduce a loss function that does appear to result in  performance beating the flat classification baseline.  The writing style flows, there is a reasonably good review of the literature, and results appear to give some insights into methods for performing and evaluating hierarchical classification methods, which are complementary to the existing literature on a problem that has been studied in various forms for decades.  The reviewers were unanimous in their opinion that the submission is just over the threshold for acceptance at NeurIPS after the rebuttal process.
confidence: Certain
Decision: Accept