Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work authors adopt hierarchical classification for OOD detection and based on hierarchy they make predictions at varying level of granularity which could provide enhance examinability w.r.t OOD. This is indeed useful for OOD detection or open-set detection, and authors provide various qualitative & quantitative analysis based on adopted hierarchical classification.
strength_and_weaknesses: Strengths:

Paper is well written, and presents good analysis by adopting & proposing various methods and metrics within in hierarchical classification such as over-prediction vs under-prediction & average hierarchy distance, etc. This is a useful direction in general to be considered for real-world use-cases. It is interesting to observe that there is no apparent benefit to visually derived hierarchies vs human-defined semantic hierarchy in current evaluation setting.

Weakness:

As the current method is fit based on likelihood in an autoregressive manner defined by hierarchy, it might important that at each level predictions are calibrated. Authors currently completely ignore calibration at intermediate levels which would result in more in-correct/over-predictions. It might be worthwhile for authors to consider calibration at each intermediate level. [3] also considers calibrated models at intermediate levels and also is evaluated for novelty detection within hierarchical classification framework, and it might be worthwhile to include this another baseline to consider.

Hierarchical classification is already adopted, and initial results are presented OOD setting in MOS baseline which authors included as one of baselines, and though proposed method is better in case of ImageNet 100 at fine-level its worse or on-par at coarse level especially for ImageNet1K.

Also authors it might be worthwhile to evaluate current method standard ImageNet openset detection benchmarks and also fine-grained/semantic OOD detection [1,2]

References: [1] https://eval.ai/web/challenges/challenge-page/1041/overview or https://www.cs.cmu.edu/~shuk/open-world-vision.html [2] Vaze et al. OPEN-SET RECOGNITION: A GOOD CLOSED-SET CLASSIFIER IS ALL YOU NEED? (ICLR 2022) Proposes imagenet benchmarks & other fine-grained OOD detection tasks. [3] K. Lee et al. Hierarchical Novelty Detection for Visual Object Recognition (CVPR 2018)
clarity,_quality,_novelty_and_reproducibility: Though the proposed evaluation setting and analysis is useful, because of lack of calibration & additional baselines, evaluations it is unclear interpret conclusive results under current evaluation. Also, it might be worthwhile to include a table for hierarchical methods to easily interpret overall results. Would encourage authors to include accuracy at each level and also percentage of in-correct, under-prediction, over-prediction & hierarchy distance for proposed method, MOS in a table.
summary_of_the_review: Authors present a good evaluation study by adopting hierarchical classification for OOD, but currently lacks novelty & further analysis would be required to interpret results/findings. I would encourage authors to address raised issues!
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper applies a hierarchical supervised classifier on which a measure whether an observed image should be considered "in-distribution" or "out of distribution."  This considered both in quality -- how specific ("fine-grained") the classification is -- and degree, by at what point in the hierarchy to consider a sample "OOD." 
strength_and_weaknesses: The idea of learning a hierarchical classifier is valuable, in terms of giving better insights into the results, scaling to larger numbers of classes and assigning a more appropriate level of granularity to the classification. 

In concept the classifier's marginal probability ascribed to the image -- the probability that it would arise in the population, irrespective of it's class should give a measure of ID versus OOD.  This would be a principled alternative to the measures used in the paper. 


clarity,_quality,_novelty_and_reproducibility: There are several places where the thread of the argument is not clear-- such as what would be a clarity test for a sample being OOD.  How could one construct an oracle?  What would it mean to say that a case does not belong to the population, as opposed to the concepts covered by the training sample? Similarly the paper speaks of "otherness." How is this defined?

The paper seems both to be concerned with the central ID / OOD question (which lacks some clarity) and the loosely related question of the performance of flat versus hierarchical classifiers. 
summary_of_the_review: Based on a cursory understanding of the work, it appears to have value, but it's hard for me to put a finger on exactly the novelty and soundness of the work.  It does illustrate some interesting characteristics of hierarchical image classification, but comes to no clear conclusions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper:  In this paper, the authors propose a hierarchical classification framework for both fine-grained and coarse-grained OOD detection. This involves creating a tree based structure from the set of classification labels to establish a hierarchy. Once established, the authors introduce a new loss function to train the model maintaining this hierarchy. The authors also propose a new OOD detection score based on prediction path entropy.

I have reviewed this paper before and the authors have addressed most of my previous concerns.

strength_and_weaknesses: Positives:

- I completely agree with the authors that such an approach is highly interpretable and this is a very relevant research direction in AI safety. 
- Also, providing a less specific correct coarse prediction is more useful than an incorrect highly specific prediction, which the method provides. In practice most encountered OOD samples would be such near-OOD samples where such a response is preferred over just calling out OOD samples.
- The paper provides a very interesting set of experiments over fine/ coarse grained OOD prediction and analyzes the result.
- Very exhaustive comparison with existing methods.

Potential improvements:

- One key advantage of this approach is the explainability/ interpretability. Some metrics on this would be useful. For example: In depth case study on "partial OOD" samples (which is OOD in fine-grain but ID in mid-level of hierarchy) and mistakes on detecting this would be interesting.
- Similarly, some downstream cost analysis (penalizing different kinds of mistakes at different hierarchies) could reflect how such a framework would be useful in the real world.
- Showing some potential failure modes and discussion on them would be very useful for the community, especially to pave the direction for future work.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written, clearly explaining the methods and experiments. The novelty of the work lies in the framework designed towards explainability aspect of OOD detection. I would strongly recommend the authors to open-source the code for reproducibility. 
summary_of_the_review: I would recommend this paper be accepted. The AI safety community has focussed on developing OOD detection methods in a binary sense accept/ reject. Most papers develop methods which improves the OOD-AUROC marginally. We are at a state where just 1-2 points improvement using a new method won't have any practical value. This paper looks beyond the traditional binary treatment of OOD detection. This is very useful in real world deployment where just showing OOD without explanation was degrade user trust and lead to a bad experience overall.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Summary:

This paper examines multi-class classification in settings with out-of-distribution (OOD) data. Here, this refers to test data where the labels are not available at training time. The proposed solution is hierarchical classification, with the leaf nodes of the tree being the classes seen in training. The tree is given, so intermediate nodes may be interpretable. For classes not seen in training, the goal is to classify as deep within the tree as possible and ultimately conclude that the sample does not match any seen labels. To achieve these goals, training is done with the proposed hierarchical OOD loss. A decision to stop classifying deeper into the tree is made by a threshold on prediction path entripy measures. Experiments are conducted on a fine-grained OOD dataset and a coarse-grained OOD dataset. The difference is that fine-grained OOD datasets may exhibit deep classifications within the tree even for OOD samples, but coarse-grained OOD datasets mostly contain OOD decisions at a shallow level. Numerical performance appears reasonable.

strength_and_weaknesses: Strengths:

* Seems like a reasonable approach.


Weaknesses

* The stopping condition, Section 3.4, merits further discussion on terminology (also for Section 5.1, 5.3). 
* Additionally, a histogram showing the distribution of entropy values for in distribution data (ID) and OOD might be informative on how well one might expect to determine the threshold.


Question

* Not really a strength or weakness, but the hypothesis seems to be that if a OOD sample is given, then at some node on the correct classification path each child should be equally likely (eqn 4). Is this really reasonable? Does this really appear to be the case when OOD (and ID) errors? Section 5.3 might be a place to expand the analysis.


clarity,_quality,_novelty_and_reproducibility: Clarity, Quality, Novelty, Reproducibility

* The paper is mostly clear. I don't like the OOD terminology, but that appears to be standard.
* The proposed methodology and results do not appear to be particularly novel. Various hierarchical losses have been proposed before. The novelty/benefit of this particular loss should be better highlighted.
* The numerical results don't seem to be huge improvements.
* Seems sufficiently reproducible.

summary_of_the_review: Summary

* There are some potentially useful, if not too interesting, ideas.
* In any case, the terminology and analysis should be improved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes fine-grain inference for hierarchical OOD detection, i.e. at varying levels of granularity. 
The reviewers found the paper well-written. The main concerns were the following:
- "Hierarchical novelty detection" and "hierarchical outlier detection" have been already investigated in several previous papers, hence the novelty appears to be a bit limited. 
- It's not clear if/how the proposed approach is superior to existing approaches (cf. the Imagenet-1k experimental results in Table 1). 
- Note: Reviewer NPqs flagged concern about OOD terminology, but that's not really specific to this paper, so I ignored that in my decision. 

Overall, the reviewers felt that the contributions are only marginally significant, and the current version falls below the acceptance threshold. I encourage the authors to revise the draft based on reviewers' feedback and resubmit to another venue.
justification_for_why_not_higher_score: Limited Novelty as several papers have already investigated hierarchical outlier detection
Contributions are only marginally significant
Decision: Reject