Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 1. Introduces a new multi headed architecture for explicitly incorporating class hierarchy into training.
2. Created a new metric to measure the impact of a miss-prediction according to class hierarchy tree distance
3. Created a new label hierarchy for a subset of ImageNet
3. Evaluated the performance of their new architecture under subpopulation shift
main_review: Strengths:
- This paper is well written. The architecture and evaluation methods are clearly explained and the research problem is well motivated.
- The proposed architecture is simple and computationally inexpensive which means it could be easy applied to any object recognition problem.
- On the datasets tested the authors demonstrated a performance benefit from using their training method.

Weaknesses:
- The proposed mulitheaded architecture is only evaluated on datasets containing images and a label hierarchy of living things. It may be the case that this limited range of object types have properties such as similar degrees of intraclass variability and the presence of distinct textures compared to other object types. Further experiments are required to prove that there are performance benefits to using this architecture.
- I would like to see more discussion and experiments evaluating the impact of the particular class hierarchy chosen for your custom dataset on your new models performance. As a baseline, what happens if subclasses are assigned to class groups randomly, what if the hierarchy is determined by visual features instead of semantic meaning (ex. feathers, scales, fur)? More experiments in this direction would provide a more convincing argument that your performance improvements are not only a result of the particular set of classes and hierarchy structure that you chose to use.
- In table 1, why is the source to source and target to target accuracy lower for Baseline-18 than it is for Subclass Level-18? Shouldn't we expect the opposite trend?
- Is there past work you could cite that supports the statement: "Humans on the other hand, seem to learn categories conceptually, progressively growing from understanding high-level concepts down to granular levels of categories." Couldn't humans also learn fine-grained categories first and then build them up into more abstract groupings?
summary_of_the_review: Overall, this paper demonstrates some promise that there could be an advantage to explicitly specifying a label hierarchy during training. More experiments are necessary to demonstrate this method really works more generally on a wider variety of object types and label hierarchies.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a new hierarchical classification method. Instead of using a flat classifier layer for all classes, the proposed approach trains a neural network with a hierarchy of classifier layers (attached to different blocks of the DNN), where each classifier layer corresponds to a specific level in the class hierarchy, and models the conditional classification probability given classes at parent level. The authors use the proposed method to tackle the subpopulation shift problem. Empirical results on a custom ImageNet dataset and the BREEDS dataset show improvements over baseline approaches.
main_review: **Strengths**
 - The problem is of great interest to the vision community.
 - Paper is well motivated, and method is technical sound.

**Weaknesses**
 - The proposal hierarchical classification method is not completely novel. For instance, [1] [2] proposed similar approaches / architectures for the hierarchical classification problem. 
* I have a particular concern of the per-level parameter sharing design. This design requires the class hierarchy be perfectly balanced (i.e. each parent class at the same level has the same number of children classes). The authors may want to comment on how the proposed approach apply to these non-perfect but practically-common situations.
 - Experimental results are insufficient.  It would be more convincing if the authors include comparison with other hierarchical classification approaches in the literature. Also, limiting the method to subpopulation shift problems seem unnecessary.


[1] HD-CNN: Hierarchical Deep Convolutional Neural Network for Large Scale Visual Recognition

[2] Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks
summary_of_the_review: Overall, contributions of the paper seem small. The proposal approach is not extremely novel and is limiting in practical settings (non-balanced class hierarchy). Experimental results lack comparison with literature hence not convincing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors address the problem of generalization under subpopulation shift where the labels remain identical in the new domain, however, their instances belong to different subcategories than in the source domain. The authors propose infusing knowledge both about these subcategories and about super-categories of the labels as additional supervision signals. For this purpose, they define a conditional training framework along with an adapted architecture and a hierarchical loss. The evaluation results demonstrate how their framework significantly improves robustness to subpopulation shift. 
main_review: I find the ideas presented are clear and straightforward to understand, and the evaluation results convincing.
The results are unsurprising given how the custom setup better addresses the goal of generalizing to subpopulation shift.
It is still informative to have a formal quantification of such generalization potential, although, such contribution renders the work as a limited experiment to address a custom problem or evaluate a specific question rather than a major contribution.

To this extent, my main concern is about the novelty of the presented work. In fact, several aspects claimed to be novel have been addressed several time in the literature over the past decade.
In particular, the idea of quantifying "misprediction impact" / "catastrophic coefficient" is very similar to the “hierarchical cost” by Deng et al. (ECCV 2010), the followup Hierarchy and Exclusion graphs (ECCV 2014). 
Also, the proposed architecture is similar in spirit to HD-CNN (Yan et al. CVPR’15).
I expected the authors to reference the above pieces of work and to compare their solution and results against them.
The authors did reference other related pieces of work, again, without comparing against them (dismissing the need by mentioning that "Contrary to these works, we utilize hierarchy as a way to mitigate the effect of subpopulation shift"). Nevertheless, these pieces share several aspects of the proposed solution even if the exact tasks they address are slight different.

With that, the current contribution is rather incremental. I would have deemed it otherwise had the authors provided theoretical insights about the problem of hierarchy, (e.g. as in the work by McClelland, Saxe and others).

References:

Deng, J., Berg, A. C., Li, K., & Fei-Fei, L. “What does classifying more than 10,000 image categories tell us?." European conference on computer vision. Springer, Berlin, Heidelberg, 2010.

Deng, J., Ding, N., Jia, Y., Frome, A., Murphy, K., Bengio, S., Li, Y., Neven, H. and Adam, H. “Large-scale object classification using label relation graphs” In European conference on computer vision (pp. 48-64). Springer, 2014.

McClelland, J. L., Z. Sadeghi, and A. M. Saxe. "A Critique of Pure Hierarchy: Uncovering Cross-Cutting Structure in a Natural Dataset." Neurocomputational Models of Cognitive Development and Processing: Proceedings of the 14th Neural Computation and Psychology Workshop. 2017.

Saxe, Andrew M., James L. McClellans, and Surya Ganguli. "Learning hierarchical categories in deep neural networks." Proceedings of the Annual Meeting of the Cognitive Science Society. Vol. 35. No. 35. 2013.

Yan, Z., Zhang, H., Piramuthu, R., Jagadeesh, V., DeCoste, D., Di, W., & Yu, Y. “HD-CNN: hierarchical deep convolutional neural networks for large scale visual recognition”. In Proceedings of the IEEE international conference on computer vision (pp. 2740-2748), 2015.

Language issues:
- at a the subpopulation
- One hot labels => One-hot labels
- if a neural networks
- are lesser catastrophic => less
- Each heads predicts
- Our networks makes 
- a batchsized binary matrix => unclear
- are superior than standard => superior to
- at level of => at the level of
- the the notion
- subclasses from the ImageNet => of ImageNet
- We now introduce some notation. => avoid vague statements involving “some”.

Formatting issues:
- Then the catastrophic coefficient, is defined => Then, …
- individually , => no space
- End all sentences with a full stop (e.g. the bullet points on page 2).
- for Toxicity classification => why capitalized?
- (Koh et al. (2021)) => avoid nested parentheses.
summary_of_the_review: The contribution is incremental, and many of the claimed novelty overlaps with the existing work (e.g. the references listed in the main review).
review_title: null

Meta-review: The paper studies subpopulation shift in object recognition when classes obey a hierarchy. It proposes an architecture, a relevant metric and a dataset (subset of imagenet).  The problem of classification in hierarchical label spaces is important and of great interest, and the effect on domain shift is interesting. Naturally, this problem was studied quite intensively over the years. 

Reviewers were concerned that the current proposal was not placed well enough in context of previous literature, both in terms of the method and in terms of experimental results.  Also, the paper would be strengthen if it provides more theoretical analysis about how the hierarchy helps with the domain shift. The authors addressed some of these issues in the rebuttal, adding references and highlighting the differences from previous methods, but the paper would need more time to make the proper experimental comparisons with previous work and subsequent analysis. As a result, the paper is still not ready for acceptance to ICLR in its current form.
Decision: Reject