Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new framework for cost-sensitive hierarchical classification. First, they decompose it into level by level learning to abstain (with different abstain costs per class) sub-problems. To solve these subproblems, authors apply deep distributionally robust learning (DRL) approach that directly minimizes the abstaining loss (based on Fathony et al. 2018). These two elements create a method named the Layer-wise Abstaining Loss Minimization method (LAM). The proposed method is compared with DARTS on two datasets and achieves attractive performance. The authors also demonstrate that this decomposition makes it easier to achieve the desired performance profile by adjusting abstaining losses of the layers.
main_review: Strengths:
+ The paper has clear motivation.
+ The organization of this paper is good, and it's easy to follow.
+ The proposed method seems to achieve attractive results compared to DARTS.
+ The decomposition into level-wise learning to abstain seems to help find the desired "performance profile" - making it easier by reducing the number of parameters.

Weaknesses:
- I lack comparison with some other methods. Authors claim that _"LAM achieves a lower hierarchical cost-sensitive loss in high accuracy regions, compared to previous methods and their modified versions for a fair comparison"_ but there is really only one baseline method - DARTS. It would be nice to see also the results for some other methods and some simple baseline to get a better idea about the benefits of the proposed approach. As mentioned in the introduction Bayse-optimal solution based on probability estimates from standard DNN would be a nice baseline. 
- The proposed prediction procedure over the hierarchy is quite simple and can be applied to any algorithm that can abstain. Comparison with other algorithms within the same prediction procedure would show the benefit of deep DRL.
- The authors use two datasets (Aves, cell classification), but most of the plots present the results only for Aves. The number of datasets is already small. At least, I would like to see all the results for these two. 
- Deep DRL is directly based on the work of Anthony et al. 2018.
- Proposed method can be only applied to hierarchies where all the leaves are on the same level.
summary_of_the_review: I believe this is a solid work but with a limited scope of contribution. I'm not that familiar with some of the related work and used datasets, but I think additional comparisons are needed to assess the attractiveness of the proposed approach correctly. That is why in my opinion, this work is now marginally below the threshold.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studys cost-sensitive hierarchical classification problems. The novelty is very limited, the experiments are vey tirial. I recommend to reject.
main_review: The paper studys cost-sensitive hierarchical classification problems, and aims to propose an efficient method. But the paper contains many issues.
1, The paper claims that "Cost-sensitive loss is hard to optimize since it is non-smooth and non-convex". But how did the paper address this issue? what is the time cost of the proposed methods? The experiments also do not report the time cost the proposed method. Since the paper focuses on the time complexity issues, but the proposed method did not show any benefit of time both from theory and experiments. This is a main major issue.
2, The distributionally robust learning approach is uesed as the optimizer. But the time cost of DRO is also very high. How can you use DRO for fast optimization?what is the time cost？can you report the time in experiments？can the proposed method converge？can you provide both theory and experiments support for convergency？
3, The data sets used in the paper is very small and trivial. Given these data sets, many existing cost-sensitive hierarchical classification methods can be efficiently optimized. Can you test your method on imagenet dataset and larg-scale NLP data sets？
4.There many methods aim to devide the cost-sensitive learning problem into some sub-problems. What is the advantage and disadvantage of the proposed method over existing methods? 
summary_of_the_review: 1, The paper claims that "Cost-sensitive loss is hard to optimize since it is non-smooth and non-convex". But how did the paper address this issue? what is the time cost of the proposed methods? The experiments also do not report the time cost the proposed method. Since the paper focuses on the time complexity issues, but the proposed method did not show any benefit of time both from theory and experiments. This is a main major issue.
2, The distributionally robust learning approach is uesed as the optimizer. But the time cost of DRO is also very high. How can you use DRO for fast optimization?what is the time cost？can you report the time in experiments？can the proposed method coverge？can you provide both theory and experiments support for convergency？
3, The data sets used in the paper is very small and trivial. Given these data sets, many existing cost-sensitive hierarchical classification methods can be efficiently optimized. Can you test your method on imagenet dataset and larg-scale NLP data sets？
4.There many methods aim to devide the cost-sensitive learning problem into some sub-problems. What is the advantage and disadvantage of the proposed method over existing methods?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents an approach for hierarchical cost sensitive classification in which abstentions are allowed. It shows a bijection between original cost sensitive problem and the set of layer wise abstaining losses. It is based on using the existing distributionally robust cost sensitive classification and extending to it to the hierarchical setup. The proposed methodology is demonstrated on birds and cell classification datasets, which are claimed to be large-scale. It is compared to a relatively old DARTS method from 2012.
main_review: =========Positives==============
+ It presents a formal approach for the hierarchical classification problems which otherwise somewhat unclear to formalize, have varying performance metrics, and based on ad-hoc methods. 


===========Negatives=========
- The significance of the work seems somewhat limited. Even though it is claimed to work on large-scale datasets, this is much smaller than classification problems for hierarchical classification consisting of thousands of labels  [1,2,3]. These datasets also have fat-tailed distribution of instances among labels, and the authors need to check the feasibility of the proposed approach on such settings as well, or discuss the limitations appropriately.
- The main premise of the proposed approach relies on abstention as an option. What if that is not an option, and one needs to make a hard choice as part of the problem formulation. 
- The paper relies heavily on existing works such as Distributionally robust cost sensitive classification has been discussed in [4], and the main contribution seems to be extending to hierarchical classification with abstention option. This does not seem enough to be a significant contribution. The formal part of the contribution in Lemma 2 needs to be proven completely instead of leaving it out mentioning simply by use of induction. 
- The experimental evaluation is performed against a 2012 approach (DARTS). Is there no work for the last ~10 years which relevant and can be used for comparison. If so, these must be used, otherwise it again seems to go back to the point about the significance of the work and potential impact in the ICLR community.

[1] On Flat versus Hierarchical Classification in Large-Scale Taxonomies, NeurIPS 2013
[2] Recursive regularization for large-scale classification with hierarchical and graphical dependencies, KDD 2013
[3] Learning taxonomy adaptation in large-scale classification, JMLR 2016
[4] Consistent Robust Adversarial Prediction for General Multiclass Classification, (https://arxiv.org/abs/1812.07526)
summary_of_the_review: The paper needs to further motivate the significance, evaluate on truly large-scale settings, discuss related works and compare with more recent methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The submission study the problem of cost-sensitive hierarchical classification (CSHC) with given label taxonomy via learning to abstentions each layer within the hierarchy. Indeed, 
1. CSHC subject has had many researches while using abstention or reject decision has also had many works in flat classification, briding both seems relatively few for which the authors present LAM to achieve a so-called new method. 
2. Using DRL framework to solve the learning to abstain problems in each layer makes optimization almost decouplable layer by layer, the rationale behind its strategy should attribute to the proved bijective correspondence between the hierarchical cost-sensitive loss and the set of abstaining losses.
3. LAM achieves better performance on limited benchmarks.
main_review: 1.  The strengths:
1.1) using a learning to abstention for CSHC and providing a LAM via efficient optimization of DRL framework layer by layer;
1.2) Comparing DARTS, LAM achieves total boost in performance.
1.3) The strategy is flexible to great extent.
2. The weaknesses:
2.1) Insufficient review on existing works.
There have existed related works devoting to this topic, however, the authors missed them, e.g., 
[m1] Cost-sensitive learning of hierarchical tree classifiers for large-scale image classification and novel category detection.
[m2]Classification with rejection based on cost-sensitive classification.
[m3]Selective Classification via One-Sided Prediction,
While, some of the works the authors have  reviewed are NOT hierarchical classification topic, e.g., (Liu et al., 2019）
2.2) Inappropriate descriptions such as
 For ”most of the previous methods only discuss the case of low-dimensional linear features and do not incorporate deep neural networks“ ，will which new challenges  yield when ”incorporate deep neural networks“ ?
2.3) Lack of the reason why to select DRL for modelling seems NOT to be given! e.g., for its consistency like "Consistent algorithms for multiclass classification with an abstain option"!
2.4) what relation between optimizations at each layer and on all layer is mathematical formulation?
2.5) Insufficient experiments
Why to just use a DARTS (2012) as a baseline, the authors should clarify the reason of just comparing the baseline. In addition, due to the use of DRL in this paper, a strategy different DARTS,  it is unclear whether the boost of performance is from such DRL! 
summary_of_the_review: In my opinion, incorporating "learning to abstention"  to CSHC is somewhat interesting and new, in particular,  the authors provide the bijective correspondence between the hierarchical cost-sensitive loss and the set of abstaining losses to make the layer-by-layer efficient optimization possible, while the proposed LAM achieves better performance comparing DARTS despite of insufficiency.
review_title: null

Meta-review: The authors tackle the problem of cost-sensitive hierarchical classification. They decompose the problem into level-wise learning-to-abstain sub-problems, and apply the distributionally-robust learning (DRL) technique to minimize the abstaining loss. The proposed approach is compared with a few competitors on several data sets. The reviewers find the key idea in the work, namely leveraging DRL as the key technique to solve the decomposed problem, to be somewhat interesting and new. Some of the reviewers find the motivations clear, while others believe that the paper could use a better positioning to connect the motivation with the significance of the technical contributions. 

While the authors have extended the discussions on related works and added some additional experiment results during the rebuttal, the reviewers generally agree that the improvements were not sufficient to warrant acceptance. Most importantly, the few baseline competitors and the small-scale data sets make it hard to convince the readers about the validity of the proposed approach. In particular, the scalability of the approach to larger-scale data sets remains questionable, and the spectrum of baseline competitors, both in terms of breadth and recency, is not sufficient. Some reviewers suggest the authors include time/efficiency/convergence analysis of the proposed approach. Furthermore, the authors are encouraged to clarify the significance of contributions, explain the choice of DRL, and deepen the discussions on related works in future revisions.
Decision: Reject