Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new method for learning structured representations from hierarchical annotations. A typical approach for learning representations in a supervised setup is to train a classifier that assigns a single label to a given image. This approach, however, might result in suboptimal representations as samples from all two classes are treated similarly, i.e. misclassifying a specific class of dog with another dog is penalized similarly to misclassifying it as a cat. The proposed method uses freely available hierarchical semantic annotation to enhance the learned representations. Their proposed method involves a new distance metric of two nodes in the hierarchical semantic tree used as an auxiliary loss function that preserves similarity between a pair of given samples on top of standard cross-entropy for classifying them. 
strength_and_weaknesses: Strengths:

- The paper follows an interesting direction, which has good potential.
- The proposed method is simple and makes sense.
- The paper is well-written and easy to follow.

Weaknesses:

- The authors aim to learn structured representations. However, there is no quantitate evaluation to show that their method results in more structure in the representation space. They only show a t-SNE plot of their method and compare it with the baseline, which is a good sign but not a thorough evaluation. They could evaluate, for instance, the OOD task, which benefits from the more structured representation [1].

- The evaluation datasets are limited. It would be good to show some results on ImageNet(or at least a smaller version of it, such as Tiny Imagenet.)

- The paper misses some baselines. It would be better to compare with $t(.,.)$ in table 1. Moreover, comparing with [1] would be interesting.

-  I am a bit confused by the term ERM. If I am not mistaken, ERM stands for empirical risk minimization, and the authors refer to Eq 3. by ERM, which is the cross-entropy loss. But all losses in this paper minimize empirical risk. Also, why it's performance is too low in Table 1.? 

[1] Hoffmann et al. Ranking Info Noise Contrastive Estimation: Boosting Contrastive Learning via Ranked Positives.




clarity,_quality,_novelty_and_reproducibility: The paper is clear and well-written and looks reproducible. The proposed method is simple, and I think the novelty would not be an issue if the authors could provide more empirical evaluation and fair comparisons. 
summary_of_the_review: Overall, the proposed method is simple and makes sense. But it needs to be supported by a better and fairer experimental setup.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper develops a cophenetic correlation coefficient (CPCC) regularizer based on the given class hierarchy for classification.  The proposed regularizer is computationally lightweight and easy to implement. They demonstrated that the regularizer can learn more interpretable representations due to the preservation of the tree metric. Experiments six real-world datasets show that the proposed method achieves better in-distribution generalization as well as under sub-population shifts.
strength_and_weaknesses: Strengths:

[+] The idea of preserving the class hierarchy in learning representations is interesting and sounds reasonable.

[+] The preliminaries give good background knowledge about the class hierarchy and the CPCC.

[+] The figure demonstrates clearly the class hierarchy.

Weaknesses:

[-] The motivation for embedding class hierarchy is not convincing. For instance, why does the model need to know whether a digit is even or odd? The pre-defined class hierarchy in MNIST is artificial, which would not make sense to the model. Intuitively, 6(even) is more similar to 9(odd) rather than 4(even).  Moreover, the class hierarchy would not always available in practice. The title of this paper lets readers expect to see how to explore the class hierarchy rather than just using it.

[-] The method is a bit short of important details about using the weights of edges in the tree graph to compute the weighted length of the shortest path.  Moreover, if there are several shortest paths between pairwise nodes in the general graph, how does the proposed tree metric deal with it?

[-] The paper argues that the tree metric is invariant to rotations of the tree and the changing of the root node. In practice, if the class hierarchy is pre-defined, is it reasonable to change the root node? With the class hierarchy, when does the tree metric work better than the dendrogrammatic distance in a real-world system?  If you wish to show the benefits of the proposed metric, extensive ablation studies are needed to compare the proposed method with other baselines embedding the class hierarchy.  And it would be better to provide more deep insights about the proposed metric. 

[-] Is there any theoretical guarantee to make sure that the proposed regularization converges? (Especially the main body of this paper is the regularization.) Is the mini-batch training strategy equivalent to the designed strategy in Eq. (2) where the regularization is constructed on the whole dataset? 

[-] In the experiments, due to the use of extra information about the class hierarchy, the better performance of the model with CPCC is not that surprising. It would be better to give more deep insights through the investigation of the regularization.

a). It would be more convincing to investigate the effects of CPCC on different objectives.

b). According to the Fine ACC and Coarse ACC in MNIST, the baselines with CPCC does not always outperform the baselines without CPCC. Here we need some explanations about the failure cases. 

c). Based on the Mid Acc and Coarser Acc, this paper provides a good way to show the one-shot generalization. However, it is necessary for baselines with CPCC to show the differences before and after the fine-tuning.   

d). What is the effect of the number of layers in the hierarchy classes in the CPCC?

e). Are there any benefits of regularization in solving the out-of-distribution problem?

clarity,_quality,_novelty_and_reproducibility: Clarity: overall good.

Quality: fair.

Novelty: weak.

Reproducibility: could be reproduced.
summary_of_the_review: The reviewer thinks the paper is marginally below the acceptance threshold. The main contribution in the method section is the proposed metric. However, the paper needs to provide more theoretical analysis and ablation studies to show the effectiveness of the proposed metric.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents CPCC loss for learning deep representation with label hierarchy. It is based on the CPCC score, while re-defining the distance metric, as a shortest path length between two nodes. Overall proposed method sounds reasonable and looks simple. In experiments, the proposed method is evaluated on MNIST (with odd/even as coarse class), CIFAR-100 (with pre-defined coarse and fine-grained label hierarchy) and BREEDS, with a few baseline models including multi-task of coarse and fine-grained loss, curriculum learning, etc. 

In experiments, authors evaluate a few metrics -- shillouette score, CPCC score, t-sne, etc -- to evaluate learned structure. W.r.t. defined metrics, the proposed method showed improved scores over baseline methods. In addition, the paper presents one-shot generalization experiments and demonstrate effectiveness on knowledge transfer.
strength_and_weaknesses: * Strength
  - The paper is clearly written.
  - The formulations are clean and well motivated.
  - Experimental results show that the method works as expected. Results on one-shot generalization particularly highlights the effectiveness of learning with CPCC score to embed label hierarchy.

* Weakness
  - While I do not closely follow this topic of learning with label hierarchy, it seems there are multiple previous works on deep representation learning with label hierarchy. For example, [[Zhang et al., 2016](https://arxiv.org/pdf/1512.02895.pdf)], [[Bertinetto et al., 2020](https://openaccess.thecvf.com/content_CVPR_2020/papers/Bertinetto_Making_Better_Mistakes_Leveraging_Class_Hierarchies_With_Deep_Networks_CVPR_2020_paper.pdf)]. In the current writeup, the related work on learning with label hierarchy is too short and narrow. Also the baseline considered seems a bit too weak and naive.
  - Experiments on one-shot generalization is not particularly interesting nor significant. For example, in experiment authors introduced "coarser" or "mid" level, which are both coarser level of "coarse" and "fine" class hierarchy. Even though one can train a new classifier using one data point from with "coarser" or "mid" level labels, aren't we supposed to do it in a "zero-shot" way by simply leveraging "coarse" and "fine" grained classifiers (i.e., for mid-level classifier, one can simply define a rule on fine-grained classifier)? Generally, I do not think this is an "one-shot" problem since 1) "coarser" and "mid" level labels can be derived from the "coarse" and "fine" level labels (e.g., "mid" level labels of odd & <= 5 is a union of 1, 3, 5 at fine-grained level labels), 2) many "coarse" and "fine" level labels are given during the model training already. 
  - In this regard, more meaningful experiments would be one-shot or few-shot generalization to unseen classes, i.e., can we learn a fine-grained representation of class "0" by leveraging knowledge of its superclass ("even") and neighbors ("2, 4, 6, 8")? 
clarity,_quality,_novelty_and_reproducibility: * The paper is well written.

* Deep representation learning with CPCC score as a regularization might be new, but the lack of comparison, both in related work and empirically, makes it difficult to evaluate its novelty.

* The paper provides sufficient details for reproducibility.
summary_of_the_review: The paper is very well written and clear. The idea of using CPCC score as a loss is well motivated. However, the paper does not fully demonstrate the effectiveness of embedding label hierarchy into the representation to downstream task. Many of the metrics are about whether the model learned label hierarchy or not. It is good that one could train a model as expected, but it is unclear what it is going to be useful for. The paper demonstrates the effectiveness on one-shot generalization for mid or coarser level classification tasks, but the task is somewhat superficial and may be easily solved by combining scores of finer-grained classifiers. Moreover, there are previous methods that regularizes deep representations differently, which should have been compared.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is about structured representation learning for image classification at different semantic level of granularity. The author proposed to leverage class hierarchy and relationships by introducing a regularization function using the CPCC in addition to the cross-entropy loss. They introduced a measurement of distance between classes structured in a tree and enforce the network to follow the rules of this distance metric in the deep space. They perform on 6 image classification datasets with different structured representation learning paradigm such as multi-task learning, curriculum learning and others.
strength_and_weaknesses: Strength:
- This paper is very well-writtent and well motivated. 
- The authors discuss the limitation of their method and provide arguments on results that do not outperform the state-of-the-art.
- The figures are very informative and provide a good insight on the ideas that motivated this work.
- The utilization od CPCC as a regularizer seems novel.
- Strong experimentation showing the strength of the method on a wide variety of datasets.

Weakness:
- There is only one contribution, while it is well detailed, it could still qualify as an incremental improvement to existing work.
- The lack of interpretability is a problem, even if it is discussed by the authors many times in the paper. For example in the MNIST dataset, the authors mentionned that the classes are sorted to the nodes as evens and odds parent classes, and that there is no visual similarity between the leaf node from the same parent. The question that arises is : Is it critical that the tree structure follow some logic ? It would have been interesting to see results on a random tree hierarchy and see if the results are the same. If they are not, then, simply breaking the class permutation invariance seems advantageous. Clearly for other datasets particularly with animals classes, visual similarity seems to be a factor in the tree hierarchy so why is it important in some cases and not others?
- There is also a concern regarding the tree structure and current capacity of deep neural networks. The tree metric is said to be weighted with respect to edge weights in section 3.1 but the difference in magnitude between this tree distance function and the euclidean distance between deep features seems to be on completely different scales. How is this addressed in the regularization function? Is there some additionnal normalization term?
clarity,_quality,_novelty_and_reproducibility: Clarity:
- This paper is clear and well structured.
Quality:
- The explanation of the main method, the figures and the appendix are proofs of a high quality report.
Novelty:
- A single contribution being the regularization term for structured image classification.
Reproducibility:
- This paper looks possible to replicate.
summary_of_the_review: The related work and citation seems to be accurate and recent. The figures are informative. I easily tend to accept this paper because of the quality of the explanation and the thoughrough experiment section. What would make this paper a candidate for being highlighted would have been an additional contribution.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Four experts reviewed the paper. Two reviewers were negative about the paper, with concerns about novelty and experimental analyses. Reviewer NCZE was initially concerned about the experiments, but the authors' rebuttal addressed it well. The last reviewer was excited about the paper and rated it "good paper". AC decided to recommend acceptance because the experiments, which the first two reviewers found weak, were strengthed by the rebuttal, and the novelty concern was not concrete. The authors are encouraged to make the necessary changes to the paper to the best of their ability following the reviewers' comments. 
justification_for_why_not_higher_score: The experiments were initially insufficient, though the rebuttal strengthened them. The novelty was fine. 
justification_for_why_not_lower_score: The overall quality is good.
Decision: Accept: poster