Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: This paper designs a multilayer connection structure for neural networks, such that the connection architecture supports implementation of a hierarchical classification scheme within these layers.  It applies this design to the task of hierarchical classification on ImageNet.  Experiments compare results with those of Deng et al. (2012), as well as baseline flat classification models.

The paper motivates the proposed approach via broad claims about what networks understand, but does not provide sufficient analysis or experimental evidence to justify these claims.  For example:

"In particular, when an existing CNN correctly identifies an image of an English Setter, the network itself does not learn that it is an instance of a dog, or more precisely, a hunting dog which is also a domestic animal and above all, a living thing"

Assuming it is trained on example images of all of these categories, how do we know that the CNN does not learn shared representations that implicitly reflect such organization of the concept space?  The paper does not employ any techniques to probe the learned representations of CNNs; without such analysis, the sweeping statements about what CNNs do or do not learn are mere speculation.

On a technical level, the design of the proposed dense classification layers appears to be quite ad-hoc.  It is not clear why a special design intermixing concept prediction nodes with hidden nodes is desirable or necessary.  How does this compare, both conceptually and experimentally, to a branching hierarchy of subnetworks?  The scheme of HD-CNN (Yan et al., 2015) is similar to the latter, but is not represented in experimental comparisons.

In fact, experiments appear to lack comparison to any recent published methods on hierarchical classification.  Deng et al. (2012) is the only prior publication that serves as a reference point.  This is far from a sufficient baseline as surely there has been other work on hierarchical classification in the past 8 years.  For example, a highly relevant work that this paper fails to even cite or discuss is:

M. Nickel and D. Kiela. Poincare Embeddings for Learning Hierarchical Representations. NeurIPS, 2017.

Together, the unsubstantiated motivating claims, ad-hoc design of questionable merit, limited experimental comparison, and missing citations to highly relevant recent work suggest that this paper is not of sufficient quality for publication.

review_title: insufficient experimental justification, missing citations

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors consider how to capture the semantic relationship among categories of a classifier. It is an important problem and has many potential applications. For example, the predicted concept chain can help people understand the performance of the classifier, the coarse-grained concepts are beneficial to the few-shot learning of new categories, and etc.
The authors incorporate WordNet as their ontology and build their neural classifier based on it. Such a tree-structured dense-connected neural architecture is not very common in the current deep learning domain. The network is bound to external ontology, so when the ontology updates, the network has to be re-built. In my opinion, the design seems not very general. Maybe the authors could consider representing restrictions among concepts in the vector space.
In the experiments, the authors used only two baselines, one is a flat single-layer classifier, the other is a work of 2012. The baselines seem too weak to demonstrate the superiority of the proposed model. The results of more recent works are necessary, even though these works "use a separate technique/tool for modeling the conceptual relations", as the authors claimed.

To sum up, the pros of this paper include:
- a valuable research topic
- a fancy model
- clear experimental details

The cons include:
- the model is not very general
- baselines are too weak
- the aspect ratio and resolution of figures seem improper
review_title: Great topic, interesting but not very general model, weak baselines.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
This paper proposes a novel module on top of ConvNet, multi-layer dense connectivity, for learning hierarchical concepts in image classification.

Pros:

This paper proposes to use the label hierarchy (with ancestor concepts from a label) instead of the label itself to learn the image recognition system. To achieve this, it has made two major contributions:
1. Building label hierarchy with a simplified set of categories, to remove the redundant and meaningless categories
2. With the constructed label hierarchy, this paper proposes a dense connectivity module to leverage the label hierarchy to model category abstractions over high-level visual embedding, on top of commonly used convolutional neural networks.

With the proposed techniques, this paper builds up its recognition system using two standard deep ConvNets and achieved strong results on large-scale image recognition benchmarks. 

Cons:

1. In general, the paper is not very well written for a few reasons: A) The motivation of the proposed method over previous methods is not clear (intro paragraph#2). B) Section 3.1 is very hard to follow. C) Some notations in Section 3.2 seems unnecessary and there are things being used before it is formally defined.

2. The design of this dense connectivity module in Section 3.2 seems quite arbitrary, there is no good explanation on why we need to use the z to multiply the output x and h.

3. Experiments of naturally adversarial examples are not motivated earlier in the introduction. It's quite hard for me to understand why using a label hierarchy would improve this task.

 
Detailed Comments:

1. Paragraph#2 in Intro: why training a neural network as multinomial/softmax logistic regression from images to labels can not acquire a comprehensive knowledge about the input entity? For instance, in some of the prior works (e.g. Hu et. al. 2018), they learn models to simultaneously classify categories on a predefined label hierarchy, including both abstracted classes such as "Dog" and concrete class such as the "English Setter".
2. It seems that from Section 3 on, it uses the term "Category" to stand for the leaf concept (most specific) and the term "Concept" as the shorthand of ` "Ancestor Concept". It would be better to mention this explicitly to avoid confusion.
3. Example in Figure 2 is not very clear and hard to follow. It might be better to simplify the figure by using a smaller hierarchy as an example. Also, it would be good to have a paragraph in section 3.1 to describe what in the right figure has been modified using the concrete examples of Figure 2. 
4. Equation 1, why do we need a \psi activation function which is linear? What it means by linear, is there an additional linear weight in \psi besides v?
5. Why are we using an MSE for the concept classifiers? I assume we can use binary cross-entropy for them?


Minor:
* The aspect ratio of Figures 2 and 3 need to be adjusted. It is hard to recognize text and symbols on the stretched figures.
* Notation \hat{h} in the text is bolded but the ones in the equation (1) is not bolded
* A recent work that also leverages hierarchical information in the label text to learn visual concept embeddings, which is closely related to the topic of this paper: Learning to Represent Image and Text with Denotation Graph. EMNLP 2020

review_title: Presentation of the papers needs improvement 

final_score: rating: 3: Clear rejection
review_content: The paper proposes a method to learn concept classes along with its concept superclasses. The proposed method relies on an ontology which they heuristically re-organize by essentially pruning nodes that have few descendants and large semantic overlap. The network proposed to model the ontology essentially just consists of a learned multiplicative gate at each level of the ontology with a standard xent loss over concepts and regularizing term that indicates if the category is an ancestor concept. The experimental results claim gains over some baselines, e.g., combined acc. of 69.05 vs chosen baseline 66.15 on ImageNet12 for ResNet-50 features at a cost of between 18.2% increase in parameters.

Overall, while some of the empirical results seem competitive, I am most concerned with the weak foundations of the motivation of the setup. The work reads like a proposed solution that's trying to look for a problem / motivation and as a result struggles to find its footing in explaining modeling choices & results. 

* The paper uses as motivation that many networks use a softmax head over semantic categories at the leaves of an ontology and claims this is therefore why models using such networks do not learn that say English Setter is a dog. This is a shallow argument for incorporating concept hierarchies since such models clearly would still not be learning deeply what the concept of a dog is, only encoding weak priors introduced by the ontology, an external knowledge base from the network. The connection to learning relationships like "is-a" relations don't ultimately fall out from the proposed method, instead you just get a list of likelihoods that correspond to superclasses that contributed to a concept prediction—the model is not learning the relation, just the co-presence of these superclasses.
* The argument that works like Deng et al (2012, 2014), which for example propose label relation priors like HEX graphs, only either predict fine-grained labels or superclasses exclusively, but not both simultaneously, is another example of where this paper falls short in its problem setup. This work doesn't answer the question of _why_ one would even want to predict both simultaneously well. If we believe that a superclass is unlikely present, why would we still predict the child classes? Even if there are reasonable arguments for this, they are absent in this paper.
* The approach to creating the "compressed concept hierarchy" largely felt like a description of what was done, again, rather than why. Unless I missed it, I also expected a baseline for ablation that doesn't use the "compressed" hierarchy, but just uses it as-is.
* It's a bit strange to me why a MSE loss is used for the indicator of whether a concept is an ancestor. Why use an unbounded error in L2, even if (or especially if) you are squeezing through a sigmoid? What is the intuition?
* I would be curious to know if the improvement in results that we see in Table 1 are just due to increased model capacity (params/compute), i.e. how does it compare to the Deng et al baselines. The comparison discussed are only made with respect to the ResNet-50 & Inception-V4 backbones.

I will also note that the paper could improve on its clarity in writing. As an example, from Figure 2, it's unclear what exactly changed from the LHS and RHS, how, and why it's meaningful; and in Figure 3, it's not obvious without work how the input, output and z-term relate.
review_title: solution looking for a problem?

Meta-review: All reviewers recommended rejection after considering the rebuttal from the authors. The main weaknesses of the submission include poorly motivated claims and designs, and insufficient experimental comparisons. The AC did not find sufficient grounds to overturn the reviewers' consensus recommendation. 


Decision: Reject