Multilayer Dense Connections for Hierarchical Concept Classification
Authors: Toufiq Parag
Hongcheng Wang
Venue:  (2021)
Keywords: Deep Learning: Applications
Methodology
and Theory
Recognition: Detection
Categorization
Retrieval and Matching
Scene Understanding
Visual Reasoning

Abstract: Classification is a pivotal function for many computer vision tasks such as image recognition, object  detection, scene segmentation. Multinomial logistic regression with a single final layer of dense connections has become the ubiquitous technique for CNN-based classification. While these classifiers project a mapping between the input and a set of output category classes, they do not typically yield  a comprehensive description of the category. In particular, when a CNN based image classifier correctly identifies the image of a Chimpanzee, its output does not clarify that Chimpanzee is a member of Primate, Mammal, Chordate families and a living thing. We propose a multilayer dense connectivity for a CNN to simultaneously predict the category \emph{and} its conceptual superclasses in hierarchical order. We experimentally demonstrate that our proposed dense connections, in conjunction with popular convolutional feature layers,  can learn to predict the  conceptual classes with minimal increase in network size while maintaining the categorical classification accuracy.

Under review as a conference paper at ICLR 2021
MULTILAYER DENSE CONNECTIONS FOR HIERARCHI- CAL CONCEPT PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Classification is pivotal function for many computer vision tasks such image recognition, object detection, scene segmentation. Multinomial logistic regression with single final layer of dense connections has become the ubiquitous technique for CNN-based classification. While these classifiers project mapping between the input and a set of output category classes, they do not typically yield com- prehensive description of the category. In particular, when a CNN based image classifier correctly identifies the image of a Chimpanzee, its output does not clar- ify that Chimpanzee is member of Primate, Mammal, Chordate families and a living thing. We propose a multilayer dense connectivity for a CNN to simulta- neously predict the category and its conceptual superclasses in hierarchical order. We experimentally demonstrate that our proposed dense connections, in conjunc- tion with popular convolutional feature layers, can learn to predict the conceptual classes with minimal increase in network size while maintaining the categorical classification accuracy.
INTRODUCTION
Classification is a core concept for numerous computer vision tasks. Given the convolutional fea-
tures, different architectures classify either the image itself (He et al., 2015 Szegedy et al. 2016),
the region/bounding boxes for object detection (He et al., 2017 Liu et al. 2015). or, at the granular
level, pixels for scene segmentation (Chen et al.] 2018)) Although early image recognition works
employed multilayer classification layers (Krizhevsky et al. 2012, Simonyan & Zisserman 2015).
the more recent models have all been using single layer dense connection (He et al. 2016 Szegedy
et al. 2016) or convolutions (Lin et al. 2017).
The vision community has invented a multitude of techniques to enhance the capacity of feature computation layers (Xie et al.,, 2017: Huang et al. 2017 Hu et al. 2018 Dai et al.) 2017 Chollet 2016 Tan & Le. 2019). But, the classification layer has mostly retained the form of multino- mial/softmax logistic regression performing a mapping from set of inputs (images) to a set of categories/labels. As such, the final output of these networks do not furnish a comprehensive depic- tion about the input entity. In particular, when existing CNN correctly identifies image of an English Setter, it is not laid out in the output that it is an instance of dog, or more extensively, hunting dog, domestic animal and a living thing. construct some internal representation of the conceptual superclasses, e.g., dog, animal etc., during training. We argue that, by appropriately harnessing such representation, one can retrieve much broader description of the input image from CNN than it is supplied by single layer output.
rational assume that convolutional layers
Extensive information about most categories are freely available in repositories such Word- Net (Fellbaum) 1998). WordNet provides the hierarchical organization of category classes (e.g., En- glish Setter) and their conceptual superclasses (e.g.. Hunting dog, Domestic animal, Living thing). However, surprisingly limited number of CNNs utilize the concept hierarchy. The primary goal of almost all existing studies is to improve the category-wise classification performance by exploiting the conceptual relations, often via separate tool.
Deng et al. 2014) and Ding et al. (2015) apply graphical models to capture the interdependence among concept labels to improve category classification accuracy. Other works either do not clarify the semantic meaning of the ancestor concepts (Yan et al.) 2015) or impose a level of complexity in the additional tool (RNN) that is perhaps unnecessary (Hu al. 2016). We have not found an

---

Under review as a conference paper at ICLR 2021
existing (deep learning) model that attempts to predict both the finer categories and the chain of ancestor concepts for an imput image by a single network. The classical hedging method (Deng et al., 2012) computes either the finer labels one of its superclasses exclusively, but not both simultaneously.
In this paper, we introduce CNN classify the category and the concept superclasses simultaneously. As illustrated in Fig- ure in order classify any category class (e.g.. English Set- ter), our model is constrained to also predict the ancestor su- perclasses (e.g., Hunting dog. Domestic animal, Living thing) in the same order as defined in a given ontology. We propose a configuration of multilayer dense connections to predict the category & concept superclasses as well as model their inter- relations based on the ontology. We also propose a simple method to prune and rearrange the label hierarchy for efficient connectivity.
Existing methods: English Setter
Propased prediction: Living thing Domestic HHinting Dog English Setter.
Capturing the hierarchical relationship within the CNN archi- tecture itself enables us to train the model end-to-end (as op- posed to attaching separate tool) by applying existing op- timization strategies for training deep networks"] We exper- imentally demonstrate that tecture using standard optimization protocols to predict the concept classes with two popular CNN backbones: ResNet and Inception V4, while maintaining their category-wise accuracy. The proposed multilayer connection is shown to further refine the learned representations of these backbone CNNs to yield better concept and category classification than multinomial logistic regression, and, 2) other existing works (that apply separate mechanisms) on standard datasets and challenging images.
The goal of the proposed algorithm. In contrast methods, our proposed CNN architec- ture predicts the chain of superclass concepts well the finer category.
Figure 1:
existing
train the proposed archi-
Predicting coarser superclasses in addition to finer level categories improves interpretability of the classifier performance. Even if an eagle is misclassified as parrot, the capability of inferring that it a bird, and not an artifact (e-g., drone), may be beneficial in some applications (e.g.. surveillance). More importantly, object detector can enhance its capability on unseen categories by adopting the proposed classífication scheme (as demonstrated equipment as a 'weapon' concept class even
Section 4.4). For example, a movie/TV
violence recognition/ddttctioo tool can recognize that particular weapon category was not in the training set. In visual question answering (VQA), encoding concept classes would expand the scope of query terms by allowing broader description ('how many vehicles are present' in addition to 'how many buses', 'how many trucks' etc.; see Cao et al. (2018); Waan al. (2016)). In Appendix G we point out how architecture can be extended to object detectors to compute the concept classes. In addition, we allude to the potential applications of our model capture label structures different from concept graph, e.g., spatial or compositional dependence.
RELEVANT WORKS
Use of hierarchical classifiers can be traced back to the early works of Torralba et al. (2004); Wu let al. (2004); Fergus et al. (2010) that shared features for improved classification. Some studies claimed a hierarchical organization of categories resembles how human cognitive system stores knowledge (Zhao et al. 2011 while others experimentally showed correspondence between struc- ture of semantic hierarchy hy and visual confusion between categories (Deng et al.) 2010). [Beggio et al. (2100) Deng et al. (201) learn label tree for efficient inference with low theoretical complexity and also suggest a label hierarchy is beneficial for datasets with tens of thousands categories.
Deng al. 2012) aim to predict either coarse level concept fine level category (but not both the same time) given an initial classical classifier. Provided an initial classifier output, this method determines category the coarse concept node (exclusively) with max reward based on aggregated probabilities in a label hierarchy. The reported results suggest the prediction of superclasses comes at the expense of the fine level category classification failure. For CNN based classification, Deng al. 20044) modeled the relationships such as subsumption, overlap and exclusion among the categories
One can also envision learning the multihyer connectivity structure from data by architecture learning techniques Zoph et al. 2017
Pham et 2018))
2

---

Under review as a conference paper at ICLR 2021
via a CRF. Although the CRF parameter can be trained via gradient descent, the inference required a separate computation of message passing. The work of Ding et al. (2015) extended this model by utilizing probabilistic label relationships.
The HDCNN framework Yann et al., 2015) groups the finer classes into coarse concept using the modules for coarse and fine categories where the
category labels. The framework comprises coarse prediction modulates the layers for finer classification. However, this work does not describe the coarser concept classes in order analyze whether they, or their descendent categories, have any semantic meaning (see Appendix IH.. Furthermore, the overlap among concept class descendants precludes mechanism predict the conceptual superclasses. [uu et al. (2016) present a structured inference model for hierarchical representation where the concepts representing scene attributes are predicted as indicator vectors of different length. bidirectional message passing, inspired by the bidirectional recurrent networks, establishes the relations among different levels of concepts. The model leads to large number of inter and intra-layer label interactions some which needed to be manually hard-coded to 0.
Appendix describes a few other studies that proposed to incorporate structural prediction tech- niques in their design (Guo et al.) 2017; Liang 2019). Unlike us, the primary objective of almost all the aforementioned papers is to improve the category prediction performance by utilizing the super- auxiliary source of information as intermediate result. With the exception of Deng et al. (202)) Hu et al. (016)), none of these studies were designed for, and demonstrate their effectiveness concept prediction. The algorithms of Deng al. (2014); [uuo et al.) (2017) require substantial modifications to be able to classify the concept classes. Furthermore, in contrast to ours, most of these studies use separate technique/tool for modeling the conceptual relations that need to be trained or applied separately with different mechanisms.
class hierarchy
It is important to distinguish our work from the hyperbolic embedding studies (Nickel & Kiela 2017 Khrulkov et al. 2020). Khrulkov et al. (2020) attempt to compute an embedding that respects ancestral hicrarchy in the hyperbolic space which implies that, ideally, a more generic image would be closer to the center than the specific ones. However, the paper does not describe and, it neither obvious nor straightforward how determine the concept classes from these embedded points. These studies, therefore, are not suitable for comparison with our method for concurrent prediction of category and concept classes.
3 PROPOSED METHOD
Given an input image, the goal of our proposed method is to determine its category (leaf node in the hierarchy) and list of its concept superclasses (i.e., ancestors in the ontology). As example, for an image of a Chimpanzee, the proposed algorithm produces predictions for 1) the category Chimpanzee, and 2) an ordered list of ancestor concepts: Living thing - Chordate - Mammal + Primate + Chimpanzee. We illustrate (and experiment with) the proposed model for image classification in this paper.
Our CNN architecture is designed to encompass the chain of relationships among the category and the predecessor concepts the dense layers. We utilize an existing label hicrarchy/ontology to guide the design of the dense layers, but do not use the hierarchy in prediction. In order to maximize the information within an ontology and to reduce the number of variables the dense layers, we condense the original label hierarchy explained in Section 3.1 In our design of multilayer dense connections, each concept is associated with a set of hidden nodes. These hidden nodes are in turn connected those of its children (both category and concept) and the output prediction nodes. Section elaboraas these connections and the loss functions to train the network.
3.1 CONDENSED CONCEPT HIERARCHY
In general, a concept class decomposes to multiple sub-concepts in an ontology, e.g., Ima- geNet12 (Deng et al. 2009) subset of WordNet (Fellbaum) 1998). However, the lineage of a parent to a single child (e.g.. Entity > Physical Entity + Abstraction is redundant and does not provide much information. Similarly, parent concepts with highly imbalanced distribution of descendants are not informative as well. Modeling the redundant and uninformative concepts will increase the network size with no information gain.
3

---

Under review as a conference paper at ICLR 2021
Entityy [1000)
Entity (1000)
Physical Entity (997)
Abstraction
Souba
Diver
Object (944)
Matter (50)
Comm. (2)
Attribute (1)
Artifact (520)
Traffic Light
Street
Bubblo
Signal [1]
Sign (i)
Shape [1]
Living Thing (407)
Matter (50)
Whole (903)
Gec. form. [10)
Locetion
Visual Signal
Round Shage (1)
Street Sign
Living Thing (407)
Artifact (520)
Natural Object
Light
Sphere (3)
Chordate (207)
Domestic Animal (123)
Invertebrate (51)
Daisy Agaric Quail
Ball (1)
Traffic Light
(4) Original ImageNot Ontology
Bubble
(b) Compressed Ontology (Proposed)
Figure 2: Partial view of the original (left) and condensed (right) label hierarchies. Concepts are enclosed in reclangular boxes, with number of all descendeants in parentheses. All the parent concepts of the categories Traffic Light, Street Sign and Bubble in the original ontology are absorbed to Entity in the compressed ontology by removing the redundant single parent-child connections and excluding nodes with descendant count < 8. Similarly, the ancestors (Physical Entity, Object, Whole) of Living Thing, Artifact are all collapsed to Entity because 7% descendants of cach parent in the ancestry were also descendants of its child.
We reorganize the given ontology to reduce such redundancy. We assume the hierarchy to be a directed acyelic graph (DAG) and perform depth first search (DFS) traversal on it. During the traversal, we first prune the label hierarchy based on the distribution of descendants of concept node. Let fy denote the number of all descendants of concept indexed by y. Any child node Yah E (chen))((7)) with Pt= 2 t, i.e., the overlap between descendants y and Yer more than By
T, is absorbed by the parent 7. This process is applied recursively to yield balanced distribution of descendants of any concept in the resulting hierarchy. In addition, we remove any concept ? in the structure with a descendant count T7 < 8 and append the children set Ch 1 (dden(r) to those of its parent Ypa. Conceptually, it is not worth modeling a concept node with only a few descendants.
We depict the differences between the original and modified ontologies in Figure 2 The distribu- tions of the descendants for children concepts are more balanced in the compressed version (right) than that in the original version (left). Our strategy removes all the redundant single parent-child connections. As the network connections are dependent on the concept hierarchy, this reduction of nodes and relationships in the ontologies are crucial for our method. It is worth noting that the proposed modification added direct concept-category relations in the middle layers of the hierarchy.
Executing a DFS on DAG ontology may lead to few ambiguous grouping of few concepts and categories as Deng et al. (0014) pointed out. We adopted DFS for simplicity to compute the com- pressed graph. which can be replaced by an unambiguous one whenever it is available; the methods of Deng et al. (2011) [Beniio et al. (2010) can also be applied to generate the ontology.
3.2
NETWORK ARCHITECTURE
Our proposed algorithm aims to model the abridged label hierarchy with dense connections. As Figure [ suggests, there are multiple kinds of dense connectivities in our proposed classification layer. Each concept in the hierarchy corresponds to one set of hidden nodes that essentially repre- sent the concept. These hidden nodes are connected to those representing its children, if any. For example, if Mammal, Bird and Reptile are the descendant concept of Chordate, there will be all to all connections from the hidden nodes representing Chordate to those accounting for Mammal, Bird and Reptile. As such, the computation of the hidden representation of a child concept is conditioned upon that of its parent.
Given the representation captured in the hidden nodes, two types of output prediction nodes detect the presence of the concept itself and any children category in the input. An additional type of connectivity explicitly constrains the concept and category predictions to follow the hierarchical organization of the ontology. We illustrate each of these connections below.
Modeling Concepts and Categories: Let us denote by z² and h² the output prediction variable and the set of hidden nodes associated with the concept The terms node and variables are used interchangeably in the description of our model. Let concept Yea and category both be children of concept y in the hierarchy and l and z; denote the hidden and the output prediction variables for them respectively. The proposed model computes the output prediction 2" and initial values h7=.
4

---

Under review as a conference paper at ICLR 2021
Ty for quantities of the children concept and categories using the following dense connections.
=² = o Σ uh ( bZ)): 2, - (
22, 12, ky -
Σ why h + +bb,)
(I)
In these equations, u, v, w and b are the weights/biases of the dense connectivity and h; corresponds to the i-th value of h. The activation functions o, w utilized for these different quantities are Sigmoid, v = Identity,
ReLU. In essence, h encodes an internal description for concept and d² predicts the presence of it in the input image. The connections between h? and h of its child concept enforces the descriptions for Ych. to be derived from, and therefore dependent on, that of its parent.
In our design, the number d' = |h"| of nodes representing a concept y > is directly proportional to the total number of descendants 7/y of We have used d' μuy, for this study with μ 2. The flattened output of the final feature layer of an existing network architecture (e.g., ResNet-50 or InceptionVe V4 etc.) is utilized to populate h and its size depends on the particular architecture used. We do not predict the root concept 2 of the hierarchy (e.g., Entity for ImageNet12) since all categories descend from it.
Concept Category Label Constraints: The values for category prediction 2; and hidden nodes for child concept h Ych calculated by multiplying initial values of these quantities with the concept prediction _
25 Ty * 2 h
h3 #2
Note that, the Sigmoid activation constrains the value of 2 to be = [0, 1]. In effect, the node =7 plays an excitatory or inhibitory role based on the predicted value the concept y. This constraint enforces that the nodes representing any child of concept y. whether it is category (zy) another downstream sub-concept (h"sh ), be activated only if the concept itself is correctly predicted.
The category predictions for an input image are computed by applying Softmax activation
all category nodes (2.... .NN] ) where N is the total number of categories. The predictions
for M concepts are given by the collection of the variables { ,..,,, ...,,*,,, The hierarchical
relationship among the variables {=1,=² ,,, z are enforced by construction. Observe that,
while an image can be classified to only one (e.g., Chimpanzee) of the N categories, multiple concepts (e.g., Primate, Mammal, Chordate) at different levels of the hicrarchy can be set to 1.
Figure 3 clarifies the proposed dense arrangement between the hidden nodes h" and its children con- cept nodes as well as the prediction outputs. The hidden nodes h² (shown in empty circles) are con- nected those of its Q children concepts and category output variables (solid circles) to com- pute the initial quantities (h"m.,..... ...., hT*4) and { } respectively. The concept predic- tion =7 (solid square) is computed by another dense connections which modulates the final values of (h .... h and {2 , Fjn for the con- cept and category variables respectively via multipli- cation.
27
h
h Toi
h Yohho
h Tolio
h?
00,,
J.
The total number of weights in proposed multilayer dense connection with o-way decomposition is < pdd + p + (I-aj) where p is the height of the hierarchy and μ is the fixed mul- tiplier used to set the number of hidden nodes for concept y (see Appendix[B] B). In comparison, conven- tional single layer CNN classifiers consist of d N variables.
Number of Variables:
C
T 3
Figure 3: Schematic view of proposed dense connections. The solid square and eircle nodes correspond to the concept and category prediction node respectively, whereas the empty circles de- picts the hidden nodes. We assume the concept has concepts and B categories as children.
Loss Functions: The proposed method minimizes
two different losses for the two types of output nodes. For the category predictions, minimize cross-entropy loss (C computed over the N category labels and the network outputs z,. j = I...., N. The concept classification loss is defined as the MSE between the concept prediction
5

---

Under review as a conference paper at ICLR 2021
-007~x77 xx) where x" = 1 if ? is an ancestor concept of the input category and x" = 0 otherwise for 7 = 0, 0,1, ,,.
variables and binary indicator quantity X: (2oN(2) Ms
Although we use MSE for learning z in this work, any loss for multilabel classification, e.g., binary cross-entropy for each 27, may work as well. The proposed method minimizes the combined loss LoE + XLOON with the balancing weight À fixed to = 5 in all our experiments. Note that, while the error for any category is backpropagated through its predecessor concepts due to the dependence imposed by construction, one needs to ensure that other concepts - that are not related to the category to remain 0 in the This is exactly the constraint enforced by LcoN(2)
EXPERIMENTS & RESULTS
4.1 EXPERIEMNTAL SETUP
We utilize the ontology provided by the ImageNet12 2 dataset Deng et al. (0099) to design our dense layers. All the labels of ImageNet12 between [1, 1000] correspond to the N 1000 category classes and labels > 1000 are assigned to the 860 concept superclasses. In all our experiments, we fixed the two quantities for compressing the concept hierarchy (Section 3.1 to be r
90%, = 20. After compressing the label hierarchy using the methods described in Section 3.1 there are 40 concept labels left in the hierarchy which has a height of p = 7.
The proposed architecture has been trained on the ImageNet12 training dataset and tested mul- tiple datasets as described in next sections. We used two popular CNN architectures for feature computation layers: ResNet-50 (He et al. 2015 2016) and Inception V4 (Szegedy al., 2015 2016 Fl
Adding the proposed multilayer dense connections increased the total number of variables of the ResNet-50 model by factor of 1.182 30.29 M). For the Inception V4 model, the increase is
1.059 46.88 M). This suggests that the increase induced by our proposed model is far lower than the 14
analytical estimate (Section 3.2 in practice and is tolerable with respect to overall network size.
Inference: During inference of the proposed network, we select the category with largest softmax probability for category classification as usual. The parent concepts are classified by thresholding z values. We also set any x if the variable for its parent 2 = 0 (i.e., lower than a confidence threshold). If more than one child of any concept is detected, we select the one with the highest confidence among them to compute the concept chain.
Evaluation measures: We report the single crop top-1 accuracy results ACccAt for categories when they are available. For concept inference. we report two measures based on hicrarchical precision and recall (Costa et al., 2007), which computes the precision an recall between the predicted and true chain of concepts: 1) the mean hierarchical precision (mhP) and recall (mhR) over all images of test set, and, 2) percentage of images AcccON that were predicted with 100% hP and hR. The combined accuracy, AcCCOMB, denotes the percentage of images with both ACCCAT = AcceoN = 1.
Baselines: As the first baseline, we build CNN classifier for N M classes with single flat dense layer for each feature computation architecture. These baselines are trained with the same loss functions of the their multilayer counterparts.
We utilize a modified version of the hedging method of Deng et al. (2012) as the second baseline. Our modified version uses ReasNet-50 and Inception V4 based classifiers instead of classical ver- sions. We observed the optimal strategy to compute category and concept chain from the output of the hedge model (which either category or max reward concept) is to use the category prediction as is and use the concept with maximum reward, along with all its ancestors, as the predicted con- cept chain. The hedge model utilizes the same underlying hierarchy as ours and we read off the prediction for concepts that overlap with our condensed hierarchy.
4.2 IMAGENET 2 DATASET RESULTS
We first demonstrate the performance of two varaints of the proposed method that were trained on ImageNet12 training set and tested on its validation set. For one variant of our proposed CNN models, MD-RN with ResNet-50 and MD-IC with Inception V4 backbones, only the weights of
pretrained versions downloade from htt.a//tttotti..ooooooooooooo/ao

---

Under review as a conference paper at ICLR 2021
Table 1: Accuracy comparison on ImageNet12 val set (single crop top-1). significantly higher AcCCON, Acceoma than the baselines.
The proposed models achieve
Method
Acccar
AcccON
AcCCOME
mhP
mhR
HG-RN (ResNet-Deng 2012)
68.97
82.9
66.15
93.28
94.57
BL-RN
76.26
62.66
53.01
92.36
89.48
BI-RN-FT
76.0
69.04
58.89
93.91
91.06
MD- RN(uur))
75.94
80.25
69.05
94.22
93.68
(ID-RN-FT(ous)
75.91
82.36
70.45
93.3
94.75
HG-IC (Inception+ at. 2012
71.49
84,28
68.87
93.75
95.07
BL-IC
80.11
72.68
63.56
95.03
91.51
MD-IC(ourS)
80.02
88.18
77.76
95.46
96.1
the multilayer dense connections were trained; the weights of the feature layers were kept at their original pretrained values. The other variant MD-RN-FTT performs finetuning, i.e., it initiates the weights in the feature layer from the pretrained ResNet-50 and trains all variables in the network. The corresponding flat dense connection baselines are referred to as BL-RN, BL-IC, BL-RN-FT respectively. The hedging model of of et al. (2012) is dubbed HG-RN and HG-IC that utilizes a ResNet-50 and Inception V4 initial classifiers respectively.
Training: We trained MD-RN, MD-RN-FT and MD-IC following standard training procedures of SZeeedyy et al. (2016); Chollet (0116). The details of training parameters and techniques are described in Appendix C for space limitation. The initial classifiers for HG-RN and HG-IC were trained from scratch on 93% of training examples for each class in ImageNet 2012 training set (achieving 74% and 76% category accuracy on validation set respectively). The remaining 7% examples were used to estimate the dual variable maximizing the reward for multiple accuracy guarantees. The results reported here correspond to the dual variable, accuracy guarantee with best overall AcCCOmB-
Table rreports the accuracies for all the aforementioned models. [ demonsttatss that the MD-RN, MD-RN-FT and MD-IC networks with proposed multilayer dense connections achieved comparable category accuracy ACCcxT of the flat baselines BL-RN, BL-RN-FT and BL-IC (which are similar to the published results). The category accuracy of HG-RN, HG-IC are significantly lower than the aforementioned models a phenomenon also observed by Deng et al. (012). This is due to the inherent design of the hedge model to only predict concept classes (and not the leaf categories) for some examples.
Evaluation:
Table also suggests that an unconstrained flat dense connection is insufficient for learning the superclass sequence. The proposed models attains > 10% higher AcCCON and even larger combined AcCсomB than the flat baselines BL-RN, BL-RN-FT, BL-IC. The baseline methods can either learn to classify the finer categories or the concept chain correctly but the proposed CNN architecture can attain the ability to do both simultancously with high accuracy. The proposed models MD-RN, MD-RN-FT and MD-IC achieves a significantly higher overall accuracy AсссомB than the hedging models HG-RN and HG-IC. The inception based model MD-IC is also more accurate than HG- IC for concept prediction indeed, MD-IC leads to the best overall accuracy AcccomB among all models.
The flat baseline methods BL-RN, BL-RN-FT and BL-IC attempts to compute the concept classes independently using the same single layer dense connections that utilized for category classi- fication. Observe that the concept accuracies AcccON of BL-RN, BL-RN-FT and BL-IC are ap- proximately 7 14% lower than category accuracies ACCCAT. This suggests that the feature rep- resentations learned (in the last convolutional layer) by ResNet-50 and Inception V4 alone are not as informative for concept prediction as they are for category classification. The proposed multi- layer design combines these representations to yield refined representation encoded in the hidden variables h to improve the concept prediction accuracy by > 13%.
The comparison implies that the proposed top-down hierarchical constraints on category classifica- tion to preserve the ancestor concept chain is superior to the hedging technique of concept prediction distilled from the aggregated probabilities accumulated bottom-up from category confidence. As we will show in next section, a top-down constrained prediction is more robust to the challenging ex- amples because it does not rely primarily on the category classification probabilities.
We provide an in-depth analysis of the failure cases of concept prediction, compared the distribution of category classifications and training times of different variants of the proposed models and baslines in Appendix D Appendix [ discusses the ablation studies with the network structure and À values in loss function.
Analysis:
7

---

Under review as a conference paper at ICLR 2021
Table 2:
Accuracy comparison on naturally adversarial images (single crop top -1).
The proposed models
achieve significantly higher AcceON, AcCeOMR than the baselines.
Method
AceCAr
AcccON
AcccomB
mhP
mhR
HG-RN (ResNet=DDeng et 2012
0.37
14.36
0.3
$7.08
55.99
BL-RN-FI
0.86
11.52
0.26
59.12
54.33
MD-RN-FT (ours)
1.06
16.13
0.77
57.6
58.0
HG-IC (Iuception Deng al. 2012
3.18
19.76
2.8
59.73
58.77
BL-IC
7.54
16.62
3.56
61.78
58.07
MD-IC (ours)
7.1
23.5
6.1
63.79
61.46
GT: Hummingbird.
GT: Fox-aquirrel
GT: Jay.
GT: Garter-snake. ResNet50: Swing
GT: Drake.
ResNet50: Mud-turtle
ResNet50. Indian cobra
ResNet50:
ResNet50: Hammerhead
Proposed:1 lving-rhing Proposed:Living thing > churdate bird >
Proposed:Living-hhin
Proposed: Living-thing chordate réptile >
Propesed Living thing chordate bbird ~ Paquitic.
chordate > mammal
> chordate bald eagle.
bird >
hummingbird.
fox fox-squirzel.
diapsid->ring Pringncok-snake.
bird + albatross.
Figure 4: Qualitative comparison between the proposed and the pretrained ResNet50 models naturally adversarial images (Hendryeks et al.
2019).
4.3 PERFORMANCE ON NATURALLY ADVERSARIAL IMAGES
Our next experiment demonstrates the robustness of the proposed network on challenging images. We use the images of naturally adversarial image dataset (Hendrycks et al. 2019) that the popular CNN architectures (e.g. ResNet) trained standard datasets classify to widely different semantic categories. Results in this section examine whether or not algorithms (c.g.. (e.g..Deng et al. (2012) that compute marginal probabilities through bottom up aggregation ancestor concepts of challenging images. There are 7500 images in this dataset that belong to 200 categories. The ResNet-50 model we downloaded from the tensorflow website (pretrained on full ImageNet 2012) could classify only 0.73% of the images. Since these 200 categories are subset of the N 1000 categories of ImageNet 2012 dataset, one can use the same concept structure for hierarchical classification.
less effective for determining the
In Table the proposed MD-RN-FT and MD-IC model accuracies are compared with those their flat dense layer counterparts BL-RN-FT and BL-IC as well as those of the hedging model (Deng al. 2012) with corresponding initial classifiers. The proposed models MD-RN-FT and MD-IC were able to correctly classify the superclass sequences of more images than the baseline models. With the exception BL-IC, their categorical accuracies ACCCAT are also higher than the baselines.
We provide qualitative comparison in Figure Siguureebetweer between the proposed MD-RN-FT method and the pretrained ResNet-50 output. One can observe that, without the constraints on predicting the concept chain correctly, the ResNet-50 can classify the images semantically very different categories. Aggregating these category-wise confidences will consequently lower the ACCcoo of the hedging model compared to those of the proposed model. The proposed MD-RN-FT, the other hand, determines the ancestor classes correctly even when the leaf level category is misclassified.
4.4 PASCAL VOC 12 CONCEPT PREDICTION
In this section, we test the generalization of the knowledge learned by the proposed architecture. Ideally, the proposed network should be able to extrapolate its understanding of the concept super- classes learned from one category set to previously unseen categories. That is, after learning that a zebra is mammal, it should be able to identify horse (of any color) as a mammal too. To test gen- cralization capability, we test classifiers learned from the ImageNet12 dataset on the trainval split of PASCAL VOC dataset (Everingham al. 2010). Note that, there is no one to one correspondence between the categories of ImageNet and voC Each of the VOC 12 categories were assigned to one of the concepts in the condensed label hierarchy of ImageNet dataset.
In Table 3 we show the average concept prediction accuracies of the different variants of the pro- posed model and the corresponding flat dense layer baselines over 16 VOC categories. Since there are no categories to predict, the second baseline predicts the concept classes by thresholding the aggregated probabilities computed in Deng et al. (012). These are referred to PrAggRN and
See http:://maagg-nttttt. .org/ehh lenges/ 2ttpg///.../0//2212///// ysis.
8

---

Under review as a conference paper at ICLR 2021
Table 3: Average concept accuracy comparison on PASCAL VOC 12 trainval subset. The proposed method achieves significantly higher AcccoN than the baseline for both architectures.
PrAgeRN
BI..RN-FT
MD-RN FT(aur)
PrAggiC:
BI-IC
MD-IC(our))
AccroN
46.2
48.32
40.3
50.38
Table 4: Accuracy comparison AwA2 dataset for simultancous category, concept, attribute classification.
Method
AcceAT
IoU Concept
IoU Attribute
Hu ct (2016)
79.36 + 0.43
84.47 = 0.38
86.92 t 0.18 89.08 1 0.01
BL-RIN
92.48 1 0.05
93.66 + 0.42
.MDRNN (ours)
92.16 1 0.17
97.01 0.05
94.1 1: 0.37
PrAggIC when computed from the initial ResNet-50 and Inception V4 classifiers respectively. As the table shows, the proposed variants MD-RN-FT, MD-RN-SC and MD-IC achieved higher aver- age concept prediction accuracy over all classes than those of the baselines. Appendix details the performances for all classes and analyzes the results on this dataset.
4.5
AwA2 DATASET RESULTS
Our next experiment applies the proposed method on the Animals with Attributes (AwA2) dataset (Lampert et al. 2014) utilized in multiple relevant studies (Deng al. 2014 Hu et al.
2016 AwA2 dataset provides the attribute as well as the category labels for 37322 images of 50 classes. For concept classes, use the same 28 superclasses and follow the same hierarchical
lations used in Hu et al. (016). Note that, this hierarchy is slightly modified from the ImageNet12 ontology (and therefore different from the compressed ontology we computed).
Following Hu et al. (016)) we use a 60-40% split of the images for train/test and report the average:tstd dev of the multiclass accuracy Accear for category and the IoU acc for concept and at- tribute labels from 3 trials. Table compprres the performances of the proposed MD-RN with that of (Hu et al. 2016 (values taken from the paper). determine the attributes, we extend our proposed architecture by adding dense connection between concept and category prediction variables to the 85 attribute classes. We also compare the performance of the flat baseline BL-RN that predicts the attributes classes independently in addition to animal and concept labels.
Hu et al. 20166) uses a weaker CNN base (Krizhevsky et al. 2012)) than ours, which is perhaps factor leading to its inferior performance. The comparison suggests that, given CNN backbone with enhanced capacity, one can attain highly correct category, concept and attribute classification from our multilayer dense connection without relying on additional complex tool, e.g., RNN. However, the lower IoU concept, IoU attribute of the flat baseline method attest that the improved claim performance of MD-RN does not arise solely from the ResNet-50 features and strengthen that the proposed mulitlayer connectivity enhances the feature representation to improve accuracy. It is interesting to observe a correlation between concept and attribute accuracies: determining the concepts correctly leads to better prediction of attributes. Appendix I examines the robustness of MD-RN trained on AwA2 on ImageNet 12 images from categories overlapping with AwA2.
5 DISCUSSION
In this paper, we have demonstrated the advantages of multilayer dense connections for classifying both the category and the ordered chain of ancestor concept classes with two popular architectures in different settings. The experimental results show that the proposed dense layers can equip the exist- ing CNN classifiers with additional capability of learning the coarse concepts without sacrificing the category-wise accuracy. Attaining this capability of determining the ancestor concept classes is rela- tively easy one can only modify the final classification layers and train using the same optimization technique, for some cases without training the full network.
Our experiments demonstrate the advantage of the proposed architecture over a) multinomial re- gression that exploits the learned feature representation of pretrained CNNs; b) methods utilizing marginal probabilities accumulated in a bottom up fashion from CNN output given an offline hier- archy (Deng et al.) 2012); and c) an algorithm that employs RNN for superclass prediction. Our top-down constraints enforces the category classification to conform to the semantically meaning- ful lineage of ancestor classes. This property not only improves the superclass computation for challenging images but also allows zero-shot concept classification for unseen categories. We be- lieve this study will encourage researchers in computer vision community to adopt and improve our multilayer dense connection to gather more descriptive output from CNNs.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Samy Bengio, Jason Weston, and David Grangier. Label embedding trees for large multi-class tasks. In NIPS. 2010.
Clemens-Alexander Brust and Joachim Denzler. Integrating domain knowledge: using hierarchies to improve deep classifiers. CoRR, abs/1811 1..01225, 2018.
Qingxing Cao, Xiaodan Liang, Bailing Li, Guanbin Li, and Liang Lin. Visual question reasoning on general dependency tree. CoRR, abs/1 80400000000, 2018.
Liang-Chies Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous 2017.
convolution for semantic image segmentation. CoRR,
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. CoRR, abs/1 has//802.02211. 2018.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1800-1807, 2016.
Eduardo Costa, A. Lorena, A.
Carvalho, and A. Freitas. A review of performance evaluation measures for hierarchical classifiers. In AAAI, 2007.
J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei. Deformable convolutional networks. In 2017 IEEE International Conference on Computer Vision (ICCV), 2017.
Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jia Deng, Alexander C. Berg, Kai Li, and Li Fei-Fei. What does classifying more than 10,000 image categories tell us? In ECCV, 2010.
Jia Deng, Sanjeev Satheesh, Alexander C. Berg, and Fei Li. Fast and balanced: Efficient label tree learning for large scale object recognition. In NIPS, 2011.
Jia Deng. Jonathan Krause, Alex Berg, and Fei-Fei. Hedging your bets: Optimizing accuracy- specificity trade-offss in large scale visual recognition. In CVPR, 2012.
Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs. In ECCV, 2014.
Nan Ding, Jia Deng, Kevin Murphy, and Hartmut Neven. Probabilistic label relation graphs with ising models. In ICCV, 2015.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The Pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
Christiane Fellbaum (ed.). WordNet: An Electronic Lexical Database. Language, Speech, and Communication. MIT Press, Cambridge, MA, 1998.
Rob Fergus, Hector Bernal, Yair Weiss, and Antonio Torralba. Semantic label sharing for learning with many categories. In ECCV, 2010.
Yanming Guo, Yu Liu, Erwin M. Bakker, Yuanhao Guo, and Michael S. Lew. Cnn-rnn: large- scale hierarchical image classification framework. Multimedia Tools and Applications, 77:10251- 10271, 2017.
K. He, G. Gkioxari, P. Dollár, and Girshick. Mask r-cnn. In ICCV, pp. 2980-2988, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, abs/15512.03388 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. networks. CoRR, abs//103.00000 2016.
Identity mappings deep residual
10

---

Under review as a conference paper at ICLR 2021
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CoRR, abs/1907.07174, 2019.
H. Hu, G. Zhou, Z. Deng, Z. Liao, and G. Mori. Learning structured inference neural networks with label relations. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Hexiang Hu, Guang-Tong Zhou, Zhiwei Deng, Zicheng Liao, and Greg Mori. Learning structured inference neural networks with label relations. CoRR, 0bs//511.00611 2015.
J. Hu, L. Shen, and G. Sun. Squeeze-and-excitatio networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
G. Huang. Z. Liu, L. v. d. Maaten, and K. Q. Weinberger. Densely connected convolutional net- works. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In CVPR, 2020.
Alex Krizhevsky, Nya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems 25. 2012.
Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. Attributebased classification for zero-shot visual object categorization. 36(3):453-465, 2014.
Xiaodan Liang. Learning personalized modular network guided by structured knowledge. In CVPR, June 2019.
Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng- Cheng-Yang Yang Fu, and Alexander C. Berg. SSD: single shot multibox detector. CoRR, abs/1512.023355, 2015.
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representa- tions. In Advances in Neural Information Processing Systems, 2017.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In ICMI., 2018.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In /CLR, 2015.
thinking the inception architecture for computer vision. CoRR, abs/1512.00555 2015. Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re-
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alex A. Alemi. Inception-v4, inception- resnet and the impact residual connections on learning. ICLR 2016 Workshop, 2016. URL hhttps://arxi:: / wwwp////............ 07261
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural net- works. In ICML, 2019.
A. Torralba, K. P. Murphy, and W. T. Freeman. Sharing features: efficient boosting procedures for multiclass object detection. In CVPR, 2004.
Zhuowen Tu, Xiangrong Chen, Alan L. Yuille, and Song-Chun Zhu. Image parsing: Unifying segmentation, detection, and recognition. Int. J. Comput. Vision, 63(2): ((2)):13-140, July 2005.
Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony R. Dick. FVQA: fact-based visual question answering. CoRR, abs/l 606.05433, 2016.

---

Under review as a conference paper at ICLR 2021
Jianxin Wu, James M Rehg, and Matthew D. Mullin. Learning rare event detection cascade by direct feature selection. In NIPS. 2004.
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. HD-CNN: Hierarchical deep convolutional neural networks for large scale visual recognition. In ICCV, pp. 2740-2745 2015.
Bin Zhao, Fei Li, and Eric P. Xing. Large-scale category structure aware image categorization. In NIPS. 2011.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. CoRR, abs/1612.01 105, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. CoRR, abs/1707.07012, 2017.
A APPENDIX: OTHER RELEVANT WORKS
Guo et al. (2177) attempted to classify the coarse labels or the conceptual superclasses of categories by augmenting an RNN to CNN output. In addition to increased complexity imposed by the RNN, it is not clear how the hierarchy among labels was generated and how the hierarchy would scale up with increasing number of categories. Liang (019) employed graph based reinforcement learning to learn features of set of modules, each of which corresponds search is completed, the activated module outputs are passed through prediction layers for generating final output. The algorithm demonstrated promising performance for classifying scene contexts for semantic segmentation.
concept. Once the network
Commercial
vision
solutions
such as
Amazon
Rekognition
(https://aws..n https://wws.amazznnn www.aaaaaaccooooo powered-by-dee seydeee-learning)) or Google Vision (https://cloud. google.cooooooo com/vision) seem produce outputs that resemble concept classes. Due to the proprietary nature of these solutions, it is not possible for us to confirm whether or not they indeed predict concept classes and their relationships. However, based on the description provided in (https://aws.aun www.:ws.aaaaoo..ooooo com/blogss/aws/amazoo rekognitonnimagg-deeeccooo
hwwp///////////azoooorrreeeeeeee www.gggtiioonmmmeddeeeeeee image-detection -ad---aadd-rrttoonn
cerpognntioo-ooweeede byy-dep-learninng! and scrutiny of results, speculate that Amazon rekognition does not any hierarchy and probably uses look-up from hard-coded undisclosed) relations strategy.
B APPENDIX: NUMBER OF VARIABLES IN MULTILAYER DENSE CONNECTIONS
This section quantifies the increase in the number of weights in the dense layers induced by our multilayer computations. The CNN classifiers typically consists of P connections in dense layer, where d and N are the size of the last feature layer and number of category classes respectively. In proposed multilayer dense connections with balanced a-way decomposition of the concepts, the
total number of weights is & μdo (N + (1))) where is the height of the hierarchy and is
the fixed multiplier used to set the number of hidden nodes for concept 7 (see Section 3.2
In a balanced decomposition, the number of hidden nodes for concepts reduces by a constant factor a. With the size of smallest set of hidden nodes d, the max layer of the dense connections is For any concept prediction z", we need d' ud
at weights at layer i.e., max number of weights for concept class prediction is ual p udo logo ( (4)) The number of concept-concept connections can be calculated as d +
log, ( dd))
+ = udd (1-o) I pd (I-a)
(1-a²)
12

---

Under review as a conference paper at ICLR 2021
In order to predict a category variable 2j Children(y)) d² weights are necessary. Since any ud! z di = o at any level balanced decomposition, the total number of weights for category class prediction must be & pddN
C APPENDIX: TRAINING DETAILS
For MD-RN, MD-RN-FT and MD-IC training, we used the RMSProp optimizer with momentum in our multi-GPU distributed training scheme similar to Szegedy et al. (2016); Chollet (2016). The initial learning rate for this experiment was 0.1 and was multiplied by 0.94 every 2 epochs. The epoch size is the same as the training set size with batch size 256, momentum value 0.9, and weight decay 0.0001. The training data were augmented by random crop and horizontal flips. For proposed network learning, the weights corresponding to the concept outputs and the concept interconnec- tions were learned for first 2 epochs before optimizing those for the category variables. We did not use label smoothing for training the Inception v4 model. The training was continued until the CNN achieved the same or close accuracy for category classification as reported in the original paper/repository.
We have also trained ResNet-50 with proposed multilayer dense connections MD-RN-SC and baseline flat dense connection BL-RN-SC from scatch on ImageNet 2012 training set. In our training from scratch, we applied stochastic gradient descent (SGD) optimizer with momentum value 0.9 and initial learning rate 0.2. The learning rate was decreased every 30 epochs by factor of 10 (similar tofSimonyan Zisserman (2015): He al. (2015) training). All other values of the hyperparameters remain the same those of MD-RN training. Table reports the accuracies of MD-RN-SC and BL- RN-SC.
(2nterrmnn(2155))))
Table Accuracy comparison for learning full network from scratch. accuracies are computed sin- gle crop top-1 setting. The proposed method achieves significantly higher AccCON, AcCCOMB than the baseline.
Method
Acccar
AcCCON
AсссOmB
BL-RN-SC
75.24
69.87
59.7
MD-RN-SC
73.5
78.14
65.74
D APPENDIX: PERFORMANCE ANALYSIS OF MD-RN AND MD-IC ON IMAGNEET
Figure 5(a) shows some example categories where the proposed MD-IC model generated differ- ent concept orders than those in the condensed ontology. Monitor and Space bar are children of Equipment and Implement concepts respectively in the condensed ontology. But, they contain many images of Desktop computer and Computer keyboards both of which are children of the Device concept. As a result, 68% and 88% Monitor and Space bar images respectively were classified Artifact-> Instrumentality- Device. Similarly, although it is grouped Matter Solid con- cepts, 75% the Mushroom images were predicted as Living thing which consists of Earthstar, Stinkhorn categories that visually very similar to Mushroom images. Motor scooter placed in Wheeled vehicle in our compressed hierarchy: but at test time, 94% of the time its images was predicted Self propelled vehicle. In general, predictions from our proposed model concur more
Monitor: Arti, (1.0)>>Instru, Devieeevcc)0 Sample Image:
Mushroom: Living (0.75). Matter (0 Solid>slid
Sample Image:
72.5
200
800
100
150
Propotudd
Boach
Concept misclassificatinn examples
Acrexr distribution
(c) Acceu progress
Figure 5: aa)
Categories where the proposed method predicted concept order different from the condensed Category-wise classification accuracy of proposed method the baseline architecture (w/
hierarchy. b
ResNet50),
c
Progession of validation accuracy of the proposed CNN (blue) and baseline (red).
13

---

Under review as a conference paper at ICLR 2021
with the concept chains for descendants of Living thing than it does for those of Artifact concept.
Since different category classes are predicted at different layers the proposed dense structure, is rational to verify whether not the category classification capability is impaired by the depth of layer. To test this, we have plotted in Figure [(b) the category prediction accuracies for each of the N = 1000 classes of the baseline against those of the proposed CNN with ResNet50 backbone. The plot implies no clear effect of the prediction depths on the classification performances on different categories they remain same very close to those of the original architecture.
Due to the increased number of dense layers and additional sigmoid activations, it is perhaps natural expect the proposed architecture require more iteration to converge. As Figure ((c) demon- strates, our model indeed takes more apochs(x-axip) to attain a category classification performance (y axis) similar to that of baseline built upon ResNet50.
E APPENDIX: ABLATION STUDIES
We have conducted ablation studies with model MD-RN. In our first ablation study, we model the dense layers based on the original uncompressed hierarchy instead of the condensed hierar- chy. Adopting the original ontology in ResNet-50 increased the number of z variables to 806, depth of dense layers to 17 and model size to 43M (compare with 40, 7, 30M resp. of compressed). Our attempts with different hyperparameters (A, Ir etc) achieved at most [AceeAt, AcCCON, ACCCOMB 1.03, 60.05, 47.6) 30 epoch (compare Acccar with ours Fig [(c) The training is perhaps smothered by 1) increased depth with many redundant single parent-child connections, and 2) imbalance in descendent distribution where smaller concept classes are under- represented.
mhP, mbR)=[(44002.11)
The second analysis is performed with varying the values of the balancing term À in the combined loss function (Section B.2). As can be inferred from the network architecture for multiple dense connections and the loss functions, with 0 the category-wise loss drive concept variables of only the ancestor concepts to 1. However, with À = O, the optimization will not force variables of the other non-ancestor concepts to Consequently, concept and combined accuracy of a model trained with À = 0 will be very low.
As Table 6 shows, we observe a very low AcCcON, AcCCOMB for MD-RN-AO while achieving close ACcCAT to the published results. It is also apparent that a small À (= 2) favors improving AccçAT while a relatively large À (=8) favors improving AcCcON compared to those the MD-RN model This is expected behavior with the loss function are optimizing.
reported Table
Table 6: Performance of MD-RN with different values. All accuracies are computed for the single crop
top-1 setting.
A
AccCAr
AcccON
AcceоMB
mhP
mhR
0
76.04
1.86
1.57
33.35
45.41
2
76.04
78.57
66.66
91.78
93.81
8
75.42
81.74
70.08
94.13
94.08
F APPENDIX: PERFORMANCE ANALYSIS ON PASCAL VOC DATASET
We have excluded the classes (Person, Pottedplant] since these categories were underrepresenteed in the ImageNet12 dataset. For example, Person is represented by three rare subclasses Scuba diver, Ballplayer, Groom. On the other hand, the (Dog, Bird) categories were over represented in Ima- geNet12 and their subcategories were assigned to multiple concepts in our hierarchy. These classes could lead to inconsistency and therefore also excluded in evaluation. After removing the images that overlaps with multiple categories, we are left with 6941 images from 16 categories. For these images, report the AcccoN in Table 7 (Accext and AcccoMB cannot be computed there is no one to one correspondence between categories of the two datasets).
The accuracy values for concept classification VOC 2012 clearly indicate the superiority of the proposed architecture to generalize the knowledge learned from ImageNet12 hierarchy of ances- tor superclasses. All the CNN models resulted in weak performances of categories (Train, Car,
14

---

Under review as a conference paper at ICLR 2021
Table 7: Concept accunicy comparison on lesling on PASCAL VOC 2012 trainval subset. The proposed method achieves significanily
higher Aencow than the baseline for hrth architectures.
Method
bosse
sheep
plane
box
mbikk
bail
Chair
diable
ava
PrAggRN
712
731
31,4
72.1
75.1
55.3
7344
34.1
61.7
24
24
36
29.3
8.5
30.8
37.8
46.2
BL-RN-FT
572
53.
43
538
72.5
50.2
65.7
35.3
56.8
1.5
21.6
17.9
1
13.1
30.7
38.06
MD
36,2
69.1
13.8
48.34
PrAnsIC
59.7
30.1
93
48.32
BC
51.9
63.2
47.2
60.5
80.3
45.9
45
2.9
71
1.9
211
25.3
19.3
3.1
24
32.3
40.3
MD-IC
74.6
84.
60.9
807
63.
77.5
8.1
65.8
38
17
46.4
39.4
1599
40.9
43.3
50.38
Diningtable). Such performance can be attributed to equivocal ancestry within the condensed ontology. For the category Train, the proposed model with Inception V4 architecture predicted Entity Artifact-> Instrumentality Container-> Wheeled vehicle - Self propelled vehicle for 56% of images whereas the compressed ontology assigns it the concept order: Entity Artifact-> Instrumentalityy>> Conveyance. It important to note though that for the ambiguous categories, both the baseline and the proposed models performed poorly i.e., the proposed method is not drawing any inequitable advantage due the label ambiguity.
G APPENDIX: POSSIBLE APPLICATIONS AND EXTENSIONS
Our dense connectivity can also be extended two-stage object detection algorithms (He et al., 2017 Lin et al. 2017) by replacing the final detection head with proposed mulitlayer connections. For SSD-type architectures (Liu al. 2015 Lin al. 2017). the dense computations can be carried out by 1 x convolutions (or extended to 3 x 3) and added to the classification branches at each scale. The convolutional form should conceptually enable the semantic segmentation techniques (Chen
et al. 2017 2018 Zhao et al.) (016) to adopt our model as well.
Furthermore, rather than capturing the conceptual class lineage, the dense layers can be modeled after a different contextual relationship such as spatial or compositional consistency (via can-coexist- with (Hu et al., (015) or is-part-of (Brust & Denzler, 2018, [Wang et al., (016) relations. These relationship graph, e.g., parsing graph (Tu et al. 2005). can be either precomputed or learned for the particular task at hand.
H APPENDIX: GROUPING CATEGORIES USING HDCNN TECHNIQUE
We also attempted to group the ImageNet 2012 categories following the strategy suggested by the HDCNN paper, Section 4 of (Yan al. 2015). For this purpose we exploited the ResNet-50 clas- sifier trained on 93% training examples of ImageNet 2012 (i.e., the initial classifier for HG-RN). The remaining 7% of the training set was used to compute 100 disjoint clusters of categories. While some of the animal and vegetable/fruit categories were correctly grouped into the same clusters, many of the clusters comprises semantically very different classes. We listing few of these clusters below. Yan et al. (2015) further created some overlapping groups for their algorithm. It is clear how to assign concept class labels to these undersegmented clusters of categories and how they can be predicted using a classifier. That is why we cannot compare our method with HDCNN.
Cluster
(n02509815 lesser-panda, n02346627 porcupine, n01872401 echidna, no2172182 dung-beetle, n02319095 sea-urchin, n03717622 manhole-cover. n04C33901 quill, n03733281 maze].
Cluster 2 [n04086273 revolver, n02804414 bassinet, n03954731
plane, n03995372 power-drill, n03483316 hand-blower, n03868863
oxygen-mask, n03759954 microphone, n02841315 binoculars, n02749479 assault-rifle, n04090263 rifle, n03602883 joystick, n04485082
tripod, n04517823 vacuum, n04554684 washer, n03724870 mask, n04069434 reflex-camera, n03424325 gasmask].
Cluster 3: (n02389026 sorrel, n02403003 ox, n02437312 Arabian-camel, n03538406 horse-cart, n03868242 oxcart, n04417672 thatch).
15

---

Under review as a conference paper at ICLR 2021
Cluster 4: (n03447447 gondola, n02951358 canoe, n04204347 shopping-cart, n03933933 pier, h04311004 steel-arch-bridge. n04366367 suspension-briddee n04532670 viaduct, n03160309 dam, n03873416 paddle}.
I APPENDIX: ROBUSTNESS OF MD-RN TRAINED ON AWA2
In order to test the robustness of the proposed MD-RN network trained on AwA2 dataset, we evaluate its performance
ImageNet 12 images with overlapping categories. There 16 AwA2 categories that are also present in ImageNet 12 dataset. We applied our MD-RN trained on AwA2 on 800 ImageNet 12 validation images and achieved category, concept and combined accuracies {AcCCAT ACCCON. ACCCOMB] = {89.87, 89.37, 85.75). 89.37. For comparison, we also ap- plied the flat baseline BL-RN on these images and achieved {AcCCAr, AceçoN, AcCCOMB) (88.87, 78. 12, 12. This experiment shows that the proposed dense layers in MD-RN learned on AwA2 can generalize better than the single layer in BL.-RN. Note that, the convolutional feature layer of both MD-RN and BL--RN have not changed because only dense layers of these two networks were trained (see Section 4.2). Figure shows few examples output from the proposed network on ImageNet 12 validation images.
ILSVRCM6
ILSVRC201E_ yal 00010022222
60089667
ILSVRC2012 08006013
40802789
Garilla
Dulmatian
Hippepotamus
Placental + primate Great ape
Placental > camivore
Placental aquatic
Placental ungulate + even-tned ungulate
Placental ungulate + odd-toed ungulate-> equine
canine > dog
mammal-> whale-
Figure 6: Qualitative performance of MD-RN trained from AwA2 datasets images from ImageNet 12 validation set. The image and MD-RN category predictions are listed on top of each image whereas concept predictions are displayed below.
16


Reviews:
[{"review_id": "wDX6DeunN1s", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper designs a multilayer connection structure for neural networks, such that the connection architecture supports implementation of a hierarchical classification scheme within these layers.  It applies this design to the task of hierarchical classification on ImageNet.  Experiments compare results with those of Deng et al. (2012), as well as baseline flat classification models.\n\nThe paper motivates the proposed approach via broad claims about what networks understand, but does not provide sufficient analysis or experimental evidence to justify these claims.  For example:\n\n\"In particular, when an existing CNN correctly identifies an image of an English Setter, the network itself does not learn that it is an instance of a dog, or more precisely, a hunting dog which is also a domestic animal and above all, a living thing\"\n\nAssuming it is trained on example images of all of these categories, how do we know that the CNN does not learn shared representations that implicitly reflect such organization of the concept space?  The paper does not employ any techniques to probe the learned representations of CNNs; without such analysis, the sweeping statements about what CNNs do or do not learn are mere speculation.\n\nOn a technical level, the design of the proposed dense classification layers appears to be quite ad-hoc.  It is not clear why a special design intermixing concept prediction nodes with hidden nodes is desirable or necessary.  How does this compare, both conceptually and experimentally, to a branching hierarchy of subnetworks?  The scheme of HD-CNN (Yan et al., 2015) is similar to the latter, but is not represented in experimental comparisons.\n\nIn fact, experiments appear to lack comparison to any recent published methods on hierarchical classification.  Deng et al. (2012) is the only prior publication that serves as a reference point.  This is far from a sufficient baseline as surely there has been other work on hierarchical classification in the past 8 years.  For example, a highly relevant work that this paper fails to even cite or discuss is:\n\nM. Nickel and D. Kiela. Poincare Embeddings for Learning Hierarchical Representations. NeurIPS, 2017.\n\nTogether, the unsubstantiated motivating claims, ad-hoc design of questionable merit, limited experimental comparison, and missing citations to highly relevant recent work suggest that this paper is not of sufficient quality for publication.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Etod9PVtLBt", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors consider how to capture the semantic relationship among categories of a classifier. It is an important problem and has many potential applications. For example, the predicted concept chain can help people understand the performance of the classifier, the coarse-grained concepts are beneficial to the few-shot learning of new categories, and etc.\nThe authors incorporate WordNet as their ontology and build their neural classifier based on it. Such a tree-structured dense-connected neural architecture is not very common in the current deep learning domain. The network is bound to external ontology, so when the ontology updates, the network has to be re-built. In my opinion, the design seems not very general. Maybe the authors could consider representing restrictions among concepts in the vector space.\nIn the experiments, the authors used only two baselines, one is a flat single-layer classifier, the other is a work of 2012. The baselines seem too weak to demonstrate the superiority of the proposed model. The results of more recent works are necessary, even though these works \"use a separate technique/tool for modeling the conceptual relations\", as the authors claimed.\n\nTo sum up, the pros of this paper include:\n- a valuable research topic\n- a fancy model\n- clear experimental details\n\nThe cons include:\n- the model is not very general\n- baselines are too weak\n- the aspect ratio and resolution of figures seem improper", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ady6P3MK6S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method to learn concept classes along with its concept superclasses. The proposed method relies on an ontology which they heuristically re-organize by essentially pruning nodes that have few descendants and large semantic overlap. The network proposed to model the ontology essentially just consists of a learned multiplicative gate at each level of the ontology with a standard xent loss over concepts and regularizing term that indicates if the category is an ancestor concept. The experimental results claim gains over some baselines, e.g., combined acc. of 69.05 vs chosen baseline 66.15 on ImageNet12 for ResNet-50 features at a cost of between 18.2% increase in parameters.\n\nOverall, while some of the empirical results seem competitive, I am most concerned with the weak foundations of the motivation of the setup. The work reads like a proposed solution that's trying to look for a problem / motivation and as a result struggles to find its footing in explaining modeling choices & results. \n\n* The paper uses as motivation that many networks use a softmax head over semantic categories at the leaves of an ontology and claims this is therefore why models using such networks do not learn that say English Setter is a dog. This is a shallow argument for incorporating concept hierarchies since such models clearly would still not be learning deeply what the concept of a dog is, only encoding weak priors introduced by the ontology, an external knowledge base from the network. The connection to learning relationships like \"is-a\" relations don't ultimately fall out from the proposed method, instead you just get a list of likelihoods that correspond to superclasses that contributed to a concept prediction—the model is not learning the relation, just the co-presence of these superclasses.\n* The argument that works like Deng et al (2012, 2014), which for example propose label relation priors like HEX graphs, only either predict fine-grained labels or superclasses exclusively, but not both simultaneously, is another example of where this paper falls short in its problem setup. This work doesn't answer the question of _why_ one would even want to predict both simultaneously well. If we believe that a superclass is unlikely present, why would we still predict the child classes? Even if there are reasonable arguments for this, they are absent in this paper.\n* The approach to creating the \"compressed concept hierarchy\" largely felt like a description of what was done, again, rather than why. Unless I missed it, I also expected a baseline for ablation that doesn't use the \"compressed\" hierarchy, but just uses it as-is.\n* It's a bit strange to me why a MSE loss is used for the indicator of whether a concept is an ancestor. Why use an unbounded error in L2, even if (or especially if) you are squeezing through a sigmoid? What is the intuition?\n* I would be curious to know if the improvement in results that we see in Table 1 are just due to increased model capacity (params/compute), i.e. how does it compare to the Deng et al baselines. The comparison discussed are only made with respect to the ResNet-50 & Inception-V4 backbones.\n\nI will also note that the paper could improve on its clarity in writing. As an example, from Figure 2, it's unclear what exactly changed from the LHS and RHS, how, and why it's meaningful; and in Figure 3, it's not obvious without work how the input, output and z-term relate.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2at9Wr4q_qW", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a novel module on top of ConvNet, multi-layer dense connectivity, for learning hierarchical concepts in image classification.\n\nPros:\n\nThis paper proposes to use the label hierarchy (with ancestor concepts from a label) instead of the label itself to learn the image recognition system. To achieve this, it has made two major contributions:\n1. Building label hierarchy with a simplified set of categories, to remove the redundant and meaningless categories\n2. With the constructed label hierarchy, this paper proposes a dense connectivity module to leverage the label hierarchy to model category abstractions over high-level visual embedding, on top of commonly used convolutional neural networks.\n\nWith the proposed techniques, this paper builds up its recognition system using two standard deep ConvNets and achieved strong results on large-scale image recognition benchmarks. \n\nCons:\n\n1. In general, the paper is not very well written for a few reasons: A) The motivation of the proposed method over previous methods is not clear (intro paragraph#2). B) Section 3.1 is very hard to follow. C) Some notations in Section 3.2 seems unnecessary and there are things being used before it is formally defined.\n\n2. The design of this dense connectivity module in Section 3.2 seems quite arbitrary, there is no good explanation on why we need to use the z to multiply the output x and h.\n\n3. Experiments of naturally adversarial examples are not motivated earlier in the introduction. It's quite hard for me to understand why using a label hierarchy would improve this task.\n\n \nDetailed Comments:\n\n1. Paragraph#2 in Intro: why training a neural network as multinomial/softmax logistic regression from images to labels can not acquire a comprehensive knowledge about the input entity? For instance, in some of the prior works (e.g. Hu et. al. 2018), they learn models to simultaneously classify categories on a predefined label hierarchy, including both abstracted classes such as \"Dog\" and concrete class such as the \"English Setter\".\n2. It seems that from Section 3 on, it uses the term \"Category\" to stand for the leaf concept (most specific) and the term \"Concept\" as the shorthand of ` \"Ancestor Concept\". It would be better to mention this explicitly to avoid confusion.\n3. Example in Figure 2 is not very clear and hard to follow. It might be better to simplify the figure by using a smaller hierarchy as an example. Also, it would be good to have a paragraph in section 3.1 to describe what in the right figure has been modified using the concrete examples of Figure 2. \n4. Equation 1, why do we need a \\psi activation function which is linear? What it means by linear, is there an additional linear weight in \\psi besides v?\n5. Why are we using an MSE for the concept classifiers? I assume we can use binary cross-entropy for them?\n\n\nMinor:\n* The aspect ratio of Figures 2 and 3 need to be adjusted. It is hard to recognize text and symbols on the stretched figures.\n* Notation \\hat{h} in the text is bolded but the ones in the equation (1) is not bolded\n* A recent work that also leverages hierarchical information in the label text to learn visual concept embeddings, which is closely related to the topic of this paper: Learning to Represent Image and Text with Denotation Graph. EMNLP 2020\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommended rejection after considering the rebuttal from the authors. The main weaknesses of the submission include poorly motivated claims and designs, and insufficient experimental comparisons. The AC did not find sufficient grounds to overturn the reviewers' consensus recommendation. 


Decision: Reject