Multilayer Dense Connections for Hierarchical Concept Classification

1 INTRODUCTION . Classification is a core concept for numerous computer vision tasks . Given the convolutional features , different architectures classify either the image itself ( He et al. , 2015 ; Szegedy et al. , 2016 ) , the region/bounding boxes for object detection ( He et al. , 2017 ; Liu et al. , 2015 ) , or , at the granular level , pixels for scene segmentation ( Chen et al. , 2018 ) . Although early image recognition works employed multilayer classification layers ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ) , the more recent models have all been using single layer dense connection ( He et al. , 2016 ; Szegedy et al. , 2016 ) or convolutions ( Lin et al. , 2017 ) . The vision community has invented a multitude of techniques to enhance the capacity of feature computation layers ( Xie et al. , 2017 ; Huang et al. , 2017 ; Hu et al. , 2018 ; Dai et al. , 2017 ; Chollet , 2016 ; Tan & Le , 2019 ) . But , the classification layer has mostly retained the form of a multinomial/softmax logistic regression performing a mapping from a set of inputs ( images ) to a set of categories/labels . As such , the final output of these networks do not furnish a comprehensive depiction about the input entity . In particular , when an existing CNN correctly identifies an image of an English Setter , it is not laid out in the output that it is an instance of a dog , or more extensively , a hunting dog , a domestic animal and a living thing . It is rational to assume that convolutional layers construct some internal representation of the conceptual superclasses , e.g. , dog , animal etc. , during training . We argue that , by appropriately harnessing such representation , one can retrieve a much broader description of the input image from a CNN than it is supplied by a single layer output . Extensive information about most categories are freely available in repositories such as WordNet ( Fellbaum , 1998 ) . WordNet provides the hierarchical organization of category classes ( e.g. , English Setter ) and their conceptual superclasses ( e.g. , Hunting dog , Domestic animal , Living thing ) . However , a surprisingly limited number of CNNs utilize the concept hierarchy . The primary goal of almost all existing studies is to improve the category-wise classification performance by exploiting the conceptual relations , often via a separate tool . Deng et al . ( 2014 ) and Ding et al . ( 2015 ) apply graphical models to capture the interdependence among concept labels to improve category classification accuracy . Other works either do not clarify the semantic meaning of the ancestor concepts ( Yan et al. , 2015 ) or impose a level of complexity in the additional tool ( RNN ) that is perhaps unnecessary ( Hu et al. , 2016 ) . We have not found an existing ( deep learning ) model that attempts to predict both the finer categories and the chain of ancestor concepts for an input image by a single network . The classical hedging method ( Deng et al. , 2012 ) computes either the finer labels or one of its superclasses exclusively , but not both simultaneously . In this paper , we introduce a CNN to classify the category and the concept superclasses simultaneously . As illustrated in Figure 1 , in order to classify any category class ( e.g. , English Setter ) , our model is constrained to also predict the ancestor superclasses ( e.g. , Hunting dog , Domestic animal , Living thing ) in the same order as defined in a given ontology . We propose a configuration of multilayer dense connections to predict the category & concept superclasses as well as model their interrelations based on the ontology . We also propose a simple method to prune and rearrange the label hierarchy for efficient connectivity . Capturing the hierarchical relationship within the CNN architecture itself enables us to train the model end-to-end ( as opposed to attaching a separate tool ) by applying existing optimization strategies for training deep networks1 . We experimentally demonstrate that one can train the proposed archi- tecture using standard optimization protocols to predict the concept classes with two popular CNN backbones : ResNet and InceptionV4 , while maintaining their category-wise accuracy . The proposed multilayer connection is shown to further refine the learned representations of these backbone CNNs to yield better concept and category classification than 1 ) multinomial logistic regression , and , 2 ) other existing works ( that apply separate mechanisms ) on standard datasets and challenging images . Predicting coarser superclasses in addition to finer level categories improves interpretability of the classifier performance . Even if an eagle is misclassified as a parrot , the capability of inferring that it is a bird , and not an artifact ( e.g. , drone ) , may be beneficial in some applications ( e.g. , surveillance ) . More importantly , an object detector can enhance its capability on unseen categories by adopting the proposed classification scheme ( as demonstrated in Section 4.4 ) . For example , a movie/TV violence recognition/detection tool can recognize an equipment as a ’ weapon ’ concept class even if that particular weapon category was not in the training set . In visual question answering ( VQA ) , encoding concept classes would expand the scope of query terms by allowing broader description ( ’ how many vehicles are present ’ in addition to ’ how many buses ’ , ’ how many trucks ’ etc . ; see Cao et al . ( 2018 ) ; Wang et al . ( 2016 ) ) . In Appendix G , we point out how our architecture can be extended to object detectors to compute the concept classes . In addition , we allude to the potential applications of our model to capture label structures different from concept graph , e.g. , spatial or compositional dependence . 2 RELEVANT WORKS . Use of hierarchical classifiers can be traced back to the early works of Torralba et al . ( 2004 ) ; Wu et al . ( 2004 ) ; Fergus et al . ( 2010 ) that shared features for improved classification . Some studies claimed a hierarchical organization of categories resembles how human cognitive system stores knowledge ( Zhao et al. , 2011 ) while others experimentally showed a correspondence between structure of semantic hierarchy and visual confusion between categories ( Deng et al. , 2010 ) . Bengio et al . ( 2010 ) ; Deng et al . ( 2011 ) learn a label tree for efficient inference with low theoretical complexity and also suggest a label hierarchy is beneficial for datasets with tens of thousands of categories . Deng et al . ( 2012 ) aim to predict either a coarse level concept or a fine level category ( but not both at the same time ) given an initial classical classifier . Provided an initial classifier output , this method determines category or the coarse concept node ( exclusively ) with max reward based on aggregated probabilities in a label hierarchy . The reported results suggest the prediction of superclasses comes at the expense of the fine level category classification failure . For CNN based classification , Deng et al . ( 2014 ) modeled the relationships such as subsumption , overlap and exclusion among the categories 1One can also envision learning the multilayer connectivity structure from data by architecture learning techniques ( Zoph et al. , 2017 ; Pham et al. , 2018 ) . via a CRF . Although the CRF parameter can be trained via gradient descent , the inference required a separate computation of message passing . The work of Ding et al . ( 2015 ) extended this model by utilizing probabilistic label relationships . The HDCNN framework ( Yan et al. , 2015 ) groups the finer classes into coarse concept using the category labels . The framework comprises two modules for coarse and fine categories where the coarse prediction modulates the layers for finer classification . However , this work does not describe the coarser concept classes in order to analyze whether they , or their descendent categories , have any semantic meaning ( see Appendix H ) . Furthermore , the overlap among concept class descendants precludes a mechanism to predict the conceptual superclasses . Hu et al . ( 2016 ) present a structured inference model for hierarchical representation where the concepts representing scene attributes are predicted as indicator vectors of different length . A bidirectional message passing , inspired by the bidirectional recurrent networks , establishes the relations among different levels of concepts . The model leads to a large number of inter and intra-layer label interactions some which needed to be manually hard-coded to 0 . Appendix A describes a few other studies that proposed to incorporate structural prediction techniques in their design ( Guo et al. , 2017 ; Liang , 2019 ) . Unlike us , the primary objective of almost all the aforementioned papers is to improve the category prediction performance by utilizing the superclass hierarchy as an auxiliary source of information or as intermediate result . With the exception of Deng et al . ( 2012 ) ; Hu et al . ( 2016 ) , none of these studies were designed for , and demonstrate their effectiveness in , concept prediction . The algorithms of Deng et al . ( 2014 ) ; Guo et al . ( 2017 ) require substantial modifications to be able to classify the concept classes . Furthermore , in contrast to ours , most of these studies use a separate technique/tool for modeling the conceptual relations that need to be trained or applied separately with different mechanisms . It is important to distinguish our work from the hyperbolic embedding studies ( Nickel & Kiela , 2017 ; Khrulkov et al. , 2020 ) . Khrulkov et al . ( 2020 ) attempt to compute an embedding that respects ancestral hierarchy in the hyperbolic space – which implies that , ideally , a more generic image would be closer to the center than the specific ones . However , the paper does not describe – and , it is neither obvious nor straightforward – how to determine the concept classes from these embedded points . These studies , therefore , are not suitable for a comparison with our method for concurrent prediction of category and concept classes . 3 PROPOSED METHOD . Given an input image , the goal of our proposed method is to determine its category ( leaf node in the hierarchy ) and a list of its concept superclasses ( i.e. , ancestors in the ontology ) . As an example , for an image of a Chimpanzee , the proposed algorithm produces predictions for 1 ) the category Chimpanzee , and 2 ) an ordered list of ancestor concepts : Living thing → Chordate → Mammal → Primate → Chimpanzee . We illustrate ( and experiment with ) the proposed model for image classification in this paper . Our CNN architecture is designed to encompass the chain of relationships among the category and the predecessor concepts in the dense layers . We utilize an existing label hierarchy/ontology to guide the design of the dense layers , but do not use the hierarchy in prediction . In order to maximize the information within an ontology and to reduce the number of variables in the dense layers , we condense the original label hierarchy as explained in Section 3.1 . In our design of multilayer dense connections , each concept is associated with a set of hidden nodes . These hidden nodes are in turn connected to those of its children ( both category and concept ) and the output prediction nodes . Section 3.2 elaborates these connections and the loss functions to train the network .

Summary: The paper proposes a method to learn concept classes along with its concept superclasses. The proposed method relies on an ontology which they heuristically re-organize by essentially pruning nodes that have few descendants and large semantic overlap. The network proposed to model the ontology essentially just consists of a learned multiplicative gate at each level of the ontology with a standard xent loss over concepts and regularizing term that indicates if the category is an ancestor concept. The experimental results claim gains over some baselines, e.g., combined acc. of 69.05 vs chosen baseline 66.15 on ImageNet12 for ResNet-50 features at a cost of between 18.2% increase in parameters.