DEEP HIERARCHICAL MODEL FOR HIERARCHICAL SELECTIVE CLASSIFICATION AND ZERO SHOT LEARNING

1 INTRODUCTION . Object recognition from images in real world scene , which requires drawing boundaries between groups of objects in a seemingly continuous space , is a highly complex problem . The visual world is populated with a vast number of diverse objects , where each object instance may pose many visual forms . Furthermore , with the growing number of classes , the similarity structures between them become complex and the distinction between classes blurs . Because of the high complexity of this task , most existing work focus on simplifying the problem to a supervised single-label classification problem . Unfortunately , since one-vs-all classifiers treat all classes as unrelated , visual recognition systems can not transfer semantic information about learned classes . One way of dealing with this issue is to respect the natural continuity of visual space instead of artificially partitioning it into disjoint categories . Classes have varying degrees of discriminability . Some classes have unique features while other classes might share similar features and are hence harder to distinguish from each other . Such similarity structures in the data are very valuable information that could potentially lead to improving classifiers ( Frome et al. , 2013 ; Rippel et al. , 2015 ; Bilal et al. , 2018 ) . Such classifier has a potential to make semantically reasonable errors . For example , in the upper image of Figure 1 the standard model choose the wrong class ’ stingray ’ while a more reasonable error can be ’ hammerhead ’ or ’ great white shark ’ . Moreover , such semantic classifiers may be less biased to irrelevant context as illustrated in the example . The standard model is biased to the wrong class because the image has people . We found that in the training set many images of stingray are with people . In order to confirm this statement we occlude1 the people . By doing this we mange to classify the image correctly with the standard classifier ( Zeiler & Fergus , 2014 ) . On the other hand , our semantic model identify the true class even in the original image . Furthermore , after 1Occlusion technique is generally used to reveal the regions which contribute most to a target response , here we used it to identify the error cause occluding the error cause our model ranks the top-5 predicated classes according to their relevance to the true class measured by the taxonomy distance . We propose an approach that addresses these shortcomings by learning a hierarchical inter-class relationship model . We present a novel hierarchical probability-based loss function , which we call soft-NLL . This loss function gives a probability weight according to the inter-class taxonomy graph distances . We evaluate the model semantic generalization ability , which means for example that the model makes more semantically reasonable errors . Our model shows a significant improvement on the hierarchical semantic metric in an order of magnitude over existing methods , while it guarantees a minor decrease in the common top-k accuracy compared to the widely used hard-NLL loss . Our approach is independent of the specific prediction model and thus can always benefit from design progress . Furthermore , we develop an algorithm where given a probabilistic model , based on its top-k predictions our algorithm returns the input corresponding super-group based on classes hierarchy without any further learning . We present two use cases for super-group retrieval and show that our soft-NLL outperforms in both applications . The first one , selective classification , deals with the problem of low-confidence classification , wherein a model is unable to make a successful exact classification . In this case , our algorithm returns a corresponding closest super-class . In the second scenario , the proposed method is used for the zero-shot learning problem . In this case , given a novel input , the algorithm returns its hierarchically related group , rather than generating a true unseen group . 2 RELATED WORK . Vision researchers have sought to exploit external semantic knowledge . Taxonomies , or class hierarchies were the first to be explored ( Griffin & Perona , 2008 ; Marszalek et al. , 2008 ) . Taxonomy depends on a human-defined criteria , and thus implicitly consists of semantic knowledge . Early works ( Bengio et al. , 2010 ; Gao & Koller , 2011 ; Griffin & Perona , 2008 ; Marszalek et al. , 2008 ) exploit the tree structure for efficient branch-and-bound training and prediction , while a few use taxonomies as sources of relational knowledge between categories ( Deng et al. , 2014 ; Hwang , 2013 ) . Our method shares the same goal with the latter group of work . Deep Convolutional Neural Networks which learn high-level features and a N-way flat softmax classifier have recently gained enormous popularity for their impressive performance on a number of visual recognition tasks . Since Krizhevsky et al . ( 2012 ) won the ImageNet ILSVRC challenge 2012 ( Deng et al. , 2009 ) , many variants of this model have been proposed . GoogLeNet ( Szegedy et al. , 2015 ) , Resnet ( He et al. , 2015 ) and VGGNet ( Simonyan & Zisserman , 2014 ) focus on increasing the network depth by adding more convolutional layers to the original model . These models generalize better than smaller ones . Our goal is to benefit form the progress of CNN designs . Some existing work has explored various approaches to learning semantic information in deep neural networks . One related approach is to learn a joint representation of images and labels in an embedding space . Frome et al . ( 2013 ) ; Norouzi et al . ( 2013 ) learn semantic relationships between classes using a semantic word embedding model , where each class represented by an embedding vector . In parallel , they pre-train a deep neural network for visual object recognition . Then , both methods develop a specific mapping of the images into the semantic embedding space . Both methods evaluated their semantic ability using a semantic metric . Moreover , they employed their models for zero-shot learning . However , Frome et al . ( 2013 ) show modest performance in its semantic relevance meaning because the word2vec embeddings are weak in the sense of capturing the inherent hierarchical semantics . Furthermore , it imposes structure only on the output space , and structure on the learned space is not explicitly enforced . However , our goal is to transfer direct inter-class hierarchical data to the learned space . Deng et al . ( 2014 ) introduce a new formalism which captures semantic relations between any two labels applied to the same object : mutual exclusion , overlap and subsumption . In the experiments part they used Wordnet heirachy and defined exclusion relation between any two labels unless they share a descendant . Each image got a single label and in this scope some leaf examples was relabeled only to theirs immediate parent ( s ) internal node label . In different to this method , we propose to formalize real mutual exclusion and subsumption relations in a continuous similarity meter calculated from taxonomy . The similarity level of labels is getting lower according to their distance from the true class . We show that their exclusion approach is too strict . That is we show that giving similarity weight even to labels which share predecessor is beneficial to performance in Section 4.2 . Secondly , our similarity concept which gives a soften-label allow to share similarity structure on all images . Moreover , the authors did not present an evaluation of how predictions ( i.e . even the incorrect ones ) are semantically meaningful which is our focus in this work . Yan et al . ( 2015 ) propose a two-staged CNN architecture named HD-CNN , which leverages the taxonomy to separate the categories into easy coarse-grained ones and confusing fine-grained ones , trained in separate networks . However , such separate learning for coarse and fine-grained categories results in larger memory footprints , while it improves performance with only a modest rate . Recently Bilal et al . ( 2018 ) designed a hierarchy-aware CNNs that accelerate model convergence and alleviate overfitting . They select AlexNet ( Krizhevsky et al. , 2012 ) as a reference architecture and created branches from multilevel deep features to perform group-level classification and backpropagate group error . The number of groups is gradually increased along the depth of the inner layers . However this scheme demands extensive analysis and consequently it can not easily benefit from the progress of CNN designs . Moreover , the latter two works did not present an evaluation of how predictions are semantically meaningful . Our goals are to directly exploit the class hierarchy information while benefit form the progress of CNN designs . We focus on large scale multi-class problems . In contrast to most previous works we evaluate the semantic ability of our model according to a semantic metric . We further show that such a metric is meaningful when we compare our soft semantic model to a regular hard-model on two applications . 3 THE PROPOSED METHOD . 3.1 SOFT NEGATIVE LOG LIKELIHOOD LOSS FORMULATION . The problem we address in this paper is a supervised multiclass classification . Following common setup , we assume the input X ∈ X and label Y ∈ Y = { 1 , ... , C } can be modelled by a joint distribution π ( X , Y ) . The labels are organized in a taxonomy graph G , where each directed edge represents an ’ is a ’ relation , and the distance between nodes reflects a semantic relationship between them . Our goal is to model these semantic relationships , so that its top-k predictions will be closely related to the true label according to G. The standard objective formulation used in the context of deep learning for a multiclass classification problem is the negative log likelihood ( NLL ) loss , which is also referred to as the cross entropy loss with the probability class indicator . Given a probabilistic model π̂ ( Y |X ) and a sample ( x0 , y0 ) , NLL loss is defined by l ( x0 , y0 ) = − C∑ y=1 π ( y|x0 ) log π̂ ( y|x0 ) , where π ( y|x0 ) = 1 ( y=y0 ) This formulation assigns the whole probability weight to a single class with all other classes having zero weight . This approach artificially partitions the visual space into disjoint categories and does not take into account semantic relationships between classes . We propose to use a ’ soft ’ ( instead of ’ hard ’ ) model π ( y|x ) as a semantic relationship measure . This model can be learned as the class graph taxonomy G. More formally , let Π ∈ RC×C represent a semantic probability taxonomy relationship ( or distance ) map , where Πi , j , π ( i|j ) . We define the soft NLL loss as : l ( x0 , y0 ) = − C∑ y=1 Πy0 , y log π̂ ( y|x0 ) ( 1 ) Π is a class row-wise i.e . each ith row resembles the ith true-class probabilities , which are inversely proportional to the on-the-graph distance defined , dG ( class node i , class node j ) , shortest path length between nodes Details about hyper-parameters selection is described in Section 4.1 . 3.2 SUPER-GROUP RETRIEVAL ALGORITHM . In this section we present a super-group retrieval algorithm . Returning a super-group can be useful in hard scenarios for example , when given object samples with high confusion between similar classes or when making an inference for novel classes . Soft-NLL has a better semantic generalization ability as we show below in Section 5.1 . Its top-k predictions are more accurate with respect to the true class . Therefore , they can be used as a coarse-grained classifier and retrieve a better super group , compared to standard models . Given the model top-k predictions the algorithm follows these steps : 1 . Clean the top-k predictions from less relevant predictions : The model top-k predictions may be noisy . Based on the graph taxonomy we can generate a cleaner subset of the top-k predictions . More specifically , given the top-k predictions , we calculate a subset of them with each ith class l-hCorrectSet as follows , Si ( l , k ) = { top-k predictions } ∩ { l-hCorrectSeti } Where for a given class its l-hCorrectSet is the l nearest set of classes gathered from the graph taxonomy as defined by Frome et al . ( 2013 ) . We choose the highest matching set , SC ( l , k ) where C = argmax i |Si ( l , k ) | . 2 . Generate super-group candidates : The set { AS } equals to super-group nodes which are ancestors for each class in SC ( l , k ) . 3 . Choose the most specific super-group : The most specific super group is the a ∈ { AS } which is the lowest common ancestor ( LCA ) of SC ( l , k ) generated by LCA = min a∈ { AS } ∑ s∈SC ( l , k ) dG ( a , s ) 2 . In Appendix 7.1 we illustrate the algorithm concepts with a simple example . In Section 4.2 we discuss ways of choosing the algorithm ’ s hyper-parameters k and l. 2The taxonomy is a graph i.e . there may be more than one route connecting two nodes , therefore generating LCA in the mentioned way is better than by taking the deepest ancestor , LCA = max a∈ { AS } ∑ dG ( a , root ) .

Summary: The paper presents a method for classification which takes into account the semantic hierarchy of output labels, rather than treating them as independent categories. In a typical classification setup, the loss penalizes the KL-divergence between the model’s predicted label distribution and a one-hot distribution placing all probability mass on the single ground-truth label for each example. The proposed method instead constructs a target distribution which places probability mass not only on leaf category nodes but also on their neighbors in a known semantic hierarchy of labels, then penalizes the KL-divergence between a model’s predicted distribution and this target distribution. This model is used for classification on ImageNet-1k, and for zero-shot classification on ImageNet-21k where a model must predict superclasses seen during training for images of leaf categories not seen during training.