NESTED LEARNING FOR MULTI-GRANULAR TASKS

1 INTRODUCTION . Deep learning is providing remarkable computational tools for the automatic analysis and understanding of complex high-dimensional problems [ Esteva et al . ( 2017 ) ; Spanhol et al . ( 2016 ) ; Parkhi et al . ( 2015 ) ] . Despite its tremendous value and versatility , methods based on Deep Neural Networks ( DNNs ) tend to be overconfident about their predictions and limited to the task and data they have been trained on [ Hein et al . ( 2018 ) ; Guo et al . ( 2017 ) ; Nguyen et al . ( 2014 ) ] . This happens , among other reasons , because the standard approach to train DNN models consists in optimizing its performance over a specific dataset and for a specific task in an end-to-end fashion Szegedy et al . ( 2014 ) . Standard DNNs are not designed to be trained with data of different quality and to simultaneously provide results at multiple granularities . Take as an example the case illustrated in Figure 1 ( left ) ; for high quality facial images , we may be able to infer the person ’ s age group and identity ; whereas for poor resolution or occluded examples , only a sub-set of these nested predictions may be achievable . We expect the network to automatically understand what can and can not predict , and this is obtained with the framework proposed in this paper ( Figure 1 , right ) . Moreover , nested learning allows us to leverage training information from diverse datasets , with varying granularity and quality of labels , and combine this information into a single model . On the other hand , when heterogeneous data with different quality and granularity of annotations ( as in the example illustrated in Figure 1 ( left ) ) is provided for training , low quality samples with coarse labels can help us to understand the structure of the coarser distributions ( person , under 50 ) while simultaneously data with finer labels can contribute to the coarse and fine tasks . This will be formalized later in the paper with tools from information theory . Recently , Alsallakh et al . ( 2017 ) showed that convolutional neural networks ( CNNs ) naturally tend to learn hierarchical high-level features that discriminate groups of classes in the early layers , while the deeper layers develop more specialized feature detectors . We explicitly enforce this behaviour by creating a sequence of nested information bottlenecks . Looking at the problem of nested learning from an information theory perspective , we design a network topology with two important properties . First , a sequence of low dimensional ( nested ) feature embeddings are enforced for each level in the taxonomy of the labels . This encourages generalization by forcing information bottlenecks [ Tishby et al . ( 1999 ) , Shwartz-Ziv & Tishby ( 2017 ) ] . Second , skipped connections allow finer embeddings to access information of the input that may be useful for finer classification but not informative on coarser categories Ronneberger et al . ( 2015 ) . Additionally , we show how the explicit calibration and combination of nested outputs can improve the finer predictions and improve robustness . Finally , having the flexibility of merging data with different levels of granularity inspired us to study which is the most efficient way of annotating data given a fixed budget that takes into account that more detailed training data is more expensive to annotate . The source code associated to this work is open source.1 The main contributions of this paper are : ( 1 ) We introduce the concept of explicit nested learning , where a given level in the hierarchy strictly contains the previous one and strictly adds information ; ( 2 ) We provide a deep learning architecture that can be trained with data from all levels of the nested structure ( all levels of labels and data quality ) , each one affecting the corresponding component of the network ; ( 3 ) We provide a model with multiple outputs , one per level of the nested hierarchy , each one with its own confidence ; the user does not need to know the “ quality ” of the data beforehand , the output confidences provide that information . 2 RELATED WORK . The problem of adapting DNNs models and training protocols to encourage nested learning shares similarities with other popular problems in machine learning such as Multi-Task Learning ( MTL ) . Though our work is related to MTL because of the similar training challenges , most of the MTL methods tackle the task in a parallel way , without imposing a task hierarchy in the architecture [ Ranjan et al . ( 2016 ) , Kokkinos ( 2017 ) , Bilen & Vedaldi ( 2016 ) ] . The idea of learning hierarchical representations to improve classification performance has been exploited prior the proliferation of DNNs , e.g. , [ Zweig & Weinshall ( 2007 ) , Fergus et al . ( 2010 ) , Zhao et al . ( 2011 ) , Liu et al . ( 2013 ) ] . Some of these ideas have been incorporated into deep learning methods [ Alsallakh et al . ( 2017 ) , Wehrmann et al . ( 2018 ) , Deng et al . ( 2014 ) , Srivastava & Salakhutdinov ( 2013 ) ] , and exploited in specific applications [ Clark et al . ( 2017 ) , Xuehong Mao et al . ( 2016 ) , Seo & shik Shin ( 2019 ) ] . 1https : //github.com/nestedlearning2019 Kim et al . ( 2018 ) proposed a nested sparse network architecture with the emphasis on having a resource-aware versatile architecture to meet ( simultaneously ) diverse resource requirements . Wehrmann et al . ( 2018 ) proposed a neural network architecture capable of simultaneously optimizing local and global loss functions to exploit local and global information while penalizing hierarchical violations . Triguero & Vens ( 2016 ) investigated different alternatives to label hierarchical multi-label problems by selecting one or multiple thresholds to map output scores to hierarchical predictions , focusing on performance measures such as the H-loss , HMC-loss and the micro-averaged F-measure . Yan et al . ( 2015 ) introduced hierarchical deep CNNs ( HD-CNNs ) which consists of embedding CNNs into a two-level category hierarchy . They propose to distinguish a coarse class using an initial classifier and then refine the classification into a second level for each individual coarse category . Although the works listed above are important , relevant , and related to the work presented here , there are notable differences between them and what we propose . For example , while Kim et al . ( 2018 ) propose a nested architecture providing different ( potentially nested ) outputs , they do not study how to combine these outputs into a refined single prediction , nor provide a reliable confidence measure associate to them . Furthermore , the architecture they propose has key differences with ours , while they propose a nested hierarchy in a end-to-end fashion ( i.e. , features associated to the coarse and fine levels are shared from the top to the bottom of the network ) , we enforce sequential information bottleneck . As we show in the following sections this sequence of coarse to fine low dimensional representations facilitate a robust calibration and combination of nested outputs . Yan et al . ( 2015 ) study the problem of nested learning for two nested levels of granularity , and optimize for a final fine prediction . Their design is specific for a two-level category hierarchy while our work generalize to any number of nested levels , and we simultaneously can train and test up to an arbitrary level of granularity . Another important difference is that the focus of their work is on the implementation details and performance of their two-hierarchy network versus traditional end-to-end learning . As they mention in the conclusion of their work , future work should aim to extend their ideas to more than two hierarchical levels and to contextualize their empirical results into a theoretical framework . Our work takes steps in these two specific directions . This is also a fundamental difference with the works developed by Triguero & Vens ( 2016 ) and Wehrmann et al . ( 2018 ) . Moreover , all these approaches suppose that every sample is annotated for all the granularity levels . In contrast with them , we can efficiently train ( and predict ) our model on dataset that provide only coarse labels , intermediate levels , or fine labels . Our approach is therefore more general in that regard . Also , we show that if testing conditions shift from the ones on training , we can still provide relatively confident coarser labels while avoiding overconfident ( erroneous ) fine predictions ( see , e.g. , Figure 1 , right ; and Figure 5 in the supplementary material ) . Also , the questions of neural network overconfidence and output readability are never addressed in those works , which is crucial for both results interpretation , and output combination . Finally , because our solution can leverage information from datasets with different granularity , we are able to analyze how different proportions of training coarse and fine data affects models cost , performance and robustness . 3 NESTED LEARNING . Let us assume we want to classify the popular hand written digits of MNIST LeCun & Cortes ( 2010 ) . An input image can be represented as a realization x of the random variableX .2 We denote as X the alphabet of X . Associated to x , there is a label y that corresponds to the actual number the person writing the character wanted to represent . The label y is a realization of the random variable Y . In this illustrative case , Y can take 10 different values : Y = { 0 , 1 , ... , 9 } . Of course Y and X are not independent random variables . Generally , Y precedes X , and the problem of classification can be stated as the problem of inferring y from an observed sample x , i.e. , Y → X → Ŷ . Ŷ denotes a new random variable ( estimated fromX ) which approximate Y . More precisely , a common practice is to find a mapping X → Ŷ such that the probability P ( Ŷ 6= Y |X ) is minimized . Nested classification . In this work we focus on the inherent hierarchical structure most classification problems have . For example , imagine now that we have hand written characters including numbers , lower case letters , and capital letters . It would be intuitive to first attempt to classify these characters into three categories : numbers , lower case letters , and capital letters . Then , depending on this coarse 2Capital letters will be used to denote random variables and lower case letters to denote the value of a particular realization . classification we can perform a finer classification , i.e. , classifying the numbers into 0 − 9 classes , the letters into a− z , and so forth . Of course , we could have an arbitrary number of nested random variables associated to different levels of labels granularity . Here , subscripts indicate the granularity of the label , for example , Yi−1 is the closest coarse level of Yi . Y ki represents the random variable associated to each k value in the closest coarse node , i.e. , Y ki represents Yi given that yi−1 = k , k ∈ Yi−1 . Definition 3.1 . We define Y1 , ... , Yn as a discrete sequence of nested labels if H ( Yi|Yi+1 ) = 0 ∀i ∈ [ 1 , n− 1 ] . H denotes the standard definition of entropy for discrete random variables . Definition 3.2 . A discrete sequence of nested labels Y1 , ... , Yn is strictly nested if H ( Yi|Yi−1 ) < H ( Yi ) ∀i ∈ [ 2 , n ] . The core of this work is to formulate classification problems such that the information of the input is extracted in a hierarchical way . To this end , intermediate ( coarse ) predictions Ŷi are jointly learned . The two key components of the proposed approach are : ( i ) nested information bottlenecks and ( ii ) a combination of the predicted coarse and fine labels . The analysis provided in this section is agnostic to most of the implementation details ( which are addressed in coming sections ) . Hierarchical information bottlenecks and the role of skipped connections . We assume the random variable X contains information about the sequence of strictly nested labels Yi , i.e. , H ( Yi|X ) < H ( Yi ) , as illustrated in Figure 2 . More precisely , H ( Yi|X ) > H ( Yi−1|X ) and H ( Yi|X ) < H ( Yi|Yi−1 , X ) . To exploit these properties , we use standard DNN layers ( convolutional , pooling , normalization , and activation layers ) . As illustrated in Figure 3 , we begin by guiding the network to find a low dimensional feature representation f1 such that H ( f1 ( X ) ) H ( X ) while , I ( f1 ( X ) , Y1 ) is close to I ( X , Y1 ) where I ( · , · ) stands for the standard mutual information . ( DNNs are remarkably efficient at compressing and extracting the mutual information between high dimensional inputs and target labels [ Shwartz-Ziv & Tishby ( 2017 ) , Moshkovitz & Tishby ( 2017 ) ] . ) The second step consists in learning the complementary information such that when combined with the representation f1 , allows us to achieve a second representation f2 from which the second hierarchical label Y2 can be inferred . To this end , skipped connections play a critical role as will be further detailed next . As we discussed before , I ( X , Yi ) = H ( X ) − H ( X|Yi ) ︸ ︷︷ ︸ > H ( X|Yi+1 ) < I ( X , Yi+1 ) . ( 1 ) On the other hand , we want each feature embedding fi to compress the information of X while I ( fi ( X ) , Yi ) ≈ I ( X , Yi ) . Equation 1 means that the finer the classification the more information from X we need . Notice that while in many DNNs , skipped connections have proved to help on the networks compactness and to mitigate vanishing gradients Szegedy et al . ( 2017 ) , in the present work they are included for a more fundamental reason . ( See the experiments presented in the supplementary material Section F.2 for complementary empirical evidence and validation . ) If we do not consider skipped connections , X → fi ( X ) → fi+1 ( X ) forms a Markov chain where I ( X , fi+1 ( X ) ) ≤ I ( X , fi ( X ) ) ( data-processing inequality ) which contradicts Equation 1 . Section 4 presents experiments illustrating the impact of skipped connections on nested learning , and complements the discussion started here . Combination of nested outputs . We will present in the following sections experimental evidence showing that nested learning leads to an improvement in performance and robustness . In addition , the explicit combination of nested predictions , since the network simultaneously produces all the outputs Yi ( with corresponding confidence ) , can improve the accuracy and robustness even further . Our motivation is to explicitly refine the fine prediction leveraging the information of all the coarser outputs , i.e. , { Ŷ1 , ... , Ŷi } → Ỹi . Let us define si ( q ) the network output score associated to the event Yi = q . In general , if si ( q ) > si ( w ) most likely P ( Yi = q ) > P ( Yi = w ) , but P ( Yi = q ) 6= si ( q ) . In other words , a score value of 0.3 does not mean the sample belongs to this class with 30 % probability . This can be addressed by calibrating the outputs , which consists of mapping output scores into an estimation of the class probability si ( q ) → PŶi ( q ) , as defined and thoroughly explained in Zadrozny & Elkan ( 2002 ) . PŶi ( q ) denotes the calibrated output of the network which approximates P ( Yi = q ) . ( We will precise how calibration is performed in the following section . ) Then , we can use the estimated probability associated to a fine label PŶi to compute the conditional probability P ( Yi = yi|Yi−1 = k ) . This is achieved by re-normalizing the finer labels associated to the same coarse label , i.e. , PŶi|Ŷi−1 ( q ) = PŶi ( q ) ∑ w∈Ykqi PŶi ( w ) , ( 2 ) where Ykqi denotes the set of labels at granularity level i that share with q the same coarser label kq . Finally , the estimated conditional probability is combined with the prior of the coarser prediction to recompute the fine prediction P ′ Ŷi ( q ) = PŶi|Ŷi−1 ( q ) PŶi−1 ( kq ) .

Summary: The problem addressed is how to train a DNN to learn a hierarchical representation of the input that has heterogeneously annotated labels at various levels of granularity. After training the network’s goal is to emit sequentially finer grained labels with corresponding confidence. The authors use information theory to propose a network typology of information bottlenecks with skip connections to achieve this nested learning problem.